The SOLR connector is a high speed mechanism for writing data to Apache Solr.
If you are seeing error messages such as Invalid version (expected 2, but 60) or the data in not in 'javabin' format compare the version of the Solr Server against the version of solrj the connector is compiled with. This error message is most likely due to a version mismatch between the server and solrj. To address this try replacing the solr-solrj-*.jar packaged with the connector with the version that matches the Solr server you are connecting to.
This connector is used to connect to SolrCloud using the Zookeeper based configuration.
The target collection for this connector is selected by the topic name. Transformations like the RegexRouter transformation can be used to change the topic name before it is sent to Solr.
The password to use for basic authentication.
Importance: High
Type: Password
Default Value: [hidden]
The username to use for basic authentication.
Importance: High
Type: String
Zookeeper hosts that are used to store solr configuration.
Importance: High
Type: List
Chroot within solr for the zookeeper configuration.
Importance: High
Type: String
Flag to determine if the connector should delete documents. General practice in Kafka is to treat a record that contains a key with a null value as a delete.
Importance: Medium
Type: Boolean
Default Value: true
Configures Solr UpdaterRequest for a commit within the requested number of milliseconds. -1 disables the commit within setting and relies on the standard Solr commit setting.
Importance: Low
Type: Int
Default Value: -1
In this example each record has an incoming topic name prefixed with solr-. Assuming that our topic is solr-customer the following example will strip the prefix of solr- allowing us to write to the collection named customer. This is accomplished by using the RegexRouter transformation that is bundled with Apache Kafka.
Select one of the following configuration methods based on how you have deployed Kafka Connect. Distributed Mode will the the JSON / REST examples. Standalone mode will use the properties based example.
{
"name" : "cloudSolrSinkConnector1",
"config" : {
"connector.class" : "com.github.jcustenborder.kafka.connect.solr.CloudSolrSinkConnector",
"tasks.max" : "1",
"topics" : "topic1,topic2,topic3",
"solr.zookeeper.hosts" : "zookeeper.example.com:2181",
"solr.username" : "freddy",
"solr.password" : "password12345",
"transforms" : "dropPrefix",
"transforms.dropPrefix.type" : "org.apache.kafka.connect.transforms.RegexRouter",
"transforms.dropPrefix.regex" : "solr-(.*)",
"transforms.dropPrefix.replacement" : "$1"
}
}connector.class=com.github.jcustenborder.kafka.connect.solr.CloudSolrSinkConnector
tasks.max=1
topics=topic1,topic2,topic3
solr.zookeeper.hosts=zookeeper.example.com:2181
solr.username=freddy
solr.password=password12345
transforms=dropPrefix
transforms.dropPrefix.type=org.apache.kafka.connect.transforms.RegexRouter
transforms.dropPrefix.regex=solr-(.*)
transforms.dropPrefix.replacement=$1
This example will connect to a Solr Cloud cluster without authentication.
Select one of the following configuration methods based on how you have deployed Kafka Connect. Distributed Mode will the the JSON / REST examples. Standalone mode will use the properties based example.
{
"name" : "cloudSolrSinkConnector1",
"config" : {
"connector.class" : "com.github.jcustenborder.kafka.connect.solr.CloudSolrSinkConnector",
"tasks.max" : "1",
"topics" : "topic1,topic2,topic3",
"solr.zookeeper.hosts" : "zookeeper.example.com:2181"
}
}connector.class=com.github.jcustenborder.kafka.connect.solr.CloudSolrSinkConnector
tasks.max=1
topics=topic1,topic2,topic3
solr.zookeeper.hosts=zookeeper.example.com:2181
This example will connect to a Solr Cloud cluster using basic authentication.
Select one of the following configuration methods based on how you have deployed Kafka Connect. Distributed Mode will the the JSON / REST examples. Standalone mode will use the properties based example.
{
"name" : "cloudSolrSinkConnector1",
"config" : {
"connector.class" : "com.github.jcustenborder.kafka.connect.solr.CloudSolrSinkConnector",
"tasks.max" : "1",
"topics" : "topic1,topic2,topic3",
"solr.zookeeper.hosts" : "zookeeper.example.com:2181",
"solr.username" : "freddy",
"solr.password" : "password12345"
}
}connector.class=com.github.jcustenborder.kafka.connect.solr.CloudSolrSinkConnector
tasks.max=1
topics=topic1,topic2,topic3
solr.zookeeper.hosts=zookeeper.example.com:2181
solr.username=freddy
solr.password=password12345
In this example we do not care about the incoming topic name. We want to force all topics to a specific collection. This is accomplished by using the RegexRouter transformation that is bundled with Apache Kafka.
Select one of the following configuration methods based on how you have deployed Kafka Connect. Distributed Mode will the the JSON / REST examples. Standalone mode will use the properties based example.
{
"name" : "cloudSolrSinkConnector1",
"config" : {
"connector.class" : "com.github.jcustenborder.kafka.connect.solr.CloudSolrSinkConnector",
"tasks.max" : "1",
"topics" : "topic1,topic2,topic3",
"solr.zookeeper.hosts" : "zookeeper.example.com:2181",
"solr.username" : "freddy",
"solr.password" : "password12345",
"transforms" : "dropPrefix",
"transforms.dropPrefix.type" : "org.apache.kafka.connect.transforms.RegexRouter",
"transforms.dropPrefix.regex" : ".*",
"transforms.dropPrefix.replacement" : "forced-collection"
}
}connector.class=com.github.jcustenborder.kafka.connect.solr.CloudSolrSinkConnector
tasks.max=1
topics=topic1,topic2,topic3
solr.zookeeper.hosts=zookeeper.example.com:2181
solr.username=freddy
solr.password=password12345
transforms=dropPrefix
transforms.dropPrefix.type=org.apache.kafka.connect.transforms.RegexRouter
transforms.dropPrefix.regex=.*
transforms.dropPrefix.replacement=forced-collection
This connector is used to connect to write directly to a Solr core.
The target collection for this connector is selected by the topic name. Transformations like the RegexRouter transformation can be used to change the topic name before it is sent to Solr.
The password to use for basic authentication.
Importance: High
Type: Password
Default Value: [hidden]
The username to use for basic authentication.
Importance: High
Type: String
Url to connect to solr with.
Importance: High
Type: String
The number of documents to batch together before sending to Solr. See ConcurrentUpdateSolrClient.Builder.withQueueSize(int)
Importance: High
Type: Int
Default Value: 100
Flag to determine if the connector should delete documents. General practice in Kafka is to treat a record that contains a key with a null value as a delete.
Importance: Medium
Type: Boolean
Default Value: true
The number of threads used to empty ConcurrentUpdateSolrClients queue. See ConcurrentUpdateSolrClient.Builder.withThreadCount(int)
Importance: Medium
Type: Int
Default Value: 1
Configures Solr UpdaterRequest for a commit within the requested number of milliseconds. -1 disables the commit within setting and relies on the standard Solr commit setting.
Importance: Low
Type: Int
Default Value: -1
This example will connect to a Solr Cloud cluster without authentication.
Select one of the following configuration methods based on how you have deployed Kafka Connect. Distributed Mode will the the JSON / REST examples. Standalone mode will use the properties based example.
{
"name" : "httpSolrSinkConnector1",
"config" : {
"connector.class" : "com.github.jcustenborder.kafka.connect.solr.HttpSolrSinkConnector",
"tasks.max" : "1",
"topics" : "topic1,topic2,topic3",
"solr.url" : "http://solr.example.com:8993/"
}
}connector.class=com.github.jcustenborder.kafka.connect.solr.HttpSolrSinkConnector
tasks.max=1
topics=topic1,topic2,topic3
solr.url=http://solr.example.com:8993/
This example will connect to a Solr Cloud cluster using basic authentication.
Select one of the following configuration methods based on how you have deployed Kafka Connect. Distributed Mode will the the JSON / REST examples. Standalone mode will use the properties based example.
{
"name" : "httpSolrSinkConnector1",
"config" : {
"connector.class" : "com.github.jcustenborder.kafka.connect.solr.HttpSolrSinkConnector",
"tasks.max" : "1",
"topics" : "topic1,topic2,topic3",
"solr.url" : "http://solr.example.com:8993/",
"solr.username" : "freddy",
"solr.password" : "password12345"
}
}connector.class=com.github.jcustenborder.kafka.connect.solr.HttpSolrSinkConnector
tasks.max=1
topics=topic1,topic2,topic3
solr.url=http://solr.example.com:8993/
solr.username=freddy
solr.password=password12345