Skip to content

Latest commit

 

History

History
341 lines (189 loc) · 11.3 KB

ConfigureKyloForGlobalSearch.rst

File metadata and controls

341 lines (189 loc) · 11.3 KB
Raw

Configure Kylo & Global Search

Kylo supports Global search via a plugin-based design. Three plugins are provided out of the box:

  1. Elasticsearch (rest client) [default]
  2. Elasticsearch (transport client)
  3. Solr

Elasticsearch 5 support

Elasticsearch 5 is supported when using NiFi 1.3 (or later) and rest client. Kylo has been tested with version 5.5.1. Please refer to the rest client configuration for additional details.

Elasticsearch (rest client) [default]

Plugin Jar:
  • Name: kylo-search-elasticsearch-rest-<version>.jar
  • Default location: /opt/kylo/kylo-services/plugin/
Plugin Property File:
  • Name: elasticsearch-rest.properties
  • Default location: /opt/kylo/kylo-services/conf/

Steps to configure Kylo with Elasticsearch engine (using rest client) are below. Both Elasticsearch versions 2 and 5 are supported via this plugin.

  1. Include search-esr profile in existing list of profiles in /opt/kylo/kylo-services/conf/application.properties

    spring.profiles.include=native,nifi-v1,auth-kylo,auth-file,search-esr

  2. Ensure that the plugin is available in /opt/kylo/kylo-services/plugin. This comes out-of-the-box at this location by default. It should have ownership as kylo:users and permissions 755.

    kylo-search-elasticsearch-rest-<version>.jar

    Note

    It is recommended to have only one search plugin in the /opt/kylo/kylo-services/plugin/ directory. If there is another search plugin (for example, kylo-search-elasticsearch-<version>.jar), move it along with its property file to /opt/kylo/setup/plugins/<plugin-name>/ for later use.

  3. Provide elasticsearch-rest properties

    Update cluster properties in /opt/kylo/kylo-services/conf/elasticsearch-rest.properties if different from the defaults provided below.

    search.rest.host=localhost
search.rest.port=9200

  4. Create Kylo Indexes

    Execute a script to create kylo indexes. If these already exist, Elasticsearch will report an index_already_exists_exception. It is safe to ignore this and continue. Change the host and port if necessary.

    /opt/kylo/bin/create-kylo-indexes-es.sh localhost 9200 1 1

  5. Restart Kylo Services

    service kylo-services restart

  6. Steps to import updated Index Text Service feed

    1. Feed Manager -> Feeds -> + orange button -> Import from file -> Choose file

    2a. [Elasticsearch version 2] Pick the index_text_service_elasticsearch.feed.zip file available at /opt/kylo/setup/data/feeds/nifi-1.0

    2b. [Elasticsearch version 5] [This requires NiFi 1.3 or later] Pick the index_text_service_v2.feed.zip file available at /opt/kylo/setup/data/feeds/nifi-1.3

    1. Leave Change the Category field blank (It defaults to System)
    2. Click Yes for these two options (1) Overwrite Feed (2) Replace Feed Template
    3. (optional) Click Yes for option (3) Disable Feed upon import only if you wish to keep the indexing feed disabled upon import (You can explicitly enable it later if required)
    4. Click Import Feed.
    5. Verify that the feed imports successfully.

Elasticsearch (transport client)

Plugin Jar:
  • Name: kylo-search-elasticsearch-<version>.jar
  • Default location: /opt/kylo/setup/plugins/search-elasticsearch-transport-client/
Plugin Property File:
  • Name: elasticsearch.properties
  • Default location: /opt/kylo/setup/plugins/search-elasticsearch-transport-client/

Steps to configure Kylo with Elasticsearch engine (using transport client) are below. Only Elasticsearch version 2 is supported via this plugin.

  1. Include search-es profile in existing list of profiles in /opt/kylo/kylo-services/conf/application.properties

    spring.profiles.include=native,nifi-v1,auth-kylo,auth-file,search-es

  2. Ensure that the plugin is available in /opt/kylo/kylo-services/plugin. The plugin comes out-of-the-box at another location /opt/kylo/setup/plugins/search-elasticsearch-transport-client/. It should have ownership as kylo:users and permissions 755.

    kylo-search-elasticsearch-<version>.jar

    Note

    It is recommended to have only one search plugin in the /opt/kylo/kylo-services/plugin/ directory. If there is another search plugin (for example, kylo-search-solr-<version>.jar), move it along with its property file to to /opt/kylo/setup/plugins/<plugin-name>/ for later use.

    Reference commands to get the plugin, and change ownership and permissions:

    cp /opt/kylo/setup/plugins/search-elasticsearch-transport-client/kylo-search-elasticsearch-<version>.jar /opt/kylo/kylo-services/plugin/
cp /opt/kylo/setup/plugins/search-elasticsearch-transport-client/elasticsearch.properties /opt/kylo/kylo-services/conf/
cd /opt/kylo/kylo-services/plugin/
chown kylo:users kylo-search-elasticsearch-<version>.jar
chmod 755 kylo-search-elasticsearch-<version>.jar
cd /opt/kylo/kylo-services/conf/
chown kylo:users elasticsearch.properties
chmod 755 elasticsearch.properties

  3. Provide elasticsearch properties

    Update cluster properties in /opt/kylo/kylo-services/conf/elasticsearch.properties if different from the defaults provided below.

    search.host=localhost
search.clusterName=demo-cluster
search.restPort=9200
search.transportPort=9300

  4. Restart Kylo Services

    service kylo-services restart

  5. Steps to import updated Index Text Service feed

    1. Feed Manager -> Feeds -> + orange button -> Import from file -> Choose file
    2. Pick the index_text_service_elasticsearch.feed.zip file available at /opt/kylo/setup/data/feeds/nifi-1.0
    3. Leave Change the Category field blank (It defaults to System)
    4. Click Yes for these two options (1) Overwrite Feed (2) Replace Feed Template
    5. (optional) Click Yes for option (3) Disable Feed upon import only if you wish to keep the indexing feed disabled upon import (You can explicitly enable it later if required)
    6. Click Import Feed.
    7. Verify that the feed imports successfully.

Solr

Plugin Jar:
  • Name: kylo-search-solr-<version>.jar
  • Default location: /opt/kylo/setup/plugins/search-solr/
Plugin Property File:
  • Name: solrsearch.properties
  • Default location: /opt/kylo/setup/plugins/search-solr/

Kylo is designed to work with Solr (SolrCloud mode) and tested with v6.5.1. This configuration assumes that you already have a running Solr instance. You can also get it from the official download page.

Steps to configure Kylo with Solr are below:

  1. Include search-solr profile in existing list of profiles in /opt/kylo/kylo-services/conf/application.properties

    spring.profiles.include=native,nifi-v1,auth-kylo,auth-file,search-solr

  2. Ensure that the plugin is available in /opt/kylo/kylo-services/plugin. The plugin comes out-of-the-box at another location /opt/kylo/setup/plugins/search-solr/. It should have ownership as kylo:users and permissions 755.

    kylo-search-solr-<version>.jar

    Note

    It is recommended to have only one search plugin in the /opt/kylo/kylo-services/plugin/ directory. If there is another search plugin (for example, kylo-search-elasticsearch-<version>.jar), move it along with its property file to /opt/kylo/setup/plugins/<plugin-name>/ for later use.

    Reference commands to get the plugin, and change ownership and permissions:

    cp /opt/kylo/setup/plugins/search-solr/kylo-search-solr-<version>.jar /opt/kylo/kylo-services/plugin/
cp /opt/kylo/setup/plugins/search-solr/solrsearch.properties /opt/kylo/kylo-services/conf/
cd /opt/kylo/kylo-services/plugin/
chown kylo:users kylo-search-solr-<version>.jar
chmod 755 kylo-search-solr-<version>.jar
cd /opt/kylo/kylo-services/conf/
chown kylo:users solrsearch.properties
chmod 755 solrsearch.properties

  3. Create a folder on the box where Kylo is running to store indexes for Kylo metadata. Ensure that Kylo can write to this folder.

    Reference commands to create this folder and give full permissions:

    mkdir /tmp/kylosolr
chmod 777 /tmp/kylosolr

  4. Provide solr properties

    Update cluster properties in /opt/kylo/kylo-services/conf/solrsearch.properties if different from the defaults provided below. The search.indexStorageDirectory should match with the folder location created in previous step.

    search.host=localhost
search.port=8983
search.indexStorageDirectory=/tmp/kylosolr
search.zooKeeperPort=9983

  5. Create collections in Solr that Kylo will use.

    Reference commands:

    bin/solr create -c kylo-datasources -s 1 -rf 1
bin/solr create -c kylo-data -s 1 -rf 1

  6. Configure Kylo collections created in previous step via Admin UI

    Reference steps:

    Navigate to Admin UI

    Configure collection for datasources

    1. Select kylo-datasources collection from the drop down on left nav area
    2. Click Schema on bottom left of nav area

    3. Click Add Field on top of right nav pane

      • name: kylo_collection
      • type: string
      • default value: kylo-datasources
      • index: no
      • store: yes

    Configure collection for data

    1. Select kylo-data collection from the drop down on left nav area
    2. Click Schema on bottom left of nav area

    3. Click Add Field on top of right nav pane

      • name: kylo_collection
      • type: string
      • default value: kylo-data
      • index: no
      • store: yes

  7. Restart Kylo Services

    service kylo-services restart

  8. Steps to import updated Index Text Service feed

    1. Feed Manager -> Feeds -> + orange button -> Import from file -> Choose file
    2. Pick the index_text_service_solr.feed.zip file available at /opt/kylo/setup/data/feeds/nifi-1.0
    3. Leave Change the Category field blank (It defaults to System)
    4. Click Yes for these two options (1) Overwrite Feed (2) Replace Feed Template
    5. (optional) Click Yes for option (3) Disable Feed upon import only if you wish to keep the indexing feed disabled upon import (You can explicitly enable it later if required)
    6. Click Import Feed.
    7. Verify that the feed imports successfully.

  9. Ensure that the box running Kylo can connect to the box running Solr (if they are on separate machines). If required, open up these ports:

    • 8983
    • 9983