Is it possible to have 2 jobs : one with worplace search output and the second with elastic search output ? #1074

LaetitiaRedFroggy · 2021-01-15T14:10:04Z

LaetitiaRedFroggy
Jan 15, 2021

Hello, I'd like to have 2 jobs : one which sends data in workplace search and one which send data directly to Elastic Search (both in the same Elastic instance), but I noticed that xpack.secury settings for workplace search are not compatible with Elastic Search indexation (where security must be disabled). Is there a solution to combine those 2 outputs ?

Answered by LaetitiaRedFroggy

Jan 21, 2021

Hi David, Here is my config file Thanx Le jeu. 21 janv. 2021 à 11:00, David Pilato <notifications@github.com> a écrit :

View full answer

dadoonet · 2021-01-15T17:05:39Z

dadoonet
Jan 15, 2021
Maintainer

No. It's not supported (yet).
and I'm unsure about the use case then. Could you explain why you would need that?

5 replies

LaetitiaRedFroggy Jan 18, 2021
Author

Thank you for your reply :-)
In my project, I have a process which indexes only office files (doc, pdf, ppt, xls), then another which indexes all filesystem files without the content, this index is used only in Kibana for statistical analyzes (I don't want these results to appear in my search results). Since the configuration of Elastic for workplace is incompatible with that of fscrawler
for Elastic, I will send everything to workplace and I will filter the sources but I would have liked to avoid sending this index to workplace.

dadoonet Jan 18, 2021
Maintainer

What about running 2 FSCrawler instances? One for WPSearch and the other one for elasticsearch?

As you have 2 different use cases, that'd make sense to me.

LaetitiaRedFroggy Jan 18, 2021
Author

I will have 2 Fscrawler instances, but my problem is that I have one Elastic instance with security settings required by Workplace Search and these settings are not compatible with Fscrawler

dadoonet Jan 18, 2021
Maintainer

I don't understand. I'm using Workplace Search deployed in cloud with security settings and that works well for me. So what do you mean?

I wrote a tutorial here: https://discuss.elastic.co/t/dec-5th-2020-en-searching-anything-anywhere-with-workplace-search/254740/

Let me know exactly what does not work for you.

LaetitiaRedFroggy Jan 18, 2021
Author

To work with workplace search, Elastic must be configured with these options:
xpack.security.enabled: true
xpack.security.authc.api_key.enabled: true

If I create an fscrawler instance that sends the data directly to Elastic (the same Elastic node used by Workplace Search), these security options don't allow fscrawler to contact the Elastic client and I get this error:
{"error": {"root_cause": [{"type": "security_exception", "reason": "missing authentication credentials
for REST request [/] "," header ": {" WWW-Authenticate ": [" ApiKey "," Basic realm = \ "security " charset = \ "U
TF-8 \ ""]}}], "type": "security_exception", "reason": "missing authentication credentials for REST requ
is [/] "," header ": {" WWW-Authenticate ": [" ApiKey "," Basic realm = \ "security " charset = \ "UTF-8 " "]}}," s
tatus ": 401}

dadoonet · 2021-01-18T13:29:04Z

dadoonet
Jan 18, 2021
Maintainer

Did you ask the username and password in FSCrawler config job as well?

what is the config file looks like?

3 replies

LaetitiaRedFroggy Jan 18, 2021
Author

Here is my configuration :

elasticsearch:
  nodes:
  - url: "http://127.0.0.1:9200"
  bulk_size: 100
  flush_interval: "5s"
  byte_size: "10mb"
  user: "elastic"
  password: "XXX"

dadoonet Jan 18, 2021
Maintainer

And what are the full logs? Could you run fscrawler also with --trace option?

LaetitiaRedFroggy Jan 19, 2021
Author

Here are the full logs with --trace option :
fscrawler.log

dadoonet · 2021-01-19T09:39:09Z

dadoonet
Jan 19, 2021
Maintainer

Your job does not contain any credentials I think. See:

---
name: "fsc-babolat-service"
fs:
  url: "/home/lrichard/clients"
  update_rate: "15m"
  includes:
  - "*/*.doc"
  - "*/*.xls"
  - "*/.pdf"
  - "*/*.ppt"
  - "*/*.docx"
  - "*/*.xlsx"
  - "*/*.pptx"
  excludes:
  - "*/~*"
  json_support: false
  filename_as_id: false
  add_filesize: true
  remove_deleted: true
  add_as_inner_object: false
  store_source: false
  index_content: true
  attributes_support: false
  raw_metadata: false
  xml_support: false
  index_folders: true
  lang_detect: true
  continue_on_error: false
  ocr:
    language: "eng"
    enabled: true
    pdf_strategy: "ocr_and_text"
  follow_symlinks: true
elasticsearch:
  nodes:
  - url: "http://127.0.0.1:9200"
  bulk_size: 100
  flush_interval: "5s"
  byte_size: "10mb"
  ssl_verification: true

That's why it fails with:

08:37:40,379 �[33mWARN �[m [f.p.e.c.f.c.v.ElasticsearchClientV7] failed to create elasticsearch client on Elasticsearch{nodes=[http://127.0.0.1:9200], index='fsc-babolat-service', indexFolder='fsc-babolat-service_folder', bulkSize=100, flushInterval=5s, byteSize=10mb, username='null', pipeline='null', pathPrefix='null', sslVerification='true'}, disabling crawler...
08:37:40,379 �[1;31mFATAL�[m [f.p.e.c.f.c.FsCrawlerCli] We can not start Elasticsearch Client. Exiting.
org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=security_exception, reason=missing authentication credentials for REST request [/]]
	at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:187) ~[elasticsearch-7.10.1.jar:7.10.1]
	at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1911) ~[elasticsearch-rest-high-level-client-7.10.1.jar:7.10.1]
	at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1888) ~[elasticsearch-rest-high-level-client-7.10.1.jar:7.10.1]
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1645) ~[elasticsearch-rest-high-level-client-7.10.1.jar:7.10.1]
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1617) ~[elasticsearch-rest-high-level-client-7.10.1.jar:7.10.1]
	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1584) ~[elasticsearch-rest-high-level-client-7.10.1.jar:7.10.1]
	at org.elasticsearch.client.RestHighLevelClient.info(RestHighLevelClient.java:785) ~[elasticsearch-rest-high-level-client-7.10.1.jar:7.10.1]
	at fr.pilato.elasticsearch.crawler.fs.client.v7.ElasticsearchClientV7.getVersion(ElasticsearchClientV7.java:179) ~[fscrawler-elasticsearch-client-v7-2.7-SNAPSHOT.jar:?]
	at fr.pilato.elasticsearch.crawler.fs.client.ElasticsearchClient.checkVersion(ElasticsearchClient.java:192) ~[fscrawler-elasticsearch-client-base-2.7-SNAPSHOT.jar:?]
	at fr.pilato.elasticsearch.crawler.fs.client.v7.ElasticsearchClientV7.start(ElasticsearchClientV7.java:152) ~[fscrawler-elasticsearch-client-v7-2.7-SNAPSHOT.jar:?]
	at fr.pilato.elasticsearch.crawler.fs.service.FsCrawlerManagementServiceElasticsearchImpl.start(FsCrawlerManagementServiceElasticsearchImpl.java:60) ~[fscrawler-core-2.7-SNAPSHOT.jar:?]
	at fr.pilato.elasticsearch.crawler.fs.FsCrawlerImpl.start(FsCrawlerImpl.java:116) ~[fscrawler-core-2.7-SNAPSHOT.jar:?]
	at fr.pilato.elasticsearch.crawler.fs.cli.FsCrawlerCli.startEsClient(FsCrawlerCli.java:307) [fscrawler-cli-2.7-SNAPSHOT.jar:?]
	at fr.pilato.elasticsearch.crawler.fs.cli.FsCrawlerCli.main(FsCrawlerCli.java:283) [fscrawler-cli-2.7-SNAPSHOT.jar:?]
	Suppressed: org.elasticsearch.client.ResponseException: method [GET], host [http://127.0.0.1:9200], URI [/], status line [HTTP/1.1 401 Unauthorized]
{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":["ApiKey","Basic realm=\"security\" charset=\"UTF-8\""]}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":["ApiKey","Basic realm=\"security\" charset=\"UTF-8\""]}},"status":401}
		at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:326) ~[elasticsearch-rest-client-7.10.1.jar:7.10.1]
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:296) ~[elasticsearch-rest-client-7.10.1.jar:7.10.1]
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:270) ~[elasticsearch-rest-client-7.10.1.jar:7.10.1]
		at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1632) ~[elasticsearch-rest-high-level-client-7.10.1.jar:7.10.1]
		at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1617) ~[elasticsearch-rest-high-level-client-7.10.1.jar:7.10.1]
		at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1584) ~[elasticsearch-rest-high-level-client-7.10.1.jar:7.10.1]
		at org.elasticsearch.client.RestHighLevelClient.info(RestHighLevelClient.java:785) ~[elasticsearch-rest-high-level-client-7.10.1.jar:7.10.1]
		at fr.pilato.elasticsearch.crawler.fs.client.v7.ElasticsearchClientV7.getVersion(ElasticsearchClientV7.java:179) ~[fscrawler-elasticsearch-client-v7-2.7-SNAPSHOT.jar:?]
		at fr.pilato.elasticsearch.crawler.fs.client.ElasticsearchClient.checkVersion(ElasticsearchClient.java:192) ~[fscrawler-elasticsearch-client-base-2.7-SNAPSHOT.jar:?]
		at fr.pilato.elasticsearch.crawler.fs.client.v7.ElasticsearchClientV7.start(ElasticsearchClientV7.java:152) ~[fscrawler-elasticsearch-client-v7-2.7-SNAPSHOT.jar:?]
		at fr.pilato.elasticsearch.crawler.fs.service.FsCrawlerManagementServiceElasticsearchImpl.start(FsCrawlerManagementServiceElasticsearchImpl.java:60) ~[fscrawler-core-2.7-SNAPSHOT.jar:?]
		at fr.pilato.elasticsearch.crawler.fs.FsCrawlerImpl.start(FsCrawlerImpl.java:116) ~[fscrawler-core-2.7-SNAPSHOT.jar:?]
		at fr.pilato.elasticsearch.crawler.fs.cli.FsCrawlerCli.startEsClient(FsCrawlerCli.java:307) [fscrawler-cli-2.7-SNAPSHOT.jar:?]
		at fr.pilato.elasticsearch.crawler.fs.cli.FsCrawlerCli.main(FsCrawlerCli.java:283) [fscrawler-cli-2.7-SNAPSHOT.jar:?]

2 replies

LaetitiaRedFroggy Jan 21, 2021
Author

I don't understand because I have declared user and password in my config file....

dadoonet Jan 21, 2021
Maintainer

Could you share your config file? Attach it here if you will.

LaetitiaRedFroggy · 2021-01-21T10:08:52Z

LaetitiaRedFroggy
Jan 21, 2021
Author

Hi David, Here is my config file Thanx Le jeu. 21 janv. 2021 à 11:00, David Pilato <notifications@github.com> a écrit :

…

Could you share your config file? Attach it here if you will. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1074 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ASO6LRZYMDUC3BWGBD7JUSDS273LNANCNFSM4WEDNR2A> .

-- Laetitia RICHARD / Consultante Data (+33) 6 26 41 36 39 / laetitia.richard@redfroggy.fr Red Froggy SAS 73 cours Albert THOMAS, 69447 LYON CEDEX 03 - FRANCE https://www.redfroggy.fr

10 replies

LaetitiaRedFroggy Jan 21, 2021
Author

BTW are you working for https://www.babolat.com/? (asking because of the name of the job)

Yes I am

dadoonet Jan 21, 2021
Maintainer

oh what a stupid mistake on my part. I'm so sorry, I was so focused on switching to workplace search that I quickly filled out the file and thought it was a security mismatch ! I'm sorry for the waste of time :-/

No worries. I'm happy we found it!

Yes I am

Great!
I believe I have been using your products for Badminton and Tennis.

Are you based in France?

LaetitiaRedFroggy Jan 21, 2021
Author

Thank you again :-)
Yes I am in France, and I work for Babolat but as a provider.

dadoonet Jan 21, 2021
Maintainer

I'm from France too ;)

I'll be happy to follow up with you on things. Could you drop me an email at david (at) elastic (dot) co ?

LaetitiaRedFroggy Jan 21, 2021
Author

Done ! Thanxxx

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Is it possible to have 2 jobs : one with worplace search output and the second with elastic search output ? #1074

{{title}}

Replies: 4 comments 20 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Is it possible to have 2 jobs : one with worplace search output and the second with elastic search output ? #1074

LaetitiaRedFroggy Jan 15, 2021

Replies: 4 comments · 20 replies

dadoonet Jan 15, 2021 Maintainer

LaetitiaRedFroggy Jan 18, 2021 Author

dadoonet Jan 18, 2021 Maintainer

LaetitiaRedFroggy Jan 18, 2021 Author

dadoonet Jan 18, 2021 Maintainer

LaetitiaRedFroggy Jan 18, 2021 Author

dadoonet Jan 18, 2021 Maintainer

LaetitiaRedFroggy Jan 18, 2021 Author

dadoonet Jan 18, 2021 Maintainer

LaetitiaRedFroggy Jan 19, 2021 Author

dadoonet Jan 19, 2021 Maintainer

LaetitiaRedFroggy Jan 21, 2021 Author

dadoonet Jan 21, 2021 Maintainer

LaetitiaRedFroggy Jan 21, 2021 Author

LaetitiaRedFroggy Jan 21, 2021 Author

dadoonet Jan 21, 2021 Maintainer

LaetitiaRedFroggy Jan 21, 2021 Author

dadoonet Jan 21, 2021 Maintainer

LaetitiaRedFroggy Jan 21, 2021 Author

LaetitiaRedFroggy
Jan 15, 2021

Replies: 4 comments 20 replies

dadoonet
Jan 15, 2021
Maintainer

LaetitiaRedFroggy Jan 18, 2021
Author

dadoonet Jan 18, 2021
Maintainer

LaetitiaRedFroggy Jan 18, 2021
Author

dadoonet Jan 18, 2021
Maintainer

LaetitiaRedFroggy Jan 18, 2021
Author

dadoonet
Jan 18, 2021
Maintainer

LaetitiaRedFroggy Jan 18, 2021
Author

dadoonet Jan 18, 2021
Maintainer

LaetitiaRedFroggy Jan 19, 2021
Author

dadoonet
Jan 19, 2021
Maintainer

LaetitiaRedFroggy Jan 21, 2021
Author

dadoonet Jan 21, 2021
Maintainer

LaetitiaRedFroggy
Jan 21, 2021
Author

LaetitiaRedFroggy Jan 21, 2021
Author

dadoonet Jan 21, 2021
Maintainer

LaetitiaRedFroggy Jan 21, 2021
Author

dadoonet Jan 21, 2021
Maintainer

LaetitiaRedFroggy Jan 21, 2021
Author