OAK-9686: replace Elastic RHLC with Elastic Java Client #490

AngelaFabregues · 2022-02-10T16:24:30Z

This is the elasticserach client migration approach that I'm following. Two clients share the same transport layer.

AngelaFabregues · 2022-02-10T18:27:57Z

I've increased the max jar file from run-elastic to try to pass the checks. Now to 120MB.

...elastic/src/main/java/org/apache/jackrabbit/oak/plugins/index/elastic/ElasticConnection.java

...astic/src/main/java/org/apache/jackrabbit/oak/plugins/index/elastic/ElasticIndexCleaner.java

…elasticknn).

…9686 # Conflicts: # oak-search-elastic/src/main/java/org/apache/jackrabbit/oak/plugins/index/elastic/index/ElasticIndexHelper.java

* trunk: OAK-9704 Fix Utils#getConnectionString OAK-9704 AzureBlobStoreBackend: empty string as null in boolean property OAK-9690 | Adding bringIndexUptoDate support for elastic index post offline reindexing (apache#492) OAK-9699 Document Elastic Indexes OAK-9701 : Additional restrictions to simplify permission setup OAK-9700: RevisionGC may fail with NPE OAK-9700: RevisionGC may fail with NPE OAK-9700: RevisionGC may fail with NPE OAK-9694: expose elastic stats in oak-index-stats and mbean (apache#498) Update oak-doc/src/site/markdown/query/elastic.md OAK-9699 Document Elastic Indexes OAK-9699 Document Elastic Indexes Update oak-doc/src/site/markdown/query/elastic.md Update oak-doc/src/site/markdown/query/elastic.md OAK-9699 Document Elastic Indexes # Conflicts: # oak-search-elastic/src/main/java/org/apache/jackrabbit/oak/plugins/index/elastic/ElasticIndexStatistics.java

…bit-oak into clean-OAK-9686 * 'OAK-9686' of https://github.com/AngelaFabregues/jackrabbit-oak: (58 commits) OAK-9686 - Executes a single elasticsearch connection to use for all the tests. OAK-9686 - Fixing spellchecker OAK-9686 - Fixing collate boolean query. (minor) removed unused code + small improvements get rid of operations using the low level client ElasticIndexHelper2: added details on why we cannot still migrate to the new client use UTF-8 to serialize SearchRequest to strings (fixes a couple of test failures) fix suggest queries use the same version for both es clients (7.17.2) ElasticFunctionIndexCommonTest: reintroduced needed overrides (minor) style improvements in similarityQuery remove XContentBuilder from ElasticRequestHandler fixed baseQuery logic fixed ElasticPropertyIndexTest fixed range queries use java client for ElasticIndexStatisticsTest fixed similarity queries (minor) improved formatting of ElasticRequestHandler fixed code OAK-9686 - Removing unnecessary overwrittes. OAK-9686 - Refactoring aggregations ... # Conflicts: # oak-search-elastic/src/test/java/org/apache/jackrabbit/oak/plugins/index/elastic/ElasticConnectionRule.java

…ements

…into OAK-9686

...elastic/src/main/java/org/apache/jackrabbit/oak/plugins/index/elastic/ElasticConnection.java

.../java/org/apache/jackrabbit/oak/plugins/index/elastic/index/ElasticBulkProcessorHandler.java

...elastic/src/main/java/org/apache/jackrabbit/oak/plugins/index/elastic/ElasticConnection.java

nit0906 · 2022-05-12T17:16:23Z

[INFO] --- maven-bundle-plugin:5.1.1:baseline (baseline) @ oak-search-elastic ---
[INFO] artifact org.apache.jackrabbit:oak-search-elastic: checking for updates from Nexus
[INFO] artifact org.apache.jackrabbit:oak-search-elastic: checking for updates from central
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  30:33 min
[INFO] Finished at: 2022-05-12T14:28:57
[INFO] ------------------------------------------------------------------------
[ERROR] GC overhead limit exceeded -> [Help 1]

The jenkins build on the PR for oak-search-elastic seems to be failing with GC overhead limit exceeded - we should probably look into this.

fabriziofortino · 2022-05-13T07:31:27Z

[ERROR] GC overhead limit exceeded -> [Help 1]
The jenkins build on the PR for oak-search-elastic seems to be failing with GC overhead limit exceeded - we should probably look into this.

@nit0906 I have seen this but it's not clear what triggers it. I would expect an OOM when the tests are executed. The OOM happens later though. Can we enable full debug on jenkins to see if that helps in understanding where this comes from?

nit0906 · 2022-05-13T07:51:58Z

@fabriziofortino

Can we enable full debug on jenkins to see if that helps in understanding where this comes from?

This seems to be some apache ci jenkins - I am not entirely sure how to change the debug logging, but it seems it runs from this jenkins file - https://github.com/apache/jackrabbit-oak/blob/trunk/Jenkinsfile

I think if we add the -X option , in the mvn clean verify command, it should log at debug level. But I think we might need to change that in the trunk branch (changing in the feature branch wouldn't work I guess since the job https://ci-builds.apache.org/job/Jackrabbit/job/oak-trunk-pr/configure here is picking from the trunk of apache repo.)

fabriziofortino · 2022-05-13T09:44:36Z

@nit0906 the problem was with the bundle. I have fine-tuned the dependencies (we can get rid of all lucene dependencies when the migration to ES Java Client is complete) and the build is now successful.

…-oak into OAK-9686 * 'trunk' of https://github.com/AngelaFabregues/jackrabbit-oak: OAK-9755 | Lucene metrics using labels (apache#565) change to info level remove duplicate fix Path Predicate OAK-9758 error out if tika dependencies are missing and improve logging to log warning instead of debug silently

angelafabreguesv and others added 4 commits February 9, 2022 21:26

OAK-9686 - Rename the old Elasticsearch client variables

f672034

OAK-9686 - Start the migration to the new Elasticsearch Client

ac9c121

Merge branch 'apache:trunk' into OAK-9686

c436330

OAK-9686 - Increase max jar size for run elastic

555ae63

angelafabreguesv added 2 commits February 10, 2022 22:13

OAK-9686 - Increase max jar size for run elastic

1c96ea4

OAK-9686 - Migration to the new Elasticsearch Client

4e595d8

thomasmueller requested changes Feb 11, 2022

View reviewed changes

angelafabreguesv and others added 22 commits February 14, 2022 13:05

Merged trunk

aed4aa3

Merge branch 'apache:trunk' into OAK-9686

1a24e31

OAK-9686 - Elasticsearch client renaming.

d255107

OAK-9686 - Enabling async client

a07776f

OAK-9686 - ElasticIndexCleaner migrated to the new client.

42f097c

OAK-9686 - ElasticIndexHelper migrated to the new client (except for …

0706bb2

…elasticknn).

OAK-9686 - ElasticIndexHelper migrated to the new client (except for …

066e51a

…elasticknn).

Merge branch 'apache:trunk' into OAK-9686

ef78174

Merge remote-tracking branch 'refs/remotes/origin/OAK-9686' into OAK-…

9b6cd0e

…9686 # Conflicts: # oak-search-elastic/src/main/java/org/apache/jackrabbit/oak/plugins/index/elastic/index/ElasticIndexHelper.java

Merge branch 'apache:trunk' into OAK-9686

8ce476f

OAK-9686 - ElasticIndexStatistics client migration.

00af3f7

OAK-9686 - Back to the high level client for ElasticIndexHelper.

97201ab

OAK-9686 - migration

c77cf4f

OAK-9686 - migration

40f383a

OAK-9686 - migration to 7.17.0

1dc8c16

OAK-9686 - migration to 7.17.0

7cb3c47

OAK-9686 - migration to 7.17.0

ea715a6

OAK-9686 - migration to 7.17.0

fb8fb95

no message

84f6efb

OAK-9686 - Execution enabled

8c657cf

merge trunk

c5c404a

angelafabreguesv and others added 11 commits May 5, 2022 18:00

Merge branch 'apache:trunk' into OAK-9686

5e3ccd4

fix plugin version (fixes similarity tests) + minor code style improv…

01d57ff

…ements

Merge remote-tracking branch 'origin/OAK-9686' into OAK-9686

fab1a78

improved spellcheck

1b6c7a0

bump es to 7.17.3

f503e5f

Merge branch 'apache:trunk' into OAK-9686

014aff9

fix size history comment in oak-run-elastic pom

9df1d87

Merge branch 'OAK-9686' of github.com:AngelaFabregues/jackrabbit-oak …

3604a6a

…into OAK-9686

ElasticIndexWriter: reduce dependency from RHLC

a16d483

last touches: cleanup + adjusted formatting

b7e45e9

fabriziofortino marked this pull request as ready for review May 12, 2022 13:52

fabriziofortino requested review from nit0906, tihom88 and thomasmueller May 12, 2022 13:52

nit0906 reviewed May 12, 2022

View reviewed changes

...elastic/src/main/java/org/apache/jackrabbit/oak/plugins/index/elastic/ElasticConnection.java Outdated Show resolved Hide resolved

nit0906 reviewed May 12, 2022

View reviewed changes

...elastic/src/main/java/org/apache/jackrabbit/oak/plugins/index/elastic/ElasticConnection.java Outdated Show resolved Hide resolved

nit0906 reviewed May 12, 2022

View reviewed changes

.../java/org/apache/jackrabbit/oak/plugins/index/elastic/index/ElasticBulkProcessorHandler.java Show resolved Hide resolved

nit0906 reviewed May 12, 2022

View reviewed changes

...elastic/src/main/java/org/apache/jackrabbit/oak/plugins/index/elastic/ElasticConnection.java Outdated Show resolved Hide resolved

fabriziofortino added 2 commits May 13, 2022 09:25

pom: remove obsolete imported packages

4d35c39

improved connection handling

b8106e8

pom: fine tune dependencies

1320a85

fabriziofortino requested a review from nit0906 May 13, 2022 11:30

fabriziofortino and others added 2 commits May 13, 2022 14:49

pom: fix import packages

7a7adda

AngelaFabregues closed this May 16, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OAK-9686: replace Elastic RHLC with Elastic Java Client #490

OAK-9686: replace Elastic RHLC with Elastic Java Client #490

AngelaFabregues commented Feb 10, 2022

AngelaFabregues commented Feb 10, 2022 •

edited

nit0906 commented May 12, 2022 •

edited

fabriziofortino commented May 13, 2022

nit0906 commented May 13, 2022 •

edited

fabriziofortino commented May 13, 2022

OAK-9686: replace Elastic RHLC with Elastic Java Client #490

OAK-9686: replace Elastic RHLC with Elastic Java Client #490

Conversation

AngelaFabregues commented Feb 10, 2022

AngelaFabregues commented Feb 10, 2022 • edited

nit0906 commented May 12, 2022 • edited

fabriziofortino commented May 13, 2022

nit0906 commented May 13, 2022 • edited

fabriziofortino commented May 13, 2022

AngelaFabregues commented Feb 10, 2022 •

edited

nit0906 commented May 12, 2022 •

edited

nit0906 commented May 13, 2022 •

edited