Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Testing ES 7.0 #61

Closed
Ark74 opened this issue Apr 28, 2019 · 34 comments

Comments

Projects
None yet
8 participants
@Ark74
Copy link

commented Apr 28, 2019

Environment

  • Nextcloud: 16.0.0.9
  • Full text search: 1.3.1
  • Full text search - Elasticsearch Platform: 1.3.0
  • Full text search - Files: 1.3.0
  • Elasticsearch: 7.0.0

The release of ElasticSearch 7.0 is now here.
https://www.elastic.co/blog/elasticsearch-7-0-0-released

I've build some testing docker images, and so far what I've tested v7.0 is not supported as the previous 6.X (6.4, 6.5, 6.6, 6.7) versions.
https://hub.docker.com/r/ark74/nc_fts
Seems like specific changes need to be integrated.

In case there are plans for supporting v7.0, I can help testing.

Version 6.7.x seems to be a LTS release with support 'til 2020-09-26, so there is plenty of time before it's necessary to migrate.

Cheers!

@Happyfeet01

This comment has been minimized.

Copy link

commented Apr 29, 2019

Maybe the same Error? nextcloud/fulltextsearch#491
Which Version is actually supported?

@Ark74

This comment has been minimized.

Copy link
Author

commented Apr 29, 2019

Not at all.
I have running ES 6.7.1 just fine.

@Happyfeet01

This comment has been minimized.

Copy link

commented Apr 29, 2019

Okay, thanks i make an downgrade and test this

@joergmschulz

This comment has been minimized.

Copy link

commented May 21, 2019

it seems to be worth switching to the recent version as soon as it is supported - we can make use of elasticsearch security options: https://www.zdnet.com/article/some-elasticsearch-security-features-are-now-free-for-everyone/

@Ark74

This comment has been minimized.

Copy link
Author

commented May 23, 2019

Well, "worth" it is. I have no idea how fulltextsearch development is funded and who is paying that "worth".

So, I offer to test it if/when it comes out, but many apps of nextcloud are developed as independent projects so I would understand if we are not bleeding edge on every single stuff.

@daita

This comment has been minimized.

Copy link
Member

commented May 31, 2019

ES7 should be supported with version of fulltextsearch_elasticsearch 1.3.1.

@jkolo

This comment has been minimized.

Copy link

commented May 31, 2019

I tried 1.3.1, and I still have an error as below:

root in ~php/nextcloud via 🐘 v7.3.6 took 2s 
➜ sudo -u nextcloud -H php occ fulltextsearch:test 
Alias tip: _ -u nextcloud -H php occ fulltextsearch:test
 
.Testing your current setup:  
Creating mocked content provider. ok  
Testing mocked provider: get indexable documents. (2 items) ok  
Loading search platform. (Elasticsearch) ok  
Testing search platform. ok  
Locking process ok  
Removing test. ok  
Pausing 3 seconds 1 2 3 ok  
Initializing index mapping. fail  
Error detected, unlocking process ok 
In Connection.php line 620:
                                                                                                                                                                                                                                                                             
  {"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"Root mapping definition has unsupported parameters:  [standard : {dynamic=true, properties={owner={type=keyword}, groups={type=keyword}, circles={type=keyword}, metatags={type=keyword}, source={ty  
  pe=keyword}, title={copy_to=combined, analyzer=keyword, term_vector=yes, type=text}, content={copy_to=combined, analyzer=analyzer, term_vector=yes, type=text}, users={type=keyword}, tags={type=keyword}, provider={type=keyword}, subtags={type=keyword}, links={type=k  
  eyword}, combined={analyzer=analyzer, term_vector=yes, type=text}, hash={type=keyword}}}]"}],"type":"mapper_parsing_exception","reason":"Failed to parse mapping [_doc]: Root mapping definition has unsupported parameters:  [standard : {dynamic=true, properties={owne  
  r={type=keyword}, groups={type=keyword}, circles={type=keyword}, metatags={type=keyword}, source={type=keyword}, title={copy_to=combined, analyzer=keyword, term_vector=yes, type=text}, content={copy_to=combined, analyzer=analyzer, term_vector=yes, type=text}, users  
  ={type=keyword}, tags={type=keyword}, provider={type=keyword}, subtags={type=keyword}, links={type=keyword}, combined={analyzer=analyzer, term_vector=yes, type=text}, hash={type=keyword}}}]","caused_by":{"type":"mapper_parsing_exception","reason":"Root mapping defi  
  nition has unsupported parameters:  [standard : {dynamic=true, properties={owner={type=keyword}, groups={type=keyword}, circles={type=keyword}, metatags={type=keyword}, source={type=keyword}, title={copy_to=combined, analyzer=keyword, term_vector=yes, type=text}, c  
  ontent={copy_to=combined, analyzer=analyzer, term_vector=yes, type=text}, users={type=keyword}, tags={type=keyword}, provider={type=keyword}, subtags={type=keyword}, links={type=keyword}, combined={analyzer=analyzer, term_vector=yes, type=text}, hash={type=keyword}  
  }}]"}},"status":400}                                                                                                                                                                                                                                                       
                                                                                                                                                                                                                                                                             

fulltextsearch:test [--output [OUTPUT]] [-j|--json] [-d|--platform_delay PLATFORM_DELAY
@Ark74

This comment has been minimized.

Copy link
Author

commented May 31, 2019

@daita Working fine with ES 7.1.0 thank you very much!!

curl -XGET "$secret@localhost:9200/"
{
  "name" : "96b7f6981a74",
  "cluster_name" : "docker-cluster",
  "cluster_uuid" : "xj_kHWBoSIqeQKN6-O1gUQ",
  "version" : {
    "number" : "7.1.0",
    "build_flavor" : "oss",
    "build_type" : "docker",
    "build_hash" : "606a173",
    "build_date" : "2019-05-16T00:43:15.323135Z",
    "build_snapshot" : false,
    "lucene_version" : "8.0.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

sudo -u www-data php ~/occ fulltextsearch:test
.Testing your current setup:  
Creating mocked content provider. ok  
Testing mocked provider: get indexable documents. (2 items) ok  
Loading search platform. (Elasticsearch) ok  
Testing search platform. ok  
Locking process ok  
Removing test. ok  
Pausing 3 seconds 1 2 3 ok  
Initializing index mapping. ok  
Indexing generated documents. ok  
Pausing 3 seconds 1 2 3 ok  
Retreiving content from a big index (license). (size: 32386) ok  
Comparing document with source. ok  
Searching basic keywords:  
 - 'test' (result: 1, expected: ["simple"]) ok  
 - 'document is a simple test' (result: 2, expected: ["simple","license"]) ok  
 - '"document is a test"' (result: 0, expected: []) ok  
 - '"document is a simple test"' (result: 1, expected: ["simple"]) ok  
 - 'document is a simple -test' (result: 1, expected: ["license"]) ok  
 - 'document is a simple +test' (result: 1, expected: ["simple"]) ok  
 - '-document is a simple test' (result: 0, expected: []) ok  
Updating documents access. ok  
Pausing 3 seconds 1 2 3 ok  
Searching with group access rights:  
 - 'license' - [] -  (result: 0, expected: []) ok  
 - 'license' - ["group_1"] -  (result: 1, expected: ["license"]) ok  
 - 'license' - ["group_1","group_2"] -  (result: 1, expected: ["license"]) ok  
 - 'license' - ["group_3","group_2"] -  (result: 1, expected: ["license"]) ok  
 - 'license' - ["group_3"] -  (result: 0, expected: []) ok  
Searching with share rights:  
 - 'license' - notuser -  (result: 0, expected: []) ok  
 - 'license' - user2 -  (result: 1, expected: ["license"]) ok  
 - 'license' - user3 -  (result: 1, expected: ["license"]) ok  
Removing test. ok  
Unlocking process ok

This deserves a tweet right?

@Happyfeet01

This comment has been minimized.

Copy link

commented May 31, 2019

it works :-)

@daita

This comment has been minimized.

Copy link
Member

commented Jun 2, 2019

@jkolo would be nice to know more about your setup: version of ES, configurations and please confirm you performed a ./occ fulltextsearch:reset first.

@Happyfeet01

This comment has been minimized.

Copy link

commented Jun 3, 2019

FTS scans the folders and files, but when i search over Nextcloud i have no Searchresult :-/

After Reset i got this Error:

`In Connection.php line 620:

{"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"Root mapping definition has unsupported parameters: [standard : {dynamic=true, pro
perties={owner={type=keyword}, groups={type=keyword}, circles={type=keyword}, metatags={type=keyword}, source={type=keyword}, title={copy_to=combined, a
nalyzer=keyword, term_vector=yes, type=text}, content={copy_to=combined, analyzer=analyzer, term_vector=yes, type=text}, users={type=keyword}, tags={typ
e=keyword}, provider={type=keyword}, subtags={type=keyword}, links={type=keyword}, combined={analyzer=analyzer, term_vector=yes, type=text}, hash={type=
keyword}}}]"}],"type":"mapper_parsing_exception","reason":"Failed to parse mapping [_doc]: Root mapping definition has unsupported parameters: [standar
d : {dynamic=true, properties={owner={type=keyword}, groups={type=keyword}, circles={type=keyword}, metatags={type=keyword}, source={type=keyword}, titl
e={copy_to=combined, analyzer=keyword, term_vector=yes, type=text}, content={copy_to=combined, analyzer=analyzer, term_vector=yes, type=text}, users={ty
pe=keyword}, tags={type=keyword}, provider={type=keyword}, subtags={type=keyword}, links={type=keyword}, combined={analyzer=analyzer, term_vector=yes, t
ype=text}, hash={type=keyword}}}]","caused_by":{"type":"mapper_parsing_exception","reason":"Root mapping definition has unsupported parameters: [standa
rd : {dynamic=true, properties={owner={type=keyword}, groups={type=keyword}, circles={type=keyword}, metatags={type=keyword}, source={type=keyword}, tit
le={copy_to=combined, analyzer=keyword, term_vector=yes, type=text}, content={copy_to=combined, analyzer=analyzer, term_vector=yes, type=text}, users={t
ype=keyword}, tags={type=keyword}, provider={type=keyword}, subtags={type=keyword}, links={type=keyword}, combined={analyzer=analyzer, term_vector=yes,
type=text}, hash={type=keyword}}}]"}},"status":400}
`

@Ark74

This comment has been minimized.

Copy link
Author

commented Jun 3, 2019

@Happyfeet01 this error happens when you have data indexed from previous versions (6.x).
This is a major release change, so I would recommend that you stop & remove the container along with the esdata volume (obviously if you are using docker), and then run again the container (and esdata volume) and index again.

If you use reset when you are on the new release (7.x), there won't be any issue so far I've tested.

About the data not being found it may be related to what I've explained above, I'll report back in a few minutes.

@Happyfeet01

This comment has been minimized.

Copy link

commented Jun 3, 2019

Okay. I try it. Have you an compose file for ES7 with ingest-attachment?

@Ark74

This comment has been minimized.

Copy link
Author

commented Jun 3, 2019

@Happyfeet01 @daita
I can confirm that if you have mixed indexed data (ES 6.x/7.x) you'll have issues,

  • No result in search (even when you have indexed "successfully", note the quotes)
  • Error when resetting fts

so is better to start from scratch on this release.

@enoch85 does the docker update script, removes the data volumes?
For this update, it might need it to be the case, in case people want to upgrade to ES 7.0.

@Happyfeet01 sorry no compose.

@Ark74

This comment has been minimized.

Copy link
Author

commented Jun 3, 2019

nextcloud - 16
fulltextsearch_elasticsearch - 1.3.1
elasticsearch - 7.1.0 (run and index from scratch)

fts-es7

@enoch85

This comment has been minimized.

Copy link
Member

commented Jun 3, 2019

@enoch85 does the docker update script, removes the data volumes?

Yes, we use watchtower in the VM that automatically updates all the dockers images with a --clean tag which removes the old images once the update is sucessfull.

There is also a script called dockerprune.sh which does a system prune once every week, just in case.

@Happyfeet01

This comment has been minimized.

Copy link

commented Jun 4, 2019

@Ark74

This comment has been minimized.

Copy link
Author

commented Jun 4, 2019

Great!, let's see if there are new reports.

@Ark74 Ark74 added the 3. to review label Jun 4, 2019

@enoch85

This comment has been minimized.

@Ark74

This comment has been minimized.

Copy link
Author

commented Jun 5, 2019

@enoch85 does the docker update script, removes the data volumes?
For this update, it might need it to be the case, in case people want to upgrade to ES 7.0.

Remove the docker volume and start index from scratch.

@enoch85

This comment has been minimized.

Copy link
Member

commented Jun 6, 2019

@Ark74 Why so keen so upgrade to ES 7.0? Are there any special futures that we need, or make things faster? Isn't it better to wait until things are stable and you don't have to do manual work? This was on a fresh install afaik.

@jurgenhaas

This comment has been minimized.

Copy link

commented Jun 7, 2019

I'm also having the issue that a new index can not be created where working with an existing index is working just fine on the same instance:

curl http://127.0.0.1:9200/
{
  "name" : "ps1",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "2OaLn25oT2yX0FsWtS_S-Q",
  "version" : {
    "number" : "7.1.1",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "7a013de",
    "build_date" : "2019-05-23T14:04:00.380842Z",
    "build_snapshot" : false,
    "lucene_version" : "8.0.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

Then, when trying to create the index:

sudo -u www-data ./occ fulltextsearch:test
 
.Testing your current setup:  
Creating mocked content provider. ok  
Testing mocked provider: get indexable documents. (2 items) ok  
Loading search platform. (Elasticsearch) ok  
Testing search platform. ok  
Locking process ok  
Removing test. ok  
Pausing 3 seconds 1 2 3 ok  
Initializing index mapping. fail  
Error detected, unlocking process ok 
In Connection.php line 620:
                                                                                                                                                          
  {"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"Root mapping definition has unsupported parameters:  [standard : {dynamic=true, p  
  roperties={owner={type=keyword}, groups={type=keyword}, circles={type=keyword}, metatags={type=keyword}, source={type=keyword}, title={copy_to=combine  
  d, analyzer=keyword, term_vector=yes, type=text}, content={copy_to=combined, analyzer=analyzer, term_vector=yes, type=text}, users={type=keyword}, tag  
  s={type=keyword}, provider={type=keyword}, subtags={type=keyword}, links={type=keyword}, combined={analyzer=analyzer, term_vector=yes, type=text}, has  
  h={type=keyword}}}]"}],"type":"mapper_parsing_exception","reason":"Failed to parse mapping [_doc]: Root mapping definition has unsupported parameters:  
    [standard : {dynamic=true, properties={owner={type=keyword}, groups={type=keyword}, circles={type=keyword}, metatags={type=keyword}, source={type=ke  
  yword}, title={copy_to=combined, analyzer=keyword, term_vector=yes, type=text}, content={copy_to=combined, analyzer=analyzer, term_vector=yes, type=te  
  xt}, users={type=keyword}, tags={type=keyword}, provider={type=keyword}, subtags={type=keyword}, links={type=keyword}, combined={analyzer=analyzer, te  
  rm_vector=yes, type=text}, hash={type=keyword}}}]","caused_by":{"type":"mapper_parsing_exception","reason":"Root mapping definition has unsupported pa  
  rameters:  [standard : {dynamic=true, properties={owner={type=keyword}, groups={type=keyword}, circles={type=keyword}, metatags={type=keyword}, source  
  ={type=keyword}, title={copy_to=combined, analyzer=keyword, term_vector=yes, type=text}, content={copy_to=combined, analyzer=analyzer, term_vector=yes  
  , type=text}, users={type=keyword}, tags={type=keyword}, provider={type=keyword}, subtags={type=keyword}, links={type=keyword}, combined={analyzer=ana  
  lyzer, term_vector=yes, type=text}, hash={type=keyword}}}]"}},"status":400}
@enoch85

This comment has been minimized.

Copy link
Member

commented Jun 7, 2019

@Ark74 Tested the updated scripts yesterday, even with the volume removed and previous index reset it can't re-index.

@daita

This comment has been minimized.

Copy link
Member

commented Jun 7, 2019

it seems that @jurgenhaas is right, I'll have a look

@daita

This comment has been minimized.

Copy link
Member

commented Jun 7, 2019

@jurgenhaas can you try this: #65 ?

@hkienlen

This comment has been minimized.

Copy link

commented Jun 7, 2019

@jurgenhaas can you try this: #65 ?

Hi, I encountered the same issue as @jurgenhaas and applied @daita suggestion
It seems working now

@jurgenhaas

This comment has been minimized.

Copy link

commented Jun 7, 2019

@daita confirmed, that fix works.

@daita daita added 4. to release and removed 3. to review labels Jun 7, 2019

@jkolo

This comment has been minimized.

Copy link

commented Jun 7, 2019

@daita also confirmed, that fix works.

@Ark74

This comment has been minimized.

Copy link
Author

commented Jun 11, 2019

Just tested on a recently upgraded instance.
#65 works just fine :)

@Happyfeet01

This comment has been minimized.

Copy link

commented Jun 11, 2019

why eher, my Container fall back to 6.5.3 .-/

After sudo docker images prune --all and recreating i got this error:

`sshadmin@larsmueller:/opt/docker-ncsearch$ sudo -uwww-data php /var/www/cloud/occ fulltextsearch:test

.Testing your current setup:
Creating mocked content provider. ok
Testing mocked provider: get indexable documents. (2 items) ok
Loading search platform. (Elasticsearch) ok
Testing search platform. ok
Locking process ok
Removing test. fail
Error detected, unlocking process ok
In Connection.php line 666:

{"error":{"root_cause":[{"type":"cluster_block_exception","reason":"blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];"}],"type":"cl
uster_block_exception","reason":"blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];"},"status":503}`

{"type": "server", "timestamp": "2019-06-11T04:57:07,748+0000", "level": "WARN", "component": "o.e.d.SeedHostsResolver", "cluster.name": "ncsearch2", "node.name": "cb52c6930848", "message": "timed out after [5s] resolving host [vm4]" } {"type": "server", "timestamp": "2019-06-11T04:57:07,748+0000", "level": "WARN", "component": "o.e.d.SeedHostsResolver", "cluster.name": "ncsearch2", "node.name": "cb52c6930848", "message": "failed to resolve host [vm5]" , "stacktrace": ["java.net.UnknownHostException: vm5", "at java.net.InetAddress$CachedAddresses.get(InetAddress.java:797) ~[?:?]", "at java.net.InetAddress.getAllByName0(InetAddress.java:1505) ~[?:?]", "at java.net.InetAddress.getAllByName(InetAddress.java:1364) ~[?:?]", "at java.net.InetAddress.getAllByName(InetAddress.java:1298) ~[?:?]", "at org.elasticsearch.transport.TcpTransport.parse(TcpTransport.java:536) ~[elasticsearch-7.1.1.jar:7.1.1]", "at org.elasticsearch.transport.TcpTransport.addressesFromString(TcpTransport.java:489) ~[elasticsearch-7.1.1.jar:7.1.1]", "at org.elasticsearch.transport.TransportService.addressesFromString(TransportService.java:744) ~[elasticsearch-7.1.1.jar:7.1.1]", "at org.elasticsearch.discovery.SeedHostsResolver.lambda$resolveHostsLists$0(SeedHostsResolver.java:143) ~[elasticsearch-7.1.1.jar:7.1.1]", "at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]", "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) ~[elasticsearch-7.1.1.jar:7.1.1]", "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]", "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]", "at java.lang.Thread.run(Thread.java:835) [?:?]"] } {"type": "server", "timestamp": "2019-06-11T04:57:10,745+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "ncsearch2", "node.name": "cb52c6930848", "message": "master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and [cluster.initial_master_nodes] is empty on this node: have discovered []; discovery will continue using [] from hosts providers and [{cb52c6930848}{gLBQfUMgTZ6fUP8Cv0tNmg}{2r50uz7AQo-98UYz6_fGEg}{188.40.63.58}{188.40.63.58:9300}{ml.machine_memory=16745021440, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0" }

@Ark74

This comment has been minimized.

Copy link
Author

commented Jun 11, 2019

You are using: 7.1.1 @Happyfeet01

[elasticsearch-7.1.1.jar:7.1.1]

Have you tried #65 ?

@Happyfeet01

This comment has been minimized.

Copy link

commented Jun 11, 2019

it works !! also updated my Compose file. must change something for ES 7.1.1

added: - discovery.type=single-node

@Ark74

This comment has been minimized.

Copy link
Author

commented Jun 13, 2019

With the release of

  • fulltextsearch 1.3.4
  • fulltextsearch_elasticsearch 1.3.2

seems this testing is complete.

Thanks all for your participation, specially to @daita

@Ark74 Ark74 closed this Jun 13, 2019

@enoch85

This comment has been minimized.

Copy link
Member

commented Jun 14, 2019

I can also confirm that the VM scripts are now working again. Thanks guys, great work!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.