Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Testing] Integration Tests: Fielddata access on the _id field is disallowed #14849

Closed
mtojek opened this issue Nov 28, 2019 · 2 comments
Closed

[Testing] Integration Tests: Fielddata access on the _id field is disallowed #14849

mtojek opened this issue Nov 28, 2019 · 2 comments
Labels
Team:Integrations Label for the Integrations team :Testing

Comments

@mtojek
Copy link
Contributor

mtojek commented Nov 28, 2019

During filebeat integration tests there can be spotted occurrences of java.lang.IllegalArgumentException, all related to Elasticsearch.

It might be related to one of last changes in Elasticsearch:
PR: elastic/elasticsearch#49166
Issue: elastic/elasticsearch#43599

Problems have been spotted since revision: docker.elastic.co/elasticsearch/elasticsearch:8.0.0-SNAPSHOT or b12aa69d54f4e2028f1f872a8bc59e939af742002cf1220e99ba1027c367d4cb

Elasticsearch logs:

task_manager_1][0], node[ZCucT673R5iUWLcg6PP4Yw], [P], s[STARTED], a[id=qohTIHwhRrm0RzWBHy_9YQ]: Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[.kibana_task_manager], indicesOptions=In
dicesOptions[ignore_unavailable=true, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false
, ignore_throttled=true], routing='null', preference='null', requestCache=null, scroll=null, maxConcurrentShardRequests=0, batchedReduceSize=512, preFilterShardSize=128, allowPartialSearchResults=true, localC
lusterAlias=null, getOrCreateAbsoluteStartMillis=-1, ccsMinimizeRoundtrips=true, source={\"query\":{\"bool\":{\"must\":[{\"term\":{\"type\":{\"value\":\"task\",\"boost\":1.0}}},{\"bool\":{\"filter\":[{\"term\
":{\"_id\":{\"value\":\"task:Lens-lens_telemetry\",\"boost\":1.0}}}],\"adjust_pure_negative\":true,\"boost\":1.0}}],\"adjust_pure_negative\":true,\"boost\":1.0}},\"sort\":[{\"task.runAt\":{\"order\":\"asc\"}}
,{\"_id\":{\"order\":\"desc\"}}]}}]", "cluster.uuid": "uWOVpf9DRm2j9LU_QeKYGA", "node.id": "ZCucT673R5iUWLcg6PP4Yw" ,
"stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [27183687308f][127.0.0.1:9300][indices:data/read/search[phase/query]]",
"Caused by: java.lang.IllegalArgumentException: Fielddata access on the _id field is disallowed, you can re-enable it by updating the dynamic cluster setting: indices.id_field_data.enabled",
"at org.elasticsearch.index.mapper.IdFieldMapper$IdFieldType$1.build(IdFieldMapper.java:172) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.index.fielddata.IndexFieldDataService.getForField(IndexFieldDataService.java:134) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.index.query.QueryShardContext.getForField(QueryShardContext.java:194) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.search.sort.FieldSortBuilder.build(FieldSortBuilder.java:358) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.search.sort.SortBuilder.buildSort(SortBuilder.java:146) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.search.SearchService.parseSource(SearchService.java:764) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.search.SearchService.createContext(SearchService.java:589) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:548) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:351) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.search.SearchService.lambda$executeQueryPhase$1(SearchService.java:343) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.action.ActionListener.lambda$map$2(ActionListener.java:146) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:58) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:73) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:688) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
"at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
"at java.lang.Thread.run(Thread.java:830) [?:?]"] }

and:

"stacktrace": ["org.elasticsearch.ElasticsearchException$1: Fielddata access on the _id field is disallowed, you can re-enable it by updating the dynamic cluster setting: indices.id_field_data.enabled",
"at org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:644) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:303) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:563) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:384) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:241) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:423) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1059) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1168) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1142) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:60) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.action.support.ChannelActionListener.onFailure(ChannelActionListener.java:56) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.action.ActionListener$1.onFailure(ActionListener.java:71) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:65) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:58) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:73) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:688) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
"at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
"at java.lang.Thread.run(Thread.java:830) [?:?]",
"Caused by: java.lang.IllegalArgumentException: Fielddata access on the _id field is disallowed, you can re-enable it by updating the dynamic cluster setting: indices.id_field_data.enabled",
"at org.elasticsearch.index.mapper.IdFieldMapper$IdFieldType$1.build(IdFieldMapper.java:172) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.index.fielddata.IndexFieldDataService.getForField(IndexFieldDataService.java:134) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.index.query.QueryShardContext.getForField(QueryShardContext.java:194) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.search.sort.FieldSortBuilder.build(FieldSortBuilder.java:358) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.search.sort.SortBuilder.buildSort(SortBuilder.java:146) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.search.SearchService.parseSource(SearchService.java:764) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.search.SearchService.createContext(SearchService.java:589) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:548) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:351) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.search.SearchService.lambda$executeQueryPhase$1(SearchService.java:343) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.action.ActionListener.lambda$map$2(ActionListener.java:146) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"... 9 more"] }

(It might be related) For some reason the container with Elasticsearch went down:

                   {
                        "Start": "2019-11-28T18:16:00.4747798Z",
                        "End": "2019-11-28T18:16:01.4025842Z",
                        "ExitCode": 0,
                        "Output": "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n                                 Dload  Upload   Total   Spent    Left  Speed\n\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r100   543  100   543    0     0  20322      0 --:--:-- --:--:-- --:--:-- 22625\n{\n  \"name\" : \"27183687308f\",\n  \"cluster_name\" : \"docker-cluster\",\n  \"cluster_uuid\" : \"uWOVpf9DRm2j9LU_QeKYGA\",\n  \"version\" : {\n    \"number\" : \"8.0.0-SNAPSHOT\",\n    \"build_flavor\" : \"default\",\n    \"build_type\" : \"docker\",\n    \"build_hash\" : \"d596585cd8a3ece712ecd98df57bebd7dc444f29\",\n    \"build_date\" : \"2019-11-28T09:34:47.236026Z\",\n    \"build_snapshot\" : true,\n    \"lucene_version\" : \"8.4.0\",\n    \"minimum_wire_compatibility_version\" : \"7.6.0\",\n    \"minimum_index_compatibility_version\" : \"7.0.0\"\n  },\n  \"tagline\" : \"You Know, for Search\"\n}\n"
                    },
                    {
                        "Start": "2019-11-28T18:16:02.432609Z",
                        "End": "2019-11-28T18:16:03.359206Z",
                        "ExitCode": 0,
                        "Output": "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n                                 Dload  Upload   Total   Spent    Left  Speed\n\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r100   543  100   543    0     0   7710      0 --:--:-- --:--:-- --:--:--  7985\n{\n  \"name\" : \"27183687308f\",\n  \"cluster_name\" : \"docker-cluster\",\n  \"cluster_uuid\" : \"uWOVpf9DRm2j9LU_QeKYGA\",\n  \"version\" : {\n    \"number\" : \"8.0.0-SNAPSHOT\",\n    \"build_flavor\" : \"default\",\n    \"build_type\" : \"docker\",\n    \"build_hash\" : \"d596585cd8a3ece712ecd98df57bebd7dc444f29\",\n    \"build_date\" : \"2019-11-28T09:34:47.236026Z\",\n    \"build_snapshot\" : true,\n    \"lucene_version\" : \"8.4.0\",\n    \"minimum_wire_compatibility_version\" : \"7.6.0\",\n    \"minimum_index_compatibility_version\" : \"7.0.0\"\n  },\n  \"tagline\" : \"You Know, for Search\"\n}\n"
                    },
                    {
                        "Start": "2019-11-28T18:16:04.4084051Z",
                        "End": "2019-11-28T18:16:57.1973408Z",
                        "ExitCode": -1,
                        "Output": "Health check exceeded timeout (30s)"
                    }
@mtojek mtojek changed the title Integration Tests: Fielddata access on the _id field is disallowed [Testing] Integration Tests: Fielddata access on the _id field is disallowed Nov 28, 2019
@mtojek
Copy link
Contributor Author

mtojek commented Nov 28, 2019

I think I identified the issue. With option indices.id_field_data.enabled set to true:
https://travis-ci.org/elastic/beats/builds/618318255?utm_source=github_status&utm_medium=notification (build successful)

without:
https://travis-ci.org/elastic/beats/jobs/618338082?utm_medium=notification&utm_source=github_status (build failed)

@mtojek mtojek added the Team:Integrations Label for the Integrations team label Nov 28, 2019
@mtojek mtojek assigned mtojek and unassigned mtojek Nov 28, 2019
@mtojek mtojek mentioned this issue Nov 28, 2019
Merged
@ChrsMark ChrsMark mentioned this issue Nov 28, 2019
Merged
@mtojek
Copy link
Contributor Author

mtojek commented Nov 28, 2019

I've merge the PR. Resolving.

@mtojek mtojek closed this as completed Nov 28, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Team:Integrations Label for the Integrations team :Testing
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@mtojek