Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

KQL removes leading zero and breaks query #62748

Merged
merged 5 commits into from
Apr 24, 2020
Merged

KQL removes leading zero and breaks query #62748

merged 5 commits into from
Apr 24, 2020

Conversation

alexwizp
Copy link
Contributor

@alexwizp alexwizp commented Apr 7, 2020 •
edited
Loading

Closes: #62217

Summary

Describe the bug:
KQL seems to silently remove leading zeros causing queries to fail.

When querying "customer: 00123" with Lucene we get results but KQL removes the leading zeros:
image
image

"filter":[{"bool":{"should":[{"match":{"customer":12282650}}]...

Lucene Query generates:
image

{"bool":{"must":[{"query_string":{"query":"customer: 0012282650","analyze_wildcard":true,"time_zone"...

Checklist

Delete any items that are not applicable to this PR.

For maintainers

@alexwizp alexwizp added this to In progress in kibana-app-arch via automation Apr 7, 2020
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-app-arch (Team:AppArch)

@alexwizp alexwizp added release_note:fix bug Fixes for quality problems that affect the customer experience labels Apr 7, 2020
@alexwizp alexwizp self-assigned this Apr 7, 2020
@alexwizp alexwizp requested a review from lukasolson April 7, 2020 13:02
@alexwizp alexwizp marked this pull request as ready for review April 7, 2020 13:02
@alexwizp alexwizp requested a review from a team as a code owner April 7, 2020 13:02
@alexwizp
Copy link
Contributor Author

alexwizp commented Apr 8, 2020

@elasticmachine merge upstream

3 similar comments
@alexwizp
Copy link
Contributor Author

@elasticmachine merge upstream

@alexwizp
Copy link
Contributor Author

@elasticmachine merge upstream

@alexwizp
Copy link
Contributor Author

@elasticmachine merge upstream

lukasolson
lukasolson reviewed Apr 13, 2020
View reviewed changes
Copy link
Member

@lukasolson lukasolson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, so at first glance I didn't like the following about this approach:

  • It treats numbers beginning with 0 differently than other numbers
  • It treats numbers beginning with . (no preceding 0 or -0) differently than other numbers

When playing around on this PR, I realized that when indexing a document into Elasticsearch, it errors out if you try to do either of the above. So I'm not super concerned with either of these things (other than the fact that the regex is a bit complicated and could be simplified with some .startsWith calls instead).

After some thought and testing, however, I think it's actually okay to always treat any numbers as strings. All of the queries I tried (including is or range queries) seem to work fine if we send to Elasticsearch as strings instead of numbers.

@alexwizp Thoughts about that approach?

If we take this approach, it will also solve this issue: #48835

I believe the number handling was just copied over from an existing grammar (I believe it was Timelion) and there wasn't any particular reason we needed to process them as numbers, rather than strings.

@alexwizp
Copy link
Contributor Author

@elasticmachine merge upstream

@alexwizp
Copy link
Contributor Author

alexwizp commented Apr 14, 2020 •
edited
Loading

@lukasolson agree, if we can pass string without any casting, we should definitely do it. PR was updated

@alexwizp
Copy link
Contributor Author

@elasticmachine merge upstream

@lukasolson lukasolson moved this from In progress to Review in progress in kibana-app-arch Apr 15, 2020
@alexwizp alexwizp requested a review from a team as a code owner April 16, 2020 16:26
@alexwizp
Copy link
Contributor Author

@elasticmachine merge upstream

@rylnd
Copy link
Contributor

rylnd commented Apr 21, 2020

Hey @alexwizp ! Could you clarify what you need review from SIEM on, here? Is it just our use of the KQL bar? Thanks!

@alexwizp alexwizp removed the request for review from a team April 21, 2020 18:00
@alexwizp
Copy link
Contributor Author

@rylnd sorry, needs a review only from AppArch team. Hope @lukasolson will help me with that.

@alexwizp
Copy link
Contributor Author

@elasticmachine merge upstream

@alexwizp
Copy link
Contributor Author

@elasticmachine merge upstream

@kibanamachine
Copy link
Contributor

💚 Build Succeeded

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@alexwizp alexwizp merged commit 321430e into elastic:master Apr 24, 2020
kibana-app-arch automation moved this from Review in progress to Done in current release Apr 24, 2020
alexwizp added a commit to alexwizp/kibana that referenced this pull request Apr 24, 2020
@alexwizp @elasticmachine
KQL removes leading zero and breaks query (elastic#62748)
a44e946 
* KQL removes leading zero and breaks query

* correctly parse numbers started from '.'

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
@alexwizp alexwizp mentioned this pull request Apr 24, 2020
Merged
gmmorris added a commit to gmmorris/kibana that referenced this pull request Apr 24, 2020
@gmmorris
Merge branch 'master' into alerting/no-explicit-any
7b89039 
* master: (70 commits)
  KQL removes leading zero and breaks query (elastic#62748)
  [FieldFormats] Cleanup: rename IFieldFormatType -> FieldFormatInstanceType (elastic#64193)
  [ML] Changes transforms wizard UI text (elastic#64150)
  [Alerting] change server log action type .log to .server-log in README (elastic#64124)
  [Metrics UI] Design Refresh: Inventory View, Episode 1 (elastic#64026)
  chore(NA): reduce siem bundle size using babel-plugin-transfor… (elastic#63269)
  chore(NA): use core-js instead of babel-polyfill on canvas sha… (elastic#63486)
  skip flaky suite (elastic#61173)
  skip flaky suite (elastic#62497)
  Renamed ilm policy for event log so it is not prefixed with dot (elastic#64262)
  [eslint] no_restricted_paths config cleanup (elastic#63741)
  Add Oil Rig Icon from @elastic/maki (elastic#64364)
  [Maps] Migrate Maps embeddables to NP (elastic#63976)
  [Ingest] Data streams list page (elastic#64134)
  chore(NA): add file-loader into jest moduleNameMapper (elastic#64330)
  [DOCS] Added images to automating report generation (elastic#64333)
  [SIEM][CASE] Api Integration Tests: Configuration (elastic#63948)
  Expose ability to check if API Keys are enabled (elastic#63454)
  [DOCS] Fixes formatting in alerting doc (elastic#64338)
  [data.search.aggs]: Create agg types function for terms agg. (elastic#63541)
  ...
alexwizp added a commit that referenced this pull request Apr 24, 2020
@alexwizp @elasticmachine
KQL removes leading zero and breaks query (#62748) (#64400)
6d2097e 
* KQL removes leading zero and breaks query

* correctly parse numbers started from '.'

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
jloleysens added a commit to jloleysens/kibana that referenced this pull request Apr 27, 2020
@jloleysens
Merge branch 'feature/ingest-node-pipelines' of github.com:elastic/ki…
42247e2 
…bana into ingest-node-pipeline/open-flyout-create-edit

* 'feature/ingest-node-pipelines' of github.com:elastic/kibana: (116 commits)
  [Ingest Node Pipelines] More lenient treatment of on-failure value (elastic#64411)
  Report Deletion via UI- functional test (elastic#64031)
  Bump handlebars dependency from 4.5.3 to 4.7.6 (elastic#64402)
  [Uptime] Update TLS settings (elastic#64111)
  [alerting] removes usage of any throughout Alerting Services code (elastic#64161)
  [CANVAS] Moves notify to a canvas service (elastic#63268)
  [Canvas] Misc NP Stuff (elastic#63703)
  update apm index pattern (elastic#64232)
  Task/hostlist pagination (elastic#63722)
  [NP] Vega migration (elastic#63849)
  Move ensureDefaultIndexPattern into data plugin (elastic#63100)
  [Fleet] Fix agent status count to not include unenrolled agents (elastic#64106)
  Migrate graph_workspace saved object registration to Kibana platform (elastic#64157)
  Index pattern management UI -> TypeScript and New Platform Ready (edit_index_pattern) (elastic#64184)
  [ML] EuiDataGrid ml/transform components. (elastic#63447)
  [ML] Moving to kibana capabilities (elastic#64057)
  Move input_control_vis into NP (elastic#63333)
  remove reference to local application service in graph (elastic#64288)
  KQL removes leading zero and breaks query (elastic#62748)
  [FieldFormats] Cleanup: rename IFieldFormatType -> FieldFormatInstanceType (elastic#64193)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lukasolson lukasolson lukasolson approved these changes

@VladLasitsa VladLasitsa Awaiting requested review from VladLasitsa

Assignees

@alexwizp alexwizp

Labels
bug Fixes for quality problems that affect the customer experience Feature:KQL KQL release_note:fix review v7.8.0 v8.0.0
Projects
kibana-app-arch
  
Done in current release
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

KQL removes leading zero and breaks query
6 participants
@alexwizp @elasticmachine @rylnd @kibanamachine @lukasolson @rayafratkina