[Ingest Manager] Split index restrictions into type,dataset, namespace parts #21406
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
michalpristas
added
Team:Ingest Management
Ingest Management:beta2
|
Pinging @elastic/ingest-management (Team:Ingest Management) |
botelastic
bot
added
needs_team
💚 Build Succeeded
Expand to view the summary
Build stats
Test stats 🧪
|
ruflin
approved these changes
Sep 30, 2020
There was a problem hiding this comment.
Left some nits, but change LGTM.
Two things which can follow later:
- We need docs to explain these limitations. @jen-huang has opened a PR for the docs for it. For now I would duplicate it but perhaps one day we have a place where we document the indexing strategy and can make it part of it
- Error message: At the moment, the check only returns true or false. Instead it would be nice to return an error we can also show the user like: namespace is too long, invalid char etc.
I guess this also needs a changelog entry.
| // The only two requirement are that it has only characters allowed in an Elasticsearch index name | ||
| // Index names must meet the following criteria: | ||
| // Not longer than 100 bytes |
There was a problem hiding this comment.
Nit: Indentation seems to be off
| return false | ||
| } | ||
|
|
||
| // Cannot include \, /, *, ?, ", <, >, |, ` ` (space character), ,, # | ||
| if strings.ContainsAny(namespace, "\\/*?\"<>| ,#") { | ||
| if strings.ContainsAny(dsType, "\\/*?\"<>| ,#:") { |
There was a problem hiding this comment.
Could we reuse the logic from dataset / namespace here to make sure it is always in sync?
| return false | ||
| } | ||
|
|
||
| if strings.ContainsAny(dataset, "\\/*?\"<>| ,#:") { |
There was a problem hiding this comment.
See above.
It looks like all 3 methods are identical besides the length limit?
michalpristas
added a commit
to michalpristas/beats
that referenced
this pull request
Oct 2, 2020
…e parts (elastic#21406) [Ingest Manager] Split index restrictions into type,dataset, namespace parts (elastic#21406)
Merged
6 tasks
v1v
added a commit
to v1v/beats
that referenced
this pull request
Oct 2, 2020
* upstream/master: (27 commits) [Ingest Manager] Split index restrictions into type,dataset, namespace parts (elastic#21406) Update Filebeat module expected logs files (elastic#21454) Edit SQL module docs and fix broken doc structure (elastic#21233) [Ingest Manager] Send snapshot flag together with metadata (elastic#21285) Revert "[JJBB] Set shallow cloning to 10 (elastic#21409)" (elastic#21447) [JJBB] Use reference repo for fast checkouts (elastic#21410) Add initial skeleton of filestream input (elastic#21427) Initial spec file for apm-server (elastic#21225) [Ingest Manager] Upgrade Action: make source URI optional (elastic#21372) Add field limit check for AWS Cloudtrail flattened fields (elastic#21388) [Winlogbeat] Move winlogbeat javascript processor to libbeat (elastic#21402) ci: pipeline to generate the changelog (elastic#21426) [JJBB] Set shallow cloning to 10 (elastic#21409) docs: add link to release notes for 7.9.2 (elastic#21405) (elastic#21419) docs: Prepare Changelog for 7.9.2 (elastic#21229) (elastic#21403) fix: mark flaky tests (elastic#21300) fix: use a fixed version of setuptools (elastic#21393) Move Kubernetes events metricset to its own block in reference config (elastic#21407) [libbeat] Enable WriteAheadLimit in the disk queue (elastic#21391) docs: fix apt/yum formatting (elastic#21362) ...
v1v
added a commit
to v1v/beats
that referenced
this pull request
Oct 2, 2020
…ne-2.0-arm * upstream/master: (54 commits) [CI] Change x-pack/auditbeat build events (comments, labels) (elastic#21463) [CI] changeset from elastic#20603 was not added to CI2.0 (elastic#21464) Add new log file reader for filestream input (elastic#21450) [CI] Send slack message with build status (elastic#21428) Remove duplicated sources url in dependencies report (elastic#21462) Add implementation of FSWatcher and FSScanner for filestream (elastic#21444) [Ingest Manager] Split index restrictions into type,dataset, namespace parts (elastic#21406) Update Filebeat module expected logs files (elastic#21454) Edit SQL module docs and fix broken doc structure (elastic#21233) [Ingest Manager] Send snapshot flag together with metadata (elastic#21285) Revert "[JJBB] Set shallow cloning to 10 (elastic#21409)" (elastic#21447) [JJBB] Use reference repo for fast checkouts (elastic#21410) Add initial skeleton of filestream input (elastic#21427) Initial spec file for apm-server (elastic#21225) [Ingest Manager] Upgrade Action: make source URI optional (elastic#21372) Add field limit check for AWS Cloudtrail flattened fields (elastic#21388) [Winlogbeat] Move winlogbeat javascript processor to libbeat (elastic#21402) ci: pipeline to generate the changelog (elastic#21426) [JJBB] Set shallow cloning to 10 (elastic#21409) docs: add link to release notes for 7.9.2 (elastic#21405) (elastic#21419) ...
v1v
added a commit
to v1v/beats
that referenced
this pull request
Oct 2, 2020
…ci-build-label-support * upstream/master: [CI] Change x-pack/auditbeat build events (comments, labels) (elastic#21463) [CI] changeset from elastic#20603 was not added to CI2.0 (elastic#21464) Add new log file reader for filestream input (elastic#21450) [CI] Send slack message with build status (elastic#21428) Remove duplicated sources url in dependencies report (elastic#21462) Add implementation of FSWatcher and FSScanner for filestream (elastic#21444) [Ingest Manager] Split index restrictions into type,dataset, namespace parts (elastic#21406) Update Filebeat module expected logs files (elastic#21454) Edit SQL module docs and fix broken doc structure (elastic#21233) [Ingest Manager] Send snapshot flag together with metadata (elastic#21285) Revert "[JJBB] Set shallow cloning to 10 (elastic#21409)" (elastic#21447) [JJBB] Use reference repo for fast checkouts (elastic#21410) Add initial skeleton of filestream input (elastic#21427) Initial spec file for apm-server (elastic#21225) [Ingest Manager] Upgrade Action: make source URI optional (elastic#21372) Add field limit check for AWS Cloudtrail flattened fields (elastic#21388) [Winlogbeat] Move winlogbeat javascript processor to libbeat (elastic#21402) ci: pipeline to generate the changelog (elastic#21426)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
This PR does not apply same restrictions set on all parts of resulting index and follows 20/100/100 bytes length requirement for type, dataset and namespace
Why is it important?
Makes rules in par with the ones applied at fleet level
Checklist
CHANGELOG.next.asciidocorCHANGELOG-developer.next.asciidoc.