feat(elasticsearch): allow bulk delete #8424
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
david-leifker
requested review from
pedro93,
anshbansal,
RyanHolstien and
iprentic
github-actions
bot
added
product
anshbansal
approved these changes
Jul 31, 2023
yoonhyejin
pushed a commit
that referenced
this pull request
Aug 24, 2023
spadhi7
added a commit
to spadhi7/datahub
that referenced
this pull request
Aug 29, 2023
* tag 'v0.10.5': (222 commits) fix(test): increase siblings.js test stability (datahub-project#8542) feat(search): Allow aggregating on facets that are not explicitly part of default filter set (datahub-project#8540) fix(ui) Make multiple small updates to new search and browse (datahub-project#8524) feat(presto-on-hive): allow v1 fieldpaths in the presto-on-hive source (datahub-project#8474) feat(cli): Adds ability to upload recipes to DataHub's UI (datahub-project#8317) feat(browseV2): add browseV2 logic to system update (datahub-project#8506) fix(ingest/json-schema): convert non-string enums to strings (datahub-project#8479) feat(ingestion/tableau): support column level lineage for custom sql (datahub-project#8466) test(ingest): test case statements with sql parser (datahub-project#8437) feat(ingest/vertica): performance improvement and bug fixes (datahub-project#8328) ci: reduce git fetch depth (datahub-project#8473) fix(ingest): remove duplication of tags (datahub-project#8532) docs: small update to homepage (datahub-project#8483) fix(ingest): pin boto3-stubs in CI (datahub-project#8527) feat(siblings): hiding non-existant siblings in FE (datahub-project#8528) fix(ingest/build): Fix sagemaker mypy and flake8 issues (datahub-project#8530) feat(metrics): add metrics for aspect write and bytes (datahub-project#8526) feat(elasticsearch): allow bulk delete (datahub-project#8424) fix(ui): use locale lowercase when filtering columns of an entity in the lineage (datahub-project#8213) fix(auth): ignore case when comparing http headers (datahub-project#8356) ...
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Batch writes to elasticsearch generally contain only create/upsert requires while deletes are typically isolated. This PR offers the ability to allow combining delete requests (including delete_by_query) with upserts. If used responsibly this works fine, however in the past delete/modify requests effecting the same document was observed in the same batch of requests. This conflict caused inconsistencies due to failed bulk requests. In general, probably useful only in extreme cases, and temporarily, for bulk delete operations.
Checklist