Add delete by query to index management #797

btaani · 2021-09-10T12:14:56Z

Description

Adding delete by query to index management, along with the API changes and cron job updates

/cc @lukas-vlcek
/assign @periklis

Links

JIRA: https://issues.redhat.com/browse/LOG-1505
Enhancement proposal: https://docs.google.com/document/d/1JjdbspwYe-aYMBH65ybc9xBroOzoobPqrxwzFg9ZRGc/edit#

btaani · 2021-09-10T14:06:24Z

/hold

periklis

Awesome job so far! It looks you got the handle from top to bottom and vice versa. 🚀

periklis · 2021-09-13T08:08:04Z

apis/logging/v1/index_management_types.go

+	Namespace []DeleteNamespaceSpec `json:"deleteNamespace,omitempty"` //#BEE: Why is this an array?
+}
+
+type DeleteNamespaceSpec struct {
+	// The unique name of the spec
+	Name string `json:"name"` // delete-<namespace_name>
+	// Namespaces to be deleted
+	NamespacesToDelete []string `json:"namespace"`
+	// Delete the records matching the namespaces which are older
+	// than this MinAge (e.g. 1d)
+	MinAge TimeUnit `json:"minAge"`


Besides that I suggest to improve naming a bit:

type IndeManagementDeleteNamespaceSpec struct { // The unique name of the spec Name string `json:"name"` // delete-<namespace_name> // Namespace to delete documents from older than minAge. Namespace string `json:"namespace"` // Delete the records matching the namespaces which are older // than this MinAge (e.g. 1d) MinAge TimeUnit `json:"minAge"` }

Consider the official kubernetes API conventions when adding/extending APIs:
https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md

I can see that you changed the datatype of Namespace from []string to string. This is because this one:
Namespaces []IndeManagementDeleteNamespaceSpec json:"namespaces,omitempty"``
is already an array, right?

Exactly each namespace spec is dedicated to a single namespace.

@periklis but the actual thought during proposal was that a user can have multiple namespace in a single spec.
For example,
Spec-1:
Name - policy1
Namespace - openshift-api, openshift-logging
Age - 1d

Spec-2:
Name - policy2
Namespace - openshift-kube-server
Age - 5h

Even the delete_by_query api is capable enough to take comma separated index names

periklis · 2021-09-13T08:13:09Z

apis/logging/v1/index_management_types.go

 	//
 	MinAge TimeUnit `json:"minAge"`
+	// +nullable
+	Namespace []DeleteNamespaceSpec `json:"deleteNamespace,omitempty"` //#BEE: Why is this an array?


I believe the array for Namespace means that you can define a delete-by-query spec per namespace, because you can define a different MinAge per Namespace. @sasagarw Do you agree? If yes, we should improve the API naming because if it is confusing to us, it will certainly be confusing for API consumers.

type IndexManagementDeletePhaseSpec struct { // The minimum age of an index before it should be deleted (e.g. 10d) // MinAge TimeUnit `json:"minAge"` // The per namesapce specification to delete documents older than a given minimum age. Namespaces []IndeManagementDeleteNamespaceSpec `json:"namespaces,omitempty"` }

Yes @periklis , you are right.
Yeah we can rename it to Namespaces to avoid confusion.

periklis · 2021-09-13T08:16:47Z

internal/indexmanagement/reconcile.go

 	}

-	if policy.Phases.Delete != nil {
+	if policy.Phases.Delete != nil { //#BEE: I think I need to read namespace value here


This does not require to check if policy.Phases.Delete.Namespaces is set. If empty, the following code compiles an empty JSON list that can be checked in the python code. If empty, the python code can skip the delete by query routine.

periklis · 2021-09-13T08:18:50Z

internal/indexmanagement/reconcile.go

 		}
 		envvars = append(envvars,
 			corev1.EnvVar{Name: "MIN_AGE", Value: strconv.FormatUint(minAgeMillis, 10)},
+			corev1.EnvVar{Name: "NAMESPACE_NAME", Value: string(namespaceNamesJson)}, //#BEE: now I can use this in the delete script


The env var name NAMESPACE_NAME does not represent the purpose of namespaceNamesJson well enough. Latter is a list of namespaces while former sounds like a single name. I suggest to rename to something in plural case: NAMESPACE_SPECS, NAMEPACES...

periklis · 2021-09-13T08:22:19Z

internal/indexmanagement/reconcile.go

 		envvars = append(envvars,
 			corev1.EnvVar{Name: "MIN_AGE", Value: strconv.FormatUint(minAgeMillis, 10)},
+			corev1.EnvVar{Name: "NAMESPACE_NAME", Value: string(namespaceNamesJson)}, //#BEE: now I can use this in the delete script
+			corev1.EnvVar{Name: "MIN_AGE_DBQ", Value: minAgeValue},


Assuming that we have a list of namespaces specs, a single MIN_AGE_DBQ is not correct here. I suggest you pass create a small dict structure called namespaceSpecsJson that gives a good handle to execute a Delete-By-Query per Namespace, e.g.:

{ "namespace_a": "1d", "namespace_b": "2d", },

periklis · 2021-09-13T08:23:29Z

internal/indexmanagement/scripts.go


+#here namespace is either a string or array of strings, minAge is also a string in the unit of days, e.g., '3d'
+def deleteByQuery(index, namespace, minAge):
+  defaultAge='7d'


This should be an env var, which we pass to the k8s cronjob. Just in case a user wants to overwrite this temporarily for all delete-by-query calls.

sasagarw · 2021-09-13T08:36:52Z

internal/indexmanagement/scripts.go

+    if namespace != None and minAge == None:
+      s = s.query('terms', kubernetes__namespace_name=namespace) #Or just #Q('terms',  tags=['kubernetes.namespace_name', namespace])
+    elif namespace == None and minAge != None:
+      s = s.filter('range', **{'@timestamp': {'lt': 'now-{}'.format(minAge)}})
+    elif namespace == None and minAge == None:
+      s = s.filter('range', **{'@timestamp': {'lt': 'now-{}'.format(defaultAge)}})
+    elif namespace != None and minAge != None:
+      s = s.query('terms', kubernetes__namespace_name=namespace)
+      s = s.filter('range', **{'@timestamp': {'lt': 'now-{}'.format(minAge)}})


This much filtering is not required. This is complicating the code.
I would suggest you to define a query as:

body = { "query" : { "terms": { "kubernetes.namespace_name": [] // Namespace }

The above query would be used when minAge is not defined. Which suggests to delete all data belonging to that namespace.

Similarly, if minAge is defined then you can append the filter part to the above query, in which case, the body will look like

body = { "query": { "bool": { "must": [ { "terms": { "kubernetes.namespace_name": [] // Namespace } } ], "filter": [ { "range": { "@timestamp": { "lt": "now-$age" } // $age=MinAge } } ] } } }

Does this means that there should always be a namespace_name given? because otherwise it would delete the entire index if no namespace nor minAge is given. (only those entries older than defaultAge, of course)

In this case I shouldn't check for this condition?

if namespace == None and minAge == None: s = s.filter('range', **{'@timestamp': {'lt': 'now-{}'.format(defaultAge)}})

If namespace_name is not given then delete_by_query function itself will be never called right. IMO there is no use of checking and generating dynamic body. WDYT?

I agree.. I asked because you wrote it as a condition in your spike document. It makes more sense this way!

That was mentioned in Spike document just to list all possible cases. You can implement in a way which is making more sense.

sasagarw · 2021-09-13T08:47:57Z

internal/indexmanagement/scripts.go

+  #BEE: Call deleteByQuwey here?
+  namespaceName=$NAMESPACE_NAME
+  minAgeDBQ=$MIN_AGE_DBQ


You can just check if these env vars is null or not. If not null then you should be calling your defined python function here and pass these variables to that function.

Call below func inside delete():

function deleteByQuery() { <-------- define variables and read them using $1, $2, ... ------------> python -c 'import indexManagementClient; print(indexManagementClient.deleteByQuery(<-- pass the read variables --->))' }

lukas-vlcek

Thanks for the proposed implementation. The following are my comments:

1. How are we going to control which indices the delete query is run against?

Which indices are we going to allow to execute delete_by_query operation against?

Proposed deleteByQuery() has three arguments. One of them being the index name. It is not clear to me where we are going to take this value from.

Proposal doc mentions:

index_pattern = mapping.Name     // app* | infra* | audit*

perhaps this means we will use hardcoded index_pattern of app*,infra*,audit*?

Related (sub-)question is which user/permission will be used when delete_by_query is run? I assume this is similar to the user that is running the job to delete indices which means pretty powerful user, right?

2. Why we do not allow single delete operation for multiple namespaces?

It seems to me that current design and implementation do not allow to run one common delete_by_query for multiple namespaces sharing common name prefix.

This is a question of both efficiency and practicality.

How about if CU will need to keep specific namespaces under control but does not have a full control over the naming of the namespace, except for common prefix?

For example an OCP user can create high number of namespaces for devel. experiments but the deal will be that all namespaces must start with John_Doe_experiment*. Like John_Doe_experiment-000001ABX, John_Doe_experiment-004207WBT, ... To make sure John does no go wild with logging all his namespaces will be pruned by delete_by_query without the need to configure job for all individual namespaces.

For this purpose we can use "Prefix query" that works on non-analyzed fields (i.e. fields that are good fit for term queries as well).

3. How do we prevent running the same delete query before the previous is finished?

The code is using the Elasticsearch Python client and is calling the delete_by_query API.

BTW, some notes regarding proposed use of this API:

it is missing the third argument – the doc_type which in our case is _doc
it is setting default_operator='AND' which is irrelevant. It would be relevant if a query of type "Query String Query" would be used. But it is not in our case (proposed implementation is using the "terms" query – BTW, given that current implementation proposal allows to run delete operation per single namespace then use of "term" query (not "terms") would be more appropriate).

The documentation specifies, that by default the call to delete\_by\_query API is blocking until it is complete. Internally, Elasticsearch uses Bulk API to delete matching documents. This means that the bulk queue throughput is critical. In other words if the cluster is under heavy ingestion pressure then the delete by query operation will compete with indexing requests coming from collectors. Given many CUs are running under-resourced cluster then risk that some delete queries may not be finished when a new one is started (by cron) is increasing.

It can also happen that there will be a need (a request) to run delete jobs more frequently.

I suggest to investigate possibility to run the delete operations in async manner (see wait_for_completion=false parameter). To prevent overlapping jobs we could use the Task API to pull list of all running actions=*/delete/byquery jobs. If there are any we can skip current cron cycle (assuming the only entity that originates these tasks is our cron job). Side effect is that this would allow us to run the delete by query operation job more frequently(?).

4. Merging segments to clear storage from deleted documents

Finally, when individual documents are deleted from index they are actually not physically removed from the disk until corresponding index segments are merged. Some segments will be merged automatically (typically smaller segments) but for some segments it can take time before they are merged.

If the ultimate goal is to reduce the size of indices then we need to control segment merging as well. But segment merging is resource intensive and time consuming operation. At this point I am not sure if we should/want to call segment force merge in the end of the cron job (which will be impossible if we decide to use delete_by_query in async manner as suggested above) or if it will be better to implement independent job that would be run more frequently and whose sole purpose would be to run some heuristics and merge segments if it is needed (and possible – for example it does not make sense to merge segments if there are no free CPUs or IO resources).

btaani · 2021-09-20T08:33:51Z

Thanks Lukas for the detailed review. Here are some comments from my side:

Which indices are we going to allow to execute delete_by_query operation against?

In the beginning I assumed that since deleteByQuery() will be called inside the delete() function, they will both work on the same index. But now after I looked at the code, delete() works on a set of indices instead of only one (which it gets from getNext25Indices.py). Let's discuss this in a call.

Related (sub-)question is which user/permission will be used when delete_by_query is run? I assume this is similar to the user that is running the job to delete indices which means pretty powerful user, right?

deleteByQuery() will be executed in the same cron job as the other functions (delete() and rollover()), which means they will all be executed under the same user permissions.

Why we do not allow single delete operation for multiple namespaces?

It seems that the delete_by_query API allows for a Prefix query:

analyze_wildcard – Specify whether wildcard and prefix queries should be analyzed (default: false)

The last 2 questions regarding async run and merging we can discuss further in a call

sasagarw · 2021-10-07T03:20:14Z

apis/logging/v1/index_management_types.go

+	// +optional
+	Name string `json:"name"` // delete-<namespace_name>


This shouldn't be marked as optional. A unique name for each spec is required to be provided.

Suggested change

// +optional

Name string `json:"name"` // delete-<namespace_name>

// +required

Name string `json:"name"` // delete-<namespace_name>

sasagarw · 2021-10-07T03:24:08Z

apis/logging/v1/index_management_types.go

+	// If MinAge for NamespaceSpec is empty, delete the records matching the namespaces which are older than this default age
+	DefaultAge TimeUnit `json:"defaultAge"`


Why this DefaultAge here? Already inside IndexManagementDeleteNamespaceSpec it has been defined right? You can just check that empty and use the default age.

sasagarw · 2021-10-07T03:26:16Z

apis/logging/v1/index_management_types.go

+	// Namespace to delete
+	Namespace string `json:"namespace"`


You can mark it as +required just for validation purpose.

sasagarw · 2021-10-07T03:27:31Z

apis/logging/v1/index_management_types.go

+	// The per namesapce specification to delete documents older than a given minimum age
+	Namespaces []IndexManagementDeleteNamespaceSpec `json:"namespaceSpec"`


Since this is optional, you can mark them +optional for validation.

sasagarw · 2021-10-07T03:28:11Z

apis/logging/v1/index_management_types.go

+	// The per namesapce specification to delete documents older than a given minimum age
+	Namespaces []IndexManagementDeleteNamespaceSpec `json:"namespaceSpec"`


Suggested change

// The per namesapce specification to delete documents older than a given minimum age

Namespaces []IndexManagementDeleteNamespaceSpec `json:"namespaceSpec"`

// The per namesapce specification to delete documents older than a given minimum age

Namespaces []IndexManagementDeleteNamespaceSpec `json:"namespaceSpec,omitempty"`

periklis

Awesome work! Really amazing progress 🚀

I suggest to add the following changes to the PR for the sake of completeness:

When we extend the API we should elaborate if we add the new fields by default in our samples per CRD. I believe this feature belong there too (https://github.com/openshift/elasticsearch-operator/blob/master/config/samples/logging_v1_elasticsearch.yaml)
The new cronjobs should be tested for suspend/unspend when ES pods are not available (https://github.com/openshift/elasticsearch-operator/blob/master/internal/indexmanagement/index_management_test.go#L55)

periklis · 2021-11-05T11:38:23Z

apis/logging/v1/index_management_types.go

+
+	// How often to run a new prune-namespaces cron job
+	// +required
+	PrunePollInterval TimeUnit `json:"prunePollInterval"`


For better visibility that this is required to be configured with Namespaces I suggest we rename this field to something like PruneNamespacesInterval, WDYT?

I agree, I went with PrunePollInterval to be consistent with PollInterval:

elasticsearch-operator/apis/logging/v1/index_management_types.go

Line 34 in 45ea70e

PollInterval TimeUnit `json:"pollInterval"`

But PruneNamespacesInterval is more descriptive

periklis · 2021-11-05T11:39:07Z

apis/logging/v1/index_management_types.go

+type IndexManagementDeleteNamespaceSpec struct {
+	// The unique name of the spec
+	// +required
+	Name string `json:"name"` // delete-<namespace_name>


What is // delete-<namespace_name> supposed to tell the reader here?

This one is from the original spike document. I honestly do not use it anywhere inside the code. I'll ask Sashank why he had it there originally. Otherwise I believe it can be omitted

periklis · 2021-11-05T11:40:36Z

apis/logging/v1/index_management_types.go

+	// Namespace to delete
+	// +required
+	Namespace string `json:"namespace"`


To minimize confusion of API docs readers, I suggest to adapt the field comment to something like:

// Target Namespace to delete logs older than MinAge (defaults to ...)

In addition we should mention that prefix queries like openshift* are allowed here.

periklis · 2021-11-05T11:41:11Z

apis/logging/v1/index_management_types.go

+	// If MinAge for NamespaceSpec is empty, delete the records matching the namespaces which are older than this default age
+	// +required
+	DefaultAge TimeUnit `json:"defaultAge"`


As mentioned in a previous call, we should drop this field for a constant in the index_management package.

periklis · 2021-11-05T11:41:59Z

bundle.Dockerfile

    summary="This is the bundle for the elasticsearch-operator" \
    maintainer="AOS Logging <aos-logging@redhat.com>"
-
+    


nit. let's not add unneeded thing into a PR's diff.

periklis · 2021-11-05T11:55:59Z

internal/indexmanagement/reconcile.go

+func pruneNamespacesCmd(policy apis.IndexManagementPolicySpec) string {
+	cmd := ""
+	if policy.Phases.Delete != nil {
+		cmd = "./prune-namespaces"
+	}
+	return cmd
+}


This function is actually obsolete as is. We should check for policy.Phases.Delete != nil in the caller site and if it is nil, we should not schedule any delete-by-query cronjob at all instead of scheduling one with an empty command.

periklis · 2021-11-05T11:57:49Z

internal/indexmanagement/scripts.go

+      #tasks = es_client.tasks.list(actions="delete/byquery")
+      #print (tasks)
+      #print (json.dumps(s.to_dict(), indent=2, sort_keys=True))


I believe we can drop these commented lines as agreed upon that this call should not be executed async, right?

periklis · 2021-11-05T11:59:42Z

internal/indexmanagement/scripts.go

+    print(e)
+    sys.stdout = open('/tmp/response.txt', 'w')
+    print(e)
+    sys.stdout = original_stdout


What is this double print good for here? Can't we just simply print the exception into the original stdout?

These were for debugging purposes, will be cleaned up soon

periklis · 2021-11-05T12:00:22Z

internal/indexmanagement/scripts.go


 for aliasBase in $writeAliases; do
-
+  echo $aliasBase


Is this debugging like still required?

periklis · 2021-11-05T12:01:10Z

internal/indexmanagement/scripts.go


 const deleteThenRolloverScript = `
 set -uo pipefail
+source /tmp/scripts/indexManagement


Is this source required? As per /tmp/scripts/delete already sourcing the same file?

I believe this was added by mistake, my bad!

periklis · 2021-11-17T14:19:19Z

config/samples/logging_v1_elasticsearch.yaml

+          - namespace: openshift-logging
+            minAge: 5h


This is a bad example actually. The Logging stack does not collect any logs from its component. This recursive collection is counter-productive. A good example would be to collect the openshift-monitoring logs.

periklis · 2021-11-17T14:20:42Z

internal/indexmanagement/reconcile.go

+		var (
+			namespaceSpecs       = make(map[string]string)
+			namespaceSpecsString = ""
+		)
+		namespaceSpecs = make(map[string]string)


Twice make(map[string]string)

periklis · 2021-11-17T14:22:40Z

internal/indexmanagement/reconcile.go

 	}

+	// prune-namespaces cron job
+	if policy.Phases.Delete != nil && policy.Phases.Delete.Namespaces != nil {


Checking len(policy.Phases.Delete.Namespaces) != 0 is more useful here. It ensures that if your slice is nil or empty at the same time, the prune job will not be scheduled.

periklis

LGTM after removing all the debug print lines in the delete-by-query script.

btaani · 2021-11-22T13:35:59Z

/hold
just found an error while testing, I'll find the cause and fix it

btaani · 2021-11-22T15:04:54Z

/unhold

sasagarw · 2021-11-23T03:26:26Z

@btaani you need to make the lint happy.
Run make lint locally and it will tell you what to change.

sasagarw · 2021-11-24T03:24:16Z

internal/indexmanagement/scripts.go

+      #print (json.dumps(s.to_dict(), indent=2, sort_keys=True))
+      print (json.dumps(response, indent=2))


Need to drop these debug lines

I think I will keep this one:
print (json.dumps(response, indent=2))
as it shows the detailed response and how many documents were delete or were in conflict

But except a developer no one would be interested in this information right? WDYT?
If it can be printed in higher log level then it makes sense to keep here IMO.

periklis

/lgtm

periklis · 2021-12-06T13:10:25Z

/approve

openshift-ci · 2021-12-06T13:10:57Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: btaani, periklis

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [periklis]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci bot assigned periklis Sep 10, 2021

openshift-ci bot requested a review from lukas-vlcek September 10, 2021 12:14

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Sep 10, 2021

periklis reviewed Sep 13, 2021

View reviewed changes

sasagarw reviewed Sep 13, 2021

View reviewed changes

btaani marked this pull request as draft September 14, 2021 10:19

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 14, 2021

lukas-vlcek reviewed Sep 17, 2021

View reviewed changes

openshift-ci bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 6, 2021

btaani force-pushed the index-mgmt branch from 6f4816b to ab64f17 Compare October 6, 2021 21:15

sasagarw suggested changes Oct 7, 2021

View reviewed changes

btaani force-pushed the index-mgmt branch 2 times, most recently from aa2d9ed to abfde6e Compare October 11, 2021 10:36

This was referenced Oct 12, 2021

add elasticsearch_dsl python package openshift/origin-aggregated-logging#2191

Merged

add delete by query permissions to index-management role openshift/origin-aggregated-logging#2193

Merged

btaani force-pushed the index-mgmt branch from abfde6e to 0460fb1 Compare October 18, 2021 13:34

openshift-ci bot added needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Oct 18, 2021

openshift-ci bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 2, 2021

periklis suggested changes Nov 5, 2021

View reviewed changes

btaani marked this pull request as ready for review November 5, 2021 12:16

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 5, 2021

btaani mentioned this pull request Nov 15, 2021

add bulk permissions to sg_role_curator openshift/origin-aggregated-logging#2205

Merged

periklis reviewed Nov 17, 2021

View reviewed changes

periklis reviewed Nov 22, 2021

View reviewed changes

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 22, 2021

sasagarw suggested changes Nov 24, 2021

View reviewed changes

btaani force-pushed the index-mgmt branch from 7a5acf7 to 4317fc3 Compare December 3, 2021 11:27

openshift-ci bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 3, 2021

btaani force-pushed the index-mgmt branch 5 times, most recently from 1033fb2 to f01f90c Compare December 6, 2021 10:29

Add prune namespaces cronjob

2934608

btaani force-pushed the index-mgmt branch from f01f90c to 2934608 Compare December 6, 2021 10:31

openshift-ci bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 6, 2021

periklis reviewed Dec 6, 2021

View reviewed changes

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Dec 6, 2021

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 6, 2021

openshift-merge-robot merged commit b031e5d into openshift:master Dec 6, 2021

sasagarw mentioned this pull request Jan 21, 2022

LOG-1507: Consume EO's delete by query api changes openshift/cluster-logging-operator#1294

Merged

		// +optional
		Name string `json:"name"` // delete-<namespace_name>

		// If MinAge for NamespaceSpec is empty, delete the records matching the namespaces which are older than this default age
		DefaultAge TimeUnit `json:"defaultAge"`

		// The per namesapce specification to delete documents older than a given minimum age
		Namespaces []IndexManagementDeleteNamespaceSpec `json:"namespaceSpec"`

		summary="This is the bundle for the elasticsearch-operator" \
		maintainer="AOS Logging <aos-logging@redhat.com>"

		#print (json.dumps(s.to_dict(), indent=2, sort_keys=True))
		print (json.dumps(response, indent=2))

Add delete by query to index management #797

Add delete by query to index management #797

Uh oh!

Conversation

btaani commented Sep 10, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Links

Uh oh!

btaani commented Sep 10, 2021

Uh oh!

periklis left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sasagarw Sep 13, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sasagarw Sep 13, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

btaani Sep 16, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sasagarw Sep 13, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lukas-vlcek left a comment

Choose a reason for hiding this comment

1. How are we going to control which indices the delete query is run against?

2. Why we do not allow single delete operation for multiple namespaces?

3. How do we prevent running the same delete query before the previous is finished?

4. Merging segments to clear storage from deleted documents

Uh oh!

btaani commented Sep 20, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

periklis left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

btaani commented Sep 10, 2021 •

edited

Loading

sasagarw Sep 13, 2021 •

edited

Loading

sasagarw Sep 13, 2021 •

edited

Loading

btaani Sep 16, 2021 •

edited

Loading

sasagarw Sep 13, 2021 •

edited

Loading