Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

IngestV2 plan apply loop is broken. #4174

Closed
2 tasks
fulmicoton opened this issue Nov 21, 2023 · 0 comments
Closed
2 tasks

IngestV2 plan apply loop is broken. #4174

fulmicoton opened this issue Nov 21, 2023 · 0 comments
Assignees
@fulmicoton
Labels
bug Something isn't working high-priority
Milestone
Distributed inges...

Comments

@fulmicoton
Copy link
Contributor

fulmicoton commented Nov 21, 2023 •
edited

IngestV2 plan apply loop is broken.

IndexingPlanDiff(missing_tasks_by_node_id={"node-1": [IndexingTask { index_uid: "gharchive4:01HFRA32Z532P0P7D6SNG5WPP4", source_id: "_ingest-source", shard_ids: [7] }]}, , unplanned_tasks_by_node_id={"node-1": [IndexingTask { index_uid: "gharchive4:01HFRA32Z532P0P7D6SNG5WPP4", source_id: "_ingest-source", shard_ids: [] }]})

my understanding is that we don't expose the list of shards in chitchat yet, and that's where the diff is.

  • fix the serialization format of indexing task in chitchat to allow shards
  • fix the exposition of the indexing task in chitchat to have shard (right now we serialize an empty vec all of the time)
@fulmicoton fulmicoton added the bug Something isn't working label Nov 21, 2023
@fulmicoton fulmicoton self-assigned this Nov 22, 2023
fulmicoton added a commit that referenced this issue Nov 22, 2023
@fulmicoton
Fixed the serialize deserialization of the indexing state to include
e62b6cb 
shard lists.

Also took it as an opportunity to clean up the code
and increase the test coverage.

Closes #4174
fulmicoton added a commit that referenced this issue Nov 22, 2023
@fulmicoton
Fixed the serialize deserialization of the indexing state to include
7a9f4a6 
shard lists.

Also took it as an opportunity to clean up the code
and increase the test coverage.

Closes #4174
fulmicoton added a commit that referenced this issue Nov 22, 2023
@fulmicoton
Fixed the serialize deserialization of the indexing state to include
0718b86 
shard lists.

Before this, there was no notion of shards and indexers were just
exposing the number of pipelines per source_uid.

With ingest v2, the list of shard is part of indexing tasks, and the
control plane wants to diff against it.

The serialization format goes
`[1,2][3]` to express two pipelines with respectively shard [1,2] and
`[3]`.

For a source that does not have notion of shards, like kafka, two
pipelines simply look as follows: `[][]`

Also took it as an opportunity to clean up the code
and increase the test coverage.

Closes #4174
fulmicoton added a commit that referenced this issue Nov 22, 2023
@fulmicoton
Fixed the serialize deserialization of the indexing state to include
b50adf7 
shard lists.

Before this, there was no notion of shards and indexers were just
exposing the number of pipelines per source_uid.

With ingest v2, the list of shard is part of indexing tasks, and the
control plane wants to diff against it.

The serialization format goes
`[1,2][3]` to express two pipelines with respectively shard [1,2] and
`[3]`.

For a source that does not have notion of shards, like kafka, two
pipelines simply look as follows: `[][]`

Also took it as an opportunity to clean up the code
and increase the test coverage.

Closes #4174
fulmicoton added a commit that referenced this issue Nov 22, 2023
@fulmicoton
Fixed the serialize deserialization of the indexing state to include
4d9d2ce 
shard lists.

Before this, there was no notion of shards and indexers were just
exposing the number of pipelines per source_uid.

With ingest v2, the list of shard is part of indexing tasks, and the
control plane wants to diff against it.

The serialization format goes
`[1,2][3]` to express two pipelines with respectively shard [1,2] and
`[3]`.

For a source that does not have notion of shards, like kafka, two
pipelines simply look as follows: `[][]`

Also took it as an opportunity to clean up the code
and increase the test coverage.

Related #4174
fulmicoton added a commit that referenced this issue Nov 22, 2023
@fulmicoton
Fixed the serialize deserialization of the indexing state to include
2f1a332 
shard lists.

Before this, there was no notion of shards and indexers were just
exposing the number of pipelines per source_uid.

With ingest v2, the list of shard is part of indexing tasks, and the
control plane wants to diff against it.

The serialization format goes
`[1,2][3]` to express two pipelines with respectively shard [1,2] and
`[3]`.

For a source that does not have notion of shards, like kafka, two
pipelines simply look as follows: `[][]`

Also took it as an opportunity to clean up the code
and increase the test coverage.

Related #4174
@fulmicoton fulmicoton mentioned this issue Nov 22, 2023
Merged
fulmicoton added a commit that referenced this issue Nov 22, 2023
@fulmicoton
Fixed the serialize deserialization of the indexing state to include
35d7cb4 
shard lists.

Before this, there was no notion of shards and indexers were just
exposing the number of pipelines per source_uid.

With ingest v2, the list of shard is part of indexing tasks, and the
control plane wants to diff against it.

The serialization format goes
`[1,2][3]` to express two pipelines with respectively shard [1,2] and
`[3]`.

For a source that does not have notion of shards, like kafka, two
pipelines simply look as follows: `[][]`

Also took it as an opportunity to clean up the code
and increase the test coverage.

Related #4174
fulmicoton added a commit that referenced this issue Nov 22, 2023
@fulmicoton @fmassot
Fixed the serialize deserialization of the indexing state to include (#…
645658e 
…4180)

* Fixed the serialize deserialization of the indexing state to include
shard lists.

Before this, there was no notion of shards and indexers were just
exposing the number of pipelines per source_uid.

With ingest v2, the list of shard is part of indexing tasks, and the
control plane wants to diff against it.

The serialization format goes
`[1,2][3]` to express two pipelines with respectively shard [1,2] and
`[3]`.

For a source that does not have notion of shards, like kafka, two
pipelines simply look as follows: `[][]`

Also took it as an opportunity to clean up the code
and increase the test coverage.

Related #4174

Co-authored-by: François Massot <francois.massot@gmail.com>
fulmicoton added a commit that referenced this issue Dec 2, 2023
@fulmicoton
Bugfix Assign shard logic.
ee214e4 
If no shard is removed, we just kill the pipeline.
If shards are added we reinitiate the shards.
The pipeline supervisor keeps track of the shard, and reassigns them if
the pipeline is respawned.

Closes #4184
Closes #4174
fulmicoton added a commit that referenced this issue Dec 2, 2023
@fulmicoton
Bugfix Assign shard logic.
36a512e 
If no shard is removed, we just kill the pipeline.
If shards are added we reinitiate the shards.
The pipeline supervisor keeps track of the shard, and reassigns them if
the pipeline is respawned.

Closes #4184
Closes #4174
fulmicoton added a commit that referenced this issue Dec 2, 2023
@fulmicoton
Bugfix Assign shard logic.
35cc8ab 
If no shard is removed, we just kill the pipeline.
If shards are added we reinitiate the shards.
The pipeline supervisor keeps track of the shard, and reassigns them if
the pipeline is respawned.

Closes #4184
Closes #4174
fulmicoton added a commit that referenced this issue Dec 4, 2023
@fulmicoton
Simple no-op refactoring.
b9dc8c0 
Bugfix: IndexPipeline remembers list of shards and applies it after respawn

Bugfix Assign shard logic.

If no shard is removed, we just kill the pipeline.
If shards are added we reinitiate the shards.
The pipeline supervisor keeps track of the shard, and reassigns them if
the pipeline is respawned.

Closes #4184
Closes #4174
fulmicoton added a commit that referenced this issue Dec 4, 2023
@fulmicoton
Simple no-op refactoring.
dfd00c2 
Bugfix: IndexPipeline remembers list of shards and applies it after respawn

Bugfix Assign shard logic.

If no shard is removed, we just kill the pipeline.
If shards are added we reinitiate the shards.
The pipeline supervisor keeps track of the shard, and reassigns them if
the pipeline is respawned.

Closes #4184
Closes #4174
fulmicoton added a commit that referenced this issue Dec 4, 2023
@fulmicoton
Simple no-op refactoring.
d2c9c2d 
Bugfix: IndexPipeline remembers list of shards and applies it after respawn

Bugfix Assign shard logic.

If no shard is removed, we just kill the pipeline.
If shards are added we reinitiate the shards.
The pipeline supervisor keeps track of the shard, and reassigns them if
the pipeline is respawned.

Closes #4184
Closes #4174
fulmicoton added a commit that referenced this issue Dec 4, 2023
@fulmicoton
Bugfix: IndexPipeline remembers list of shards and applies it after r…
094f91f 
…espawn

Bugfix Assign shard logic.

If no shard is removed, we just kill the pipeline.
If shards are added we reinitiate the shards.
The pipeline supervisor keeps track of the shard, and reassigns them if
the pipeline is respawned.

Closes #4184
Closes #4174
fulmicoton added a commit that referenced this issue Dec 4, 2023
@fulmicoton
Bugfix: IndexPipeline remembers list of shards and applies it after r…
fb92c7a 
…espawn

Bugfix Assign shard logic.

If no shard is removed, we just kill the pipeline.
If shards are added we reinitiate the shards.
The pipeline supervisor keeps track of the shard, and reassigns them if
the pipeline is respawned.

Closes #4184
Closes #4174
fulmicoton added a commit that referenced this issue Dec 4, 2023
@fulmicoton
Bugfix: IndexPipeline remembers list of shards and applies it after r…
d67b52a 
…espawn

Bugfix Assign shard logic.

If no shard is removed, we just kill the pipeline.
If shards are added we reinitiate the shards.
The pipeline supervisor keeps track of the shard, and reassigns them if
the pipeline is respawned.

Closes #4184
Closes #4174
fulmicoton added a commit that referenced this issue Dec 5, 2023
@fulmicoton
Bugfix: IndexPipeline remembers list of shards and applies it after r…
64fca08 
…espawn

Bugfix Assign shard logic.

If no shard is removed, we just kill the pipeline.
If shards are added we reinitiate the shards.
The pipeline supervisor keeps track of the shard, and reassigns them if
the pipeline is respawned.

Closes #4184
Closes #4174
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@fulmicoton fulmicoton

Labels
bug Something isn't working high-priority
Projects
No open projects
Quickwit 0.7
Status: Done
Milestone
Distributed ingestion and indexing
Development

No branches or pull requests
1 participant
@fulmicoton