Issue/4184 assign shard logic #4223

fulmicoton · 2023-12-01T07:54:53Z

No description provided.

quickwit/quickwit-indexing/src/source/ingest/mod.rs

quickwit/quickwit-indexing/src/actors/indexing_pipeline.rs

guilload · 2023-12-01T14:48:18Z

quickwit/quickwit-indexing/src/actors/indexing_pipeline.rs

        if let Some(handles) = &mut self.handles_opt {
            info!(
-                shard_ids=?message.0.shard_ids,
+                shard_ids=?assign_shards_message.0.shard_ids,
                "assigning shards to indexing pipeline."


Suggested change

"assigning shards to indexing pipeline."

"assigning shards to indexing pipeline"

quickwit/quickwit-indexing/src/actors/indexing_service.rs

guilload · 2023-12-01T14:52:29Z

quickwit/quickwit-indexing/src/actors/indexing_service.rs

+        ctx: &ActorContext<Self>,
+        apply_plan_req: ApplyIndexingPlanRequest,
+    ) -> Result<(), IndexingError> {
+        let tasks: &[IndexingTask] = &apply_plan_req.indexing_tasks[..];


Suggested change

let tasks: &[IndexingTask] = &apply_plan_req.indexing_tasks[..];

let tasks: &[IndexingTask] = &apply_plan_req.indexing_tasks;

quickwit/quickwit-indexing/src/actors/indexing_pipeline.rs

quickwit/quickwit-indexing/src/source/ingest/mod.rs

…espawn Bugfix Assign shard logic. If no shard is removed, we just kill the pipeline. If shards are added we reinitiate the shards. The pipeline supervisor keeps track of the shard, and reassigns them if the pipeline is respawned. Closes #4184 Closes #4174

Co-authored-by: Adrien Guillo <adrien@quickwit.io>

fulmicoton requested a review from guilload December 1, 2023 07:54

fulmicoton force-pushed the issue/4184-assign-shard-logic branch 4 times, most recently from b7672cc to 9541f3e Compare December 2, 2023 04:59

fulmicoton marked this pull request as draft December 2, 2023 06:41

fulmicoton force-pushed the issue/4184-assign-shard-logic branch from 9541f3e to ee214e4 Compare December 2, 2023 08:11

fulmicoton marked this pull request as ready for review December 2, 2023 08:11

fulmicoton commented Dec 2, 2023

View reviewed changes

quickwit/quickwit-indexing/src/source/ingest/mod.rs Show resolved Hide resolved

fulmicoton marked this pull request as draft December 2, 2023 08:39

fulmicoton force-pushed the issue/4184-assign-shard-logic branch 6 times, most recently from d2c9c2d to 094f91f Compare December 4, 2023 07:14

fulmicoton marked this pull request as ready for review December 4, 2023 07:14

fulmicoton force-pushed the issue/4184-assign-shard-logic branch 2 times, most recently from fb92c7a to d67b52a Compare December 4, 2023 08:16

guilload approved these changes Dec 4, 2023

View reviewed changes

fulmicoton and others added 4 commits December 5, 2023 13:01

Apply suggestions from code review

04c2190

Co-authored-by: Adrien Guillo <adrien@quickwit.io>

Apply suggestions from code review

a87a6ef

Co-authored-by: Adrien Guillo <adrien@quickwit.io>

Clearer contract for reset_if_needed and fixing unit test

7075d3f

fulmicoton force-pushed the issue/4184-assign-shard-logic branch from c48488c to 7075d3f Compare December 5, 2023 04:01

fulmicoton merged commit e8f852c into main Dec 5, 2023
4 checks passed

fulmicoton deleted the issue/4184-assign-shard-logic branch December 5, 2023 05:09

guilload mentioned this pull request Dec 5, 2023

Remove IndexingStatus from IngestSource #4141

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Issue/4184 assign shard logic #4223

Issue/4184 assign shard logic #4223

fulmicoton commented Dec 1, 2023

guilload Dec 1, 2023

guilload Dec 1, 2023

	"assigning shards to indexing pipeline."
	"assigning shards to indexing pipeline"

	let tasks: &[IndexingTask] = &apply_plan_req.indexing_tasks[..];
	let tasks: &[IndexingTask] = &apply_plan_req.indexing_tasks;

Issue/4184 assign shard logic #4223

Issue/4184 assign shard logic #4223

Conversation

fulmicoton commented Dec 1, 2023

guilload Dec 1, 2023

Choose a reason for hiding this comment

guilload Dec 1, 2023

Choose a reason for hiding this comment