Skip to content

feat: derivation pipeline worker graceful shutdown #357

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Oct 13, 2025
Merged

feat: derivation pipeline worker graceful shutdown #357

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
2e376fa
derivation pipeline worker graceful shutdown
frisitano Oct 13, 2025
dbbfc3f
Merge branch 'main' into feat/derivation-pipeline-graceful-shutdown
frisitano Oct 13, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 20 additions & 6 deletions crates/derivation-pipeline/src/lib.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -173,16 +173,30 @@ where
tokio::select! {
biased;

Some(batch_info) = self.batch_receiver.recv(), if self.futures.len() < DERIVATION_PIPELINE_WORKER_CONCURRENCY => {
let fut = self.derivation_future(batch_info);
self.futures.push_back(fut);
maybe_batch = self.batch_receiver.recv(), if self.futures.len() < DERIVATION_PIPELINE_WORKER_CONCURRENCY => {
match maybe_batch {
Some(batch_info) => {
let fut = self.derivation_future(batch_info);
self.futures.push_back(fut);
}
None => {
tracing::info!(target: "scroll::derivation_pipeline", "Batch channel closed, shutting down derivation pipeline worker");
break;
}
}

}
Some(result) = self.futures.next() => {
match result {
Ok(res) => self.result_sender.send(res).expect("Failed to send batch derivation result"),
Ok(res) => {
if self.result_sender.send(res).is_err() {
tracing::info!(target: "scroll::derivation_pipeline", "Result channel closed, shutting down derivation pipeline worker");
break;
}
}
Err((batch_info, err)) => {
tracing::error!(target: "scroll::derivation_pipeline", ?batch_info, ?err, "Failed to derive payload attributes");
self.futures.push_front(self.derivation_future(batch_info));
tracing::error!(target: "scroll::derivation_pipeline", ?batch_info, ?err, "Failed to derive payload attributes");
self.futures.push_front(self.derivation_future(batch_info));
}
}
}
Expand Down