Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion TODO-CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,10 @@ Minigun is a high-performance data processing pipeline framework for Ruby with s
- [ ] producers inside IPC/COW forks
- [ ] routing with multiple forked processes - round-robin via IPC workers
- [ ] start of IPC/COW stage should not require await - added await: true option
- [ ] :worker_finished event seems like it should not work like it does. It resends back into the master... hmmm
- [X] :worker_finished event seems like it should not work like it does. It resends back into the master... hmmm
- [ ] cleanup pipeline, etc constructor args
- [ ] wait_for_first_item implmentation look wonky
- [ ] make StageContext and actual class
- [ ] Transmit stats across forks
- [ ] Transmit logs across forks--look at Puma
- [ ] Support MINIGUN_LOG_LEVEL var
Expand Down
3 changes: 2 additions & 1 deletion lib/minigun/pipeline.rb
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,8 @@ module Minigun
# A Pipeline can be standalone or part of a multi-pipeline Task
class Pipeline
attr_reader :name, :config, :stages, :hooks, :dag, :output_queues, :stats,
:context, :stage_hooks, :runtime_edges, :input_queues, :parent_pipeline, :task
:context, :stage_hooks, :runtime_edges, :input_queues, :parent_pipeline, :task,
:entrance_router

def initialize(name, task, parent_pipeline, config = {}, stages: nil, hooks: nil, stage_hooks: nil, dag: nil, stats: nil)
@name = name
Expand Down
5 changes: 2 additions & 3 deletions lib/minigun/stage.rb
Original file line number Diff line number Diff line change
Expand Up @@ -59,12 +59,11 @@ def initialize(name, pipeline, block = nil, options = {})
end

def task
return unless @pipeline&.respond_to?(:task)
@pipeline.task
end

def root_pipeline
pipeline&.root_pipeline
@pipeline.root_pipeline
end

# Get the queue size for this stage
Expand Down Expand Up @@ -179,7 +178,7 @@ def send_end_signals(stage_ctx)
task = stage_ctx.stage.task

all_targets.each do |target|
queue = task&.find_queue(target)
queue = task.find_queue(target)
next unless queue

queue << EndOfSource.new(stage_ctx.stage)
Expand Down
11 changes: 4 additions & 7 deletions lib/minigun/worker.rb
Original file line number Diff line number Diff line change
Expand Up @@ -113,12 +113,9 @@ def handle_disconnected_stage(stage_ctx) # rubocop:disable Naming/PredicateMetho

# Wait for first item to arrive via dynamic routing
# Returns true if timed out (should shutdown), false if item received (continue)
# TODO: This implementation looks wonky, consider alternatives
def wait_for_first_item(timeout:, stage_ctx:)
input_queue = stage_ctx.input_queue
return false unless input_queue # Safety check for mocked contexts

raw_queue = input_queue.instance_variable_get(:@queue) if input_queue.respond_to?(:instance_variable_get)
return false unless raw_queue # Safety check for mocked queues
raw_queue = stage_ctx.input_queue

# Try to pop with timeout using Timeout module
begin
Expand Down Expand Up @@ -174,8 +171,8 @@ def create_stage_context
Set.new
else
# Check if this stage is an entrance router or single entry stage for nested pipeline
input_queues = @pipeline.instance_variable_get(:@input_queues)
entrance_router = @pipeline.instance_variable_get(:@entrance_router)
input_queues = @pipeline.input_queues
entrance_router = @pipeline.entrance_router

if @stage == entrance_router && input_queues
# For entrance router, use sources from parent pipeline
Expand Down
Loading
Loading