Complete Pipeline Redesign for GoBatch v0.2.0

[MasterOfBinary]

Complete Redesign for Chained Processor Architecture in GoBatch v0.2.0

This pull request implements a major redesign of the GoBatch architecture to enable a chained processor design. The changes provide a more flexible, robust batch processing experience with improved error handling and better context cancellation support.

Key Improvements

• Chained Processor Design:

  • Complete reimagining of how processors connect and interact
  • Redesigned the Processor interface to use slices of items instead of individual items
  • Updated the Source interface for simpler usage and clearer ownership of channels
  • Removed the older PipelineStage in favor of explicit slice and channel-based interfaces
  • Processors can now be linked together seamlessly in a single pipeline
  • Improved context cancellation handling throughout the codebase

• New Processors:

  • Added a Filter processor for filtering items based on custom predicates
  • Added a Transform processor for applying transformations to item data
  • Added Error and Nil processors for testing and simulation

• Enhanced Source Implementations:

  • Improved Channel source with configurable buffer sizes
  • Added better error handling to Error source with nil error filtering
  • Fixed timer handling in Nil source for more accurate timing tests

• Error Handling Improvements:

  • Added per-item error tracking via the new Error field in the Item struct
  • Standardized error reporting through error channels
  • Improved error propagation from processors to batch consumers

• Comprehensive Documentation:

  • Updated all interface and implementation documentation
  • Added detailed descriptions of error handling patterns
  • Prepared CHANGELOG for the v0.2.0 release with a comprehensive summary

Technical Changes

• Minimum supported Go version updated to 1.18 to leverage generics
• Rewritten core functions for clearer responsibility boundaries
• Added consistency in error handling patterns across all implementations
• Comprehensive test coverage for all components

These changes significantly improve GoBatch's usability, reliability, and flexibility while maintaining backward compatibility where possible.

[Copilot]

Pull Request Overview

This pull request implements a major redesign of GoBatch to support a chained processor design that improves flexibility, error handling, and context cancellation. Key changes include a complete overhaul of processor interfaces (now handling slices of items), the introduction of new built-in processors (Filter, Transform, Error, Nil), and significant updates to source implementations and documentation.

Reviewed Changes

Copilot reviewed 29 out of 30 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
processor/transform.go	Added a Transform processor for applying custom transformations on item data.
processor/processor.go	Updated processor package comments to reflect all available processor implementations.
processor/nil.go & nil_test.go	Redesigned Nil processor to properly handle sleep duration and context cancellation with tests.
processor/filter.go & filter_test.go	Introduced Filter processor with predicate/inversion logic and its accompanying tests.
processor/error.go & error_test.go	Updated Error processor to support configurable failure fractions and improved error propagation.
Batch package files (pipeline_stage, item, helpers, example, errors, config, README, CHANGELOG)	Removed deprecated PipelineStage and Item mocks, updated configuration, examples, and documentation to support the new design.

Files not reviewed (1)

• go.mod: Language not supported

Comments suppressed due to low confidence (1)

processor/error.go:51

• [nitpick] The variable name 'failEvery' could be more descriptive; consider renaming it to 'errorInterval' to better convey that it determines the interval at which errors are applied.

failEvery := int(1.0 / failFraction)