Home

pytransflow is a simple library for record-level processing using flows of transformations defined as YAML files.

Some of the features pytransflow provides include:

• Define processing flows using YAML files
• Use all kinds of flow configurations to fine-tune the flow
• Leverage pydantic's features for data validation
• Apply transformations only if defined condition is met
• Build your own library of transformations
• Use multiple input and output datasets
• Ignore specific errors during processing
• Set conditions for output datasets
• Track failed records
• Define flow fail scenarios
• Process records in parallel
• Use flow level variables

The following sections provide more details on how to use this library:

• Getting Started
  • Installation
  • Simple Flow
  • Path Separator
  • Conditions
  • Datasets
  • Failed Record
  • Ignore Errors
  • Schema Validation
• Flow Configuration
  • Instant Fail
  • Fail Scenarios
  • Parallelization
  • Flow Variables
• How It Works
• Global Configuration
• Custom Transformations
• Custom Evaluation
• Contributing
• Next Steps

Acknowledgment

I want to express my gratitude to Bas Van Kortenhof for sparking the idea that led to this personal project. His work had a significant impact on both my own work and the direction of my ideas in general, including this project. Working alongside Bas was a rewarding experience, and I learned a great deal from our collaboration.

I’d also like to thank my colleagues for our discussions on potential features and their practicality within the library. The interchange of ideas and feedback played a significant role in shaping the direction and functionality of the features I implemented.

License

MIT