feat(logging): pluggable custom logger support#73
Merged
Conversation
added 6 commits
April 17, 2026 17:14
Introduce pre_init and post_init hook folders executed via runpy around initialize_pipeline(), with extensions/libraries on sys.path and legacy flat extensions/ deprecated. - Add extension_loader (add_extensions_libraries_to_sys_path, run_init_hooks) - Extend constants for libraries/pre_init/post_init paths; resolve python function files under extensions/libraries before legacy extensions/ - Wire hooks in DLTPipelineBuilder; bump version; add unit tests - Update docs, scaffold, samples (bronze), Cursor assets/SKILL, VERSION
…tensions/ Replaces the single-directory extensions/ convention with a structured src/ layout that separates cluster libraries, spec-referenced Python, and lifecycle init scripts. Renames extension_loader.py to bundle_loader.py to reflect its broader role. Layout changes - Pipeline bundles: src/libraries/ (wheels + sys.path), src/python/ (spec Python), src/init/pre/ and src/init/post/ (lifecycle scripts) - Framework bundle: custom code lives exclusively under src/local/ (libraries/, python/, init/pre/, init/post/) — no top-level src/ equivalents - FrameworkPaths constants renamed to LOCAL_* to reflect local/ prefix; all *sourcePath confs resolve to src/ so constants omit the src/ segment Deprecation - extensions/ on sys.path deprecated (v0.13.0), removed in v1.0.0 - extensions/ is registered unconditionally alongside src/python/ when both exist - extensions/libraries/ (never shipped) dropped with no deprecation window Code - extension_loader.py → bundle_loader.py; uses FrameworkPaths.LOCAL_* for framework paths and PipelineBundlePaths.* for bundle paths - dataflow_spec_builder: search paths updated to LOCAL_* framework constants - constants.py: FrameworkPaths gains LOCAL_* paths; PipelineBundlePaths paths corrected to omit redundant src/ prefix Samples & template - bronze_sample migrated: extensions/libraries/ → src/python/, extensions/pre_init|post_init/ → src/init/pre|post/ - New libraries demo: phonenumbers wheel in src/libraries/, installed via environment.dependencies; demonstrates full install→import chain - pipeline_bundle_template scaffolded with README.md for each new src/ directory - src/local/ scaffolded in framework src/ with README.md files Docs & ADRs - feature_python_extensions.rst rewritten: new title, bundle context table, src/python/ replaces src/extensions/ throughout, environment.dependencies replaces libraries: whl/pypi syntax - feature_python_functions.rst, feature_python_source.rst: updated to src/python/ with deprecation notices for src/extensions/ - build_pipeline_bundle_steps.rst, build_pipeline_bundle_structure.rst: updated directory diagrams and added steps for libraries/python/init - docs/decisions/0001-bundle-src-layout.md, 0002-extensions-deprecation.md: new ADRs
…tdout example Replaces the hardcoded utility.set_logger() call with a merge-driven resolution pipeline that supports optional custom loggers configured via logger.json in both the framework bundle (config/override/) and the pipeline bundle (pipeline_configs/). Core changes: - src/logger.py: new module — create_default_logger, load/merge logger.json, resolve_logger, CompositeLogger (custom primary + stdout mirror), and the top-level resolve_pipeline_logger entry point - src/config/default/logger.json: shipped default (enabled: false) - src/dlt_pipeline_builder.py: wire resolve_pipeline_logger; call register_bundle_sys_paths() before logger init so src/local/libraries/ modules are importable when the factory is loaded - src/bundle_loader.py: make logger optional to break the init chicken-and-egg - src/utility.py: convert debug print() calls to logger.debug(); add optional logger param to get_data_from_files_parallel - tests/test_logger.py: unit tests for merge, resolve, and CompositeLogger - docs/source/feature_logging.rst: full rewrite; includes custom logger contract and step-by-step structured JSON logger example
1 task
rederik76
changed the base branch from
main
to
feature/init-hooks-extensions-layout
added 4 commits
May 17, 2026 16:26
Contributor
|
Big fan of this @rederik76, thanks for your efforts! |
haillew
approved these changes
May 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces the hardcoded utility.set_logger() call with a merge-driven resolution
pipeline that supports optional custom loggers configured via logger.json in both
the framework bundle (config/override/) and the pipeline bundle (pipeline_configs/).
Core changes:
resolve_logger, CompositeLogger (custom primary + stdout mirror), and the
top-level resolve_pipeline_logger entry point
register_bundle_sys_paths() before logger init so src/local/libraries/
modules are importable when the factory is loaded
logger param to get_data_from_files_parallel
and step-by-step structured JSON logger example