Skip to content

Conversation

@koenvo
Copy link
Contributor

@koenvo koenvo commented Oct 7, 2025

Add debug_source() helper for easy source development

Adds a new debug_source() function that simplifies testing and debugging sources during development.

Key features:

  • Single function call to test any source with minimal setup
  • Automatic ephemeral storage in /tmp (or persistent if needed)
  • Auto-configures logging
  • Returns engine for further inspection
  • Works with both simple sources and those using discover_selectors()

Example usage:

from ingestify import debug_source

source = MySource(name="test", ...)
engine = debug_source(
    source,
    dataset_type="match",
    data_spec_versions={"match": "v1"}
)

Also includes:

  • engine.run() alias for engine.load() (more intuitive)
  • Optional dev_dir parameter for custom storage location
  • Updated README with development workflow guide

Replaces boilerplate setup code in every source file with a single helper function.

koenvo added 4 commits October 7, 2025 20:45
- Add debug_source() function for quick source testing
- Add engine.run() alias for engine.load()
- Add get_dev_engine() internal helper
- Update README with development workflow guide
- Add test for dev engine functionality
New public API additions warrant minor version bump:
- debug_source() helper function
- engine.run() alias
- Remove redundant use_discovered_selectors parameter
- Behavior is now automatic based on source having discover_selectors()
- Update docstrings and examples
- Cleaner API with one less parameter to worry about
@koenvo koenvo merged commit c7786cf into main Oct 7, 2025
8 checks passed
@koenvo koenvo deleted the feat/dev-engine branch October 7, 2025 20:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants