Skip to content

fix: reduce driver/base import overhead#1615

Open
akashmalbari wants to merge 2 commits into
apache:mainfrom
akashmalbari:1246-import-overhead
Open

fix: reduce driver/base import overhead#1615
akashmalbari wants to merge 2 commits into
apache:mainfrom
akashmalbari:1246-import-overhead

Conversation

@akashmalbari
Copy link
Copy Markdown

Closes #1246.

This PR reduces import-time overhead for from hamilton import driver, base by deferring import-time work that is not needed during the initial driver/base import path.

Changes

  • Profiled from hamilton import driver, base using python -X importtime and cProfile.
  • Identified unnecessary import-time overhead in the driver/base import path.
  • Deferred the relevant import work so it only happens when needed.
  • Preserved existing public import behavior for:
    • from hamilton import driver, base
    • from hamilton.driver import Driver
    • from hamilton import base

How I tested this

Import smoke test:

python -c "from hamilton import driver, base; print(driver.Driver); print(base.DictResult)"

## Profiling

Command used:

```bash
python -X importtime -c "from hamilton import driver, base"
python -c "import cProfile; cProfile.run('from hamilton import driver, base', '/tmp/hamilton_import_before.prof')"
python -c "import cProfile; cProfile.run('from hamilton import driver, base', '/tmp/hamilton_import_after.prof')"

Before: 1.068s importtime, 0.894s cProfile.
After: 0.111s importtime, 0.188s cProfile.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[ergonomics] profile driver and base module imports

1 participant