1.0.16 (core) / 0.16.16 (libraries)
New
- [dagit] The new Overview and Workspace pages have been enabled for all users, after being gated with a feature flag for the last several releases. These changes include design updates, virtualized tables, and more performant querying.
- The top navigation has been updated to improve space allocation, with main nav links moved to the left.
- “Overview” is the new Dagit home page and “factory floor” view, were you can find the run timeline, which now offers time-based pagination. The Overview section also contains pages with all of your jobs, schedules, sensors, and backfills. You can filter objects by name, and collapse or expand repository sections.
- “Workspace” has been redesigned to offer a better summary of your repositories, and to use the same performant table views, querying, and filtering as in the Overview pages.
@asset
and@multi_asset
now accept aretry_policy
argument. (Thanks Adam Bloom!)- When loading an input that depends on multiple partitions of an upstream asset, the
fs_io_manager
will now return a dictionary that maps partition keys to the stored values for those partitions. (Thanks andrewgryan!). JobDefinition.execute_in_process
now accepts arun_config
argument even when the job is partitioned. If supplied, the run config will be used instead of any config provided by the job’sPartitionedConfig
.- The
run_request_for_partition
method on jobs now accepts arun_config
argument. If supplied, the run config will be used instead of any config provided by the job’sPartitionedConfig
. - The new
NotebookMetadataValue
can be used to report the location of executed jupyter notebooks, and Dagit will be able to render the notebook. - Resolving asset dependencies within a group now works with multi-assets, as long as all the assets within the multi-asset are in the same group. (Thanks @peay!)
- UPathIOManager, a filesystem-agnostic IOManager base class has been added - (Thanks @danielgafni!)
- A threadpool option has been added for the scheduler daemon. This can be enabled via your
dagster.yaml
file; check out the docs. - The default LocalComputeLogManager will capture compute logs by process instead of by step. This means that for the
in_process
executor, where all steps are executed in the same process, the captured compute logs for all steps in a run will be captured in the same file. - [dagster-airflow]
make_dagster_job_from_airflow_dag
now supports airflow 2, there is also a new mock_xcom parameter that will mock all calls to made by operators to xcom. - [helm] volume and volumeMount sections have been added for the dagit and daemon sections of the helm chart.
Bugfixes
- For partitioned asset jobs whose config is a hardcoded dictionary (rather than a
PartitionedConfig
), previouslyrun_request_for_partition
would produce a run with no config. Now, the run has the hardcoded dictionary as its config. - Previously, asset inputs would be resolved to upstream assets in the same group that had the same name, even if the asset input already had a key prefix. Now, asset inputs are only resolved to upstream assets in the same group if the input path only has a single component.
- Previously, asset inputs could get resolved to outputs of the same
AssetsDefinition
, through group-based asset dependency resolution, which would later error because of a circular dependency. This has been fixed. - Previously, the “Partition Status” and “Backfill Status” fields on the Backfill page in dagit were always incomplete and showed missing partitions. This has been fixed to accurately show the status of the backfill runs.
- [dagit] When viewing the config dialog for a run with a very long config, scrolling was broken and the “copy” button was not visible. This has been fixed.
- Executors now compress step worker arguments to avoid CLI length limits with large DAGs.
- [dagster-msteams] Longer messages can now be used in Teams HeroCard - (Thanks @jayhale!)
Documentation
- API docs for InputContext have been improved - (Thanks @peay!)
- [dagster-snowflake] Improved documentation for the Snowflake IO manager
All Changes
See All Contributors
- ee39fcd -
rm all_types from config_context (#10203)
by @alangenfeld - bad2e44 -
Also skip docs next (#10178)
by @jmsanders - bd6e269 -
Make handleLaunchResult agnostic to the query that returned the data (#10179)
by @salazarm - b5212c7 -
Make PythonPackages.get() more flexible to _/- (#10184)
by @jmsanders - 16bc4cf -
[dagit] Invert stored state for expand/collapse in Overview pages (#10212)
by @hellendag - af58bf9 -
[config] memoize ConfigType snap creation (#10210)
by @alangenfeld - 2eebc3c -
[config] avoid double init on field cache objects (#10214)
by @alangenfeld - fcda5f3 -
Override default io manager in more places (#10202)
by @johannkm - 59ae17e -
Fix black (#10216)
by @johannkm - d4334f1 -
type annotations on backfill-related code paths (#9402)
by @sryza - b56013e -
Fix lint (#10218)
by @johannkm - 3176dcc -
Add docs for customizing the serverless base image (#9571)
by @petehunt - a24298a -
Skip test-project image builds (#10099)
by @jmsanders - f7c058a -
[dagit] Updated asset details event view (#10143)
by @bengotow - 4ef9d27 -
[dagit] Fix AssetView.test key warnings (#10226)
by @hellendag - 2bbf998 -
Also trigger builds when tox.ini changes (#10209)
by @jmsanders - c1ff910 -
1.0.15 changelog (#10227)
by @jamiedemaria - 688a5d8 -
Automation: versioned docs for 1.0.15
by @elementl-devtools - 0478723 -
add materialization property on dagster event (#10230)
by @prha - cf8ad4a -
Parse HEAD commit in addition to BUILDKITE_MESSAGE (#10208)
by @jmsanders - 58ee17c -
Allow default io_manager load_input to support partitions of differing frequencies (#10172)
by @andrewgryan - 67313e8 -
Install the correct dagster-buildkite CLI (#10211)
by @jmsanders - cf84896 -
Type annotations in dagster-graphql (#10005)
by @smackesey - 174dd99 -
fix black, mypy (#10235)
by @prha - 3c89b70 -
[freshness-policy] [1/n] FreshnessPolicy object (#10024)
by @OwenKephart - 281595b -
Break up sensor_definition.py (#10181)
by @sryza - ffa01e1 -
Highlight config entry when being hovered in yaml editor (#10239)
by @salazarm - a5f0fa9 -
Resolve multi-asset deps when they have the same group (#10222)
by @peay - 77faa3c -
update apidoc for postgres (#10241)
by @prha - 35b8f8b -
fix schedules threading config (#10247)
by @alangenfeld - 8296cf7 -
[dagit] Ship Overview/Workspace (#10245)
by @hellendag - 24af700 -
Fix stray references to define_assets_job (#10199)
by @bengotow - 631c3ef -
captured log manager (#9429)
by @prha - 65b47a0 -
Allow for longer messages in Teams HeroCard (#10234)
by @jayhale - ee43f93 -
[dagit] Clean up code post-Overview changes (#10249)
by @hellendag - a097c43 -
[dagit] Move the refresh indicator on asset details pages to avoid flicker (#10253)
by @bengotow - ce7afb7 -
[dagit] Fix issues with embedding fonts in downloaded DAG SVGs (#10252)
by @bengotow - 37f2c95 -
[dagit] Fix daylight savings issue in humanCronString.test (#10263)
by @hellendag - 4ca8da0 -
[dagit] Make 50-level colors opaque (#10186)
by @hellendag - 997fdbb -
[dagit] Add query countdown/refresh to Timeline (#10250)
by @hellendag - da7555c -
Support overriding run config for partitions with execute_in_process (#10246)
by @sryza - a2bb0cf -
Add calendar icon (#10267)
by @salazarm - 5de98f5 -
UPathIOManager - filesystem-agnostic IOManager base (#10193)
by @danielgafni - 40bf6c7 -
rename logKey to fileKey / logFileKey for disambiguation with new API (#9956)
by @prha - dc667ef -
fix some cases in group-based asset dep resolution (#10266)
by @sryza - ef678e6 -
dagster-io/ui release notes (#10269)
by @salazarm - 966bfb7 -
fix UnresolvedAssetJobDefinition.run_request_for_partition when confi… (#10238)
by @sryza - 36f7c38 -
Improve doc for
InputContext.{dagster_type,metadata}(#10242)
by @peay - 69bfd0c -
Fetch asset materialization planned event from index shard in Sqlite (#10248)
by @clairelin135 - cb62bc4 -
fix #9193: add retry policy to @asset and @multi_asset (#10150)
by @adam-bloom - ca18259 -
adds dagster-aws install to dockerfile in docker deployment guide (#10225)
by @jamiedemaria - e6444be -
keep storage name for serialized event backcompat (#10280)
by @prha - 16b3cc3 -
graphql for captured log subscription (#9957)
by @prha - 9ccfe7c -
Add NotebookMetadataValue (#10278)
by @jamiedemaria - b88f309 -
Add volumes and volumeMounts to dagit and daemon in OSS helm chart (#10285)
by @gibsondan - 1830880 -
support new captured log API for process-based execution (#9958)
by @prha - a0a1340 -
add new capture APIs in frontend queries (#9959)
by @prha - 621b57a -
specify schema in pandas to_sql (#10289)
by @jamiedemaria - 7dd241f -
[dagit] Fix AssetView flakiness (#10293)
by @hellendag - fc6ea04 -
[dagit] Middle-truncate asset key path in virtualized list (#10275)
by @hellendag - b4c0930 -
Snowflake IO Manager API docs (#10175)
by @jamiedemaria - 2b3280f -
Compress execute_step args (#10244)
by @johannkm - 1dd602a -
fix black formatting (#10298)
by @prha - 9a3b5bc -
fix backfill table status (#10295)
by @prha - f7f1e13 -
run_config argument on run_request_for_partition (#10279)
by @sryza - 0e5386b -
[dagit] NotebookMetadataValue support (#10287)
by @jamiedemaria - f966490 -
[dagit] Fix very tall configs in run config dialog (#10301)
by @hellendag - f466eb7 -
Observable source asset decorator (#9899)
by @smackesey - 70c5f8d -
convert NoOpComputeLogManager to support captured log API (#9960)
by @prha - 4df6e55 -
convert S3 compute log manager to support new captured log API (#9961)
by @prha - c8dee2d -
Test createdBefore run filter (#10270)
by @benpankow - 7c38a8c -
fix build; isort; black (#10303)
by @prha - 082b407 -
Fix example scaffold (#10003)
by @smackesey - 6459b0f -
Add version to asset decorator (#10167)
by @smackesey - 3f99335 -
Fix logic to run lints on builds (#10304)
by @jmsanders - 98fe688 -
Don't repeat skip logs (#10307)
by @jmsanders - 0fc9f84 -
[jog] Add a utility method to execute op inside of graph (#10255)
by @dpeng817 - 6b333d5 -
[jog] execution_tests,hook_tests (#10254)
by @dpeng817 - 1283e63 -
Revert "convert S3 compute log manager to support new captured log API (#9961) (#10311)
by @prha - 834abba -
[docs] Automatically toggle tab components to display URL hash/anchor (#10231)
by @benpankow - 5b0d959 -
[dagstermill] notebook backed assets (#10277)
by @jamiedemaria - 345f0b3 -
notebook and assets example project (#10315)
by @jamiedemaria - 9157ebb -
[dagster-airbyte][managed-elements] Explicit handling of reconciling secret values (#10195)
by @benpankow - 251b666 -
[dagster-airflow] add airflow 2 support to
make_dagster_job_from_airflow_dag+ xcom mock option (#10337)
by @Ramshackle-Jamathon - 89948a3 -
1.0.16 changelog (#10340)
by @alangenfeld - afb20aa -
1.0.16
by @elementl-devtools