changelog 1.1.11 pt 2 (#11800)

CHANGES.md

@@ -4,23 +4,35 @@
 
 ### New
 
-- [dagster-snowflake] Snowflake resource and IO manager now support private key authentication with unencrypted keys
+- Added a new `dagster dev` command that can be used to run both Dagit and the Dagster daemon in the same process during local development. See the new Running Dagster Locally guide in the docs for more information.
+- Added instructions for installing the `dagster` package on M1 and M2 Macs that avoids installation errors when building the `grpcio` package. See [the Installing Dagster guide in the docs](https://docs.dagster.io/getting-started/install) for more information.
+- `create_repository_using_definitions_args` has been added for users to backport their repository definitions to the new `Definitions` API
+- When running Dagit on your local machine, a prompt will now appear that allows you to optionally enter an email address to receive Dagster security updates or subscribe to the Dagster newsletter. This prompt can be dismissed in the UI, or permanently disabled by adding the following to your dagster.yaml file:
+
+```
+nux:
+  enabled: false
+```
+
 - [dagit] Improved query performance on Scheduled Runs page.
 - [dagit] Added or improved document (browser tab) titles throughout Dagit.
+- [dagster-snowflake] Snowflake resource and IO manager now support private key authentication with unencrypted keys
 - [dagster-dbt] The op created when using `load_assets_from_dbt*` is now configurable, allowing you to pass in specific parameters to the underlying dbt command (such as `--full-refresh`). To do so, supply config of the form `{"ops": {"run_dbt_<abcde>": {"config": {"full_refresh": True}}}}` when defining your asset job, or in Dagit.
-- A `dagster dev` which spins up both dagit and the daemon, and requires no `$DAGSTER_HOME` env var to be set
-- `create_repository_using_definitions_args` has been added for users to backport their repository definitions to the new `Definitions` API
 
 ### Bugfixes
 
-- [dagit] Fixed broken code location names in the “Definitions” filter dialog accessible from the left navigation.
 - For time-partitioned assets, the `build_asset_reconciliation_sensor` will now only materialize partitions from the past day. Previously, if a new asset was added with a large number of missing partitions, a run for each of the missing partitions would be launched immediately.
 - A variety of performance improvements to the `build_asset_reconciliation_sensor`, which should help significantly speed up sensor evaluation in cases where there is a large number of assets or partitions.
+- [dagit] Fixed broken code location names in the “Definitions” filter dialog accessible from the left navigation.
 
 ### Breaking Changes
 
 - Strings with escaped characters are now invalid substrings of partition keys.
 
+### Experimental
+
+- [dagster-dbt] Added a utility to cache compilations from dbt Cloud jobs, allowing software-defined assets to be loaded faster from dbt Cloud jobs.
+
 ### Community Contributions
 
 - In dagster-airbyte, keys containing credentials are now considered a secret (thanks [joel-olazagasti](https://github.com/joel-olazagasti))!
@@ -181,7 +193,6 @@
 - [dagster-dbt] Fixed an error where dbt Cloud jobs with `dbt run` and `dbt run-operation` were incorrectly validated.
 - [dagster-airflow] `use_ephemeral_airflow_db` now works when running within a PEX deployment artifact.
 
-
 ### Documentation
 
 - New documentation for [Code locations](https://docs.dagster.io/concepts/code-locations) and how to define one using `Definitions`
@@ -201,7 +212,7 @@
 - Fixed a bug that caused `load_asset_value` to error with the default IO manager when a `partition_key` argument was provided.
 - Previously, trying to access `context.partition_key` or `context.asset_partition_key_for_output` when invoking an asset directly (e.g. in a unit test) would result in an error. This has been fixed.
 - Failure hooks now receive the original exception instead of `RetryRequested` when using a retry policy.
-- The LocationStateChange GraphQL subscription has been fixed (thanks @****[roeij](https://github.com/roeij) !)**
+- The LocationStateChange GraphQL subscription has been fixed (thanks @\***\*[roeij](https://github.com/roeij) !)**
 - Fixed a bug where a `sqlite3.ProgrammingError` error was raised when creating an ephemeral `DagsterInstance`, most commonly when `build_resources` was called without passing in an instance parameter.
 - [dagstermill] Jupyter notebooks now correctly render in Dagit on Windows machines.
 - [dagster-duckdb-pyspark] New `duckdb_pyspark_io_manager` helper to automatically create a DuckDB I/O manager that can store and load PySpark DataFrames.
@@ -214,7 +225,6 @@
 - [dagit] In op and asset descriptions in Dagit, rendered markdown no longer includes extraneous escape slashes.
 - Assorted typos and omissions fixed in the docs — thanks @[C0DK](https://github.com/C0DK) and @[akan72](https://github.com/akan72)!
 
-
 ### Experimental
 
 - As an optional replacement of the workspace/repository concepts, a new `Definitions` entrypoint for tools and the UI has been added. A single `Definitions` object per code location may be instantiated, and accepts typed, named arguments, rather than the heterogenous list of definitions returned from an `@repository`-decorated function. To learn more about this feature, and provide feedback, please refer to the [Github Discussion](https://github.com/dagster-io/dagster/discussions/10772).
@@ -228,7 +238,6 @@
 - Added docs for asset code versioning: [https://docs.dagster.io/concepts/assets/software-defined-assets#asset-code-versions](https://docs.dagster.io/concepts/assets/software-defined-assets#asset-code-versions)
 - Added docs for observable source assets: [https://docs.dagster.io/concepts/assets/asset-observations#observable-source-assets](https://docs.dagster.io/concepts/assets/asset-observations#observable-source-assets)
 
-
 # 1.1.5 (core) / 0.17.5 (libraries)
 
 ### Bugfixes
@@ -252,8 +261,8 @@
 - Fixed inconsistent io manager behavior when using `execute_in_process` on a `GraphDefinition` (it would use the `fs_io_manager` instead of the in-memory io manager)
 - Compute logs will now load in Dagit even when websocket connections are not supported.
 - [dagit] A handful of changes have been made to our URLs:
-    - The `/instance` URL path prefix has been removed. E.g. `/instance/runs` can now be found at `/runs`.
-    - The `/workspace` URL path prefix has been changed to `/locations`. E.g. the URL for job `my_job` in repository `foo@bar` can now be found at `/locations/foo@bar/jobs/my_job`.
+  - The `/instance` URL path prefix has been removed. E.g. `/instance/runs` can now be found at `/runs`.
+  - The `/workspace` URL path prefix has been changed to `/locations`. E.g. the URL for job `my_job` in repository `foo@bar` can now be found at `/locations/foo@bar/jobs/my_job`.
 - [dagit] The “Workspace” navigation item in the top nav has been moved to be a tab under the “Deployment” section of the app, and is renamed to “Definitions”.
 - [dagstermill] Dagster events can now be yielded from asset notebooks using `dagstermill.yield_event`.
 - [dagstermill] Failed notebooks can be saved for inspection and debugging using the new `save_on_notebook_failure` parameter.
@@ -270,13 +279,14 @@
 - [dagit] The run filter typeahead no longer surfaces key-value pairs when searching for `tag:`. This resolves an issue where retrieving the available tags could cause significant performance problems. Tags can still be searched with freeform text, and by adding them via click on individual run rows.
 - [dagit] Fixed an issue in the Runs tab for job snapshots, where the query would fail and no runs were shown.
 - [dagit] Schedules defined with cron unions displayed “Invalid cron string” in Dagit. This has been resolved, and human-readable versions of all members of the union will now be shown.
+
 ### Breaking Changes
 
 - You can no longer set an output’s asset key by overriding `get_output_asset_key` on the `IOManager` handling the output. Previously, this was experimental and undocumented.
 
 ### Experimental
 
-- Sensor and schedule evaluation contexts now have an experimental `log` property, which log events that can later be viewed in Dagit.  To enable these log views in dagit, navigate to the user settings and enable the `Experimental schedule/sensor logging view` option.  Log links will now be available for sensor/schedule ticks where logs were emitted.  Note: this feature is not available for users using the `NoOpComputeLogManager`.
+- Sensor and schedule evaluation contexts now have an experimental `log` property, which log events that can later be viewed in Dagit. To enable these log views in dagit, navigate to the user settings and enable the `Experimental schedule/sensor logging view` option. Log links will now be available for sensor/schedule ticks where logs were emitted. Note: this feature is not available for users using the `NoOpComputeLogManager`.
 
 # 1.1.3 (core) / 0.17.3 (libraries)
 
@@ -308,12 +318,12 @@
 - The new `@multi_asset_sensor` decorator enables defining custom sensors that trigger based on the materializations of multiple assets. The context object supplied to the decorated function has methods to fetch latest materializations by asset key, as well as built-in cursor management to mark specific materializations as “consumed”, so that they won’t be returned in future ticks. It can also fetch materializations by partition and mark individual partitions as consumed.
 - `RepositoryDefinition` now exposes a `load_asset_value` method, which accepts an asset key and invokes the asset’s I/O manager’s `load_input` function to load the asset as a Python object. This can be used in notebooks to do exploratory data analysis on assets.
 - With the new `asset_selection` parameter on `@sensor` and `SensorDefinition`, you can now define a sensor that directly targets a selection of assets, instead of targeting a job.
-- When running `dagit` or `dagster-daemon` locally, **environment variables included in a `.env` file** in the form  `KEY=value` in the same folder as the command will be automatically included in the environment of any Dagster code that runs, allowing you to easily use environment variables during local development.
+- When running `dagit` or `dagster-daemon` locally, **environment variables included in a `.env` file** in the form `KEY=value` in the same folder as the command will be automatically included in the environment of any Dagster code that runs, allowing you to easily use environment variables during local development.
 
 ### Dagit
 
 - The Asset Graph has been redesigned to make better use of color to communicate asset health. New status indicators make it easy to spot missing and stale assets (even on large graphs!) and the UI updates in real-time as displayed assets are materialized.
-- The Asset Details page has been redesigned and features a new side-by-side UI that makes it easier to inspect event metadata. A color-coded timeline on the partitions view  allows you to drag-select a time range and inspect the metadata and status quickly. The new view also supports assets that have been partitioned across multiple dimensions.
+- The Asset Details page has been redesigned and features a new side-by-side UI that makes it easier to inspect event metadata. A color-coded timeline on the partitions view allows you to drag-select a time range and inspect the metadata and status quickly. The new view also supports assets that have been partitioned across multiple dimensions.
 - The new Workspace page helps you quickly find and navigate between all your Dagster definitions. It’s also been re-architected to load significantly faster when you have thousands of definitions.
 - The Overview page is the new home for the live run timeline and helps you understand the status of all the jobs, schedules, sensors, and backfills across your entire deployment. The timeline is now grouped by repository and shows a run status rollup for each group.
 
@@ -326,8 +336,8 @@
 ### Database migration
 
 - Optional database schema migration, which can be run via `dagster instance migrate`:
-    - Improves Dagit performance by adding database indexes which should speed up the run view as well as a range of asset-based queries.
-    - Enables multi-dimensional asset partitions and asset versioning.
+  - Improves Dagit performance by adding database indexes which should speed up the run view as well as a range of asset-based queries.
+  - Enables multi-dimensional asset partitions and asset versioning.
 
 ### Breaking Changes and Deprecations
 
@@ -348,16 +358,16 @@
 - `build_asset_reconciliation_sensor` now launches runs to keep assets in line with their defined FreshnessPolicies.
 - The `FreshnessPolicy` object is now exported from the top level dagster package.
 - For assets with a `FreshnessPolicy` defined, their current freshness status will be rendered in the asset graph and asset details pages.
-- The AWS, GCS, and Azure compute log managers now take an additional config argument `upload_interval` which specifies in seconds, the interval in which partial logs will be uploaded to the respective cloud storage.  This can be used to display compute logs for long-running compute steps.
-- When running `dagit` or `dagster-daemon` locally, environment variables included in a `.env` file in the form  `KEY=value` in the same folder as the command will be automatically included in the environment of any Dagster code that runs, allowing you to easily test environment variables during local development.
+- The AWS, GCS, and Azure compute log managers now take an additional config argument `upload_interval` which specifies in seconds, the interval in which partial logs will be uploaded to the respective cloud storage. This can be used to display compute logs for long-running compute steps.
+- When running `dagit` or `dagster-daemon` locally, environment variables included in a `.env` file in the form `KEY=value` in the same folder as the command will be automatically included in the environment of any Dagster code that runs, allowing you to easily test environment variables during local development.
 - `observable_source_asset` decorator creates a `SourceAsset` with an associated `observation_fn` that should return a `LogicalVersion`, a new class that wraps a string expressing a version of an asset’s data value.
 - [dagit] The asset graph now shows branded compute_kind tags for dbt, Airbyte, Fivetran, Python and more.
 - [dagit] The asset details page now features a redesigned event viewer, and separate tabs for Partitions, Events, and Plots. This UI was previously behind a feature flag and is now generally available.
 - [dagit] The asset graph UI has been revamped and makes better use of color to communicate asset status, especially in the zoomed-out view.
 - [dagit] The asset catalog now shows freshness policies in the “Latest Run” column when they are defined on your assets.
 - [dagit] The UI for launching backfills in Dagit has been simplified. Rather than selecting detailed ranges, the new UI allows you to select a large “range of interest” and materialize only the partitions of certain statuses within that range.
 - [dagit] The partitions page of asset jobs has been updated to show per-asset status rather than per-op status, so that it shares the same terminology and color coding as other asset health views.
-- [dagster-k8s] Added an `execute_k8s_job` function that can be called within any op to run an image within a Kubernetes job. The implementation is similar to the build-in `k8s_job_op` , but allows additional customization - for example, you can incorporate the output of a previous op into the launched Kubernetes job by passing it into `execute_k8s_job`.  See the [dagster-k8s API docs](https://docs.dagster.io/_apidocs/libraries/dagster-k8s#ops) for more information.
+- [dagster-k8s] Added an `execute_k8s_job` function that can be called within any op to run an image within a Kubernetes job. The implementation is similar to the build-in `k8s_job_op` , but allows additional customization - for example, you can incorporate the output of a previous op into the launched Kubernetes job by passing it into `execute_k8s_job`. See the [dagster-k8s API docs](https://docs.dagster.io/_apidocs/libraries/dagster-k8s#ops) for more information.
 - [dagster-databricks] Environment variables used by dagster cloud are now automatically set when submitting databricks jobs if they exist, thank you @zyd14!
 - [dagstermill] `define_dagstermill_asset` now supports `RetryPolicy` . Thanks @**[nickvazz](https://github.com/dagster-io/dagster/commits?author=nickvazz)!**
 - [dagster-airbyte] When loading assets from an Airbyte instance using `load_assets_from_airbyte_instance`, users can now optionally customize asset names using `connector_to_asset_key_fn`.