Skip to content

Conversation

@neverett
Copy link
Contributor

@neverett neverett commented Apr 22, 2025

Summary & Motivation

Preview: https://dgify-docs-and-examples.archive.dagster-docs.io

This PR makes the following changes:

  • ETL pipeline tutorial updated to follow new project creation protocol and structure, and include Components
  • Installation doc updated to enable users to use/install create-dagster CLI
  • Quickstart updated to follow new project creation protocol and structure
  • Components docs moved from Labs to Build > Components
  • Docs in Build, Automate, Operate, Monitor & debug, and Test sections updated to follow new project structure:
    • Code examples updated to remove defs = dg.Definitions() lines and move resource definitions to separateresources.py files
    • Titles updated to reflect new project structure
    • Notes added about using the dg CLI to scaffold Dagster definitions, where appropriate
  • New dg CLI API docs section added, which includes create-dagster CLI documentation

How I Tested These Changes

Local build

Changelog

Insert changelog entry or delete this section.

Signed-off-by: nikki everett <nikki@dagsterlabs.com>
@neverett neverett requested a review from a team as a code owner April 22, 2025 22:31
@github-actions
Copy link

github-actions bot commented Apr 22, 2025

Deploy preview for dagster-docs ready!

Preview available at https://dagster-docs-fvnxx66tl-elementl.vercel.app
https://dgify-docs-and-examples.archive.dagster-docs.io

Direct link to changed pages:

@neverett neverett changed the title [WIP] dg-ify docs and examples [WIP - DO NOT MERGE] dg-ify docs and examples Apr 23, 2025
neverett and others added 23 commits April 24, 2025 14:05
## Summary & Motivation
Update the llm fine tuning example to `dg`

## How I Tested These Changes

## Changelog

> Insert changelog entry or delete this section.
## Summary & Motivation
Update the prompt engineering example to `dg`

## How I Tested These Changes

## Changelog

> Insert changelog entry or delete this section.
## Summary & Motivation
Updating hello world example on the main page by removing the explicit
`Definitions` object. This example would still load probably with
`dagster dev -f hello-world.py` and would not need the assets set in the
Definitions if scaffolded with `dg`.

## How I Tested These Changes

## Changelog

> Insert changelog entry or delete this section.
neverett and others added 5 commits June 20, 2025 12:57
## Summary & Motivation
Update examples consistency by removing `setup.py` from already merged
examples

## How I Tested These Changes

## Changelog

> Insert changelog entry or delete this section.
## Summary & Motivation
* Update navigation to use component language
* Include `definitions.py` in trees

#29506 (comment)

## How I Tested These Changes

## Changelog

> Insert changelog entry or delete this section.
## Summary & Motivation

* Remove dg plugin doc for now (created followup ticket to document
registry modules)
* Remove preview note from most Components docs (left on index page)
* Use `<project-name>` in project docs for consistency
* Update `tree` output in project file reference doc to include both
`uv` and `pip` output

## How I Tested These Changes

Local build

## Changelog

> Insert changelog entry or delete this section.

---------

Signed-off-by: nikki everett <nikki@dagsterlabs.com>
yuhan and others added 4 commits June 20, 2025 18:25
## Summary & Motivation

## How I Tested These Changes

## Changelog

> Insert changelog entry or delete this section.
## Summary & Motivation
* Rename components for better clarity
* Small typos and cleanup

## How I Tested These Changes

## Changelog

> Insert changelog entry or delete this section.
## Summary & Motivation

Addresses feedback on Quickstart and incorporates those changes into the
"Creating a project" doc as well.

## How I Tested These Changes

## Changelog

> Insert changelog entry or delete this section.

---------

Signed-off-by: nikki everett <nikki@dagsterlabs.com>
@cmpadden
Copy link
Contributor

cmpadden commented Jun 22, 2025

@dehume @neverett , below you can find my feedback for the ETL pipeline tutorial.

/etl-pipeline-tutorial

  • I'm of the opinion "ETL pipeline tutorial" should be called "Tutorial"

  • "Imports data into DuckDB" consider "Ingests data into DuckDB"

  • In overview consider making DuckDB, Sling, dbt, and Evidence links to those tools landing pages for users to learn more

  • "You will learn to: How to set up a Dagster project" might roll off the tongue better as "You will learn:"

  • I was prompted with "A uv installation was detected"; we should have an admonition warning someone of this message (EDIT: noticed we do this on /getting-started/quickstart)

    A uv installation was detected. Run uv sync? This will create a uv.lock file and the virtual environment you need to activate in order to work on this project. If you wish to use a non-uv package manager, choose "n". (y/n) [y]:

  • "Change into that project" may sound more natural as "Change directory into your new project"

/etl-pipeline-tutorial/extract-data

  • Agreed with Christian that uv pip install should be a toggle between uv add and pip install
  • I was confused for a split second that the dg list components in the docs had one entry whereas mine had 3
  • At this point it crossed my mind that we've pivoted pretty hard on Sling which may not be a tool familiar to most
  • Following the instructions to "Navigate to deployment and reload" I do not see the six assets as stated, we may want to remove that sentence and just say "You can alunch these assets in the UI"
  • The dg launch --assets * command failed for me until I quoted it dg launch --assets "*"
  • We should consider explaining how to see the fruits of the launch, by either looking at the duckdb file, or just informing them that the file is created

/etl-pipeline-tutorial/transform-data

  • No file title on first code block in step "3. Configure the dbt..."
  • Consider line highlighting the "one addition" we made, and using one code block
  • dg list defs was quite poorly formatted at this point, we may need some improved dynamic sizing / truncation still
  • Should we consider launching assets again at this point, and/or at the end of each chapter in the tutorial?

/etl-pipeline-tutorial/join-customer-and-order-data

  • Consider renaming title / sidebar of "Join customer and order data" to the higher level concept we're demonstrating of creating an asset for custom processing
  • I was unable to copy-and-paste this asset in working form as it's missing imports from dagster_duckdb import DuckDBResource
  • Typo in "2. Define the DuckDB resource" file name src/etl_tutorial/defs/resources/py

/etl-pipeline-tutorial/partition-asset

  • Backfill of payment_performance failed with the error:

    dagster._core.errors.DagsterExecutionStepExecutionError: Error occurred while executing op "payment_performance"::

    duckdb.duckdb.ParserException: Parser Error: syntax error at or near "credit_card"

/etl-pipeline-tutorial/create-a-sensor

  • Typo in command dg scaffold dagster.sensor sensors.py should be dg scaffold defs dagster.sensor sensors.py

/etl-pipeline-tutorial/visualize-data

  • We should consider informing the user to run the assets and view the results of their hard work

@dehume dehume mentioned this pull request Jun 23, 2025
@dehume
Copy link
Contributor

dehume commented Jun 23, 2025

Incorporated most of Colton's feedback into the ETL tutorial except holding off on:

Agreed with Christian that uv pip install should be a toggle between uv add and pip install

Elsewhere in the documentation it seemed like we were coalescing around just uv pip install.... So maybe more of a followup

@cmpadden
Copy link
Contributor

Incorporated most of Colton's feedback into the ETL tutorial except holding off on:

Agreed with Christian that uv pip install should be a toggle between uv add and pip install

Elsewhere in the documentation it seemed like we were coalescing around just uv pip install.... So maybe more of a followup

Sounds good - thank you Dennis.

neverett and others added 9 commits June 23, 2025 09:38
)

## Summary & Motivation

See title

## How I Tested These Changes

Local build

## Changelog

> Insert changelog entry or delete this section.

---------

Signed-off-by: nikki everett <nikki@dagsterlabs.com>
## Summary

Enable rendering nested command info in the Sphinx docs for `dg` CLI.
Enable for all commands except scaffold - this compiles correctly but
produces invalid `.md` files for some reason, investigating.
Signed-off-by: nikki everett <nikki@dagsterlabs.com>
## Summary & Motivation

Colton feedback

#29506 (comment)

And remove Sling

## How I Tested These Changes

## Changelog

> Insert changelog entry or delete this section.

---------

Signed-off-by: nikki everett <nikki@dagsterlabs.com>
Co-authored-by: Nikki Everett <neverett@users.noreply.github.com>
Co-authored-by: nikki everett <nikki@dagsterlabs.com>
Copy link
Member

@schrockn schrockn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

party approve

neverett added 2 commits June 23, 2025 21:36
Signed-off-by: nikki everett <nikki@dagsterlabs.com>
@neverett neverett merged commit e251e18 into master Jun 24, 2025
7 of 8 checks passed
@neverett neverett deleted the dgify-docs-and-examples branch June 24, 2025 14:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: docs Related to documentation in general

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants