[DOC-430] Add Managing Concurrency Guide #24296

cnolanminich · 2024-09-06T19:20:16Z

Adapted / edited the original doc for the new guide format.

I had a couple of questions as I was working on it:

Should we keep the following sections or move them to a "concepts" page?
- glossary
- Setting priority for ops/assets
- Freeing concurrency slots
- Throttling concurrency-limited runs
- Troubleshooting
tag_concurrency_limits could use a major look -- is bug noted here accurate, or does this still work with Postgres / MySQL?
question: include Dagster insance settings (dagster.yaml / Dagster+ Deployment settings) examples as inline YAML, keep the folder with the dagster.yaml? To repro, need to export DAGSTER_HOME=$(pwd)/global_concurrency` (and for tag_concurrency, need an OSS version using Postgres / MYSQL)
(from Linear ticket) How can I set concurrency limits on a resource? -- I don't think this is possible, other than via using the concurrency tags.

…cy-in-your-pipelines

github-actions · 2024-09-06T19:22:48Z

Deploy preview for dagster-docs-beta ready!

Preview available at https://dagster-docs-beta-br53lag1f-elementl.vercel.app

Direct link to changed pages:

PedramNavid · 2024-09-25T02:27:43Z

@cnolanminich I noticed this is a draft, would this be ready for review you think?

cnolanminich · 2024-09-27T19:25:25Z

@PedramNavid yes I think so -- I wrote this during the week after docathon and wasn't sure if I should mark it ready for review or as a draft until some of the questions in my comment were addressed. Will mark it ready for review!

PedramNavid · 2024-09-27T22:21:21Z

I think you will need to tag a reviewer to help get this over the finish line as well.
As to these questions:

Should we keep the following sections or move them to a "concepts" page?

glossary

i don't see this in the original or old docs, prefer not to have one.

Setting priority for ops/assets
Freeing concurrency slots
Throttling concurrency-limited runs

i think the above 3 can be part of a reference of configurations

Troubleshooting

Can be included here as well yes.

graphite-app · 2024-09-28T20:10:14Z

Graphite Automations

"Label and add CE on all Docs" took an action on this PR • (09/28/24)

3 reviewers were added to this PR based on Pedram Navid's automation.

cnolanminich · 2024-09-30T20:37:19Z

Setting priority for ops/assets
Freeing concurrency slots
Throttling concurrency-limited runs

i think the above 3 can be part of a reference of configurations

Do you mean added as a reference of configuration to this page, or as a separate page? Will add a troubleshooting section and then tag reviewers, thanks!

PedramNavid · 2024-10-01T05:30:00Z

Setting priority for ops/assets
Freeing concurrency slots
Throttling concurrency-limited runs

i think the above 3 can be part of a reference of configurations

Do you mean added as a reference of configuration to this page, or as a separate page? Will add a troubleshooting section and then tag reviewers, thanks!

A separate page, fine to link to a [/todo] for now!

cnolanminich · 2024-10-18T20:24:40Z

@PedramNavid the deploy docs revamp action is failing, it had succeeded previously and I ran yarn build successfully locally -- anything I might be missing on why the build isn't working?

@prha it looks like you were the original author of the limiting concurrency docs page -- would you be able to review this from a content perspective or recommend someone who can?

cmpadden · 2024-10-18T20:34:02Z

@PedramNavid the deploy docs revamp action is failing, it had succeeded previously and I ran yarn build successfully locally -- anything I might be missing on why the build isn't working?

@prha it looks like you were the original author of the limiting concurrency docs page -- would you be able to review this from a content perspective or recommend someone who can?

Here's the full error message, @cnolanminich:

[ERROR] Error: Unable to build website for locale en.
    at tryToBuildLocale (/vercel/path0/docs/docs-beta/node_modules/@docusaurus/core/lib/commands/build.js:54:19)
    at async /vercel/path0/docs/docs-beta/node_modules/@docusaurus/core/lib/commands/build.js:65:9
    at async mapAsyncSequential (/vercel/path0/docs/docs-beta/node_modules/@docusaurus/utils/lib/jsUtils.js:21:24)
    at async Command.build (/vercel/path0/docs/docs-beta/node_modules/@docusaurus/core/lib/commands/build.js:63:5) {
  [cause]: Error: Docusaurus found broken links!
  
  Please check the pages of your site in the list below, and make sure you don't reference any path that does not exist.
  Note: it's possible to ignore broken links with the 'onBrokenLinks' Docusaurus configuration, and let the build pass.
  
  Exhaustive list of all broken links found:
  - Broken link on source page path = /guides/tbd/managing-concurrency:
     -> linking to /guides/tbd/running-local-ui-development

…cy-in-your-pipelines

cnolanminich · 2024-10-18T20:59:40Z

@PedramNavid the deploy docs revamp action is failing, it had succeeded previously and I ran yarn build successfully locally -- anything I might be missing on why the build isn't working?
@prha it looks like you were the original author of the limiting concurrency docs page -- would you be able to review this from a content perspective or recommend someone who can?

Here's the full error message, @cnolanminich:

[ERROR] Error: Unable to build website for locale en.
    at tryToBuildLocale (/vercel/path0/docs/docs-beta/node_modules/@docusaurus/core/lib/commands/build.js:54:19)
    at async /vercel/path0/docs/docs-beta/node_modules/@docusaurus/core/lib/commands/build.js:65:9
    at async mapAsyncSequential (/vercel/path0/docs/docs-beta/node_modules/@docusaurus/utils/lib/jsUtils.js:21:24)
    at async Command.build (/vercel/path0/docs/docs-beta/node_modules/@docusaurus/core/lib/commands/build.js:63:5) {
  [cause]: Error: Docusaurus found broken links!
  
  Please check the pages of your site in the list below, and make sure you don't reference any path that does not exist.
  Note: it's possible to ignore broken links with the 'onBrokenLinks' Docusaurus configuration, and let the build pass.
  
  Exhaustive list of all broken links found:
  - Broken link on source page path = /guides/tbd/managing-concurrency:
     -> linking to /guides/tbd/running-local-ui-development

Thanks! Fixed the link and it built successfully 🎉

prha · 2024-10-22T17:09:18Z

docs/docs-beta/docs/guides/managing-concurrency.md

+    <CodeExample filePath="guides/tbd/concurrency-tag-key-asset.py" language="python" title="No more than 1 asset running with a tag of 'database'" />
+
+  </TabItem>
+  <TabItem value="Op Tag" label="Asset tag concurrency limits">


fixed the label

prha · 2024-10-22T17:10:01Z

docs/docs-beta/docs/guides/managing-concurrency.md

+
+<Tabs>
+  <TabItem value="Asset Tag" label="Asset tag concurrency limits">
+    <CodeExample filePath="guides/tbd/concurrency-tag-key-asset.py" language="python" title="No more than 1 asset running with a tag of 'database'" />


maybe too wordy, but consider adding "across all runs"

it's more wordy but I think precision is good for this since it's a complex topic

prha · 2024-10-22T17:10:15Z

docs/docs-beta/docs/guides/managing-concurrency.md

+
+  </TabItem>
+  <TabItem value="Op Tag" label="Asset tag concurrency limits">
+  <CodeExample filePath="guides/tbd/concurrency-tag-key-op.py" language="python" title="No more than 1 op running with a tag of 'database'" />


ditto ("across all runs")

prha · 2024-10-22T17:11:40Z

docs/docs-beta/docs/guides/managing-concurrency.md

+Or you can configure it in the job definition:
+
+<Tabs>
+  <TabItem value="Asset Tag with Job" label="Asset tag concurrency limits in a job">


in a job is a little bit misleading... It's really within a run.

e.g. I can kick off two runs of job foo, and each run will have 1 "database" tag asset running.

yes, that was sloppy, corrected!

prha · 2024-10-22T17:14:33Z

docs/docs-beta/docs/guides/managing-concurrency.md

+  </TabItem>
+</Tabs>
+
+Or you can configure it in the job definition:


It might be helpful to explain why you might use one versus the other.

For example... if you have some resource-constraints on something like BigQuery, you might want to configure global concurrency limits so that you don't tax this resource which is used across runs.

The run-scoped tag concurrency limits are useful when limiting compute resources within the run, if it's using something like the multiprocess executor.

good point, reworded / added some context on when you might want to do this

prha · 2024-10-22T17:16:46Z

docs/docs-beta/docs/guides/managing-concurrency.md

+
+Need screenshot here
+
+## Prevent runs from starting if another run is already occurring (advanced)


should we talk about block_op_concurrency_limited_runs at all? There's a section here about it in the old docs: https://docs.dagster.io/guides/limiting-concurrency-in-data-pipelines#throttling-concurrency-limited-runs

Pedram and I decided to punt that and a few other items to a "reference" page to be written

cnolanminich

thanks @prha! updated per your feedback

cnolanminich · 2024-10-23T17:17:40Z

docs/docs-beta/docs/guides/managing-concurrency.md

+Or you can configure it in the job definition:
+
+<Tabs>
+  <TabItem value="Asset Tag with Job" label="Asset tag concurrency limits in a job">


yes, that was sloppy, corrected!

cnolanminich · 2024-10-23T17:18:09Z

docs/docs-beta/docs/guides/managing-concurrency.md

+  </TabItem>
+</Tabs>
+
+Or you can configure it in the job definition:


good point, reworded / added some context on when you might want to do this

cnolanminich · 2024-10-23T17:18:26Z

docs/docs-beta/docs/guides/managing-concurrency.md

+    <CodeExample filePath="guides/tbd/concurrency-tag-key-asset.py" language="python" title="No more than 1 asset running with a tag of 'database'" />
+
+  </TabItem>
+  <TabItem value="Op Tag" label="Asset tag concurrency limits">


fixed the label

cnolanminich · 2024-10-23T17:18:30Z

docs/docs-beta/docs/guides/managing-concurrency.md

+
+<Tabs>
+  <TabItem value="Asset Tag" label="Asset tag concurrency limits">
+    <CodeExample filePath="guides/tbd/concurrency-tag-key-asset.py" language="python" title="No more than 1 asset running with a tag of 'database'" />


it's more wordy but I think precision is good for this since it's a complex topic

cnolanminich · 2024-10-23T17:19:02Z

docs/docs-beta/docs/guides/managing-concurrency.md

+
+Need screenshot here
+
+## Prevent runs from starting if another run is already occurring (advanced)


Pedram and I decided to punt that and a few other items to a "reference" page to be written

cmpadden

I think that this guide has come a long way, and I'd love to get this merged. I've been sharing the PR to this to some community members, and would love to get this on docs-preview.dagster.io.

We can definitely iterate more, and will likely do another pass of everything again.

cnolanminich · 2024-10-29T19:04:27Z

Per @cmpadden, the oss-internal-compat buildkite failure is flaky and is ok to merge

cnolanminich added 4 commits September 6, 2024 15:05

add managing concurrency

7386b1d

clean up comments

4a9d7a2

vale lint

0f5ff67

Merge branch 'master' into christian/doc-430-write-managing-concurren…

0cfb549

…cy-in-your-pipelines

cnolanminich marked this pull request as ready for review September 27, 2024 19:25

graphite-app bot added the area: docs Related to documentation in general label Sep 27, 2024

graphite-app bot requested review from cmpadden, PedramNavid and C00ldudeNoonan September 28, 2024 20:06

cnolanminich added 3 commits October 18, 2024 09:31

add troubleshooting

994ff8d

formatting

9953d26

vale suggestion

ce3c5f7

cnolanminich requested a review from prha October 18, 2024 20:15

add next steps

2d4a066

cnolanminich added 3 commits October 18, 2024 16:49

Merge branch 'master' into christian/doc-430-write-managing-concurren…

6909fb6

…cy-in-your-pipelines

merge from master

b1011f4

only 1 guide

985a5c6

prha reviewed Oct 22, 2024

View reviewed changes

updates from review

64a310e

cnolanminich commented Oct 23, 2024

View reviewed changes

cmpadden approved these changes Oct 29, 2024

View reviewed changes

appease ruff

b765d9d

cnolanminich merged commit 910f5c5 into master Oct 29, 2024
1 of 3 checks passed

cnolanminich deleted the christian/doc-430-write-managing-concurrency-in-your-pipelines branch October 29, 2024 19:08

garethbrickman mentioned this pull request Nov 7, 2024

It's unclear how/where I should configure job concurrency in Dagster OSS based on the documentation #19402

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[DOC-430] Add Managing Concurrency Guide #24296

[DOC-430] Add Managing Concurrency Guide #24296

cnolanminich commented Sep 6, 2024

github-actions bot commented Sep 6, 2024 •

edited

Loading

PedramNavid commented Sep 25, 2024

cnolanminich commented Sep 27, 2024

PedramNavid commented Sep 27, 2024

graphite-app bot commented Sep 28, 2024

cnolanminich commented Sep 30, 2024

PedramNavid commented Oct 1, 2024

cnolanminich commented Oct 18, 2024

cmpadden commented Oct 18, 2024

cnolanminich commented Oct 18, 2024

prha Oct 22, 2024

cnolanminich Oct 23, 2024

prha Oct 22, 2024

cnolanminich Oct 23, 2024

prha Oct 22, 2024

prha Oct 22, 2024

cnolanminich Oct 23, 2024

prha Oct 22, 2024

cnolanminich Oct 23, 2024

prha Oct 22, 2024

cnolanminich Oct 23, 2024

cnolanminich left a comment

cnolanminich Oct 23, 2024

cnolanminich Oct 23, 2024

cnolanminich Oct 23, 2024

cnolanminich Oct 23, 2024

cnolanminich Oct 23, 2024

cmpadden left a comment •

edited

Loading

cnolanminich commented Oct 29, 2024


		Need screenshot here

		## Prevent runs from starting if another run is already occurring (advanced)

[DOC-430] Add Managing Concurrency Guide #24296

[DOC-430] Add Managing Concurrency Guide #24296

Conversation

cnolanminich commented Sep 6, 2024

github-actions bot commented Sep 6, 2024 • edited Loading

PedramNavid commented Sep 25, 2024

cnolanminich commented Sep 27, 2024

PedramNavid commented Sep 27, 2024

graphite-app bot commented Sep 28, 2024

Graphite Automations

cnolanminich commented Sep 30, 2024

PedramNavid commented Oct 1, 2024

cnolanminich commented Oct 18, 2024

cmpadden commented Oct 18, 2024

cnolanminich commented Oct 18, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cnolanminich left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cmpadden left a comment • edited Loading

Choose a reason for hiding this comment

cnolanminich commented Oct 29, 2024

github-actions bot commented Sep 6, 2024 •

edited

Loading

cmpadden left a comment •

edited

Loading