Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
1 parent
e3b1036
commit 4c0f7d1
Showing
4 changed files
with
116 additions
and
124 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
python_modules/dagster/README.md |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,4 +1,4 @@ | ||
include README.rst | ||
include README.md | ||
include LICENSE | ||
include COPYING | ||
include dagster/py.typed | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,113 @@ | ||
<p align="center"> | ||
<a target="_blank" href="https://dagster.io"> | ||
<picture> | ||
<source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/dagster-io/dagster/master/.github/dagster-logo-dark.svg"> | ||
<img alt="dagster logo" src="https://raw.githubusercontent.com/dagster-io/dagster/master/.github/dagster-logo-light.svg" width="auto" height="120"> | ||
</picture> | ||
</a> | ||
<br /><br /> | ||
<a target="_blank" href="https://twitter.com/dagsterio"> | ||
<img src="https://img.shields.io/twitter/follow/dagsterio?labelColor=4F43DD&color=163B36&logo=twitter&style=flat"> | ||
</a> | ||
<a target="_blank" href="https://dagster.io/slack"> | ||
<img src="https://dagster-slackin.herokuapp.com/badge.svg?bg=163B36"> | ||
</a> | ||
<a target="_blank" href="https://github.com/dagster-io/dagster"> | ||
<img src="https://img.shields.io/github/stars/dagster-io/dagster?labelColor=4F43DD&color=163B36&logo=github"> | ||
<br /><br /> | ||
<a target="_blank" href="https://github.com/dagster-io/dagster/blob/master/LICENSE"> | ||
<img src="https://img.shields.io/badge/License-Apache_2.0-blue.svg?label=license&labelColor=4F43DD&color=163B36"> | ||
</a> | ||
<a target="_blank" href="https://pypi.org/project/dagster/"> | ||
<img src="https://img.shields.io/pypi/v/dagster?labelColor=4F43DD&color=163B36"> | ||
</a> | ||
<a target="_blank" href="https://coveralls.io/github/dagster-io/dagster?branch=master"> | ||
<img src="https://img.shields.io/coveralls/github/dagster-io/dagster/master?labelColor=4F43DD&color=163B36"> | ||
</a> | ||
<img src="https://img.shields.io/pypi/pyversions/dagster?labelColor=4F43DD&color=163B36"> | ||
</p> | ||
|
||
Dagster is an orchestrator that's designed for developing and maintaining data assets, such as tables, data sets, machine learning models, and reports. | ||
|
||
You declare functions that you want to run and the data assets that those functions produce or update. Dagster then helps you run your functions at the right time and keep your assets up-to-date. | ||
|
||
Dagster is built to be used at every stage of the data development lifecycle - local development, unit tests, integration tests, staging environments, all the way up to production. | ||
|
||
If you're new to Dagster, we recommend reading about its [core concepts](https://docs.dagster.io/concepts) or learning with the hands-on [tutorial](https://docs.dagster.io/tutorial). | ||
|
||
An asset graph defined in Python: | ||
|
||
```python | ||
from dagster import asset | ||
from pandas import DataFrame, read_html, get_dummies | ||
from sklearn.linear_model import LinearRegression | ||
|
||
@asset | ||
def country_populations() -> DataFrame: | ||
df = read_html("https://tinyurl.com/mry64ebh")[0] | ||
df.columns = ["country", "continent", "rg", "pop2018", "pop2019", "change"] | ||
df["change"] = df["change"].str.rstrip("%").str.replace("−", "-").astype("float") | ||
return df | ||
|
||
@asset | ||
def continent_change_model(country_populations: DataFrame) -> LinearRegression: | ||
data = country_populations.dropna(subset=["change"]) | ||
return LinearRegression().fit( | ||
get_dummies(data[["continent"]]), data["change"] | ||
) | ||
|
||
@asset | ||
def continent_stats( | ||
country_populations: DataFrame, continent_change_model: LinearRegression | ||
) -> DataFrame: | ||
result = country_populations.groupby("continent").sum() | ||
result["pop_change_factor"] = continent_change_model.coef_ | ||
return result | ||
``` | ||
|
||
The graph loaded into Dagster's web UI: | ||
|
||
<p align="center"> | ||
<img width="478" alt="image" src="https://user-images.githubusercontent.com/654855/183537484-48dde394-91f2-4de0-9b17-a70b3e9a3823.png"> | ||
</p> | ||
|
||
## Installation | ||
|
||
Dagster is available on PyPI and officially supports Python 3.7+. | ||
|
||
```bash | ||
pip install dagster dagit | ||
``` | ||
|
||
This installs two modules: | ||
|
||
- **Dagster**: The core programming model. | ||
- **Dagit**: The web interface for developing and operating Dagster jobs and assets. | ||
|
||
## Documentation | ||
|
||
You can find the full Dagster documentation [here](https://docs.dagster.io). | ||
|
||
## Community | ||
|
||
Connect with thousands of other data practitioners building with Dagster. Share knowledge, get help, | ||
and contribute to the open-source project. To see featured material and upcoming events, check out | ||
our [Dagster Community](https://dagster.io/community) page. | ||
|
||
Join our community here: | ||
|
||
- 🌟 [Star us on Github](https://github.com/dagster-io/dagster) | ||
- 🐦 [Follow us on Twitter](https://twitter.com/dagsterio) | ||
- 📺 [Subscribe to our YouTube channel](https://www.youtube.com/channel/UCfLnv9X8jyHTe6gJ4hVBo9Q) | ||
- 📚 [Read our blog posts](https://dagster.io/blog) | ||
- 👋 [Join us on Slack](https://dagster.io/slack) | ||
- ✏️ [Start a Github Discussion](https://github.com/dagster-io/dagster/discussions) | ||
|
||
## Contributing | ||
|
||
For details on contributing or running the project for development, check out our [contributing | ||
guide](https://docs.dagster.io/community/contributing/). | ||
|
||
## License | ||
|
||
Dagster is [Apache 2.0 licensed](https://github.com/dagster-io/dagster/blob/master/LICENSE). |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters