Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate types concepts docs to crag #4806

Merged
merged 2 commits into from Sep 13, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
50 changes: 25 additions & 25 deletions docs/content-crag/concepts/types.mdx
@@ -1,11 +1,11 @@
---
title: Types | Dagster
description: The Dagster type system helps you describe what kind of values your solids accept and produce.
description: The Dagster type system helps you describe what kind of values your ops accept and produce.
---

# Dagster Types

The Dagster type system helps you describe what kind of values your solids accept and produce.
The Dagster type system helps you describe what kind of values your ops accept and produce.

## Relevant APIs

Expand All @@ -19,19 +19,19 @@ The Dagster type system helps you describe what kind of values your solids accep

## Overview

Each solid input and output can be given a Dagster Type.
Each op input and output can be given a Dagster Type.

The type system:

- Is _gradual_ and _optional_. Pipelines can run without types specified explicitly, and specifying types in some places doesn't require that types be specified everywhere. Inputs and outputs default to the <PyObject module="dagster" object="Any" /> type.
- Is _gradual_ and _optional_. Jobs can run without types specified explicitly, and specifying types in some places doesn't require that types be specified everywhere. Inputs and outputs default to the <PyObject module="dagster" object="Any" /> type.

- Happens at solid execution time - each type defines a `type_check_fn` that knows how to check whether values match what it expects. When a type is specified for a solid's input, then the type check occurs immediately before the solid is executed. When a type is specified for a solid's output, then the type check occurs immediately after the solid is executed.
- Happens at op execution time - each type defines a `type_check_fn` that knows how to check whether values match what it expects. When a type is specified for a op's input, then the type check occurs immediately before the op is executed. When a type is specified for a op's output, then the type check occurs immediately after the op is executed.

- Is complimentary to the [PEP 484](https://www.python.org/dev/peps/pep-0484/) Python type system. PEP 484 annotations enable static checks that verify variables and return values match particular Python types, while the Dagster type system enables runtime checks that include arbitrary validation logic.

### DagsterTypes vs Python Types (mypy type-checking)

You should not use manually-defined <PyObject module="dagster" object="DagsterType" /> objects in your python type annotations. The simple rule of thumb is to annotate solid bodies just as you would a regular function, and if you wish to use a <PyObject module="dagster" object="DagsterType" /> alongside your annotations (to perform more complex validation than the default runtime typechecks), you should include this information in the relevant <PyObject module="dagster" object="InputDefinition" /> or <PyObject module="dagster" object="OutputDefinition" />.
You should not use manually-defined <PyObject module="dagster" object="DagsterType" /> objects in your python type annotations. The simple rule of thumb is to annotate op bodies just as you would a regular function, and if you wish to use a <PyObject module="dagster" object="DagsterType" /> alongside your annotations (to perform more complex validation than the default runtime typechecks), you should include this information in the relevant <PyObject module="dagster" object="InputDefinition" /> or <PyObject module="dagster" object="OutputDefinition" />.

You can see an example of this pattern [below](#using-dagster-types-with-pep-484-type-annotations).

Expand All @@ -48,12 +48,12 @@ EvenDagsterType = DagsterType(
)
```

Once created, types can be attached to solid <PyObject module="dagster" object="InputDefinition" pluralize /> and <PyObject module="dagster" object="OutputDefinition" pluralize />.
Once created, types can be attached to op <PyObject module="dagster" object="InputDefinition" pluralize /> and <PyObject module="dagster" object="OutputDefinition" pluralize />.

```python file=/concepts/types/types.py startafter=start_basic_even_type_no_annotations endbefore=end_basic_even_type_no_annotations
@solid(
input_defs=[InputDefinition("num", EvenDagsterType)],
output_defs=[OutputDefinition(EvenDagsterType)],
@op(
ins={"num": In(EvenDagsterType)},
out=Out(EvenDagsterType),
)
def double_even(num):
return num
Expand All @@ -65,22 +65,22 @@ The type system truly shines once the type check expresses richer behavior, such

If a Python input or output has a PEP 484 type annotation, and a DagsterType is not provided on the corresponding input or output definition, then Dagster will automatically generate a DagsterType that corresponds to the annotated Python type.

In this example, the defined solid will end up with a DagsterType named "MyClass" that:
In this example, the defined op will end up with a DagsterType named "MyClass" that:

- Shows up in Dagit in the representation of the solid.
- Is checked at runtime on the value returned by the solid.
- Shows up in Dagit in the representation of the op.
- Is checked at runtime on the value returned by the op.

```python file=/concepts/types/types.py startafter=start_auto_type endbefore=end_auto_type
class MyClass:
pass


@solid
def my_solid() -> MyClass:
@op
def my_op() -> MyClass:
return MyClass()
```

If the solid in the above example returned an object that was not an instance of MyClass, Dagster would raise an error after executing the solid.
If the op in the above example returned an object that was not an instance of MyClass, Dagster would raise an error after executing the op.

## Built-in Types

Expand Down Expand Up @@ -129,12 +129,12 @@ def test_dagster_type():

### Using Dagster Types with PEP 484 Type Annotations

Dagster types peacefully coexist with Python type annotations. In this example, the inputs and outputs of the solid compute function are integers, and the type check function for `EvenDagsterType` will be invoked at runtime to verify that they are even.
Dagster types peacefully coexist with Python type annotations. In this example, the inputs and outputs of the op compute function are integers, and the type check function for `EvenDagsterType` will be invoked at runtime to verify that they are even.

```python file=/concepts/types/types.py startafter=start_basic_even_type_with_annotations endbefore=end_basic_even_type_with_annotations
@solid(
input_defs=[InputDefinition("num", EvenDagsterType)],
output_defs=[OutputDefinition(EvenDagsterType)],
@op(
ins={"num": In(EvenDagsterType)},
out=Out(EvenDagsterType),
)
def double_even_with_annotations(num: int) -> int:
return num
Expand All @@ -144,7 +144,7 @@ def double_even_with_annotations(num: int) -> int:

### Associating Dagster Types with Python Types

As mentioned earlier, any Python type used to annotate an argument or return value of solid-decorated function has a corresponding Dagster type that will be used for the corresponding input or output definition.
As mentioned earlier, any Python type used to annotate an argument or return value of op-decorated function has a corresponding Dagster type that will be used for the corresponding input or output definition.

By default, Dagster will generate Dagster types for Python types that it's not aware of, but you can also explicitly control the Dagster type that will be used for a particular Python type.

Expand All @@ -160,7 +160,7 @@ This is designed for importing python types libraries that cannot be altered and
#### Decorating class declarations

```python file=/concepts/types/usable_as.py
from dagster import solid, usable_as_dagster_type
from dagster import op, usable_as_dagster_type


@usable_as_dagster_type
Expand All @@ -170,15 +170,15 @@ class EvenType:
self.num = num


@solid
@op
def double_even(even_num: EvenType) -> EvenType:
return EvenType(even_num.num * 2)
```

#### Mapping existing classes

```python file=/concepts/types/make_usable.py
from dagster import PythonObjectDagsterType, make_python_type_usable_as_dagster_type, solid
from dagster import PythonObjectDagsterType, make_python_type_usable_as_dagster_type, op


class EvenType:
Expand All @@ -192,7 +192,7 @@ EvenDagsterType = PythonObjectDagsterType(EvenType, name="EvenDagsterType")
make_python_type_usable_as_dagster_type(EvenType, EvenDagsterType)


@solid
@op
def double_even(even_num: EvenType) -> EvenType:
return EvenType(even_num.num * 2)
```
Expand Down
@@ -1,4 +1,4 @@
from dagster import PythonObjectDagsterType, make_python_type_usable_as_dagster_type, solid
from dagster import PythonObjectDagsterType, make_python_type_usable_as_dagster_type, op


class EvenType:
Expand All @@ -12,6 +12,6 @@ def __init__(self, num):
make_python_type_usable_as_dagster_type(EvenType, EvenDagsterType)


@solid
@op
def double_even(even_num: EvenType) -> EvenType:
return EvenType(even_num.num * 2)
@@ -1,4 +1,4 @@
from dagster import PythonObjectDagsterType, solid
from dagster import PythonObjectDagsterType, op


# start_object_type
Expand All @@ -12,7 +12,7 @@ def __init__(self, num):
# end_object_type

# start_use_object_type
@solid
@op
def double_even(even_num: EvenDagsterType) -> EvenDagsterType:
return EvenType(even_num.num * 2)

Expand Down
@@ -1,6 +1,6 @@
"""isort:skip_file"""

from dagster import DagsterType, InputDefinition, OutputDefinition, solid
from dagster import DagsterType, In, Out, op


# start_basic_even_type
Expand All @@ -12,9 +12,9 @@


# start_basic_even_type_no_annotations
@solid(
input_defs=[InputDefinition("num", EvenDagsterType)],
output_defs=[OutputDefinition(EvenDagsterType)],
@op(
ins={"num": In(EvenDagsterType)},
out=Out(EvenDagsterType),
)
def double_even(num):
return num
Expand All @@ -23,9 +23,9 @@ def double_even(num):
# end_basic_even_type_no_annotations

# start_basic_even_type_with_annotations
@solid(
input_defs=[InputDefinition("num", EvenDagsterType)],
output_defs=[OutputDefinition(EvenDagsterType)],
@op(
ins={"num": In(EvenDagsterType)},
out=Out(EvenDagsterType),
)
def double_even_with_annotations(num: int) -> int:
return num
Expand All @@ -41,8 +41,8 @@ class MyClass:
pass


@solid
def my_solid() -> MyClass:
@op
def my_op() -> MyClass:
return MyClass()


Expand Down
@@ -1,4 +1,4 @@
from dagster import solid, usable_as_dagster_type
from dagster import op, usable_as_dagster_type


@usable_as_dagster_type
Expand All @@ -8,6 +8,6 @@ def __init__(self, num):
self.num = num


@solid
@op
def double_even(even_num: EvenType) -> EvenType:
return EvenType(even_num.num * 2)
@@ -1,50 +1,44 @@
import pytest
from dagster import DagsterTypeCheckDidNotPass, execute_solid
from dagster import DagsterTypeCheckDidNotPass
from docs_snippets_crag.concepts.types.types import test_dagster_type


def test_basic_even_type():
from docs_snippets_crag.concepts.types.types import double_even

assert execute_solid(double_even, input_values={"num": 2}).success
double_even(num=2)

with pytest.raises(DagsterTypeCheckDidNotPass):
execute_solid(double_even, input_values={"num": 3})

assert not execute_solid(double_even, input_values={"num": 3}, raise_on_error=False).success
double_even(num=3)


def test_basic_even_type_with_annotations():
from docs_snippets_crag.concepts.types.types import double_even_with_annotations

assert execute_solid(double_even_with_annotations, input_values={"num": 2}).success
double_even_with_annotations(num=2)

with pytest.raises(DagsterTypeCheckDidNotPass):
execute_solid(double_even_with_annotations, input_values={"num": 3})

assert not execute_solid(
double_even_with_annotations, input_values={"num": 3}, raise_on_error=False
).success
double_even_with_annotations(num=3)


def test_python_object_dagster_type():
from docs_snippets_crag.concepts.types.object_type import EvenType, double_even

assert execute_solid(double_even, input_values={"even_num": EvenType(2)}).success
double_even(even_num=EvenType(2))
with pytest.raises(AssertionError):
execute_solid(double_even, input_values={"even_num": EvenType(3)})
double_even(even_num=EvenType(3))


def test_usable_as_dagster_type():
from docs_snippets_crag.concepts.types.usable_as import EvenType, double_even

assert execute_solid(double_even, input_values={"even_num": EvenType(2)}).success
double_even(even_num=EvenType(2))


def test_make_python_type_usable_as_dagster_type():
from docs_snippets_crag.concepts.types.make_usable import EvenType, double_even

assert execute_solid(double_even, input_values={"even_num": EvenType(2)}).success
double_even(even_num=EvenType(2))


def test_unit_test():
Expand Down