Skip to content

Commit

Permalink
add api doc for op_def, graph_def (#6776)
Browse files Browse the repository at this point in the history
* add api doc for op_def, graph_def

* build
  • Loading branch information
prha committed Feb 24, 2022
1 parent 83643b2 commit 472c4dc
Show file tree
Hide file tree
Showing 5 changed files with 108 additions and 3 deletions.
2 changes: 1 addition & 1 deletion docs/content/api/modules.json

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion docs/content/api/searchindex.json

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion docs/content/api/sections.json

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
Expand Up @@ -118,6 +118,59 @@ def visit(solid_name):


class GraphDefinition(NodeDefinition):
"""Defines a Dagster graph.
A graph is made up of
- Nodes, which can either be an op (the functional unit of computation), or another graph.
- Dependencies, which determine how the values produced by nodes as outputs flow from
one node to another. This tells Dagster how to arrange nodes into a directed, acyclic graph
(DAG) of compute.
End users should prefer the :func:`@graph <graph>` decorator. GraphDefinition is generally
intended to be used by framework authors or for programatically generated graphs.
Args:
name (str): The name of the graph. Must be unique within any :py:class:`GraphDefinition`
or :py:class:`JobDefinition` containing the graph.
description (Optional[str]): A human-readable description of the pipeline.
node_defs (Optional[List[NodeDefinition]]): The set of ops / graphs used in this graph.
dependencies (Optional[Dict[Union[str, NodeInvocation], Dict[str, DependencyDefinition]]]):
A structure that declares the dependencies of each op's inputs on the outputs of other
ops in the graph. Keys of the top level dict are either the string names of ops in the
graph or, in the case of aliased ops, :py:class:`NodeInvocations <NodeInvocation>`.
Values of the top level dict are themselves dicts, which map input names belonging to
the op or aliased op to :py:class:`DependencyDefinitions <DependencyDefinition>`.
input_mappings (Optional[List[InputMapping]]): Defines the inputs to the nested graph, and
how they map to the inputs of its constituent ops.
output_mappings (Optional[List[OutputMapping]]): Defines the outputs of the nested graph,
and how they map from the outputs of its constituent ops.
config (Optional[ConfigMapping]): Defines the config of the graph, and how its schema maps
to the config of its constituent ops.
tags (Optional[Dict[str, Any]]): Arbitrary metadata for any execution of the graph.
Values that are not strings will be json encoded and must meet the criteria that
`json.loads(json.dumps(value)) == value`. These tag values may be overwritten by tag
values provided at invocation time.
Examples:
.. code-block:: python
@op
def return_one():
return 1
@op
def add_one(num):
return num + 1
graph_def = GraphDefinition(
name='basic',
node_defs=[return_one, add_one],
dependencies={'add_one': {'num': DependencyDefinition('return_one')}},
)
"""

def __init__(
self,
name: str,
Expand Down
52 changes: 52 additions & 0 deletions python_modules/dagster/dagster/core/definitions/op_definition.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,58 @@


class OpDefinition(SolidDefinition):
"""
Defines an op, the functional unit of user-defined computation.
For more details on what a op is, refer to the
`Ops Overview <../../concepts/ops-jobs-graphs/ops>`_ .
End users should prefer the :func:`@op <op>` decorator. OpDefinition is generally intended to be
used by framework authors or for programatically generated ops.
Args:
name (str): Name of the op. Must be unique within any :py:class:`GraphDefinition` or
:py:class:`JobDefinition` that contains the op.
input_defs (List[InputDefinition]): Inputs of the op.
compute_fn (Callable): The core of the op, the function that performs the actual
computation. The signature of this function is determined by ``input_defs``, and
optionally, an injected first argument, ``context``, a collection of information
provided by the system.
This function will be coerced into a generator or an async generator, which must yield
one :py:class:`Output` for each of the op's ``output_defs``, and additionally may
yield other types of Dagster events, including :py:class:`AssetMaterialization` and
:py:class:`ExpectationResult`.
output_defs (List[OutputDefinition]): Outputs of the op.
config_schema (Optional[ConfigSchema): The schema for the config. If set, Dagster will check
that the config provided for the op matches this schema and will fail if it does not. If
not set, Dagster will accept any config provided for the op.
description (Optional[str]): Human-readable description of the op.
tags (Optional[Dict[str, Any]]): Arbitrary metadata for the op. Frameworks may
expect and require certain metadata to be attached to a op. Users should generally
not set metadata directly. Values that are not strings will be json encoded and must meet
the criteria that `json.loads(json.dumps(value)) == value`.
required_resource_keys (Optional[Set[str]]): Set of resources handles required by this op.
version (Optional[str]): (Experimental) The version of the op's compute_fn. Two ops should
have the same version if and only if they deterministically produce the same outputs
when provided the same inputs.
retry_policy (Optional[RetryPolicy]): The retry policy for this op.
Examples:
.. code-block:: python
def _add_one(_context, inputs):
yield Output(inputs["num"] + 1)
OpDefinition(
name="add_one",
input_defs=[InputDefinition("num", Int)],
output_defs=[OutputDefinition(Int)], # default name ("result")
compute_fn=_add_one,
)
"""

@property
def node_type_str(self) -> str:
return "op"
Expand Down

0 comments on commit 472c4dc

Please sign in to comment.