Skip to content

Commit

Permalink
chore(docs): refactor flow basic docs
Browse files Browse the repository at this point in the history
  • Loading branch information
hanxiao committed Aug 20, 2022
1 parent 885914a commit f8a63cc
Show file tree
Hide file tree
Showing 10 changed files with 210 additions and 85 deletions.
1 change: 1 addition & 0 deletions docs/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@
'tests',
'page_templates',
'.github',
'api'
]
pygments_style = 'rainbow_dash'
html_theme = 'furo'
Expand Down
4 changes: 2 additions & 2 deletions docs/fundamentals/client/client.md
Original file line number Diff line number Diff line change
Expand Up @@ -219,7 +219,7 @@ This feature is intended for the case where there are multiple Executors that ta
This is often the case for Executors from the Hub, since they tend to share a common interface for parameters.


(callback-functions)=

## Async send

There also exists an async version of the Python Client which works with {meth}`~jina.clients.mixin.PostMixin.post` and {meth}`~jina.clients.mixin.MutateMixin.mutate`.
Expand Down Expand Up @@ -316,7 +316,7 @@ with f: # Using it as a Context Manager will start the Flow
This will send the request to all Executors whose names start with 'bar', such as 'barExecutor'.
In the simplest case, you can specify a precise Executor name, and the request will be sent only to that single Executor.


(callback-functions)=
## Callbacks

After performing {meth}`~jina.clients.mixin.PostMixin.post`, you may want to further process the obtained results.
Expand Down
2 changes: 1 addition & 1 deletion docs/fundamentals/executor/hub/push-executor.md
Original file line number Diff line number Diff line change
Expand Up @@ -132,7 +132,7 @@ For example:
```
````

For multiple enviroment variables, we can pass it in this way:
For multiple environment variables, we can pass it in this way:

```bash
jina hub push --build-env FIRST=foo --build-env SECOND=bar
Expand Down
2 changes: 1 addition & 1 deletion docs/fundamentals/executor/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ executor-methods
monitoring-executor
executor-run
executor-serve
yaml-spec
executor-files
containerize-executor
yaml-spec
```
2 changes: 1 addition & 1 deletion docs/fundamentals/executor/yaml-spec.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
(executor-yaml-spec)=
# YAML specification
# {octicon}`file-code` YAML specification

This page outlines the specification for valid Executor YAML files.

Expand Down
147 changes: 71 additions & 76 deletions docs/fundamentals/flow/create-flow.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,65 +4,70 @@

{class}`~jina.Flow` defines how your Executors are connected together and how your data *flows* through them.

Every {class}`~jina.Flow` can be defined either purely in Python, or be loaded from a YAML file.

````{admonition} Best practice
:class: hint
For production use we recommend YAML files to configure your Flows. This is because YAML files are:
- independent of Python source code
- easy to edit, maintain and extend
- human-readable
````

## Create

The most trivial {class}`~jina.Flow` is the empty Flow and, like any other Flow, it can be instantiated purely in Python, or from a
YAML file:
The most trivial {class}`~jina.Flow` is the empty Flow. It can be defined purely in Python or from a YAML file:

````{tab} Python
```python
from jina import Flow
f = Flow() # Create the empty Flow
with f: # Using it as a Context Manager will start the Flow
f.post(on='/search') # This sends a request to the /search endpoint of the Flow
f = Flow()
```
````

`````{tab} YAML
`flow.yml`:
```yaml
jtype: Flow
```
`````

```{tip}
An empty Flow contains only {ref}`the Gateway<flow>`.
```

For production, we recommend YAML files to define the Flows. This is because YAML files are independent of Python logic code and easy to maintain.




### Conversion between Python and YAML

Python Flow definition can be easily converted to/from YAML definition.

To load a Flow from a YAML file, use the {meth}`~jina.Flow.load_config`:

```python
from jina import Flow

f = Flow.load_config('flow.yml') # Load the Flow definition from Yaml file
with f: # Using it as a Context Manager will start the Flow
f.post(on='/search') # This sends a request to the /search endpoint of the Flow
f = Flow.load_config('flow.yml')
```

````{admonition} Hint: Dump Flow configuration
:class: hint
To export an existing Flow definition to a YAML file use {meth}`~jina.Flow.save_config`:

In addition to loading a Flow from a YAML file, you can also save an existing Flow configuration to YAML. To do so, execute `f.save_config('path/to/flow.yml')`.
````
`````
```python
from jina import Flow

f = Flow().add().add() # Create a Flow with two Executors

f.save_config('flow.yml')
```

## Start and stop

When a {class}`~jina.Flow` starts, all its {ref}`added Executors <flow-add-executors>` will start as well, making it possible to {ref}`reach the service through its API <access-flow-api>`.

Jina Flows are context managers and can be started and stopped using Pythons `with` notation:
There are three ways to start a Flow. Depending on the use case, you can start a Flow either in Python, or from a YAML file, or from the terminal.

- Generally in Python: use Flow as a context manager in Python.
- As an entrypoint from terminal: use Jina CLI and a Flow YAML.
- As an entrypoint from Python code: use Flow as a context manager inside `if __name__ == '__main__'`
- No context manager: manually call {meth}`~jina.Flow.start` and {meth}`~jina.Flow.close`.


````{tab} General in Python
```python
from jina import Flow
Expand All @@ -71,69 +76,49 @@ f = Flow()
with f:
pass
```
````

The statement `with f:` starts the Flow, and exiting the indented `with` block stops the Flow, including all Executors defined in it.


### Start inside `__main__`

If applicable, always start the Flow inside `if __name__ == '__main__'`. For example:

````{tab} ✅ Do
```{code-block} python
---
emphasize-lines: 13, 14
---
from jina import Flow, Executor, requests
````{tab} Jina CLI entrypoint
```bash
jina flow --uses flow.yml
```
````

class CustomExecutor(Executor):
@requests
async def foo(self, **kwargs):
...
````{tab} Python entrypoint
```python
from jina import Flow
f = Flow().add(uses=CustomExecutor)
f = Flow()
if __name__ == '__main__':
with f:
...
pass
```
````

````{tab} 😔 Don't
```{code-block} python
---
emphasize-lines: 2
---
````{tab} Python no context manager
```python
from jina import Flow
from jina import Flow, Executor, requests
f = Flow()
class CustomExecutor(Executor):
@requests
def foo(self, **kwargs):
...
f.start()
f = Flow().add(uses=CustomExecutor)
with f:
...
f.close()
```
````

"""
# error
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
The statement `with f:` starts the Flow, and exiting the indented `with` block stops the Flow, including all Executors defined in it.

if _name_ == '_main_':
freeze_support()
...

The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
A successful start of a Flow looks like this:

"""
```{figure} success-flow.png
:scale: 70%
```

````
Your addresses and entrypoints can be found in the output. When enabling more features such as monitoring, HTTP gateway, TLS encryption, this display will also expand to contain more information.


### Set multiprocessing `spawn`

Expand All @@ -152,19 +137,27 @@ There's no need to set this for Windows, as it only supports spawn method for mu
## Serve forever

In most scenarios, a Flow should remain reachable for prolonged periods of time.
This can be achieved by *blocking* the execution:
This can be achieved by `jina flow --uses flow.yml` from terminal.


Or if you are serving a Flow from Python:

```python
from jina import Flow

f = Flow()

with f:
f.block()
```

The `.block()` method blocks the execution of the current thread or process, which enables external clients to access the Flow.

In this case, the Flow can be stopped by interrupting the thread or process. Alternatively, a `multiprocessing` or `threading` `Event` object can be passed to `.block()`, which stops the Flow once set.
In this case, the Flow can be stopped by interrupting the thread or process.

### Server until an event

Alternatively, a `multiprocessing` or `threading` `Event` object can be passed to `.block()`, which stops the Flow once set.

```python
from jina import Flow
Expand Down Expand Up @@ -221,6 +214,8 @@ One can also do it in the terminal via:
jina export flowchart flow.yml flow.svg
```

One can also visualize a remote Flow by passing the URL to `jina export flowchart`.

## Export

A {class}`~jina.Flow` YAML can be exported as a Docker Compose YAML or a Kubernetes YAML bundle.
Expand Down
Binary file added docs/fundamentals/flow/success-flow.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 1 addition & 1 deletion docs/fundamentals/flow/yaml-spec.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
(flow-yaml-spec)=
# YAML specification
# {octicon}`file-code` YAML specification

This page outlines the specification for valid {class}`~jina.Executor` YAML files.

Expand Down
6 changes: 3 additions & 3 deletions docs/how-to/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,12 +48,12 @@ can put that into practice you can find {ref}`here <kubernetes>`.
```{toctree}
:hidden:
google-colab
../fundamentals/clean-code
realtime-streaming
flow-switch
scale-out
../fundamentals/clean-codegoogle-colab
gpu-executor
external-executor
flow-switch
docker-compose
kubernetes
monitoring
Expand Down
Loading

0 comments on commit f8a63cc

Please sign in to comment.