An HTTP API for the `looper run` command that works with the `pydantic`-based command models by zz1874 · Pull Request #441 · pepkit/looper

zz1874 · 2024-01-19T08:55:56Z

Warning

Currently based on #440. Once that is merged, we'll make a new branch, feature-http-api, that will be based on the then updated dev branch and target feature-http-api with this PR. So don't merge just quite yet 🙂

This PR introduces a first version of an HTTP API to execute looper commands (#433).

Changes made:

Implemented a new HTTP API using FastAPI to run the looper run command.
Slightly modified the enrich_vars_cfg() function in looper/utils.py module to enable execution via both the CLI and the HTTP API.
Slightly refactor cli_pydantic.py

Endpoints

/ (POST): allows users to trigger looper commands asynchronously . For now, because we only have the run command modeled as a pydantic model, only the run command is supported. Returns a unique job ID that can be used in the /status endpoint.
This endpoint accepts a JSON payload with the run model and other top-level parameters such as looper_config, see the auto-generated documentation on http://127.0.0.1:8000/docs and / or looper/command_models/commands.py.
/status/<job ID> (GET): retrieve the status / results of a previously started job. Returns a JSON with, notably, a field console_output that contains all stdout / stderr from the performed looper run, once the job has finished.

Usage:

Run the app:

looper-serve [--host <host IP address>] [--port <port>]

Note

This assumes that all files specified in the arguments are available on the file system of the machine that is running the HTTP API server. Best make sure you use absolute file paths in all looper YAML configuration files.

To test this, you can clone the hello_looper repository and then run (for example) the following in a second terminal:

curl -X POST -H "Content-Type: application/json" -d '{"run": {"time_delay": 5}, "looper_config": "/path/to/hello_looper/.looper.yaml"}' "http://127.0.0.1:8000"

This will return a six-letter job ID, say abc123. Then get the result / output of the run with

curl -X GET -v localhost:8000/status/abc123

For better visualization / readability, you can post-process the output by piping it to jq ( | jq -r .console_output).

nsheff · 2024-01-19T17:42:43Z

can you put the usage information into a readme?

That's because this endpoint will support _all_ commands, and not only `run`.

This allows, together with the hacked, threadable (but _not_ thread-safe) `yacman` version, to run `looper` commands in a non-blocking way.

Everything will probably still work - after all, `logmuse.logger_via_cli()` is called in `cli_looper.py` / `cli_pydantic.py` which also sets up a logger.

Before this commit, the logging stdout we captured will be mixed if we submit several jobs at the same time. This captures outputs for each thread (job) separately. See https://stackoverflow.com/questions/14890997/redirect-stdout-to-a-file-only-for-a-specific-thread. Co-authored-by: Simeon Carstens <simeon.carstens@tweag.io>

This allows us to capture the output of the Bash scripts or commands `looper` executes.

Co-authored-by: Zhihan Zhang <zhihan.zhang@tweag.io>

The module we use to capture console output does not seem to distinguish easily between `stdout` and `stderr`. So we rather use a generic `console_output` field in the job model that subsumes both.

This makes the Swagger documentation show the job schema for that endpoint.

* Apply formatter * Add documentation for `POST` and `GET` requests * Update looper/api/main.py Co-authored-by: Simeon Carstens <simeon.carstens@tweag.io> * Update looper/api/main.py Co-authored-by: Simeon Carstens <simeon.carstens@tweag.io> * Add where to access the API documentation --------- Co-authored-by: Simeon Carstens <simeon.carstens@tweag.io>

donaldcampbelljr

This looks good. Please proceed with merging #440 into its feature branch and then this PR into its feature branch when you are able.

simeoncarstens · 2024-02-08T17:27:04Z

Done 🙂

zz1874 requested review from nsheff and simeoncarstens January 19, 2024 08:55

zz1874 mentioned this pull request Jan 26, 2024

Add version of YAMLConfigManager that allows to run in a non-main thread databio/yacman#63

Open

simeoncarstens force-pushed the tweag/run-hello-world branch from 6b81058 to cd5ebcf Compare January 26, 2024 13:01

simeoncarstens force-pushed the tweag/http-api branch from a083b01 to f92be03 Compare January 27, 2024 11:34

simeoncarstens force-pushed the tweag/run-hello-world branch from cd5ebcf to d71763b Compare January 29, 2024 10:05

simeoncarstens mentioned this pull request Jan 29, 2024

reconsidering suggested use databio/logmuse#26

Open

simeoncarstens force-pushed the tweag/http-api branch from 5c3a47c to 643779c Compare January 29, 2024 17:34

simeoncarstens mentioned this pull request Jan 30, 2024

Add remaining arguments to run command model #448

Merged

simeoncarstens force-pushed the tweag/run-hello-world branch 3 times, most recently from 1d5d17f to c21d8ed Compare February 1, 2024 15:00

simeoncarstens mentioned this pull request Feb 1, 2024

First iteration of a CLI based on pydantic models that allows to run the hello_looper example #440

Merged

zz1874 and others added 16 commits February 1, 2024 17:20

HTTP API settings

6e4e9d4

Create an argparse.Namespace

eab5127

Add run function from cli_pydantic

72087ee

Adjust enrich_args_via_cfg to http api

1734c80

Run adjusted enrich_args_via_cfg in http api

e0e3a6f

Re-organize cli_pydantic.py to run looper run via CLI and http-api

67182dd

Slight refactor of create_argparse_namespace

6346654

Remove run from route

e1f7308

That's because this endpoint will support _all_ commands, and not only `run`.

Capture stderr / stdout and return in HTTP response

dd978c8

Rename run_endpoint -> main_endpoint

e010f75

Add response model

8af2bb2

Add a comment about the endpoint likely being blocking

a89e7bc

Apply formatter

1880372

Add logger def to be captured by API and also CLI

42119f0

Add README for the API

f0c749d

Add endpoint "\status" to capture UUID

6d146b5

simeoncarstens and others added 16 commits February 1, 2024 17:45

Add lower bound for uvicorn version

a82a8f7

Make background task function non-async

f995b47

This allows, together with the hacked, threadable (but _not_ thread-safe) `yacman` version, to run `looper` commands in a non-blocking way.

[DELETE ME] hack to use local yacman copy

3c54546

Don't call logmuse.init_logger() in looper.__init__.py

9b3a1da

Everything will probably still work - after all, `logmuse.logger_via_cli()` is called in `cli_looper.py` / `cli_pydantic.py` which also sets up a logger.

Explicitly initialize logmuse logger with sys.stderr as stream

16f0ab5

Add source for stdout_redirects.py

8ffaef7

Add a comment about not calling stdout_redirect.stop_redirect()

d8ae6ec

Remove superfluous import

db9f8f5

Remove progress field from Job model

ad621c6

Capture subprocess output to sys.stdout/sys.stderr

95278b3

This allows us to capture the output of the Bash scripts or commands `looper` executes.

Make CLI for HTTP API server

6870bcd

Co-authored-by: Zhihan Zhang <zhihan.zhang@tweag.io>

Add entry point console script for HTTP API server

8ead693

Replace stdout / stderr job fields with job_output field

8b1b2ca

The module we use to capture console output does not seem to distinguish easily between `stdout` and `stderr`. So we rather use a generic `console_output` field in the job model that subsumes both.

Add return type to /status endpoint

0d6b016

This makes the Swagger documentation show the job schema for that endpoint.

simeoncarstens force-pushed the tweag/http-api branch from 4e5090d to ffefcd1 Compare February 1, 2024 16:55

simeoncarstens changed the title ~~[WIP] An HTTP API for the looper run command that works with the pydantic-based CLI~~ [WIP] An HTTP API for the looper run command that works with the pydantic-based command models Feb 2, 2024

Run formatter

6619440

simeoncarstens force-pushed the tweag/http-api branch from ffefcd1 to 6619440 Compare February 2, 2024 09:37

Make HTTP API code Python 3.8 compatible

b3aa4aa

simeoncarstens marked this pull request as ready for review February 2, 2024 10:24

simeoncarstens requested review from donaldcampbelljr and khoroshevskyi February 2, 2024 10:25

simeoncarstens changed the title ~~[WIP] An HTTP API for the looper run command that works with the pydantic-based command models~~ An HTTP API for the looper run command that works with the pydantic-based command models Feb 2, 2024

Update README with more detailed usage instructions

d75942c

donaldcampbelljr approved these changes Feb 7, 2024

View reviewed changes

simeoncarstens changed the base branch from tweag/run-hello-world to feature-http-api February 8, 2024 17:25

simeoncarstens merged commit ebd2fb8 into pepkit:feature-http-api Feb 8, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

An HTTP API for the `looper run` command that works with the `pydantic`-based command models#441

An HTTP API for the `looper run` command that works with the `pydantic`-based command models#441
simeoncarstens merged 46 commits intopepkit:feature-http-apifrom
tweag:tweag/http-api

zz1874 commented Jan 19, 2024 •

edited by simeoncarstens

Loading

Uh oh!

nsheff commented Jan 19, 2024

Uh oh!

donaldcampbelljr left a comment

Uh oh!

simeoncarstens commented Feb 8, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

zz1874 commented Jan 19, 2024 • edited by simeoncarstens Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes made:

Endpoints

Usage:

Uh oh!

nsheff commented Jan 19, 2024

Uh oh!

donaldcampbelljr left a comment

Choose a reason for hiding this comment

Uh oh!

simeoncarstens commented Feb 8, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

zz1874 commented Jan 19, 2024 •

edited by simeoncarstens

Loading