An HTTP API for the `looper run` command that works with the `pydantic`-based command models #441

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

simeoncarstens merged 46 commits into pepkit:feature-http-api from tweag:tweag/http-api

Feb 8, 2024

looper/__init__.py

-Original file line number
+Diff line change
@@ Expand Up / @@ -7,10 +7,6 @@ @@
     """
-    import logmuse
-    logmuse.init_logger("looper")
     from .divvy import ComputingConfiguration, select_divvy_config
     from .divvy import DEFAULT_COMPUTE_RESOURCES_NAME
     from .divvy import NEW_COMPUTE_KEY as COMPUTE_KEY
@@ Expand Down @@

looper/api/README.md

-Original file line number
+Diff line change
@@ -0,0 +1,34 @@
+    # Looper HTTP API
+    ## Overview
+    This API provides an HTTP interface for running the `looper` commands, allowing users to interact with Looper via HTTP requests.
+    ## Usage
+    ### Running the server
+    Run the app:
+    ```bash
+    looper-serve [--host <host IP address>] [--port <port>]
+    ```
+    > [!NOTE]
+    This assumes that all files specified in the arguments are available on the file system of the machine that is running the HTTP API server. Best make sure you use absolute file paths in all `looper` YAML configuration files.
+    ### Sending requests
+    To test this, you can clone the [`hello_looper`](https://github.com/pepkit/hello_looper) repository and then run (for example) the following in a second terminal:
+    ```bash
+    curl -X POST -H "Content-Type: application/json" -d '{"run": {"time_delay": 5}, "looper_config": "/path/to/hello_looper/.looper.yaml"}' "http://127.0.0.1:8000"
+    ```
+    This will return a six-letter job ID, say `abc123`. Then get the result / output of the run with
+    ```bash
+    curl -X GET -v localhost:8000/status/abc123
+    ```
+    For better visualization / readability, you can post-process the output by piping it to `jq` (` | jq -r .console_output`).
+    ## API Documentation
+    The API documentation is automatically generated and can be accessed in your web browser:
+    Swagger UI: http://127.0.0.1:8000/docs
+    ReDoc: http://127.0.0.1:8000/redoc
+    Explore the API documentation to understand available endpoints, request parameters, and response formats.

looper/api/__init__.py

Empty file.

looper/api/main.py

-Original file line number
+Diff line change
@@ -0,0 +1,126 @@
+    from argparse import ArgumentParser, Namespace
+    import secrets
+    from typing import Dict, TypeAlias
+    import fastapi
+    import pydantic
+    import uvicorn
+    from fastapi import FastAPI
+    from looper.cli_pydantic import run_looper
+    from looper.command_models.commands import SUPPORTED_COMMANDS, TopLevelParser
+    from looper.api import stdout_redirects
+    stdout_redirects.enable_proxy()
+    class Job(pydantic.BaseModel):
+        id: str = pydantic.Field(
+            default_factory=lambda: secrets.token_urlsafe(4),
+            description="The unique identifier of the job",
+        )
+        status: str = pydantic.Field(
+            default="in_progress",
+            description="The current status of the job. Can be either `in_progress` or `completed`.",
+        )
+        console_output: str | None = pydantic.Field(
+            default=None,
+            description="Console output produced by `looper` while performing the requested action",
+        )
+    app = FastAPI(validate_model=True)
+    jobs: Dict[str, Job] = {}
+    def background_async(top_level_model: TopLevelParser, job_id: str) -> None:
+        argparse_namespace = create_argparse_namespace(top_level_model)
+        output_stream = stdout_redirects.redirect()
+        run_looper(argparse_namespace, None, True)
+        # Here, we should call `stdout_redirects.stop_redirect()`, but that fails for reasons discussed
+        # in the following issue: https://github.com/python/cpython/issues/80374
+        # But this *seems* not to pose any problems.
+        jobs[job_id].status = "completed"
+        jobs[job_id].console_output = output_stream.getvalue()
+    def create_argparse_namespace(top_level_model: TopLevelParser) -> Namespace:
+        """
+        Converts a TopLevelParser instance into an argparse.Namespace object.
+        This function takes a TopLevelParser instance, and converts it into an
+        argparse.Namespace object. It includes handling for supported commands
+        specified in SUPPORTED_COMMANDS.
+        :param TopLevelParser top_level_model: An instance of the TopLevelParser
+            model
+        :return argparse.Namespace: An argparse.Namespace object representing
+            the parsed command-line arguments.
+        """
+        namespace = Namespace()
+        for argname, value in vars(top_level_model).items():
+            if argname not in [cmd.name for cmd in SUPPORTED_COMMANDS]:
+                setattr(namespace, argname, value)
+            else:
+                command_namespace = Namespace()
+                command_namespace_args = value
+                for command_argname, command_arg_value in vars(
+                    command_namespace_args
+                ).items():
+                    setattr(
+                        command_namespace,
+                        command_argname,
+                        command_arg_value,
+                    )
+                setattr(namespace, argname, command_namespace)
+        return namespace
+    @app.post(
+        "/",
+        status_code=202,
+        summary="Run Looper",
+        description="Start a `looper` command with arguments specified in "
+        "`top_level_model` in the background and return a job identifier.",
+    )
+    async def main_endpoint(
+        top_level_model: TopLevelParser, background_tasks: fastapi.BackgroundTasks
+    ) -> Dict:
+        job = Job()
+        jobs[job.id] = job
+        background_tasks.add_task(background_async, top_level_model, job.id)
+        return {"job_id": job.id}
+    @app.get(
+        "/status/{job_id}",
+        summary="Get job status",
+        description="Retrieve the status of a job based on its unique identifier.",
+    )
+    async def get_status(job_id: str) -> Job:
+        return jobs[job_id]
+    def main() -> None:
+        parser = ArgumentParser("looper-serve", description="Run looper HTTP API server")
+        parser.add_argument(
+            "--host",
+            type=str,
+            default="0.0.0.0",
+            help="Host IP address to use (127.0.0.1 for local access only)",
+        )
+        parser.add_argument(
+            "--port", type=int, default=8000, help="Port the server listens on"
+        )
+        args = parser.parse_args()
+        uvicorn.run(app, host=args.host, port=args.port)
+    if __name__ == "__main__":
+        main()

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

An HTTP API for the `looper run` command that works with the `pydantic`-based command models #441

Uh oh!

Diff view

Diff view

There are no files selected for viewing

Uh oh!

An HTTP API for the looper run command that works with the pydantic-based command models #441

Uh oh!

An HTTP API for the looper run command that works with the pydantic-based command models #441

Uh oh!

Uh oh!

Diff view

Diff view

There are no files selected for viewing

Uh oh!

An HTTP API for the `looper run` command that works with the `pydantic`-based command models #441

An HTTP API for the `looper run` command that works with the `pydantic`-based command models #441