Add router management functions to SDK (#155)

* Refactor swagger/openapi specs for sdk * Add autogenerated openapi classes for routers * Rewrote OpenAPI spec for ExperimentConfig to avoid using oneOf * Remove dependency of ExperimentConfig schema on ExperimentEngineType * Add additional fields for enricher component * Add additional nullable fields to OpenAPI spec * Reference ensembler, event and router id to common id schema * Rename Ensembler in OpenAPI spec for routers to RouterEnsemblerConfig * Reorganise router related tests * Refactor route tests to utilise predefined fixtures * Add resource_request sdk classes and tests * Add RouterEnsemblerConfig sdk and classes * Move common router components to separate file * Refactor RouterEnsemblerConfig to use common schemas * Add Enricher sdk classes and tests * Add ExperimentEngine sdk class * Add fix to enforce project_id in ExperimentConfig to be read as int * Refactor regex checks for cpu and memory requests * Add create router tests * Refactor string checks and add docstrings * Rewrite router test case * Modify min/max replica comparison check to allow equality * Refactor swagger/openapi specs for sdk * Add autogenerated openapi classes for routers * Rewrote OpenAPI spec for ExperimentConfig to avoid using oneOf * Remove dependency of ExperimentConfig schema on ExperimentEngineType * Add additional fields for enricher component * Add additional nullable fields to OpenAPI spec * Reference ensembler, event and router id to common id schema * Rename Ensembler in OpenAPI spec for routers to RouterEnsemblerConfig * Refactor route tests to utilise predefined fixtures * Add resource_request sdk classes and tests * Add RouterEnsemblerConfig sdk and classes * Move common router components to separate file * Refactor RouterEnsemblerConfig to use common schemas * Add Enricher sdk classes and tests * Add ExperimentEngine sdk class * Add fix to enforce project_id in ExperimentConfig to be read as int * Refactor regex checks for cpu and memory requests * Add create router tests * Refactor string checks and add docstrings * Rewrite router test case * Modify min/max replica comparison check to allow equality * Refactor Enricher, LogConfig, Route, RouterConfig, RouterEnsemblerConfig into dataclasses * Remove regex validation from SDK and move them to OpenAPI spec * Reinsert missing semi-colon in js file * Correct regex strings in OpenApi spec * Rearrange modules in SDK * Add all router API endpoints and reformat response schema for delete router operation * Add delete router method to router sdk class * Add SDK method for getting router by router_id * Add SDK method for updating router config * Add deploy and undeploy router SDK functions * Add tests for create/delete/deploy/undeploy router SDK methods * Add tests for undeploy router SDK method * Add get/delete router version SDK methods * Add tests for get/delete router version SDK methods * Add deploy version/get events router SDK methods * Add list router versions SDK method * Reorganise router module * Add sample file to demonstrate the use of the router class in the SDK * Add docstrings to RouterVersion * Replace print with logging statements and fix comments * Regroup import statements * Add global numbers to implementation of to_dict * Add wait methods for routers to sample; move location of RouterStatus * Replace kwargs with empty dict when created OpenAPI ExperimentConfig * Refactor log_config fixture * Implement logging and tests for blocking functions * Add additional debugging statements for sample * Refactor blocking functions to take floats * Add dataclass implementation to config classes * Remove extra comma in class definition * Add additional sleep duration to ensure router dependencies are undeployed in sample * Add support for experiment plugin configs * Remove plugin_config from SDK and OpenAPI spec * Remove plugin_config from autogenerated classes * Fix update method of Router class to ensure all attributes are updated * Replace images with generic images from Docker Hub * Use generic images and names for tests * Add additional sample to demonstrate creating a new router from an existing one * Remove redundant imports * Wrap sleep statements into router blocking functions Co-authored-by: ewezy <ziyi.ewe@gojek.com>
caraml-dev · Jan 25, 2022 · 6b67695 · 6b67695
1 parent 6c16861
commit 6b67695
Show file tree

Hide file tree

Showing 27 changed files with 3,523 additions and 66 deletions.
diff --git a/api/api/openapi-sdk.yaml b/api/api/openapi-sdk.yaml
@@ -25,3 +25,17 @@ paths:
   # R O U T E R S
   "/projects/{project_id}/routers":
     $ref: "specs/routers.yaml#/paths/~1projects~1{project_id}~1routers"
+  "/projects/{project_id}/routers/{router_id}":
+    $ref: "specs/routers.yaml#/paths/~1projects~1{project_id}~1routers~1{router_id}"
+  "/projects/{project_id}/routers/{router_id}/deploy":
+    $ref: "specs/routers.yaml#/paths/~1projects~1{project_id}~1routers~1{router_id}~1deploy"
+  "/projects/{project_id}/routers/{router_id}/undeploy":
+    $ref: "specs/routers.yaml#/paths/~1projects~1{project_id}~1routers~1{router_id}~1undeploy"
+  "/projects/{project_id}/routers/{router_id}/versions":
+    $ref: "specs/routers.yaml#/paths/~1projects~1{project_id}~1routers~1{router_id}~1versions"
+  "/projects/{project_id}/routers/{router_id}/versions/{version}":
+    $ref: "specs/routers.yaml#/paths/~1projects~1{project_id}~1routers~1{router_id}~1versions~1{version}"
+  "/projects/{project_id}/routers/{router_id}/versions/{version}/deploy":
+    $ref: "specs/routers.yaml#/paths/~1projects~1{project_id}~1routers~1{router_id}~1versions~1{version}~1deploy"
+  "/projects/{project_id}/routers/{router_id}/events":
+    $ref: "specs/routers.yaml#/paths/~1projects~1{project_id}~1routers~1{router_id}~1events"
diff --git a/api/api/specs/experiment-engines.yaml b/api/api/specs/experiment-engines.yaml
@@ -296,6 +296,10 @@ components:
           $ref: "#/components/schemas/ExperimentEngineType"
         config:
           type: "object"
+      additionalProperties:
+        type: "object"
+        readOnly: true
+        nullable: true
 
     # Enums
     ExperimentEngineType:

diff --git a/api/api/specs/routers.yaml b/api/api/specs/routers.yaml
@@ -154,7 +154,7 @@ paths:
               schema:
                 type: "object"
                 properties:
-                  router_id:
+                  id:
                     <<: *id
         400:
           description: "Invalid project_id or router_id"
@@ -320,7 +320,7 @@ paths:
             <<: *id
           required: true
       responses:
-        200:
+        202:
           description: "OK"
           content:
             application/json:
@@ -401,9 +401,12 @@ paths:
           content:
             application/json:
               schema:
-                type: "array"
-                items:
-                  $ref: "#/components/schemas/Event"
+                type: "object"
+                properties:
+                  events:
+                    type: "array"
+                    items:
+                      $ref: "#/components/schemas/Event"
         400:
           description: "Invalid project_id or router_id"
         404:

diff --git a/sdk/samples/router/create_from_existing_router.py b/sdk/samples/router/create_from_existing_router.py
@@ -0,0 +1,182 @@
+import turing
+import turing.batch
+import turing.batch.config
+import turing.router.config.router_config
+from turing.router.config.route import Route
+from turing.router.config.router_config import RouterConfig
+from turing.router.config.router_version import RouterStatus
+from turing.router.config.resource_request import ResourceRequest
+from turing.router.config.log_config import LogConfig, ResultLoggerType
+from turing.router.config.traffic_rule import TrafficRule, HeaderTrafficRuleCondition
+from turing.router.config.enricher import Enricher
+from turing.router.config.router_ensembler_config import DockerRouterEnsemblerConfig
+from turing.router.config.common.env_var import EnvVar
+from turing.router.config.experiment_config import ExperimentConfig
+
+
+def main(turing_api: str, project: str):
+    # Initialize Turing client
+    turing.set_url(turing_api)
+    turing.set_project(project)
+
+    # Build a router for the sake of showing how you can retrieve one from the API
+    # Create some routes
+    routes = [
+        Route(
+            id='route-1',
+            endpoint='http://paths.co/route-1',
+            timeout='20ms'
+        ),
+        Route(
+            id='route-2',
+            endpoint='http://paths.co/route-2',
+            timeout='20ms'
+        )
+    ]
+
+    # Create some traffic rules
+    rules = [
+        TrafficRule(
+            conditions=[
+                HeaderTrafficRuleCondition(
+                    field='turns',
+                    values=['left']
+                )
+            ],
+            routes=[
+                'route-1'
+            ]
+        ),
+        TrafficRule(
+            conditions=[
+                HeaderTrafficRuleCondition(
+                    field='turns',
+                    values=['right']
+                )
+            ],
+            routes=[
+                'route-2'
+            ]
+        )
+    ]
+
+    # Create an experiment config (
+    experiment_config = ExperimentConfig(
+        type="xp",
+        config={
+            'variables':
+                [
+                    {'name': 'latitude', 'field': 'farm_lat', 'field_source': 'header'},
+                    {'name': 'longitude', 'field': 'farm_long', 'field_source': 'header'}
+                ],
+            'project_id': 102
+        }
+    )
+
+    # Create a resource request config for the router
+    resource_request = ResourceRequest(
+        min_replica=0,
+        max_replica=2,
+        cpu_request="500m",
+        memory_request="512Mi"
+    )
+
+    # Create a log config for the router
+    log_config = LogConfig(
+        result_logger_type=ResultLoggerType.NOP
+    )
+
+    # Create an enricher for the router
+    enricher = Enricher(
+        image="ealen/echo-server:0.5.1",
+        resource_request=ResourceRequest(
+            min_replica=0,
+            max_replica=2,
+            cpu_request="500m",
+            memory_request="512Mi"
+        ),
+        endpoint="/",
+        timeout="60ms",
+        port=3000,
+        env=[
+            EnvVar(
+                name="NODES",
+                value="2"
+            )
+        ]
+    )
+
+    # Create an ensembler for the router
+    ensembler = DockerRouterEnsemblerConfig(
+        id=1,
+        image="ealen/echo-server:0.5.1",
+        resource_request=ResourceRequest(
+            min_replica=1,
+            max_replica=3,
+            cpu_request="500m",
+            memory_request="512Mi"
+        ),
+        endpoint="/echo",
+        timeout="60ms",
+        port=3000,
+        env=[],
+    )
+
+    # Create the RouterConfig instance
+    router_config = RouterConfig(
+        environment_name="id-dev",
+        name="my-router-1",
+        routes=routes,
+        rules=rules,
+        default_route_id="test",
+        experiment_engine=experiment_config,
+        resource_request=resource_request,
+        timeout="100ms",
+        log_config=log_config,
+        enricher=enricher,
+        ensembler=ensembler
+    )
+
+    # Create a router using the RouterConfig object
+    router = turing.Router.create(router_config)
+    print(f"You have created a router with id: {router.id}")
+
+    # Wait for the router to get deployed; note that a router that is PENDING will have None as its router_config
+    try:
+        router.wait_for_status(RouterStatus.DEPLOYED)
+    except TimeoutError:
+        raise Exception(f"Turing API is taking too long for router {router.id} to get deployed.")
+
+    # Imagine we only have the router's id, and would like to retrieve it
+    router_1 = turing.Router.get(router.id)
+
+    # Now we'd like to create a new router that's similar to router_1, but with some configs modified
+    # Get the router config from router_1
+    router_config = router_1.config
+
+    # Make your desired changes to the config
+    # Note that router_config.enricher.env is a regular Python list; so you can use methods such as append or extend
+    router_config.enricher.env.append(
+        EnvVar(
+            name="WORKERS",
+            value="2"
+        )
+    )
+
+    router_config.resource_request.max_replica = 5
+
+    # NOTE: If you are using this config (extracted from an existing router) to create a NEW router, remember to give it
+    # a new name (this will end up being registered as the router name and router names MUST be unique)
+    router_config.name = "my-router-2"
+
+    # Create your new router with the router_config object
+    router_2 = turing.Router.create(router_config)
+
+    # Check the routers that you now have
+    for r in turing.Router.list():
+        print(r)
+
+
+if __name__ == '__main__':
+    import fire
+    fire.Fire(main)