Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add router management functions to SDK #155

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
83 commits
Select commit Hold shift + click to select a range
3f7f61d
Refactor swagger/openapi specs for sdk
deadlycoconuts Dec 24, 2021
a553730
Add autogenerated openapi classes for routers
deadlycoconuts Dec 24, 2021
a69c676
Rewrote OpenAPI spec for ExperimentConfig to avoid using oneOf
deadlycoconuts Jan 4, 2022
7bb5c51
Remove dependency of ExperimentConfig schema on ExperimentEngineType
deadlycoconuts Jan 5, 2022
ff99859
Add additional fields for enricher component
deadlycoconuts Jan 5, 2022
ec6f708
Add additional nullable fields to OpenAPI spec
deadlycoconuts Jan 5, 2022
39c312d
Reference ensembler, event and router id to common id schema
deadlycoconuts Jan 6, 2022
a44fa55
Rename Ensembler in OpenAPI spec for routers to RouterEnsemblerConfig
deadlycoconuts Jan 6, 2022
a8fd945
Reorganise router related tests
deadlycoconuts Jan 7, 2022
d0dbfce
Refactor route tests to utilise predefined fixtures
deadlycoconuts Jan 10, 2022
97cff4c
Add resource_request sdk classes and tests
deadlycoconuts Jan 10, 2022
e105658
Add RouterEnsemblerConfig sdk and classes
deadlycoconuts Jan 11, 2022
25a5e9a
Move common router components to separate file
deadlycoconuts Jan 11, 2022
158204a
Refactor RouterEnsemblerConfig to use common schemas
deadlycoconuts Jan 11, 2022
8797e2f
Add Enricher sdk classes and tests
deadlycoconuts Jan 11, 2022
b7d9a89
Add ExperimentEngine sdk class
deadlycoconuts Jan 12, 2022
dc64ddc
Add fix to enforce project_id in ExperimentConfig to be read as int
deadlycoconuts Jan 12, 2022
6379eb4
Refactor regex checks for cpu and memory requests
deadlycoconuts Jan 12, 2022
46d4175
Add create router tests
deadlycoconuts Jan 12, 2022
909c4c5
Refactor string checks and add docstrings
deadlycoconuts Jan 12, 2022
ee11286
Rewrite router test case
deadlycoconuts Jan 12, 2022
13ddbbd
Modify min/max replica comparison check to allow equality
deadlycoconuts Jan 12, 2022
82bdd6d
Refactor swagger/openapi specs for sdk
deadlycoconuts Dec 24, 2021
bf41bab
Add autogenerated openapi classes for routers
deadlycoconuts Dec 24, 2021
caefe14
Rewrote OpenAPI spec for ExperimentConfig to avoid using oneOf
deadlycoconuts Jan 4, 2022
c8032e9
Remove dependency of ExperimentConfig schema on ExperimentEngineType
deadlycoconuts Jan 5, 2022
02d0476
Add additional fields for enricher component
deadlycoconuts Jan 5, 2022
a47431c
Add additional nullable fields to OpenAPI spec
deadlycoconuts Jan 5, 2022
9ad6135
Reference ensembler, event and router id to common id schema
deadlycoconuts Jan 6, 2022
ebaf83c
Rename Ensembler in OpenAPI spec for routers to RouterEnsemblerConfig
deadlycoconuts Jan 6, 2022
a125da5
Refactor route tests to utilise predefined fixtures
deadlycoconuts Jan 10, 2022
3838ee2
Add resource_request sdk classes and tests
deadlycoconuts Jan 10, 2022
56df7be
Add RouterEnsemblerConfig sdk and classes
deadlycoconuts Jan 11, 2022
ae4a169
Move common router components to separate file
deadlycoconuts Jan 11, 2022
911b861
Refactor RouterEnsemblerConfig to use common schemas
deadlycoconuts Jan 11, 2022
22e0d89
Add Enricher sdk classes and tests
deadlycoconuts Jan 11, 2022
c63a636
Add ExperimentEngine sdk class
deadlycoconuts Jan 12, 2022
fb398ae
Add fix to enforce project_id in ExperimentConfig to be read as int
deadlycoconuts Jan 12, 2022
786d4f6
Refactor regex checks for cpu and memory requests
deadlycoconuts Jan 12, 2022
138e7a3
Add create router tests
deadlycoconuts Jan 12, 2022
36f6f65
Refactor string checks and add docstrings
deadlycoconuts Jan 12, 2022
937e09e
Rewrite router test case
deadlycoconuts Jan 12, 2022
bcb6452
Modify min/max replica comparison check to allow equality
deadlycoconuts Jan 12, 2022
902b471
Refactor Enricher, LogConfig, Route, RouterConfig, RouterEnsemblerCon…
deadlycoconuts Jan 13, 2022
730c457
Remove regex validation from SDK and move them to OpenAPI spec
deadlycoconuts Jan 14, 2022
d608ce7
Reinsert missing semi-colon in js file
deadlycoconuts Jan 14, 2022
4c3c7b9
Correct regex strings in OpenApi spec
deadlycoconuts Jan 14, 2022
b21bee2
Rearrange modules in SDK
deadlycoconuts Jan 14, 2022
59e5efd
Add all router API endpoints and reformat response schema for delete …
deadlycoconuts Jan 13, 2022
9543d37
Add delete router method to router sdk class
deadlycoconuts Jan 13, 2022
0d7387a
Add SDK method for getting router by router_id
deadlycoconuts Jan 14, 2022
4cf9717
Add SDK method for updating router config
deadlycoconuts Jan 17, 2022
f5f7a7d
Add deploy and undeploy router SDK functions
deadlycoconuts Jan 17, 2022
1da7036
Add tests for create/delete/deploy/undeploy router SDK methods
deadlycoconuts Jan 17, 2022
e73674f
Add tests for undeploy router SDK method
deadlycoconuts Jan 17, 2022
fffa0cd
Add get/delete router version SDK methods
deadlycoconuts Jan 18, 2022
09dc23f
Add tests for get/delete router version SDK methods
deadlycoconuts Jan 18, 2022
af97ecb
Add deploy version/get events router SDK methods
deadlycoconuts Jan 18, 2022
7864981
Add list router versions SDK method
deadlycoconuts Jan 18, 2022
7d6c331
Reorganise router module
deadlycoconuts Jan 18, 2022
2c70ab2
Add sample file to demonstrate the use of the router class in the SDK
deadlycoconuts Jan 18, 2022
8c0c5b8
Add docstrings to RouterVersion
deadlycoconuts Jan 19, 2022
d95dffa
Replace print with logging statements and fix comments
deadlycoconuts Jan 19, 2022
64cde0e
Regroup import statements
deadlycoconuts Jan 19, 2022
d982e35
Add global numbers to implementation of to_dict
deadlycoconuts Jan 19, 2022
93f0acb
Add wait methods for routers to sample; move location of RouterStatus
deadlycoconuts Jan 19, 2022
8f1d9da
Replace kwargs with empty dict when created OpenAPI ExperimentConfig
deadlycoconuts Jan 20, 2022
01a0936
Refactor log_config fixture
deadlycoconuts Jan 20, 2022
0867468
Implement logging and tests for blocking functions
deadlycoconuts Jan 20, 2022
5b391a2
Add additional debugging statements for sample
deadlycoconuts Jan 20, 2022
85de070
Refactor blocking functions to take floats
deadlycoconuts Jan 20, 2022
b0152bf
Add dataclass implementation to config classes
deadlycoconuts Jan 20, 2022
109cedc
Remove extra comma in class definition
deadlycoconuts Jan 20, 2022
e81483d
Add additional sleep duration to ensure router dependencies are undep…
deadlycoconuts Jan 20, 2022
158ff88
Add support for experiment plugin configs
deadlycoconuts Jan 21, 2022
a63795c
Remove plugin_config from SDK and OpenAPI spec
deadlycoconuts Jan 21, 2022
082bfa0
Remove plugin_config from autogenerated classes
deadlycoconuts Jan 21, 2022
8e6ea9b
Fix update method of Router class to ensure all attributes are updated
deadlycoconuts Jan 24, 2022
6f3f1e5
Replace images with generic images from Docker Hub
deadlycoconuts Jan 24, 2022
c22c380
Use generic images and names for tests
deadlycoconuts Jan 24, 2022
38f7c2f
Add additional sample to demonstrate creating a new router from an ex…
deadlycoconuts Jan 24, 2022
99193e5
Remove redundant imports
deadlycoconuts Jan 24, 2022
225f256
Wrap sleep statements into router blocking functions
deadlycoconuts Jan 25, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions api/api/openapi-sdk.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -25,3 +25,17 @@ paths:
# R O U T E R S
"/projects/{project_id}/routers":
$ref: "specs/routers.yaml#/paths/~1projects~1{project_id}~1routers"
"/projects/{project_id}/routers/{router_id}":
$ref: "specs/routers.yaml#/paths/~1projects~1{project_id}~1routers~1{router_id}"
"/projects/{project_id}/routers/{router_id}/deploy":
$ref: "specs/routers.yaml#/paths/~1projects~1{project_id}~1routers~1{router_id}~1deploy"
"/projects/{project_id}/routers/{router_id}/undeploy":
$ref: "specs/routers.yaml#/paths/~1projects~1{project_id}~1routers~1{router_id}~1undeploy"
"/projects/{project_id}/routers/{router_id}/versions":
$ref: "specs/routers.yaml#/paths/~1projects~1{project_id}~1routers~1{router_id}~1versions"
"/projects/{project_id}/routers/{router_id}/versions/{version}":
$ref: "specs/routers.yaml#/paths/~1projects~1{project_id}~1routers~1{router_id}~1versions~1{version}"
"/projects/{project_id}/routers/{router_id}/versions/{version}/deploy":
$ref: "specs/routers.yaml#/paths/~1projects~1{project_id}~1routers~1{router_id}~1versions~1{version}~1deploy"
"/projects/{project_id}/routers/{router_id}/events":
$ref: "specs/routers.yaml#/paths/~1projects~1{project_id}~1routers~1{router_id}~1events"
4 changes: 4 additions & 0 deletions api/api/specs/experiment-engines.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -296,6 +296,10 @@ components:
$ref: "#/components/schemas/ExperimentEngineType"
config:
type: "object"
additionalProperties:
type: "object"
readOnly: true
nullable: true

# Enums
ExperimentEngineType:
Expand Down
13 changes: 8 additions & 5 deletions api/api/specs/routers.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -154,7 +154,7 @@ paths:
schema:
type: "object"
properties:
router_id:
id:
<<: *id
400:
description: "Invalid project_id or router_id"
Expand Down Expand Up @@ -320,7 +320,7 @@ paths:
<<: *id
required: true
responses:
200:
202:
description: "OK"
content:
application/json:
Expand Down Expand Up @@ -401,9 +401,12 @@ paths:
content:
application/json:
schema:
type: "array"
items:
$ref: "#/components/schemas/Event"
type: "object"
properties:
events:
type: "array"
items:
$ref: "#/components/schemas/Event"
romanwozniak marked this conversation as resolved.
Show resolved Hide resolved
400:
description: "Invalid project_id or router_id"
404:
Expand Down
182 changes: 182 additions & 0 deletions sdk/samples/router/create_from_existing_router.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,182 @@
import turing
import turing.batch
import turing.batch.config
import turing.router.config.router_config
from turing.router.config.route import Route
from turing.router.config.router_config import RouterConfig
from turing.router.config.router_version import RouterStatus
from turing.router.config.resource_request import ResourceRequest
from turing.router.config.log_config import LogConfig, ResultLoggerType
from turing.router.config.traffic_rule import TrafficRule, HeaderTrafficRuleCondition
from turing.router.config.enricher import Enricher
from turing.router.config.router_ensembler_config import DockerRouterEnsemblerConfig
from turing.router.config.common.env_var import EnvVar
from turing.router.config.experiment_config import ExperimentConfig


def main(turing_api: str, project: str):
# Initialize Turing client
turing.set_url(turing_api)
turing.set_project(project)

# Build a router for the sake of showing how you can retrieve one from the API
# Create some routes
routes = [
Route(
id='route-1',
endpoint='http://paths.co/route-1',
timeout='20ms'
),
Route(
id='route-2',
endpoint='http://paths.co/route-2',
timeout='20ms'
)
]

# Create some traffic rules
rules = [
TrafficRule(
conditions=[
HeaderTrafficRuleCondition(
field='turns',
values=['left']
)
],
routes=[
'route-1'
]
),
TrafficRule(
conditions=[
HeaderTrafficRuleCondition(
field='turns',
values=['right']
)
],
routes=[
'route-2'
]
)
]

# Create an experiment config (
experiment_config = ExperimentConfig(
type="xp",
config={
'variables':
[
{'name': 'latitude', 'field': 'farm_lat', 'field_source': 'header'},
{'name': 'longitude', 'field': 'farm_long', 'field_source': 'header'}
],
'project_id': 102
}
)

# Create a resource request config for the router
resource_request = ResourceRequest(
min_replica=0,
max_replica=2,
cpu_request="500m",
memory_request="512Mi"
)

# Create a log config for the router
log_config = LogConfig(
result_logger_type=ResultLoggerType.NOP
)

# Create an enricher for the router
enricher = Enricher(
image="ealen/echo-server:0.5.1",
resource_request=ResourceRequest(
min_replica=0,
max_replica=2,
cpu_request="500m",
memory_request="512Mi"
),
endpoint="/",
timeout="60ms",
port=3000,
env=[
EnvVar(
name="NODES",
value="2"
)
]
)

# Create an ensembler for the router
ensembler = DockerRouterEnsemblerConfig(
id=1,
image="ealen/echo-server:0.5.1",
resource_request=ResourceRequest(
min_replica=1,
max_replica=3,
cpu_request="500m",
memory_request="512Mi"
),
endpoint="/echo",
timeout="60ms",
port=3000,
env=[],
)

# Create the RouterConfig instance
router_config = RouterConfig(
environment_name="id-dev",
name="my-router-1",
routes=routes,
rules=rules,
default_route_id="test",
experiment_engine=experiment_config,
resource_request=resource_request,
timeout="100ms",
log_config=log_config,
enricher=enricher,
ensembler=ensembler
)

# Create a router using the RouterConfig object
router = turing.Router.create(router_config)
print(f"You have created a router with id: {router.id}")

# Wait for the router to get deployed; note that a router that is PENDING will have None as its router_config
try:
router.wait_for_status(RouterStatus.DEPLOYED)
except TimeoutError:
raise Exception(f"Turing API is taking too long for router {router.id} to get deployed.")

# Imagine we only have the router's id, and would like to retrieve it
router_1 = turing.Router.get(router.id)

# Now we'd like to create a new router that's similar to router_1, but with some configs modified
# Get the router config from router_1
router_config = router_1.config

# Make your desired changes to the config
# Note that router_config.enricher.env is a regular Python list; so you can use methods such as append or extend
router_config.enricher.env.append(
EnvVar(
name="WORKERS",
value="2"
)
)

router_config.resource_request.max_replica = 5

# NOTE: If you are using this config (extracted from an existing router) to create a NEW router, remember to give it
# a new name (this will end up being registered as the router name and router names MUST be unique)
router_config.name = "my-router-2"

# Create your new router with the router_config object
router_2 = turing.Router.create(router_config)

# Check the routers that you now have
for r in turing.Router.list():
print(r)


if __name__ == '__main__':
import fire
fire.Fire(main)
Loading