Fix dependencies and migrate benchmark auth #490
Conversation
Noting some authorization methods don't require the ACCESS_KEY and SECRET_KEY.
| colorama==0.4.1 # required for structlog | ||
| connexion==1.4 | ||
| connexion[swagger-ui]==2.6 |
| @@ -24,7 +24,7 @@ | |||
| import time | |||
| import os | |||
|
|
|||
| from clkhash import rest_client | |||
| from anonlinkclient.rest_client import RestClient | |||
| @@ -1,6 +1,6 @@ | |||
| anonlink-client==0.0.1 | |||
There was a problem hiding this comment.
This will be 0.1.0 pretty soon
There was a problem hiding this comment.
Sweet. Our depandabot will pester us when it is released :-)
| @@ -232,11 +234,11 @@ def compose_result(status, tt, experiment, sizes, threshold): | |||
| return result | |||
|
|
|||
|
|
|||
| def delete_resources(config, credentials, run): | |||
| def delete_resources(credentials, run): | |||
There was a problem hiding this comment.
it would be great to see what key-value pairs might be in config in doc
| rest_client.run_delete(config['server'], credentials['project_id'], run['run_id'], credentials['result_token']) | ||
| rest_client.project_delete(config['server'], credentials['project_id'], credentials['result_token']) | ||
| rest_client.run_delete(credentials['project_id'], run['run_id'], credentials['result_token']) | ||
| rest_client.project_delete(credentials['project_id'], credentials['result_token']) |
There was a problem hiding this comment.
Is it true that currently there is only server in config?
There was a problem hiding this comment.
No, there is lots of stuff in config all set around line 84. the change is because the new rest_client already knows the server address
| result = {} | ||
| credentials = {} | ||
| run = {} | ||
| logger.info("Starting time: {}".format(time.asctime())) | ||
| nb_parties = len(sizes) | ||
| try: | ||
| credentials = rest_client.project_create(server, config['schema'], 'groups', | ||
| credentials = rest_client.project_create(config['schema'], 'groups', | ||
| "benchy_{}".format(experiment), parties=nb_parties) | ||
| # upload clks | ||
| upload_binary_clks(config, sizes, credentials) |
There was a problem hiding this comment.
ideally, that should be done by the rest_client as well.
There was a problem hiding this comment.
Agree => data61/anonlink-client#23
| status = rest_client.wait_for_run(server, project_id, run_id, | ||
| credentials['result_token'], timeout=config['timeout']) | ||
| status = rest_client.wait_for_run(project_id, run_id, | ||
| credentials['result_token'], timeout=config['timeout'], update_period=5) |
There was a problem hiding this comment.
will that lead to a new line in the logs every 5 seconds? what about those huge benchmarks? How big's the log file going to be?
There was a problem hiding this comment.
Not on the benchmark container side - we just wait until the whole thing is done (perhaps we should be outputting progress but that is another question).
On the server side we would log the request in both nginx and the flask app.
| arrow | ||
| boto3 | ||
| clkhash==0.14.0 | ||
| jsonschema | ||
| pandas | ||
| requests |
There was a problem hiding this comment.
shoule we pin all of those as well?
| secretKeyRef: | ||
| name: anonlink-benchmark-aws-credentials | ||
| key: OBJECT_STORE_SECRET_KEY | ||
| - name: OBJECT_STORE_BUCKET |
There was a problem hiding this comment.
don't you also have to define OBJECT_STORE_SERVER?
There was a problem hiding this comment.
No we default to assuming S3 is used
This PR makes a few small changes to the benchmark container/script:
rest_clientapi which.This depandabot flurry of dependency updates somehow managed to update Flask to a version that isn't compatible with our code. The integration tests passed on the PR, but not once in develop 👎
So I've also upped the version of connexion just to get the CI to pass, in theory this would be done in the PR #479 that Wilko is working on.