Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update benchmarking user guide for reading private datasets #427

Merged
merged 1 commit into from
May 14, 2021
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 23 additions & 1 deletion docs/user_guides/benchmarking/run.rst
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,10 @@ The ``sdgym.run`` function accepts the following arguments:
- ``output_path (str)``:: If an ``output_path`` is given, the generated
leaderboard will be stored in the indicated path as a CSV file. The
given path must be a complete path including the ``.csv`` filename.
- ``aws_key (str)``:: If an ``aws_key`` is provided, the given access
key id will be used to read from the specified bucket.
- ``aws_secret (str)``:: If an ``aws_secret`` is provided, the given
secret access key will be used to read from the specified bucket.

Synthesizers
~~~~~~~~~~~~
Expand Down Expand Up @@ -167,6 +171,24 @@ For example, if we want to evaluate our synthesizer function on only the

In [15]: scores = sdgym.run(my_synthesizer_function, datasets=datasets)

Running on Private Datasets
~~~~~~~~

If we want to run sdgym on datasets in a private S3 bucket for which we do not have
`system-wide credentials configured <https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html>`__,
we can pass in the s3 bucket name, along with the aws access key ID and the aws secret
access key. For example:

.. code:: python3

In [16]: scores = sdgym.run(
...: my_synthesizer_function,
...: datasets=datasets,
...: bucket='my-bucket',
...: aws_key='my-key',
...: aws_secret='my-secret'
...: )

Iterations
~~~~~~~~~~

Expand All @@ -181,4 +203,4 @@ combination can be altered by passing a different value to the

.. code:: python3

In [16]: scores = sdgym.run(my_synthesizer_function, iterations=10)
In [17]: scores = sdgym.run(my_synthesizer_function, iterations=10)