Commands are available to help :ref:`configure <storage_setup>`, :ref:`test <storage_test>` and :ref:`upgrade <storage_upgrade>` the storage of Oríon. There is additionally commands to :ref:`delete <storage_rm>` experiment and trials or :ref:`update <storage_set>` values in the storage.
For more flexibility, there is the :ref:`storage_python_apis`.
The setup command helps creating a global configuration file
for the configuration of Oríon's storage. For more details on its usage
see :ref:`Database Configuration` in the database
installation and configuration section.
The test command provides a simple and efficient way of testing the storage configuration. For
more details on its usage see :ref:`Test Connection` in the database installation and configuration
section.
Command to delete experiments and trials.
To delete an experiment and its trials, simply give the experiment's name.
orion db rm my-exp-nameTo delete only trials that are broken, simply add --status broken.
Note that the experiment will not be deleted, only the trials.
orion db rm my-exp-name --status brokenOr --status * to delete all trials of the experiment.
orion db rm my-exp-name --status *By default, the last version of the experiment is deleted. Add --version
to select a prior version. Note that all child of the selected version
will be deleted as well. You cannot delete a parent experiment without
deleting the child experiments.
orion db rm my-exp-name --version 1Command to update trial attributes.
To change a trial status, simply give the experiment name, trial id and status. (use orion status --all to get trial ids)
orion db set my-exp-name id=3cc91e851e13281ca2152c19d888e937 status=interruptedTo change all trials from a given status to another, simply give the two status
orion db set my-exp-name status=broken status=interruptedOr * to apply the change to all trials
orion db set my-exp-name '*' status=interruptedBy default, trials of the last version of the experiment are selected. Add --version to select a prior version. Note that the modification is applied recursively to all child experiment, but not to the parents.
orion db set my-exp-name --version 1 status=broken status=interruptedDatabase scheme may change from one version of Oríon to another. If such change happens, you will get the following error after upgrading Oríon.
The database is outdated. You can upgrade it with the command `orion db upgrade`.Make sure to create a backup of your database before upgrading it. You should also make sure that no process writes to the database during the upgrade otherwise the latter could fail. When ready, simply run the upgrade command.
orion db upgradeIn short, users are expected to only use the :py:class:`ExperimentClient <orion.client.experiment.ExperimentClient>` to interact with the storage client, to fetch and register trials. Creation of experiments should always be done through :py:func:`create_experiment() <orion.client.create_experiment>`.
If you need to access the storage with more flexibility, you can do so using the methods of the storage client directly. See :ref:`storage_backend` section for more details.
Finally, legacy databases supported by Oríon can also be accessed directly in last resort if the storage backend is not flexible enough. See :ref:`database_backend` section for more details.
The experiment client must be created with the helper function :py:func:`get_experiment() <orion.client.get_experiment>` which will take care of initiating the storage backend and load the corresponding experiment from the storage. To create a new experiment use :py:func:`create_experiment() <orion.client.create_experiment>`.
There is a small subset of methods to fetch trials or register new ones. We focus here on the methods for loading or creation of trials in particular, see :py:class:`ExperimentClient <orion.client.experiment.ExperimentClient>` for documentation of all methods.
The experiment client can be loaded in read-only or read/write mode. Make sure to load the experiment with the proper mode if you want to edit the database. For full read/write/execution rights, use :py:func:`create_experiment() <orion.client.create_experiment>`.
Here is a short example to fetch trials or insert a new one.
from orion.client import create_experiment
# Create the ExperimentClient
experiment = create_experiment('exp-name', space=dict(x='uniform(0, 1)'))
# To fetch all trials from an experiment
trials = experiment.fetch_trials()
# To fetch trials in a form on panda dataframe
df = experiment.to_pandas()
# Insert a new trial in storage
experiment.insert(dict(x=0.5))
# Insert a new trial and reserve to execute
trial = experiment.insert(dict(x=0.6), reserve=True).. automethod:: orion.client.experiment.ExperimentClient.to_pandas :noindex:
.. automethod:: orion.client.experiment.ExperimentClient.fetch_trials :noindex:
.. automethod:: orion.client.experiment.ExperimentClient.fetch_trials_by_status :noindex:
.. automethod:: orion.client.experiment.ExperimentClient.fetch_noncompleted_trials :noindex:
.. automethod:: orion.client.experiment.ExperimentClient.get_trial :noindex:
.. automethod:: orion.client.experiment.ExperimentClient.insert :noindex:
Warning
The storage backends are not meant to be used directly by users. Be careful if you use any method which modifies the data in storage or you may break your experiment or trials.
The storage backend is used by the :py:class:`ExperimentClient <orion.client.experiment.ExperimentClient>` to read and write persistant records of the experiment and trials. Although we recommand using the experiment client, we document the storage backend here for users who may need more flexibility.
There is two ways for creating the storage client. If you already created an experiment client, the storage was already created during the process of creating the experiment client and you can get it with :py:func:`orion.storage.base.get_storage`. Otherwise, you can create the storage client with :py:func:`orion.storage.base.setup_storage` before fetching it with :py:func:`get_storage() <orion.storage.base.get_storage>`. To recap, you can create it indirectly with :py:func:`create_experiment() <orion.client.create_experiment>` or directly with :py:func:`setup_storage() <orion.storage.base.setup_storage>`. In both case, you can access it with :py:func:`get_storage() <orion.storage.base.get_storage>`.
from orion.client import create_experiment
from orion.storage.base import get_storage, setup_storage
# Create the ExperimentClient and storage implicitly
experiment = create_experiment('exp-name', space=dict(x='uniform(0, 1)'))
# Or create storage explicitly using setup_storage
setup_storage(dict(
type='legacy',
database=dict(
type='pickleddb',
host='db.pkl')
)
)
)
# Get the storage client
storage = get_storage()
# fetch trials
trials = storage.fetch_trials(uid=experiment.id)
# Update trial status
storage.set_trial_status(trials[0], 'interrupted')Note
The function :py:func:`setup_storage() <orion.storage.base.setup_storage>` reads the global configuration like :py:func:`create_experiment() <orion.client.create_experiment>` does if there is missing information. Therefore, it is possible to call it without any argument the same way it is possible to call :py:func:`create_experiment() <orion.client.create_experiment>` without specifying storage configuration.
.. automethod:: orion.storage.base.BaseStorageProtocol.update_experiment :noindex:
.. automethod:: orion.storage.base.BaseStorageProtocol.fetch_experiments :noindex:
.. automethod:: orion.storage.base.BaseStorageProtocol.delete_experiment :noindex:
.. automethod:: orion.storage.base.BaseStorageProtocol.register_trial :noindex:
.. automethod:: orion.storage.base.BaseStorageProtocol.reserve_trial :noindex:
.. automethod:: orion.storage.base.BaseStorageProtocol.fetch_trials :noindex:
.. automethod:: orion.storage.base.BaseStorageProtocol.delete_trials :noindex:
.. automethod:: orion.storage.base.BaseStorageProtocol.get_trial :noindex:
.. automethod:: orion.storage.base.BaseStorageProtocol.update_trials :noindex:
.. automethod:: orion.storage.base.BaseStorageProtocol.update_trial :noindex:
.. automethod:: orion.storage.base.BaseStorageProtocol.fetch_lost_trials :noindex:
.. automethod:: orion.storage.base.BaseStorageProtocol.fetch_pending_trials :noindex:
.. automethod:: orion.storage.base.BaseStorageProtocol.fetch_noncompleted_trials :noindex:
.. automethod:: orion.storage.base.BaseStorageProtocol.fetch_trials_by_status :noindex:
.. automethod:: orion.storage.base.BaseStorageProtocol.count_completed_trials :noindex:
.. automethod:: orion.storage.base.BaseStorageProtocol.count_broken_trials :noindex:
.. automethod:: orion.storage.base.BaseStorageProtocol.set_trial_status :noindex:
Warning
The database backends are not meant to be used directly by users. Be careful if you use any method which modifies the data in database or you may break your experiment or trials.
The database backend used to be the sole database support
initially. An additional abstraction layer, the storage protocol,
has been added with the goal to support various storage types
such as third-party experiment management platforms which
could not be supported using the basic methods read
and write.
This is why the database backend has been turned into
a legacy storage procotol. Because it is the default
storage protocol, we document it here for users
who may need even more flexibility than what the
storage protocol provides.
There is two ways for creating the database client. If you already created an experiment client, the database was already created during the process of creating the experiment client and you can get it with :py:func:`orion.storage.legacy.get_database`. Otherwise, you can create the database client with :py:func:`orion.storage.legacy.setup_database` before fetching it with :py:func:`get_database() <orion.storage.legacy.get_database>`. To recap, you can create it indirectly with :py:func:`create_experiment() <orion.client.create_experiment>` or directly with :py:func:`setup_database() <orion.storage.legacy.setup_database>`. In both case, you can access it with :py:func:`get_database() <orion.storage.legacy.get_database>`.
Here's an example on how you could remove an experiment
from orion.client import create_experiment
from orion.storage.legacy import get_database, setup_database
# Create the ExperimentClient and database implicitly
experiment = create_experiment('exp-name', space=dict(x='uniform(0, 1)'))
# Or create database explicitly using setup_database
setup_database(dict(
type='pickleddb',
host='db.pkl'
)
)
# This gets the db singleton that was already instantiated within the experiment object.
db = get_database()
# To remove all trials of an experiment
db.remove('trials', dict(experiment=experiment.id))
# To remove the experiment
db.remove('experiments', dict(_id=experiment.id)).. automethod:: orion.core.io.database.Database.read
.. automethod:: orion.core.io.database.Database.write
.. automethod:: orion.core.io.database.Database.remove
.. automethod:: orion.core.io.database.Database.read_and_write