Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

no singleton test #942

Merged
merged 98 commits into from
Jul 29, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
98 commits
Select commit Hold shift + click to select a range
f9fb609
no singleton test
Delaunay Jun 6, 2022
508a6a3
use config to create the storage
Delaunay Jun 6, 2022
6fd116d
Run test to check
Delaunay Jun 6, 2022
d548376
Remove reliance on singleton when possible
Delaunay Jun 7, 2022
fec2323
Forward storage to resolution
Delaunay Jun 7, 2022
be01c47
remove some get_storage imports
Delaunay Jun 8, 2022
5cf114b
-
Delaunay Jun 8, 2022
25f0021
-
Delaunay Jun 8, 2022
2a25e5f
-
Delaunay Jun 8, 2022
236ef3d
Hunt all setup_storage
Delaunay Jun 8, 2022
964d89b
Fix test setup
Delaunay Jun 9, 2022
180e3db
-
Delaunay Jun 9, 2022
0170030
Make sure state sets the config on init
Delaunay Jun 13, 2022
db9ad69
Patch pickleddb setup issue
Delaunay Jun 14, 2022
7968209
Add config reset option
Delaunay Jun 15, 2022
cb38531
-
Delaunay Jun 15, 2022
2aa4086
-
Delaunay Jun 16, 2022
3dd17e1
Make sure exception are printed in debug at the very least, pass stor…
Delaunay Jun 16, 2022
ca0a449
-
Delaunay Jun 16, 2022
2e3895b
Propagate more storage instance
Delaunay Jun 21, 2022
231d2ae
Fix some tests
Delaunay Jun 21, 2022
3151380
-
Delaunay Jun 21, 2022
b660aa2
Use orion state for storage setups
Delaunay Jun 21, 2022
06454c6
-
Delaunay Jun 21, 2022
9901b09
Make sure ExperimentBranchBuilder use a storage
Delaunay Jun 22, 2022
8288c08
-
Delaunay Jun 22, 2022
9c9b3af
-
Delaunay Jun 23, 2022
456e4b6
Merge branch 'develop' of github.com:Epistimio/orion into no_singleton
Delaunay Jun 23, 2022
cc6433e
Resolve some EVC failures
Delaunay Jun 23, 2022
9ee1cbf
-
Delaunay Jun 23, 2022
7bb08c2
Update config to use pickleddb
Delaunay Jul 4, 2022
9cd205b
Make sure storage is using tmp paths
Delaunay Jul 4, 2022
00023d5
Fix missing storage config inside global conf file
Delaunay Jul 5, 2022
6c82d24
-
Delaunay Jul 5, 2022
0431fee
-
Delaunay Jul 6, 2022
d5d785d
-
Delaunay Jul 6, 2022
559f5e6
Fix serving tests
Delaunay Jul 7, 2022
ea93839
Merge branch 'develop' of github.com:Epistimio/orion into no_singleton
Delaunay Jul 7, 2022
53fd6ca
-
Delaunay Jul 7, 2022
656ce7b
-
Delaunay Jul 7, 2022
bec3768
-
Delaunay Jul 7, 2022
c1eace5
-
Delaunay Jul 7, 2022
3a99332
Tweak Plot test setup
Delaunay Jul 11, 2022
10965e1
-
Delaunay Jul 11, 2022
231b564
-
Delaunay Jul 11, 2022
20cfe97
-
Delaunay Jul 11, 2022
b6f729d
Pylint
Delaunay Jul 11, 2022
b1e9f9e
Remove useless function
Delaunay Jul 11, 2022
f2503f3
Remove debug trace
Delaunay Jul 11, 2022
9f57d74
-
Delaunay Jul 11, 2022
f97b316
fix ci
Delaunay Jul 11, 2022
d6b1422
Update serve code
Delaunay Jul 11, 2022
2f5f0f9
Merge branch 'no_singleton' of github.com:Delaunay/orion into no_sing…
Delaunay Jul 11, 2022
5831a98
Update src/orion/core/io/database/pickleddb.py
Delaunay Jul 12, 2022
2b66e92
Update src/orion/core/io/database/pickleddb.py
Delaunay Jul 12, 2022
bad830a
-
Delaunay Jul 12, 2022
4c6c1b8
Improve ExperimentBuilder docstring
bouthilx Jul 12, 2022
a3ddb5a
Use branch_builder's storage
bouthilx Jul 12, 2022
27b9e30
Remove all SingletonAlreadyInstantiatedError exception
Delaunay Jul 12, 2022
da8363d
-
Delaunay Jul 12, 2022
735bb11
Merge branch 'no_singleton' of github.com:Delaunay/orion into no_sing…
Delaunay Jul 12, 2022
d06bbc4
-
Delaunay Jul 12, 2022
152f4f0
-
Delaunay Jul 12, 2022
b3cdd12
Merge branch 'develop' of github.com:Epistimio/orion into no_singleton
Delaunay Jul 13, 2022
ee32040
Add test
Delaunay Jul 13, 2022
c5b9f44
Remove dead code
Delaunay Jul 14, 2022
5425793
Factor out storage from get_or_create_benchmark
Delaunay Jul 15, 2022
ee35552
Merge branch 'develop' of github.com:Epistimio/orion into no_singleton
Delaunay Jul 15, 2022
9770f94
-
Delaunay Jul 15, 2022
d4505a9
-
Delaunay Jul 15, 2022
ed99c02
-
Delaunay Jul 15, 2022
965a895
-
Delaunay Jul 15, 2022
999d7e2
-
Delaunay Jul 15, 2022
8c6b775
-
Delaunay Jul 15, 2022
1fca8a6
-
Delaunay Jul 15, 2022
99db807
-
Delaunay Jul 15, 2022
2a76d1e
-
Delaunay Jul 15, 2022
ec16fd9
-
Delaunay Jul 15, 2022
1129838
-
Delaunay Jul 15, 2022
560f81a
-
Delaunay Jul 18, 2022
677a7b1
-
Delaunay Jul 18, 2022
4dd8e67
-
Delaunay Jul 18, 2022
f9082ad
=
Delaunay Jul 18, 2022
833db87
-
Delaunay Jul 18, 2022
45b2c56
-
Delaunay Jul 28, 2022
f5e1f56
Update src/orion/core/worker/experiment.py
Delaunay Jul 28, 2022
9b85923
Update tests/functional/configuration/test_all_options.py
Delaunay Jul 28, 2022
f5b43d2
Add tests for Configuration.from_dict
Delaunay Jul 28, 2022
1194a43
Merge branch 'no_singleton' of github.com:Delaunay/orion into no_sing…
Delaunay Jul 28, 2022
04efcb2
Update tests/unittests/storage/test_legacy.py
Delaunay Jul 28, 2022
c85d3b7
Update tests/unittests/storage/test_legacy.py
Delaunay Jul 28, 2022
4958421
-
Delaunay Jul 28, 2022
ebdeea7
Merge branch 'no_singleton' of github.com:Delaunay/orion into no_sing…
Delaunay Jul 28, 2022
a36817f
-
Delaunay Jul 28, 2022
9520c85
-
Delaunay Jul 28, 2022
7ce98f8
Add comment for why we need to override setup_storage for certain tests
Delaunay Jul 28, 2022
b2be35c
-
Delaunay Jul 28, 2022
2b9d745
-
Delaunay Jul 28, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ jobs:
strategy:
matrix:
toxenv: [pylint, doc8, docs]

steps:
- uses: actions/checkout@v2
- name: Set up Python 3.9
Expand Down
4 changes: 2 additions & 2 deletions dashboard/src/src/__tests__/flattenObject.test.js
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ test('test flatten object', () => {
const input = {
a: 1,
b: {
ba: 'world',
be: 'world',
bb: 1.5,
bc: {
bd: {
Expand All @@ -30,7 +30,7 @@ test('test flatten object', () => {
const keys = Object.keys(output);
expect(keys.length).toBe(12);
expect(output.hasOwnProperty('a')).toBeTruthy();
expect(output.hasOwnProperty('b.ba')).toBeTruthy();
expect(output.hasOwnProperty('b.be')).toBeTruthy();
expect(output.hasOwnProperty('b.bb')).toBeTruthy();
expect(output.hasOwnProperty('b.bc.bd.a key')).toBeTruthy();
expect(output.hasOwnProperty('b.bc.bd.another key.x')).toBeTruthy();
Expand Down
10 changes: 3 additions & 7 deletions docs/scripts/build_database_and_plots.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,7 @@
import subprocess

from orion.client import get_experiment
from orion.core.utils.singleton import update_singletons
from orion.storage.base import get_storage, setup_storage
from orion.storage.base import setup_storage

ROOT_DIR = os.path.abspath(os.path.dirname(__file__))
DOC_SRC_DIR = os.path.join(ROOT_DIR, "..", "src")
Expand Down Expand Up @@ -69,9 +68,8 @@ def prepare_dbs():

def setup_tmp_storage(host):
# Clear singletons
update_singletons()

setup_storage(
return setup_storage(
storage={
"type": "legacy",
"database": {
Expand All @@ -81,8 +79,6 @@ def setup_tmp_storage(host):
}
)

return get_storage()


def load_data(host):
print("Loading data from", host)
Expand All @@ -102,7 +98,7 @@ def copy_data(data, host=TMP_DB_HOST):
storage = setup_tmp_storage(host)
for exp_id, experiment in data["experiments"].items():
del experiment["_id"]
storage.create_experiment(experiment)
storage.create_experiment(experiment, storage=storage)
assert exp_id != experiment["_id"]
trials = []
for trial in data["trials"][exp_id]:
Expand Down
6 changes: 2 additions & 4 deletions docs/scripts/filter_database.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,11 @@
import shutil

from orion.core.io.orion_cmdline_parser import OrionCmdlineParser
from orion.storage.base import get_storage, setup_storage
from orion.storage.base import setup_storage

shutil.copy("./examples/plotting/database.pkl", "./examples/base_db.pkl")

setup_storage(
storage = setup_storage(
dict(
type="legacy",
database=dict(type="pickleddb", host="./examples/base_db.pkl"),
Expand All @@ -23,8 +23,6 @@
("lateral-view-multitask3", 1): "3-dim-cat-shape-exp",
}

storage = get_storage()


def update_dropout(experiment_config):
metadata = experiment_config["metadata"]
Expand Down
13 changes: 9 additions & 4 deletions examples/benchmark/profet_benchmark.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@
ProfetXgBoostTask,
)
from orion.benchmark.task.profet.profet_task import MetaModelConfig, ProfetTask
from orion.storage.base import setup_storage

try:
from simple_parsing.helpers import choice
Expand Down Expand Up @@ -103,7 +104,15 @@ def main(config: ProfetExperimentConfig):

print(f"Storage file used: {config.storage_pickle_path}")

storage = setup_storage(
{
"type": "legacy",
"database": {"type": "pickleddb", "host": str(config.storage_pickle_path)},
}
)

benchmark = get_or_create_benchmark(
storage,
name=config.name,
algorithms=config.algorithms,
targets=[
Expand All @@ -114,10 +123,6 @@ def main(config: ProfetExperimentConfig):
],
}
],
storage={
"type": "legacy",
"database": {"type": "pickleddb", "host": str(config.storage_pickle_path)},
},
debug=config.debug,
)
benchmark.setup_studies()
Expand Down
22 changes: 17 additions & 5 deletions src/orion/benchmark/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
import orion.core
from orion.client import create_experiment
from orion.executor.base import executor_factory
from orion.storage.base import BaseStorageProtocol


class Benchmark:
Expand All @@ -20,8 +21,12 @@ class Benchmark:

Parameters
----------
storage: Storage
Delaunay marked this conversation as resolved.
Show resolved Hide resolved
Instance of the storage to use

name: str
Name of the benchmark

algorithms: list, optional
Algorithms used for benchmark, and for each algorithm, it can be formats as below:

Expand Down Expand Up @@ -49,19 +54,26 @@ class Benchmark:
task: list
Task objects

storage: dict, optional
Configuration of the storage backend.
executor: `orion.executor.base.BaseExecutor`, optional
Executor to run the benchmark experiments
"""

def __init__(self, name, algorithms, targets, storage=None, executor=None):
def __init__(
self,
storage,
name,
algorithms,
targets,
executor=None,
):
assert isinstance(storage, BaseStorageProtocol)

self._id = None
self.name = name
self.algorithms = algorithms
self.targets = targets
self.metadata = {}
self.storage_config = storage
self.storage = storage
self._executor = executor
self._executor_owner = False

Expand Down Expand Up @@ -353,7 +365,7 @@ def setup_experiments(self):
space=space,
algorithms=algorithm.experiment_algorithm,
max_trials=max_trials,
storage=self.benchmark.storage_config,
storage=self.benchmark.storage,
executor=executor,
)
self.experiments_info.append((task_index, experiment))
Expand Down
50 changes: 27 additions & 23 deletions src/orion/benchmark/benchmark_client.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,19 +11,24 @@
from orion.benchmark.task.base import bench_task_factory
from orion.core.io.database import DuplicateKeyError
from orion.core.utils.exceptions import NoConfigurationError
from orion.storage.base import get_storage, setup_storage

logger = logging.getLogger(__name__)


def get_or_create_benchmark(
name, algorithms=None, targets=None, storage=None, executor=None, debug=False
storage,
name,
algorithms=None,
targets=None,
executor=None,
):
"""
Create or get a benchmark object.

Parameters
----------
storage: BaseStorageProtocol
Instance of the storage to use
name: str
Name of the benchmark
algorithms: list, optional
Expand All @@ -35,30 +40,25 @@ def get_or_create_benchmark(
Assessment objects
task: list
Task objects
storage: dict, optional
Configuration of the storage backend.
executor: `orion.executor.base.BaseExecutor`, optional
Executor to run the benchmark experiments
debug: bool, optional
If using in debug mode, the storage config is overridden with legacy:EphemeralDB.
Defaults to False.

Returns
-------
An instance of `orion.benchmark.Benchmark`
"""
setup_storage(storage=storage, debug=debug)

# fetch benchmark from db
db_config = _fetch_benchmark(name)
db_config = _fetch_benchmark(storage, name)

benchmark_id = None
input_configure = None

if db_config:
if algorithms or targets:
input_benchmark = Benchmark(name, algorithms, targets)
input_benchmark = Benchmark(storage, name, algorithms, targets)
input_configure = input_benchmark.configuration

benchmark_id, algorithms, targets = _resolve_db_config(db_config)

if not algorithms or not targets:
Expand All @@ -68,7 +68,11 @@ def get_or_create_benchmark(
)

benchmark = _create_benchmark(
name, algorithms, targets, storage=storage, executor=executor
storage,
name,
algorithms,
targets,
executor=executor,
)

if input_configure and input_benchmark.configuration != benchmark.configuration:
Expand All @@ -80,7 +84,7 @@ def get_or_create_benchmark(
if benchmark_id is None:
logger.debug("Benchmark not found in DB. Now attempting registration in DB.")
try:
_register_benchmark(benchmark)
_register_benchmark(storage, benchmark)
logger.debug("Benchmark successfully registered in DB.")
except DuplicateKeyError:
logger.info(
Expand All @@ -89,7 +93,11 @@ def get_or_create_benchmark(
)
benchmark.close()
benchmark = get_or_create_benchmark(
name, algorithms, targets, storage, executor, debug
storage,
name,
algorithms,
targets,
executor,
)

return benchmark
Expand Down Expand Up @@ -132,9 +140,9 @@ def _resolve_db_config(db_config):
return benchmark_id, algorithms, targets


def _create_benchmark(name, algorithms, targets, storage, executor):
def _create_benchmark(storage, name, algorithms, targets, executor):

benchmark = Benchmark(name, algorithms, targets, storage, executor)
benchmark = Benchmark(storage, name, algorithms, targets, executor)
benchmark.setup_studies()

return benchmark
Expand All @@ -147,22 +155,18 @@ def _create_study(benchmark, algorithms, assess, task):
return study


def _fetch_benchmark(name):

if name:
configs = get_storage().fetch_benchmark({"name": name})
else:
configs = get_storage().fetch_benchmark({})
def _fetch_benchmark(storage, name):
configs = storage.fetch_benchmark({"name": name})

if not configs:
return {}

return configs[0]


def _register_benchmark(benchmark):
def _register_benchmark(storage, benchmark):
benchmark.metadata["datetime"] = datetime.datetime.utcnow()
config = benchmark.configuration
# This will raise DuplicateKeyError if a concurrent experiment with
# identical (name, metadata.user) is written first in the database.
get_storage().create_benchmark(config)
storage.create_benchmark(config)
Loading