Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor algorithms observe/suggest to work on trial objects directly #686

Merged
merged 34 commits into from
Nov 16, 2021

Commits on Oct 22, 2021

  1. Configuration menu
    Copy the full SHA
    c267fe6 View commit details
    Browse the repository at this point in the history
  2. Adjust Transformers space to using Trials

    Why:
    
    The algorithms will be working with trial objects instead of tuple of
    values. This means the space needs to sample trials, and space
    transformations should be applied on trials as well instead of tuples of
    values.
    
    How:
    
    For simplicity, only interface of the space classes (TransformedSpace
    and ReshapedSpace) will be working with trials. The transformations per
    dimension will be applied using tuple of values so that, in particular,
    reshaping operations remain straightforward.
    
    To facilitate debugging, transformed trials are wrapped so that the
    original trial can still be accessible. This will prove handy in
    algorithms if we need access to original trial objects, and also because
    the ID of the transformed trial should always be based on the original
    parameters (otherwise the ID gets incoherent with the database).
    bouthilx committed Oct 22, 2021
    Configuration menu
    Copy the full SHA
    567a392 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    8fdd52f View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    b64d1ae View commit details
    Browse the repository at this point in the history

Commits on Nov 5, 2021

  1. Configuration menu
    Copy the full SHA
    9a25e52 View commit details
    Browse the repository at this point in the history
  2. Test that TransformedTrial can be copied.

    The use of `__getattr__` to copy the TransformedTrial was causing an
    infinite recursion error.
    bouthilx committed Nov 5, 2021
    Configuration menu
    Copy the full SHA
    c3c50a8 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    72022e1 View commit details
    Browse the repository at this point in the history
  4. Handle reshaped dimension with diff. name ordering

    Why:
    
    If the names of the dimensions have the same prefix, but some have a
    shape, a transformed space with have a different ordering of dimension.
    For example, the following dimensions will have their named sorted
    differently:
    
      dim (shape 2), dim1 (no shape) -->  dim1, dim[0], dim[1]
    
    This is causing an issue when we try to restore the shape of the
    transformed dimension, with the names being swapped.
    
    How:
    
    When restoring shape, keep track of the original keys and their order,
    and reassign the restored dimensions to the correct index (correct dim
    name).
    bouthilx committed Nov 5, 2021
    Configuration menu
    Copy the full SHA
    629f748 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    e6ec755 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    3d4f54b View commit details
    Browse the repository at this point in the history

Commits on Nov 8, 2021

  1. Configuration menu
    Copy the full SHA
    ed70273 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    e15723c View commit details
    Browse the repository at this point in the history
  3. Support reshaped transform on dim + default_value

    Reshaped transform on dim:
    
    Sometimes we need to apply a transformation on a dimension only. So far
    the transformation of View could only be applied on a full point,
    because the ReshapedDimension wrapping the view was expected to fetch
    the corresponding dimension at self.index. This commit pushes up the
    fetch at self.index at the level of the ReshapedSpace instead, the only
    place where we are expected to work on a full point, not only on a
    dimension.
    
    default value:
    
    Transformed dimensions should support default value like original
    dimensions.
    bouthilx committed Nov 8, 2021
    Configuration menu
    Copy the full SHA
    038da12 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    c41fb1e View commit details
    Browse the repository at this point in the history

Commits on Nov 9, 2021

  1. Add branch() method to Trial

    Why:
    
    The branching method will be used by algorithm generating new trials
    based on existing one, such as Hyperband increasing the fidelity, and
    later on Population Based Training which could also change the working
    directory.
    
    How:
    
    When calling branch(), a new trial is created with same parameters as
    current one, except for the parameter values passed to
    (ex: ``branch(params={'some': 'values'})``).
    bouthilx committed Nov 9, 2021
    Configuration menu
    Copy the full SHA
    4445570 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    5455a85 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    7c42f8c View commit details
    Browse the repository at this point in the history

Commits on Nov 10, 2021

  1. Get rid of TransformedTrial

    Why:
    
    The wrapper makes it cumbersome to keep a coherent interface with the
    base Trial object. Also it does not bring much value beside providing
    access to the original trial and its corresponding id. Algorithms can
    revert the trial to access the trial id.
    
    How:
    
    Return a trial object with params overwritten instead. Like it was
    already done when reverting a transformed trial anyway. Now base
    algo.get_id() method compute the trial hash based on the reverted trial.
    bouthilx committed Nov 10, 2021
    Configuration menu
    Copy the full SHA
    30c6337 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    08dd775 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    f457515 View commit details
    Browse the repository at this point in the history

Commits on Nov 11, 2021

  1. Configuration menu
    Copy the full SHA
    531c5c8 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    f7d0dd8 View commit details
    Browse the repository at this point in the history
  3. Remove pack/unpack_point

    Why:
    
    Deprecated and to be removed for v0.2.0. The newest versions
    regroup_dims and flatten_dims are also useless now due to the flattened
    space. They are also removed.
    bouthilx committed Nov 11, 2021
    Configuration menu
    Copy the full SHA
    5c53fbd View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    b8228c9 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    72d4535 View commit details
    Browse the repository at this point in the history
  6. isort

    bouthilx committed Nov 11, 2021
    Configuration menu
    Copy the full SHA
    85fa11a View commit details
    Browse the repository at this point in the history
  7. pylint

    bouthilx committed Nov 11, 2021
    Configuration menu
    Copy the full SHA
    7ad527b View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    5cee651 View commit details
    Browse the repository at this point in the history
  9. Fix doc reference

    bouthilx committed Nov 11, 2021
    Configuration menu
    Copy the full SHA
    cf1ac0b View commit details
    Browse the repository at this point in the history
  10. Dims of shape (1, ) should use Views

    Why:
    
    When flattening the space, dims of shape (1, ) should be flattened
    as well otherwise the parameters will be a list of one element.
    bouthilx committed Nov 11, 2021
    Configuration menu
    Copy the full SHA
    6b06eb9 View commit details
    Browse the repository at this point in the history
  11. Configuration menu
    Copy the full SHA
    2d43e2c View commit details
    Browse the repository at this point in the history
  12. Configuration menu
    Copy the full SHA
    e9b3c08 View commit details
    Browse the repository at this point in the history

Commits on Nov 12, 2021

  1. Take a snapshot of trials_info to deepcopy it

    Why:
    
    The deepcopy is failing on github-actions with error
    `RuntimeError: dictionary changed size during iteration`. I have been
    unable to reproduce the issue locally both with python 3.6 and 3.7. It
    does fail on 3.7 on github-actions. Taking a copy of the dictionary to
    do the deep copy should fix the issue only a dictionary inside some
    trials is the source of the issue. The stack trace seams to hint towards
    trials_info as the culprit however.
    
    ```
     tests/unittests/benchmark/test_benchmark_client.py:345:
      _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
      src/orion/benchmark/__init__.py:90: in process
          study.execute(n_workers)
      src/orion/benchmark/__init__.py:341: in execute
          experiment.workon(self.task, n_workers=n_workers, max_trials=max_trials)
      src/orion/client/experiment.py:767: in workon
          for _ in range(n_workers)
      src/orion/executor/joblib_backend.py:32: in wait
          return joblib.Parallel(n_jobs=self.n_workers)(futures)
      .tox/py/lib/python3.7/site-packages/joblib/parallel.py:1056: in __call__
          self.retrieve()
      .tox/py/lib/python3.7/site-packages/joblib/parallel.py:935: in retrieve
          self._output.extend(job.get(timeout=self.timeout))
      /opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/multiprocessing/pool.py:657: in get
          raise self._value
      /opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/multiprocessing/pool.py:121: in worker
          result = (True, func(*args, **kwds))
      .tox/py/lib/python3.7/site-packages/joblib/_parallel_backends.py:595: in __call__
          return self.func(*args, **kwargs)
      .tox/py/lib/python3.7/site-packages/joblib/parallel.py:263: in __call__
          for func, args, kwargs in self.items]
      .tox/py/lib/python3.7/site-packages/joblib/parallel.py:263: in <listcomp>
          for func, args, kwargs in self.items]
      src/orion/client/experiment.py:781: in _optimize
          with self.suggest(pool_size=pool_size) as trial:
      src/orion/client/experiment.py:560: in suggest
          trial = reserve_trial(self._experiment, self._producer, pool_size)
      src/orion/client/experiment.py:54: in reserve_trial
          producer.produce(pool_size)
      src/orion/core/worker/producer.py:115: in produce
          self.algorithm.set_state(self.naive_algorithm.state_dict)
      src/orion/core/worker/primary_algo.py:47: in state_dict
          return self.algorithm.state_dict
      src/orion/algo/tpe.py:265: in state_dict
          _state_dict = super(TPE, self).state_dict
      src/orion/algo/base.py:132: in state_dict
          return {"_trials_info": copy.deepcopy(self._trials_info)}
      /opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/copy.py:150: in deepcopy
          y = copier(x, memo)
      _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
    
      x = {'0ef99dad51485ac9518b49c19b43f4ec': (Trial(experiment=2, status='completed', params=x[0]:0.5497,x[1]:2.397,x[2]:-4.29...params=x[0]:-0.8816,x[1]:2.087,x[2]:1.176), {'constraint': [], 'gradient': None, 'objective': 1187.240632192948}), ...}
      memo = {140575410255504: datetime.datetime(2021, 11, 11, 22, 57, 32, 364584), 140575419437584: datetime.datetime(2021, 11, 11....datetime(2021, 11, 11, 22, 57, 32, 177829), 140575419438016: datetime.datetime(2021, 11, 11, 22, 57, 32, 241378), ...}
      deepcopy = <function deepcopy at 0x7fdac6dfb680>
    
          def _deepcopy_dict(x, memo, deepcopy=deepcopy):
              y = {}
              memo[id(x)] = y
      >       for key, value in x.items():
      E       RuntimeError: dictionary changed size during iteration
    ```
    bouthilx committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    e519a06 View commit details
    Browse the repository at this point in the history
  2. Remove trailing TODO notes

    bouthilx committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    63eda39 View commit details
    Browse the repository at this point in the history