Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

User can request managerial info, like trial string identifiers, from cli #69

Merged
merged 11 commits into from
May 10, 2018

Commits on May 9, 2018

  1. Trial: Add hash_name property and params_repr function

    - Also, explicitly fill `trial._id` with `Trial.NoID` during init.
    - Minor renaming from self._type to self.type for getters in
      `Trial.Value`.
    - Change representation of a `Trial` to better print out contained
      parameters.
    
    Why:
       I need to way to identify saved stuff in local disk from the
       execution of my script given a particular trial. This way I can have
       a reproducible deterministic string identifier that I intend to pass
       appropriately to a command line argument (if specified from ``orion
       hunt``) of my executable. This will disambiguate saved states,
       snapshots, or logs (i.e. from tensorboard), among different trials of
       the same executable.
    
       The actual implementation of all save/load is left to the executable
       itself, as well as configuring the absolute path where the stuff will
       be saved to. Most (responsible) ML programs I have personally encountered
       have a cli way to specify some naming for the experiment to run.
    tsirif committed May 9, 2018
    Configuration menu
    Copy the full SHA
    78f9942 View commit details
    Browse the repository at this point in the history
  2. Add properties hash_name and full_name to Trial

    - Replace name property
    - Planning to match all flavours of experiment naming
    tsirif committed May 9, 2018
    Configuration menu
    Copy the full SHA
    113cb49 View commit details
    Browse the repository at this point in the history
  3. Command args builder parses expression that substitute from trial

    Copy pasting from documentation of `SpaceBuilder.build_from`:
       This method is also responsible for parsing semantics which allow
       information from the `Experiment` or a `Trial` object to be passed
       to user's script. For example, a command line option like
       ``--name~trial.hash_name`` will be parsed to mean that a unique hash
       identifier of the trial that defines an execution shall be passed to
       the option ``--name``. Usage of these properties are not obligatory,
       but it helps integrating with solutions which help experiment
       reproducibility and resumability. See :attr:`EXPOSED_PROPERTIES` to
       check which properties are supported.
    tsirif committed May 9, 2018
    Configuration menu
    Copy the full SHA
    6607c89 View commit details
    Browse the repository at this point in the history
  4. Refactor SpaceBuilder cli parsing, support substitutions

     - `SpaceBuilder.userargs_tmpl` is now an OrderedDict.
     - Built command line args respect **the order** and **the way**
       in which they where parsed.
     - Less acrobatics regarding configuration file path.
     - Tackles the cases where ``~`` in a cli arg means home folder as
       in a shell path name.
     - Logs warning wherever implicit arguable decisions are taken.
     - TODO: First positional may not be a config file => it's a bad
       restriction.
     - TODO: Import parsing regex elements perhaps from a orion config file
    
    Copying from documentation of `SpaceBuilder._build_from_args`:
    
       Build templates from arguments found in the original cli.
    
       Rules for namespacing:
          1. Prefix ``'/'`` is given to parameter dimensions.
          2. Prefix ``'$'`` is given to substitutions from exposed properties.
          3. Prefix ``'_'`` is given to arguments which do not interact with.
             Suffix is a unique integer.
          4. ``'config'`` is given to user's script configuration file template
    
       User script configuration file argument is treated specially, trying to
       recognise a ``--config`` option or by checking the first positional
       argument, if not found elsewhere. Only one is allowed.
    
       .. note:: Currently positional arguments cannot define a parameter
          dimension. And I don't think they will; it wouldn't be good for
          your health down the line. Please be responsible and give a good
          descriptive name to your inputs.
    
      .. note:: Templates preserve given argument order.
    tsirif committed May 9, 2018
    Configuration menu
    Copy the full SHA
    a27cc0e View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    66a18aa View commit details
    Browse the repository at this point in the history
  6. Fix test: Remove expectation msg in native mongo connection error

    Reason: MongoDB version changes, message in exception changes
    tsirif committed May 9, 2018
    Configuration menu
    Copy the full SHA
    6ce1e14 View commit details
    Browse the repository at this point in the history
  7. Add exp.name as exported property

    I got greedy. Support it with a functional test.
    Remove also dead code from ``SpaceBuilder._build_from_args``.
    Add a test to show how it fails when it does.
    tsirif committed May 9, 2018
    Configuration menu
    Copy the full SHA
    6b77c8b View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    ed23ca3 View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    ed294ee View commit details
    Browse the repository at this point in the history
  10. Replace '/' with '.' in Trial.full_name

    Reason is that '/' means special thing in UNIX path names
    tsirif committed May 9, 2018
    Configuration menu
    Copy the full SHA
    fcb01f2 View commit details
    Browse the repository at this point in the history

Commits on May 10, 2018

  1. Fix flake8 and docstring

    tsirif committed May 10, 2018
    Configuration menu
    Copy the full SHA
    54d1090 View commit details
    Browse the repository at this point in the history