Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add config filter parameter to simulate_factors() #14

Merged
merged 8 commits into from
May 5, 2017

Commits on May 1, 2017

  1. New only_factor parameter for simulate_factors()

    Add a new optional `only_factor` parameter for the simulate_factors()
    method.
    
    The only_factor parameter is specified, only a single simulation will be
    run. The simulation that runs is for the factor at the index specified
    by the value of the only_factor argument.
    
    For example, if the snowbird command-line arguments look like this:
    
    	meta.cmd_line: sb -f host.queue_depth 4,16 -f fm.fim.xfer_rate_MBps 400,800 -f fm.ase.t_wait_rd 4,5,6 -s duration 10 ms --only-factor 4
    
    Then snowbird will pass in only_factor=4 as the argument to desmod and
    the meta.sim.index and meta.sim.workspace will be set to 4 and
    workspace/4 respectively.
    
    Additionally, meta.sim.special will look like this:
    
        meta.sim.special:
        - [host.queue_depth, 4]
        - [fm.fim.xfer_rate_MBps, 800]
        - [fm.ase.t_wait_rd, 5]
    
    That is, only a single simulation, for the factor indicated by
    meta.sim.special, was run.
    
    One use for this new feature is orchestrating the execution of multiple,
    per-factor, simulations on separate machines (e.g., in the cloud) with a
    single command (with multiple factors).
    bgmerrell committed May 1, 2017
    Configuration menu
    Copy the full SHA
    3142c63 View commit details
    Browse the repository at this point in the history
  2. Optionally sync workspaces to s3

    Some new config options have been added:
     - sim.workspace.s3_sync: When true, syncs any workspaces to s3
     - sim.workspace.s3_bucket: The s3 bucket to sync to
     - sim.workspace.s3_sync_prefix: A string to act as the root path in the
       s3 bucket.  By default, this is set the user's username.
    
    Here's an example of running snowbird with these new options:
    
    $ sb -s sim.workspace ~/locker/test -s sim.workspace.s3_sync_prefix
    bgmerrell -n test-rd-rnd -s sim.duration 10ms -s sim.workspace.s3_sync
    True -f host.queue_depth 4,16
    
    This would result in the following being synced to the default bucket in
    s3:
    
    /bgmerrell/test/0/config.yaml
    /bgmerrell/test/0/result.yaml
    /bgmerrell/test/0/sim.log
    /bgmerrell/test/1/config.yaml
    /bgmerrell/test/1/result.yaml
    /bgmerrell/test/1/sim.log
    
    The method to do the sync work is called when the _Workspace context
    manager is exiting.
    
    The uploading to s3 is done using the concurrent.features package.  This
    package isn't available in the Python 2 library, so the requirements.txt
    file has been updated to install the package for Python 2 only.
    bgmerrell committed May 1, 2017
    Configuration menu
    Copy the full SHA
    ab8ab48 View commit details
    Browse the repository at this point in the history
  3. Update unit tests

    bgmerrell committed May 1, 2017
    Configuration menu
    Copy the full SHA
    190ff12 View commit details
    Browse the repository at this point in the history
  4. flake8 fixes

    bgmerrell committed May 1, 2017
    Configuration menu
    Copy the full SHA
    3383cf5 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    006567d View commit details
    Browse the repository at this point in the history

Commits on May 3, 2017

  1. Updates based on review

    * only_factor -> config_filter
    * use setdefault() for some workspace sync config keys
    * default s3 sync prefix is now ""
    * don't importa boto3 or concurrent.futures until needed.
    * sim.workspace.s3_sync -> sim.sync.s3.enable (and similar for other s3
      sync-related keys)
    bgmerrell committed May 3, 2017
    Configuration menu
    Copy the full SHA
    49242a8 View commit details
    Browse the repository at this point in the history

Commits on May 5, 2017

  1. Remove code for s3 sync

    The sync code will not live in desmod.  We've decided to take a
    different approach.
    bgmerrell committed May 5, 2017
    Configuration menu
    Copy the full SHA
    a1a33b8 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    aa35322 View commit details
    Browse the repository at this point in the history