Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve utility of configuration specification #5148

Open
jmchilton opened this issue Dec 7, 2017 · 5 comments

Comments

@jmchilton
Copy link
Member

commented Dec 7, 2017

This issue a follow up to #5105 to improve what is there beyond a MVP.

  • Allow more annotations of Galaxy options in config_schema.yml and use when generating documentation.
    • Specify an optional minimum Galaxy version that implements the options.
    • Specify how common it would be to modify the options - so can build a version of the docs that hides un-common options.
    • Specify unsafe / beta options and automatically warn on them.
    • Specify paths and path defaults more intelligently.
  • Pre-process the options when loading into Galaxy to not need to duplicate type information like converting strings to integers when parsing ini options.
  • Implement a modality of Galaxy that prevents loading non-lint passing configurations.
  • Rework various other configuration files and allow them to be embedded in a hierarchal way into galaxy.yml.
    • job_conf.yml
    • per-IE options
    • datatypes
    • dependency resolvers (can already by yaml)
    • job metrics (can already by yaml)
    • containers (is already yaml)
@natefoo

This comment has been minimized.

Copy link
Member

commented Dec 4, 2018

John and I worked on a specification for a YAML job configuration and came up with the following. There's some incomplete stuff in there in regards to templating but we didn't want to lose this since a fair amount of thought went in to it.

job_config:
  runners:
    local:
      class: galaxy.jobs.runners.local:LocalJobRunner
      workers: 4
      default: true
  handling:
    default: handlers
    method:
      - uwsgi_mule_message
      - db_preassign
    processes:
      handler0:
        tags: 
          - handlers
      handler1:  # no config
  execution:
    default: local
    environments:
      # could also be a dict keyed on ID, list is used to support the templating idea
      - id: local
        runner: local
      - common:
          native_spec: '-cores $cores -mem $mem' 
          mem: 12
          runner: drmaa
        environemnts:
          - id: bigmem
            mem: 45
      # dict version
      local:
          runner: local
  tools:
    # one option for specifying mapping defaults, instead of in "handling" and "execution"
    defaults:
      environment: foo
      handler: local
    mappings:
      cat1:
        environment: bar
  tools:
    # alternative tools implementation
    cat1:
      environment: bar
@jmchilton

This comment has been minimized.

Copy link
Member Author

commented May 2, 2019

@natefoo

I think I want to make:

runners:
  local:
      load: 'galaxy.jobs.runners.local:LocalJobRunner'

into

running:
  workers: 4
  dynamic: {}
  runners:
    local:
      load: 'galaxy.jobs.runners.local:LocalJobRunner'

so it matches execution and we have a place to configure default worker count and dynamic parameters? (why continue to pretend the dynamic runner is a runner - right?). Do you recall if this was something we discussed and dismissed?

At the cost of consistency we could make either runners or running available at the top-level and just fill in defaults if running is absent but runners are not.

@natefoo

This comment has been minimized.

Copy link
Member

commented May 2, 2019

I don't recall discussing it - seems fine to me. I assume the dict under the dynamic key is for options to the dynamic runner? I use this currently to set the module path to my rules.

@jmchilton

This comment has been minimized.

Copy link
Member Author

commented May 3, 2019

@natefoo At first glance that is the only parameter, I wonder if just adding dynamic_rules_module inside running would be better than:

running:
  dynamic:
    rules_module: galaxy.rules

since there is only one parameter.

@natefoo

This comment has been minimized.

Copy link
Member

commented May 6, 2019

It could be, although it'd make things messy in the (admittedly unlikely) event that we ever want to add more params for the dynamic so-called runner.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.