Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add cross-product matrix strategy #20

Open
mikeharder opened this Issue Oct 19, 2018 · 5 comments

Comments

Projects
None yet
3 participants
@mikeharder
Copy link

mikeharder commented Oct 19, 2018

By "cross-product matrix", I mean the scenario where you want to use the matrix strategy on a cross-product of multiple dimensions. For example, say you want to run tests on 2 OS (Linux, Windows) and 2 versions of Python (3.6, 3.7). Currently, you'd need to manually create a matrix with all the combinations:

strategy:
  matrix:
    Linux_Python36:
      VM_IMAGE: 'ubuntu-16.04'
      PYTHON_VERSION: '3.6'
    Linux_Python37:
      VM_IMAGE: 'ubuntu-16.04'
      PYTHON_VERSION: '3.7'
    Windows_Python36:
      VM_IMAGE: 'vs2017-win2016'
      PYTHON_VERSION: '3.6'
    Windows_Python37:
      VM_IMAGE: 'vs2017-win2016'
      PYTHON_VERSION: '3.7'

While this isn't too bad with a small number of dimensions and a small number of variables in each dimension, it can quickly become unmaintainable as the dimensions and variables grow. It would be nice if there was a way to express this more succinctly, something like:

strategy:
  cross-product:
    PYTHON_VERSION: [ '3.6', '3.7' ]
    VM_IMAGE: [ `ubuntu-16.04`, `vs2017-win2016` ]
@vtbassmatt

This comment has been minimized.

Copy link
Member

vtbassmatt commented Oct 19, 2018

The concept is good. A handful of issues to think on.

I need to predictably generate job names (in the current unrolled syntax, Windows_Python37 is the job name) for downstream dependencies and accessing output variables. I don't love "Job1" - "JobN" but that seems to be the most straightforward. We could define and document what order we generate them in.

I also need to handle exceptions as a subkey of cross-product. So the syntax might end up more like:

strategy:
  cross-product:
    vars:
      PYTHON_VERSION: [ '3.6', '3.7', '2.7' ]
      VM_IMAGE: [ 'ubuntu-16.04', 'vs2017-win2016' ]
    except:
    - PYTHON_VERSION: '2.7'  # skip Python 2.7 on Windows
      VM_IMAGE: 'vs2017-win2016'
@mikeharder

This comment has been minimized.

Copy link
Author

mikeharder commented Oct 19, 2018

Overall this looks great. In combination with #4, it should be perfect for our current requirements.

A few questions about the job name:

  1. Would it make sense to allow optionally specifying a format string for the job name (example "{0}_{1}"), to override the default?
  2. How would dependsOn work? With the current matrix strategy, it appears that dependsOn just references the base job name:
- job: 'Test'
  strategy:
    matrix:
      Python36:
        python.version: '3.6'
      Python37:
        python.version: '3.7'

- job: 'Publish'
  dependsOn: 'Test'

Would cross-product work the same, so another job can depend on the entire cross product with a single dependency?

@vtbassmatt

This comment has been minimized.

Copy link
Member

vtbassmatt commented Oct 22, 2018

Hmm, I need to think more about dependsOn. I thought you could "reach into" the matrix and depend on a single resulting job, but if not, then no reason to introduce that here. (In fact, this would likely be implemented as syntactic sugar on top of matrix, so we'd get the exact same level of support.)

Regarding a name: we have restrictions on what you can name jobs (most obvious one, no spaces) but wouldn't want to restrict values that way. I guess if we added some new functions - maybe make_job_name(string) - that you could manipulate the values with, that might cover it.

@mikeharder

This comment has been minimized.

Copy link
Author

mikeharder commented Oct 22, 2018

A default job name would be sufficient for our needs -- I was just throwing out the idea of a format string. I also agree this should be syntactic sugar on top of matrix to minimize the number of fundamental concepts.

@jbergstroem

This comment has been minimized.

Copy link

jbergstroem commented Mar 13, 2019

Shameless bump! This would be a very good quality-of-life feature.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.