dvc exp run unavailable in CML

[datacubeR]

Is there any way to run DVC exp run in CML?

My use case includes training several experiments in a Cloud or self hosted Server and publish in the PR the results of my work.

For now when trying to do this using the setup action I get an unexpected error.

I understand that commit hides experiment results and I read that CML is not intended for experimentation but that is not quite clear in official documentation.

[casperdcl]

I think this is an upstream bug - #7531.

What error are you getting? Are you using setup-cml? In which case you should also use setup-dvc.

Or are you using the CML Docker images?

[casperdcl]

commit hides experiment results and I read that CML is not intended for experimentation

Sorry I don't quite follow. How do commits hide results? CML is intended for provisioning & reporting experiments.

[datacubeR]

commit hides experiment results and I read that CML is not intended for experimentation

Sorry I don't quite follow. How do commits hide results? CML is intended for provisioning & reporting experiments.

When I say experiments I refer to using dvc exp. Once you commit your changes dvc exp show no longer shows experiments table.

In my specific case I wanted to try the following:

name: experiments
on: [push]
jobs:
  train-model:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: iterative/setup-cml@v1
      - uses: actions/setup-python@v2
        with:
          python-version: '3.x'
      - name: Train model
        env:
          repo_token: ${{ secrets.GITHUB_TOKEN }}
        run: |
          pip3 install -r requirements.txt
          #bash exp_file.sh
          dvc exp run -S train.C=0.005
          dvc exp run -S train.C=100

          echo "## Resultados del Experimento" >> report.md
          dvc exp show --only-changed --drop 'assets|src' --no-pager --md >> report.md
          
          echo "## Parallel Plot\n" >> report.md
          dvc exp show --only-changed --drop 'assets|src' --pcp --sort-by test_recall >> report.md

          cml send-comment report.md

In this particular case I wanted to report the results of 2 experiments with dvc exp run. I've tried also using a bash file (that you can see commented out) with the following:

dvc exp run --queue -S train.C=5
dvc exp run --queue -S train.C=30
dvc exp run --queue -S train.C=60
dvc exp run --queue -S train.C=120

dvc exp run --run-all

All of the experiments resulted in something like:

So my question would be: Is it allowed to run something like what I'm proposing?
Can I run my experiments and report it in the PR?

I can't remember when I read that Experimentation was meant to be done locally and only dvc repro on the CI Pipeline. Can you clarify that please?

Thanks in advance,

Alfonso

[dacbd]

My brain has stopped working for the day already, but I don't think there isn't any reason you shouldn't be able to do this. Without having tested anything like this I would blindly expand your checkout step to see if that fixes it:

- uses: actions/checkout@v3
  with:
     fetch-depth: 0

https://github.com/actions/checkout#fetch-all-history-for-all-tags-and-branches

[casperdcl]

(Transferred to DVC because it looks like a config/setup issue unrelated to CML)