Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix: Cache correctly identifies when it needs to be updated #987

Merged
merged 7 commits into from
Mar 30, 2023
Merged

fix: Cache correctly identifies when it needs to be updated #987

merged 7 commits into from
Mar 30, 2023

Conversation

adamrtalbot
Copy link
Contributor

@adamrtalbot adamrtalbot commented Mar 29, 2023 •
edited
Loading

When running on custom runners and the paths to the FASTQ would not be correct because it had not been updated. We identified that the data was being downloaded from the cache instead of creating a new data source if github.workspace had changed.

To fix, we use a hash of the github.workspace as a key value (plus some identifying features for the repository and release). When github.workspace value changes (such as when using a different base directory), it will download the data again and fix the CSV input file.

PR checklist

  • This comment contains a description of changes (with reason).
  • If you've fixed a bug or added code that should be tested, add tests!
  • If you've added a new tool - have you followed the pipeline conventions in the contribution docs- [ ] If necessary, also make a PR on the nf-core/rnaseq branch on the nf-core/test-datasets repository.
  • Make sure your code lints (nf-core lint).
  • Ensure the test suite passes (nextflow run . -profile test,docker --outdir <OUTDIR>).
  • Usage Documentation in docs/usage.md is updated.
  • Output Documentation in docs/output.md is updated.
  • CHANGELOG.md is updated.
  • README.md is updated (including new tool citations and authors/contributors).

@github-actions
Copy link

github-actions bot commented Mar 29, 2023 •
edited
Loading

nf-core lint overall result: Passed ✅ ⚠️

Posted for pipeline commit 1333a7a

+| ✅ 146 tests passed       |+
#| ❔   5 tests were ignored |#
!| ❗   5 tests had warnings |!

❗ Test warnings:

  • files_exist - File not found: .github/workflows/awstest.yml
  • files_exist - File not found: .github/workflows/awsfulltest.yml
  • nextflow_config - Config manifest.version should end in dev: '3.11.0'
  • readme - README did not have a Nextflow minimum version mentioned in Quick Start section.
  • pipeline_todos - TODO string in methods_description_template.yml: #Update the HTML below to your prefered methods description, e.g. add publication citation for this pipeline

❔ Tests ignored:

  • files_unchanged - File ignored due to lint config: assets/email_template.html
  • files_unchanged - File ignored due to lint config: assets/email_template.txt
  • files_unchanged - File ignored due to lint config: lib/NfcoreSchema.groovy
  • files_unchanged - File ignored due to lint config: lib/NfcoreTemplate.groovy
  • actions_awstest - 'awstest.yml' workflow not found: /home/runner/work/rnaseq/rnaseq/.github/workflows/awstest.yml

✅ Tests passed:

Run details

  • nf-core/tools version 2.7.2
  • Run at 2023-03-30 16:24:10

@adamrtalbot
Copy link
Contributor Author

@nf-core-bot fix linting

@adamrtalbot adamrtalbot changed the title [WIP] Use relative paths in input CSV files fix: Cache correctly identifies when it needs to be updated Mar 30, 2023
@maxulysse maxulysse mentioned this pull request Mar 30, 2023
Closed
maxulysse
maxulysse reviewed Mar 30, 2023
View reviewed changes
CHANGELOG.md Outdated Show resolved Hide resolved
@maxulysse
Copy link
Member

@nf-core-bot fix linting pretty please 🙏

@adamrtalbot
Copy link
Contributor Author

Proof it correctly restores the cache:
image

@drpatelh drpatelh merged commit 93d36d1 into dev Mar 30, 2023
@maxulysse
Copy link
Member

cc @emiller88 @ewels we had issue on the custom runner due to caching, here's how @adamrtalbot fixed everything 🚀

@maxulysse maxulysse mentioned this pull request Mar 31, 2023
Merged
9 tasks
@adamrtalbot adamrtalbot deleted the ci_relative_paths branch March 31, 2023 08:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@maxulysse maxulysse maxulysse approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@adamrtalbot @maxulysse @drpatelh @nf-core-bot