feat: Add token auth to GitLab/GitHub hosting providers #1761

tdido · 2022-07-08T11:13:33Z

Description

This PR adds ~~an optional "token" argument~~ the ability to get a token from environment variables to the gitlab() and github() code hosting provider. This allows for the usage of modules stored in private repositories.

QC

The PR contains a test case for the changes or the changes are already covered by an existing test case.
The documentation (docs/) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).

Closes #1301

tdido · 2022-07-18T14:12:31Z

Converting to draft as during some testing I realised the "conda_env" path doesn't contain the token e.g. when using a module in a private repo that contains a conda environment. Looking into it now.

@johanneskoester any comments would be appreciated.

tdido · 2022-07-18T14:20:56Z

Example error message:

Not a conda environment: /my/workflow/.test/https:/gitlab.com/api/v4/projects/myproject/repository/files/workflow%2Fenvs%2Fenvironment.yaml/raw?ref=main

So it's treating the environment URL as local. I guess the module import requires redefining the env in this case?

docs/snakefiles/modularization.rst

johanneskoester

Nice improvement! For the sake of code coverage, may I ask you to extend the corresponding testcase to also check the token parameter? You can just use the environment variable GITHUB_TOKEN for this, and simply still point to a public file.

Co-authored-by: Johannes Köster <johannes.koester@uni-due.de>

sonarcloud · 2022-07-19T11:31:57Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

tdido · 2022-07-29T08:52:35Z

@johanneskoester seems like conda env files are still fetched without the token. I've tested hardcoding it in the conda deployment file [1] and it works. Any pointers on the best way to make the token available there?

[1] https://github.com/tdido/snakemake/blob/435913d63cc42ac7b9e0cf438673e339aa25d055/snakemake/deployment/conda.py#L843

sonarcloud · 2022-10-20T13:45:51Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

sonarcloud · 2022-10-20T14:04:17Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

tdido · 2022-10-20T14:06:21Z

Hey @johanneskoester. I've had to change the approach since I couldn't find a way to get environment files to use the token with the previous approach. It now would seem like all cases are covered, although it implies adding a command line argument for the hosting provider token. Please let me know what you think.

johanneskoester · 2022-12-13T09:07:54Z

A cli arg for the token is not an option IMHO, because then the token will be inside of the bash history, which is a security risk. Can you explain what exactly fails when just fetching the token from os.environ?

tdido · 2022-12-13T14:44:26Z

Right, yes, that makes sense. The failure I was referring to was the one when adding the argument to the gitlab() function, where it doesn't pass the token to the conda environments it fetches.

I'll change the cli arg to os.environ instead, should be straightforward.

tdido · 2022-12-15T16:18:13Z

@johanneskoester I've switched to using an env variable for the token and it seems to be working fine. Please let me know whether the implementation looks OK.

tdido

Some requested changes not needed anymore with env vars approach, not sure how to mark as solved...

snakemake/workflow.py

snakemake/deployment/conda.py

johanneskoester · 2022-12-16T09:44:59Z

Can you please run black on your changes?

johanneskoester

Almost forgot: could you also add a sentence about the use of the two env vars to the respective section in the docs? (this one)

sonarcloud · 2022-12-16T11:06:52Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

🤖 I have created a release *beep* *boop* --- ## [7.20.0](v7.19.1...v7.20.0) (2023-01-18) ### Features * add tes token ([#1966](#1966)) ([59a8fa0](59a8fa0)) * Add token auth to GitLab/GitHub hosting providers ([#1761](#1761)) ([e03a3b4](e03a3b4)), closes [#1301](#1301) * allow for human friendly resource definitions (e.g. mem="5GB", runtime="1d") ([#1861](#1861)) ([24610ac](24610ac)) ### Bug Fixes * 🐛 - fix hyperlink ([#2046](#2046)) ([9519d31](9519d31)) * Catch missing error stream in Slurm executor ([#2063](#2063)) ([c21fc7e](c21fc7e)) * correctly parse empty values in config cli ([#2032](#2032)) ([1b0689d](1b0689d)) * Correctly parse UserDicts in executors ([#2016](#2016)) ([e3926fa](e3926fa)) * Fix handling of --jobs in no-exec state ([#2029](#2029)) ([e8e8222](e8e8222)) * make `--show-failed-logs` handle empty log files ([#2039](#2039)) ([683c6f2](683c6f2)), closes [#2023](#2023) * make python version check more robust ([#2058](#2058)) ([e685621](e685621)) * parsing error when last line is comment ([#2054](#2054)) ([a928dd4](a928dd4)) * prevent overriding of retries when set to 0 ([#2053](#2053)) ([a328f3e](a328f3e)) * propagate attempt count from group to subjobs ([#2052](#2052)) ([da3f1c0](da3f1c0)) * remove overflow from rulegraph div in report ([9a0aaa7](9a0aaa7)) * skip type checks of missing dir in touch mode ([#2051](#2051)) ([ae00c25](ae00c25)) * slurm default_resources quoting ([#2043](#2043)) ([47d3fc3](47d3fc3)) * Update list of python versions in classifiers ([#2020](#2020)) ([7a98100](7a98100)) * use short argument name for `--chdir` for compatibility with Slurm <=v17 ([#2040](#2040)) ([a9ed3ec](a9ed3ec)) ### Documentation * Fix typo in SLURM help text ([#2049](#2049)) ([79b7025](79b7025)) * mention XDG_CACHE_HOME ([#2057](#2057)) ([ec2ef45](ec2ef45)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

🤖 I have created a release *beep* *boop* --- ## [7.20.0](snakemake/snakemake@v7.19.1...v7.20.0) (2023-01-18) ### Features * add tes token ([snakemake#1966](snakemake#1966)) ([59a8fa0](snakemake@59a8fa0)) * Add token auth to GitLab/GitHub hosting providers ([snakemake#1761](snakemake#1761)) ([e03a3b4](snakemake@e03a3b4)), closes [snakemake#1301](snakemake#1301) * allow for human friendly resource definitions (e.g. mem="5GB", runtime="1d") ([snakemake#1861](snakemake#1861)) ([24610ac](snakemake@24610ac)) ### Bug Fixes * 🐛 - fix hyperlink ([snakemake#2046](snakemake#2046)) ([9519d31](snakemake@9519d31)) * Catch missing error stream in Slurm executor ([snakemake#2063](snakemake#2063)) ([c21fc7e](snakemake@c21fc7e)) * correctly parse empty values in config cli ([snakemake#2032](snakemake#2032)) ([1b0689d](snakemake@1b0689d)) * Correctly parse UserDicts in executors ([snakemake#2016](snakemake#2016)) ([e3926fa](snakemake@e3926fa)) * Fix handling of --jobs in no-exec state ([snakemake#2029](snakemake#2029)) ([e8e8222](snakemake@e8e8222)) * make `--show-failed-logs` handle empty log files ([snakemake#2039](snakemake#2039)) ([683c6f2](snakemake@683c6f2)), closes [snakemake#2023](snakemake#2023) * make python version check more robust ([snakemake#2058](snakemake#2058)) ([e685621](snakemake@e685621)) * parsing error when last line is comment ([snakemake#2054](snakemake#2054)) ([a928dd4](snakemake@a928dd4)) * prevent overriding of retries when set to 0 ([snakemake#2053](snakemake#2053)) ([a328f3e](snakemake@a328f3e)) * propagate attempt count from group to subjobs ([snakemake#2052](snakemake#2052)) ([da3f1c0](snakemake@da3f1c0)) * remove overflow from rulegraph div in report ([9a0aaa7](snakemake@9a0aaa7)) * skip type checks of missing dir in touch mode ([snakemake#2051](snakemake#2051)) ([ae00c25](snakemake@ae00c25)) * slurm default_resources quoting ([snakemake#2043](snakemake#2043)) ([47d3fc3](snakemake@47d3fc3)) * Update list of python versions in classifiers ([snakemake#2020](snakemake#2020)) ([7a98100](snakemake@7a98100)) * use short argument name for `--chdir` for compatibility with Slurm <=v17 ([snakemake#2040](snakemake#2040)) ([a9ed3ec](snakemake@a9ed3ec)) ### Documentation * Fix typo in SLURM help text ([snakemake#2049](snakemake#2049)) ([79b7025](snakemake@79b7025)) * mention XDG_CACHE_HOME ([snakemake#2057](snakemake#2057)) ([ec2ef45](snakemake@ec2ef45)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

tdido requested a review from johanneskoester as a code owner July 8, 2022 11:13

tdido changed the title ~~Add token authentication to GitLab hosting provider~~ [feat] Add token authentication to GitLab hosting provider Jul 8, 2022

tdido changed the title ~~[feat] Add token authentication to GitLab hosting provider~~ feat: Add token authentication to GitLab hosting provider Jul 8, 2022

feat: Add token auth to GitLab hosting provider

89538f0

tdido changed the title ~~feat: Add token authentication to GitLab hosting provider~~ feat: Add token auth to GitLab hosting provider Jul 8, 2022

tdido added 2 commits July 8, 2022 13:23

Make example token less real

5daecd0

Add support for GitHub

68681bc

tdido changed the title ~~feat: Add token auth to GitLab hosting provider~~ feat: Add token auth to GitLab/GitHub hosting providers Jul 8, 2022

Update docs with GitHub examples

ac169cb

tdido mentioned this pull request Jul 8, 2022

Use modules from private github repository #1301

Closed

tdido marked this pull request as draft July 18, 2022 14:11

johanneskoester requested changes Jul 19, 2022

View reviewed changes

docs/snakefiles/modularization.rst Outdated Show resolved Hide resolved

johanneskoester requested changes Jul 19, 2022

View reviewed changes

tdido and others added 2 commits July 19, 2022 12:36

Add message about not hardcoding tokens

db0a7f3

Co-authored-by: Johannes Köster <johannes.koester@uni-due.de>

Extend test to include token

435913d

tdido requested a review from johanneskoester July 29, 2022 11:12

tdido and others added 7 commits September 30, 2022 10:00

Merge branch 'main' into gitlab_auth

e897842

Apply token to envs in GitHub

9d07e0a

Apply token to envs in GitLab

bf11502

Use token from cmd line in includes

40285f0

Remove token argument from hosting provider functions

0f7dad2

Raise error when not using gitlab.org or github.com

a35c1f4

Merge branch 'snakemake:main' into gitlab_auth

7925f95

tdido added 2 commits October 20, 2022 15:50

Fix typo

b9c7693

Remove docs for function-based token args

f29f49e

Remove rogue print()

4466512

tdido marked this pull request as ready for review October 20, 2022 14:04

Merge branch 'main' into gitlab_auth

0205e27

tdido and others added 2 commits December 15, 2022 16:53

Merge branch 'snakemake:main' into gitlab_auth

1c78e49

Switch cli arg for env var

9e4a459

tdido commented Dec 15, 2022

View reviewed changes

johanneskoester requested changes Dec 15, 2022

View reviewed changes

snakemake/workflow.py Outdated Show resolved Hide resolved

snakemake/deployment/conda.py Outdated Show resolved Hide resolved

tdido added 2 commits December 16, 2022 10:39

Use specifiv env vars and remove redundant code

bdce3dc

Remove unused import

a9cbe22

johanneskoester approved these changes Dec 16, 2022

View reviewed changes

Restore empty line in docs

bbce34a

tdido requested a review from johanneskoester December 16, 2022 10:05

Lint

e5b2650

johanneskoester requested changes Dec 16, 2022

View reviewed changes

Add note to docs

034c590

tdido requested a review from johanneskoester December 16, 2022 11:07

johanneskoester approved these changes Dec 16, 2022

View reviewed changes

johanneskoester merged commit e03a3b4 into snakemake:main Dec 16, 2022

github-actions bot mentioned this pull request Dec 16, 2022

chore(main): release 7.20.0 #2018

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Add token auth to GitLab/GitHub hosting providers #1761

feat: Add token auth to GitLab/GitHub hosting providers #1761

tdido commented Jul 8, 2022 •

edited

tdido commented Jul 18, 2022

tdido commented Jul 18, 2022 •

edited

johanneskoester left a comment

sonarcloud bot commented Jul 19, 2022

tdido commented Jul 29, 2022

sonarcloud bot commented Oct 20, 2022

sonarcloud bot commented Oct 20, 2022

tdido commented Oct 20, 2022

johanneskoester commented Dec 13, 2022

tdido commented Dec 13, 2022

tdido commented Dec 15, 2022

tdido left a comment •

edited

johanneskoester commented Dec 16, 2022

johanneskoester left a comment

sonarcloud bot commented Dec 16, 2022

feat: Add token auth to GitLab/GitHub hosting providers #1761

feat: Add token auth to GitLab/GitHub hosting providers #1761

Conversation

tdido commented Jul 8, 2022 • edited

Description

QC

tdido commented Jul 18, 2022

tdido commented Jul 18, 2022 • edited

johanneskoester left a comment

Choose a reason for hiding this comment

sonarcloud bot commented Jul 19, 2022

tdido commented Jul 29, 2022

sonarcloud bot commented Oct 20, 2022

sonarcloud bot commented Oct 20, 2022

tdido commented Oct 20, 2022

johanneskoester commented Dec 13, 2022

tdido commented Dec 13, 2022

tdido commented Dec 15, 2022

tdido left a comment • edited

Choose a reason for hiding this comment

johanneskoester commented Dec 16, 2022

johanneskoester left a comment

Choose a reason for hiding this comment

sonarcloud bot commented Dec 16, 2022

tdido commented Jul 8, 2022 •

edited

tdido commented Jul 18, 2022 •

edited

tdido left a comment •

edited