Usage in GitLab

[dieterc]

Dear,
Can this uploader also been used inside GitLab?
Kind regards

[atrisovic]

Hi there!
I think yes with some minor changes.
The communication with Dataverse is happening here: https://github.com/IQSS/dataverse-uploader/blob/master/dataverse.py and this script can be slightly modified to run from GitLab.

[PennyHow]

We have just been experimenting with this and have a script that works with the Gitlab CI/CD.

# Gitlab-to-Dataverse uploader 
dataverse
image: python:3.9

workflow:
    rules:
        - if: $CI_PIPELINE_SOURCE == "push"
        
dataverse_uploader:
  variables:
    DATAVERSE_TOKEN: "$DATAVERSE_API_TOKEN"
    DATAVERSE_SERVER: "https://example.dataverse.org"
    DATAVERSE_DATASET_DOI: "doi:10.12345/EXAMPLE"
    GITHUB_DIR: "./"
    DELETE: "false"
    PUBLISH: "false"

  script:
    - apt-get update && apt-get install -y git python3-pip
    - git clone https://github.com/IQSS/dataverse-uploader.git dataverse-uploader
    - cd dataverse-uploader
    - pip install -r requirements.txt
    - echo "$DATAVERSE_TOKEN" "$DATAVERSE_SERVER" "doi:$DATAVERSE_DATASET_DOI" "$CI_PROJECT_URL" "$DELETE" "$PUBLISH"
    - python dataverse.py "$DATAVERSE_TOKEN" "$DATAVERSE_SERVER" "doi:$DATAVERSE_DATASET_DOI" "$CI_PROJECT_URL" -r "$DELETE" -p "$PUBLISH"

The $DATAVERSE_API_TOKEN is the API key, which can be set in the repo's CI/CD variables. The scheduling of this pipeline can either be set in the repo's schedules options, or with the workflow rules flag

I don't know if this should be an example made readily available in something similar to the repo readme or a Gitlab equivalent of this place? Might be useful!

[atrisovic]

OK this is awesome! Yeah, if there is a GitLab marketplace that would be the right place, or in any case we should add it to Dataverse documentation (https://guides.dataverse.org/en/latest/admin/integrations.html) :-)

[dieterc]

@PennyHow That's interesting to hear! Did you change something to the dataverse.py script to make it work?
Our job successfully runs but without doing anything, it seems that the files are not seen by the script.
Do you know which version you have installed of GitLab? We use GitLab Community Edition 15.6.2. I know that the Community Edition has less features.

[PennyHow]

I didn't have to change anything in dataverse.py and we are using GitLab Community Edition 15.6.0.

We have some issues with the transference of large files, but at least it sees the files.

[dieterc]

@PennyHow Which kind of GitLab runner are you using? A shared, specific or group runner?
If I print(list(walk(path))) right after setting the path variable I get an empty array in GitLab while in GitHub I get the files of the repository