Github Actions

Overview of Github Actions

Production data depends on one bespoke workflow that pulls data in, processes it, and returns a zipped csv file. This workflow triggers the generic gh-pages deploy workflow that updates the data on the gh-pages branch.

flowchart TB

%% Nodes
    GMPD2("GMPD2")
    HP3("HP3")
    Shaw("Shaw")
    EID2("EID2 (2015)")

    RECON1("`
__Reconciliation 1__
• names reconciled to NCBI taxonomy
• temporal and sampling metadata standardized
• manual curation of unmatched names
`")

    style RECON1 text-align:left, fill:#efefef, stroke:#a0a0a0;
    style RECON2 text-align:left, fill:#efefef, stroke:#a0a0a0;

    CLOVER("CLOVER")

    GenBank("🔄 GenBank")

    PREDICT("📦 PREDICT")

    RECON2("`__Reconciliation 2__
• update of dynamic datasets (GenBank)
• names reconciled to NCBI and ICTV
• additional quality checks
• community sourcing for validation
`")


    subgraph virion["`__Virion flat files__`"]
        Virion["💾 Virion.csv.gz"]
        db_table["💾 db_table.csv"]
        tax_table["💾 tax_table.csv.gz"]
        ncbi_accession["💾 ncbi_accession.csv.gz"]
        direction TB
        db_table --"One to Many"--- Virion
        tax_table --"One to Many"--- Virion
        Virion --"One to One"---> ncbi_accession
    end

    style virion stroke:#a0a0a0, fill:#efefef;

    ZenodoDeposit("`
__Zenodo Deposit__
• Creates frictionless data package for Virion
• Adds descriptive and structural metadata
• Deposits a new version of Virion into Zenodo
`")

%% Edge connections between nodes
    GMPD2 --> RECON1;
    HP3 --> RECON1;
    Shaw --> RECON1;
    EID2 --> RECON1;
    RECON1 --> CLOVER;

    CLOVER --> RECON2;
    GenBank --> RECON2;
    PREDICT --> RECON2;

    RECON2 --> virion;

    virion --> ZenodoDeposit;

%% Individual node styling. Try the visual editor toolbar for easier styling!
    style GMPD2 color:#FFFFFF, fill:#AA00FF, stroke:#AA00FF
    style HP3 color:#FFFFFF, fill:#AA00FF, stroke:#AA00FF
    style Shaw color:#FFFFFF, fill:#AA00FF, stroke:#AA00FF
    style EID2 color:#FFFFFF, fill:#AA00FF, stroke:#AA00FF

    style CLOVER color:#FFFFFF, stroke:#00C853, fill:#00C853

    style GenBank color:#FFFFFF, stroke:#2962FF, fill:#2962FF
    style PREDICT color:#FFFFFF, stroke:#2962FF, fill:#2962FF

%% You can add notes with two "%" signs in a row!

Loading

Description of each gha file

main_targets.yml

jobs: targets

Sets up R and julia on ubuntu, installing necessary packages and dependencies. Runs R code and uses upload-artifact - docs for artifacts - to add csvs to the repo. Runs a gh-pages deploy step to update the gh-pages branch.

Using ACT to run github actions locally

Sometimes, it can take a very long time to run a particular github action because of issues around builds etc. Using ACT you can run the action on your machine before pushing things up to the github runners.

• Note that if you're using docker desktop, you may need to edit your config file from credsStore to credStore .

# if you have an apple silicon mac make sure you specify --container-architecture linux/amd64

# run the GenBank job (j) 
act -j 'targets'

# run the GenBank job (j) and reuse the containers (r) 
act -jr 'targets'