diff --git a/discourse-topic-check.sh b/discourse-topic-check.sh index a68d721d..7d477da3 100644 --- a/discourse-topic-check.sh +++ b/discourse-topic-check.sh @@ -31,10 +31,14 @@ fi for file in $(git diff --name-only "$baseBranch".. | grep "\.rst$" | grep -Fvxf $DOES_NOT_REQUIRE_DISCOURSE_TOPIC) do - if ! containsDiscourseTopic "$file" + # The file may be in the base branch but not the current branch. Skip in that case. + if [[ -f "$file" ]] then - echo "${file} does not have a discourse topic" - RETURN_VALUE=1 + if ! containsDiscourseTopic "$file" + then + echo "${file} does not have a discourse topic" + RETURN_VALUE=1 + fi fi done diff --git a/docs/_static/language-support.html b/docs/_static/language-support.html index 0d261f3c..91ce6727 100644 --- a/docs/_static/language-support.html +++ b/docs/_static/language-support.html @@ -103,6 +103,13 @@ Yes Yes + + Open Data + Yes [11] + Yes[12] + No + No + Dockstore CLI @@ -192,7 +199,7 @@ No - Notifications + Notifications Yes Yes No diff --git a/docs/advanced-topics/docker-alternatives.rst b/docs/advanced-topics/docker-alternatives.rst index f060826c..4ad93fea 100644 --- a/docs/advanced-topics/docker-alternatives.rst +++ b/docs/advanced-topics/docker-alternatives.rst @@ -122,7 +122,7 @@ Podman Podman is a daemon-less alternative of Docker that allows end users to run Docker/Open Container Initiative (OCI) containers without root privileges. Podman can be installed following the instructions -`here `__. +`here `__. cwltool ~~~~~~~ diff --git a/docs/advanced-topics/dockstore-cli/advanced-features.rst b/docs/advanced-topics/dockstore-cli/advanced-features.rst index a6073025..b396fb85 100644 --- a/docs/advanced-topics/dockstore-cli/advanced-features.rst +++ b/docs/advanced-topics/dockstore-cli/advanced-features.rst @@ -520,7 +520,7 @@ The table below summarizes some of the WES endpoints available: +-----------+------------------------------------------------------+-----------+ -.. note:: WES SUPPORT IS IN BETA AT THIS TIME. RESULTS MAY BE UNPREDICTABLE. +.. note:: WES SUPPORT IS IN PREVIEW MODE AT THIS TIME. RESULTS MAY BE UNPREDICTABLE. .. discourse:: :topic_identifier: 1274 diff --git a/docs/advanced-topics/dockstore-cli/dockstore-cli-faq.rst b/docs/advanced-topics/dockstore-cli/dockstore-cli-faq.rst index d8b28893..e9d0734b 100644 --- a/docs/advanced-topics/dockstore-cli/dockstore-cli-faq.rst +++ b/docs/advanced-topics/dockstore-cli/dockstore-cli-faq.rst @@ -1,3 +1,5 @@ +.. _topCLIFAQ: + Dockstore CLI FAQ ================= @@ -17,6 +19,8 @@ FTP, S3, and GCS. As of Release 1.12, the Dockstore CLI has support for running client `__. Please see `file provisioning plugins `__ for more information on these two file transfer sources. +:ref:`(back to top) ` + .. _how-do-i-use-the-dockstore-cli-on-a-mac: How do I use the Dockstore CLI on a Mac? @@ -34,6 +38,8 @@ can change what it allocates using the Docker for Mac GUI under The default allocation can cause workflows or tools to fail without informing the user with a memory or resource related error message. If you find that your workflow or tool is behaving differently on a Mac compared to a similarly resourced Ubuntu environment, you can try increasing the resources allocated to Docker on the Mac to resolve the discrepancy. If you are using WDL, see also our notes on :doc:`local Cromwell configuration files``. +:ref:`(back to top) ` + Why am I getting "Mount denied" errors when launching workflows on Mac? ----------------------------------------------------------------------- Docker behaves a bit differently on a `Mac `__ than on a typical Ubuntu machine. On a Mac, by default, the only shared volumes Docker can access are /Users, /Volumes, /tmp, and /private. This is problematic because many workflow executors, such as `cwltool`, uses your TMPDIR (the :ref:`dict environment variable`) to set up volumes with Docker. Since `$TMDIR` on a Mac can default to a subdirectory of /var, which Docker usually cannot access, you may run into an error. When using cwltool, the error will look something like this: @@ -66,6 +72,7 @@ Depending on the permissions available to your machine, you may be able to give $TMPDIR might be set to a subfolder of /private/var/folders. If you are still having issues, try adding /var/folders to Docker's list of accessible directories instead. +:ref:`(back to top) ` How do I launch tools/workflows without internet access on compute nodes? ------------------------------------------------------------------------- @@ -82,6 +89,8 @@ follow these steps: The Dockstore CLI will automatically load all Docker images in the directory specified prior to a ``launch --local-entry`` command. +:ref:`(back to top) ` + .. _return-code-wdl: How do I find the return code for a WDL task? @@ -89,8 +98,9 @@ How do I find the return code for a WDL task? The numeric return code for a WDL task will be in that task's execution folder. It is a single file named `rc` with no extension. Generally speaking, a 0 is a success, and anything else is a failure. -Let's say you are running [this vcf-to-gds file conversion workflow](https://dockstore.org/workflows/github.com/DataBiosphere/analysis_pipeline_WDL/vcf-to-gds-wdl:v7.1.1), which runs the check-gds task as a scattered task on an array of three files. Cromwell will refer to each instance of that scattered task as a "shard" and will name them starting with 0. If you notice that shard 0 seems to have failed, look for `/cromwell-executions/[workflow ID]/call-check_gds/shard-0/execution/rc` keeping in mind that the workflow ID will usually be a long mix of numbers, letters, and dashes such as 18a85cc0-aa59-4749-b1b9-e2580ed5e557. +Let's say you are running [this vcf-to-gds file conversion workflow](https://dockstore.org/workflows/github.com/DataBiosphere/analysis_pipeline_WDL/vcf-to-gds-wdl:v7.1.1), which runs the check-gds task as a scattered task on an array of three files. Cromwell will refer to each instance of that scattered task as a "shard" and will name them starting with 0. If you notice that shard 0 seems to have failed, look for `/cromwell-executions/[workflow ID]/call-check_gds/shard-0/execution/rc` keeping in mind that the workflow ID will usually be a long mix of numbers, letters, and dashes such as 18a85cc0-aa59-4749-b1b9-e2580ed5e557. +:ref:`(back to top) ` .. _cromwell-docker-lockup: @@ -103,7 +113,9 @@ If a Docker lockup happens, you will notice in-progress WDL tasks do not progres The other issue we often see is some instances of scattered tasks getting `sigkilled `__ by the operating system. You will know when this happens because the `rc` (return code) file will read 137. If it reads anything except 137, then you can assume that it wasn't actually a resource management error and look in stderr or stdout for the true culprit. For more on return codes, see :ref:`this FAQ ` entry. -To prevent these issues from happening, we recommend setting up your Cromwell configuration file to limit how many scattered tasks run at once, and then setting up the Dockstore CLI to make use of that Cromwell configuration file. :doc:`A step-by-step tutorial is available here. ` +To prevent these issues from happening, we recommend setting up your Cromwell configuration file to limit how many scattered tasks run at once, and then setting up the Dockstore CLI to make use of that Cromwell configuration file. :doc:`A step-by-step tutorial is available here. ` + +:ref:`(back to top) ` The CLI is failing with Java 8 or 11 ------------------------------------ @@ -121,7 +133,9 @@ The Dockstore CLI as of 1.7.0 is compiled and tested using Java 11 due to the Java 8 EOL. You will need to upgrade from Java 8 to use CLI versions betweenn 1.7 and 1.13. The Dockstore CLI as of 1.14.0 is compiled and tested using Java 17 due -to the approaching Java 11 EOL. You will need to update to Java 17 to use the CLI version 1.14.0+. +to the approaching Java 11 EOL. You will need to update to Java 17 to use the CLI version 1.14.0+. + +:ref:`(back to top) ` .. discourse:: :topic_identifier: 6481 diff --git a/docs/advanced-topics/dockstore-tools-overhaul.rst b/docs/advanced-topics/dockstore-tools-overhaul.rst index 08ab6a34..be4e3714 100644 --- a/docs/advanced-topics/dockstore-tools-overhaul.rst +++ b/docs/advanced-topics/dockstore-tools-overhaul.rst @@ -13,6 +13,8 @@ At a Glance .. include:: /advanced-topics/legacy/table--legtool-vs-ghatool.rst +.. _legacy_tools: + Dockstore Tools Prior to 1.12 (Legacy Tools) -------------------------------------------- diff --git a/docs/advanced-topics/metrics.rst b/docs/advanced-topics/metrics.rst new file mode 100644 index 00000000..bccb231f --- /dev/null +++ b/docs/advanced-topics/metrics.rst @@ -0,0 +1,154 @@ +Workflow Metrics +================ + +.. note:: Dockstore Workflow Metrics is currently in preview mode. Also note that the screenshots below were taken on our staging site. + +What are workflow metrics? +------------------------------------ +Workflow metrics are metrics of workflow executions on a platform. Metrics include things like the resources used during the execution, and how often workflow executions succeeded or failed. + +Users are able to execute workflows on various platforms using Dockstore's Launch With feature, shown on the right side in the screenshot below. + +.. figure:: /assets/images/docs/submit-metrics/workflow-launch-with.png + :alt: Launch With Partners + +Platforms are able to submit metrics of workflows executed on their platform to Dockstore and we aggregate the metrics and display them in the UI to the users in the :ref:`Metrics tab `. + +Why would I want to submit workflow metrics? +-------------------------------------------- +As a platform owner, workflow metrics indicate to others that your platform is compatible with many workflows on Dockstore. Workflow metrics provide valuable information to users, including information about the resources and time needed to run the workflow. It helps the user determine if the workflow is high quality and is very likely to work for others. + +.. _How to view metrics: + +How do I view workflow metrics? +------------------------------- + +To view workflow metrics for a workflow on Dockstore, you will need to use the ``metrics`` feature flag. + +To use the ``metrics`` feature flag, append ``metrics`` to the Dockstore URL as a query parameter. You only need to do this once, unless you refresh/close your browser. + +For example, if you're on the https://dockstore.org page, append ``metrics`` such that it looks like this: https://dockstore.org?metrics. + +If you're on a page that already contains query parameters, indicated by the presence of a question mark, append ``metrics`` to the URL using an ampersand. For example, if you're on the https://dockstore.org/workflows/github.com/gatk-workflows/seq-format-conversion/BAM-to-Unmapped-BAM:3.0.0?tab=info page, append ``metrics`` such that it looks like https://dockstore.org/workflows/github.com/gatk-workflows/seq-format-conversion/BAM-to-Unmapped-BAM:3.0.0?tab=info&metrics. + +After applying the ``metrics`` feature flag, a Metrics tab will appear when viewing a workflow on Dockstore. Click on the Metrics tab and you will see workflow metrics if they are available. + +.. figure:: /assets/images/docs/submit-metrics/metrics-tab.png + :alt: Metrics tab + +How do I submit workflow metrics? +--------------------------------- + +.. note:: Submitting workflow metrics is only available for admins, curators, and platform partners. If you're a platform owner and you don't have a platform partner user, please contact us via our `GitHub `_ issues or open a helpdesk ticket on `Discourse `_ and we will help you get set up. + +Go to https://dockstore.org/api/static/swagger-ui/index.html#/extendedGA4GH/executionMetricsPost. This is the endpoint used to submit metrics to Dockstore. Click the "Try it out” button. + +First, fill out the query parameters. This is where you'll input information about which workflow was executed and what platform it was executed on. + +For the ``id`` parameter, provide the TRS ID of the workflow that you want to submit metrics for. For example, the `dockstore-tool-bamstats `__ workflow has the TRS ID ``#workflow/github.com/dockstore/dockstore-tool-bamstats/wdl`` as shown in the “Info” tab with the label “TRS". + +For the ``version_id`` parameter, provide the name of the workflow version that was executed. It can be any version listed in the "Versions" tab of the workflow. For example, `dockstore-tool-bamstats `__ has the following versions currently: 1.25-9 and develop. It is recommended to submit execution metrics for versions that are unlikely to change, like tags. + +Select the ``platform`` that the workflow was executed on from the available values. + +For the ``description``, you may provide an optional description about the metrics you're submitting. + +.. figure:: /assets/images/docs/submit-metrics/query-parameters.png + :alt: Query parameters for submitting metrics + +In the request body, provide the workflow execution metrics that you want to submit. Click on Schema to view the schema of the request body. + +.. figure:: /assets/images/docs/submit-metrics/request-body-schema.png + :alt: Request body schema for submitting metrics + +You can provide a individual executions through ``runExecutions`` and ``validationExecutions``. + +A run execution is an execution that runs the workflow. The only required metric for run executions is ``executionStatus``. If you want to provide additional metrics that are not defined in the schema, use the ``additionalProperties`` key to provide your metric. + +.. figure:: /assets/images/docs/submit-metrics/run-executions-schema.png + :alt: Schema for run executions + +A validation execution is an execution of a validator tool, like miniwdl, on the workflow. The following are required for a validation execution: + +- ``validatorTool`` +- ``validatorToolVersion`` +- ``isValid`` +- ``dateExecuted`` + +If you want to provide additional metrics that are not defined in the schema, use the ``additionalProperties`` key to provide your metric. + +.. figure:: /assets/images/docs/submit-metrics/validation-executions-schema.png + :alt: Schema for validation executions + +Instead of providing individual executions, you may also provide aggregated metrics that summarize data for multiple executions through ``aggregatedExecutions``. There are no required fields, but you must provide at least one metric. If you want to provide aggregated metrics that are not defined in the schema, use the ``additionalAggregatedMetrics`` key to provide your metric. + +.. figure:: /assets/images/docs/submit-metrics/aggregated-metrics-schema.png + :alt: Schema for aggregated metrics + +Lastly, provide your Dockstore token using the lock icon at the top right of the endpoint. + +Below is an example of what a request for submitting individual execution metrics looks like. The request is for a workflow that was executed on Terra. The request body submits one run execution that was successful and took 30 seconds to execute, and one validation execution of miniwdl version 1.9.1 which validated the workflow successfully. + +.. figure:: /assets/images/docs/submit-metrics/individual-executions-example.png + :alt: Example request for submitting individual run executions and validation executions + + +The curl command results in something like: + +.. code:: bash + + curl -X 'POST' \ + 'https://dockstore.org/api/api/ga4gh/v2/extended/%23workflow%2Fgithub.com%2Fdockstore%2Fdockstore-tool-bamstats%2Fwdl/versions/1.25-9/executions?platform=TERRA' \ + -H 'accept: */*' \ + -H 'Authorization: Bearer iamafakebearertoken' \ + -H 'Content-Type: application/json' \ + -d '{ + "runExecutions": [ + { + "executionStatus": "SUCCESSFUL", + "executionTime": "PT30S" + } + ], + "validationExecutions": [ + { + "validatorTool": "miniwdl", + "validatorToolVersion": "1.9.1", + "isValid": true, + "dateExecuted": "2023-03-31T15:06:49.888745366Z" + } + ] + }' + +If it was submitted successfully, you should receive a ``204`` response code. + +Below is an example of what a request for submitting aggregated execution metrics looks like. The request body submits an aggregated execution status metric, indicating that the workflow was successfully executed 5 times, and it failed twice, once because it was run time invalid and once because it was semantically invalid. + +.. figure:: /assets/images/docs/submit-metrics/aggregated-executions-example.png + :alt: Example request for submitting aggregated metrics + +The curl command results in something like: + +.. code:: bash + + curl -X 'POST' \ + 'https://dockstore.org/api/api/ga4gh/v2/extended/%23workflow%2Fgithub.com%2Fdockstore%2Fdockstore-tool-bamstats%2Fwdl/versions/1.25-9/executions?platform=TERRA' \ + -H 'accept: */*' \ + -H 'Authorization: Bearer iamafakebearertoken' \ + -H 'Content-Type: application/json' \ + -d '{ + "aggregatedExecutions": [ + { + "executionStatusCount": { + "count": { + "SUCCESSFUL": 5, + "FAILED_RUNTIME_INVALID": 1, + "FAILED_SEMANTIC_INVALID": 1 + }, + "numberOfSuccessfulExecutions": 5, + "numberOfFailedExecutions": 2 + } + } + ] + }' + +If it was submitted successfully, you should receive a ``204`` response code. diff --git a/docs/advanced-topics/organizations-and-collections.rst b/docs/advanced-topics/organizations-and-collections.rst index 0902f72e..07cd30af 100644 --- a/docs/advanced-topics/organizations-and-collections.rst +++ b/docs/advanced-topics/organizations-and-collections.rst @@ -20,11 +20,30 @@ Creating an organization To create an organization request, go to the `organizations `__ page and select ``Create Organization Request``. Any user can request to create an -organization by filling out the following form. For now, the request +organization by filling out the following form. The request must be approved by a Dockstore curator in order to be public. Until it is approved, you are still able to edit it, add collections, add members, etc. +To create an organization request, you must have at least 2 external accounts +linked to your Dockstore account. Linked accounts help the Dockstore curators +verify that a legitimate user is creating the request. + +When you click the ``Create Organization Request`` button, you are prompted with +a dialog that shows the number of external accounts you have linked. If you have +at least 2 accounts linked, you can create an organization request. If +you have fewer than two accounts linked, you cannot create an +organization request. In both cases, the dialog has a ``Link Accounts`` button +that takes you to the Accounts page, where you can link additional accounts. + +.. figure:: /assets/images/docs/UserVerification.png + :alt: User Verification + + User Verification + +If you have at least 2 accounts linked, clicking the ``Next`` button brings up the +Create Organization Request dialog. + .. figure:: /assets/images/docs/CreateOrganizationRequest.png :alt: Create Organization Request diff --git a/docs/advanced-topics/verification.rst b/docs/advanced-topics/verification.rst index 04cd14be..98e9178e 100644 --- a/docs/advanced-topics/verification.rst +++ b/docs/advanced-topics/verification.rst @@ -1,6 +1,8 @@ Verification ============ +.. important:: Verification will eventually be replaced by :doc:`Workflow Metrics `. Workflow Metrics achieve the same goal as verification by tracking the platforms that the workflow was successfully run on. Workflow Metrics are preferred because it is a more scalable way of tracking that a workflow was successfully run on a platform. + What is a verified tool/workflow? --------------------------------- A verified tool/workflow indicates that it was successfully run and verified by either: @@ -109,7 +111,7 @@ Verification Process verified. The path of the test parameter file is relative to the primary descriptor. This path can be found using the `files endpoint`_ or by viewing the files tab of a tool/workflow such as: - https://dockstore.org/containers/quay.io/briandoconnor/dockstore-tool-md5sum:1.0.4?tab=files + https://dockstore.org/workflows/github.com/DataBiosphere/topmed-workflows/UM_variant_caller_wdl:1.32.0?tab=files and then further selecting the Test Parameter Files tab and view the right-most “File” dropdown. This relative path must be a test parameter file, providing a descriptor will not work. diff --git a/docs/advanced-topics/wes/wes-wdl-agc-tutorial.rst b/docs/advanced-topics/wes/wes-wdl-agc-tutorial.rst index fdddbb01..e205e926 100644 --- a/docs/advanced-topics/wes/wes-wdl-agc-tutorial.rst +++ b/docs/advanced-topics/wes/wes-wdl-agc-tutorial.rst @@ -38,8 +38,9 @@ The following WDL workflow tutorials will cover: "workflowInputs": } - 2. AWS AGC references descriptor files by URLs. The Dockstore CLI references the primary descriptor in an execution request as a GA4GH Tool Registry Service (`TRS `_) URL. - The AGC infrastructure will only be able to access the file referenced by the TRS URL if the workflow is published (i.e. public) on Dockstore. + 2. AWS AGC references descriptor files by URLs. The Dockstore CLI references the primary descriptor in an execution request as a GA4GH Tool Registry Service (`TRS `_) URL. The AGC infrastructure will only be able to access the file referenced by the TRS URL if the workflow is published (i.e. public) on Dockstore. + + 3. On AWS AGC, the test parameter file for a WDL workflow running in Cromwell must reference other files using S3 URLs. Other types of URLs, including `https:`, are not supported. Configuring AGC and the Dockstore CLI ---------------------------------------- diff --git a/docs/assets/images/docs/UserVerification.png b/docs/assets/images/docs/UserVerification.png new file mode 100644 index 00000000..a092aa90 Binary files /dev/null and b/docs/assets/images/docs/UserVerification.png differ diff --git a/docs/assets/images/docs/add-tool-button.png b/docs/assets/images/docs/add-tool-button.png index dccdde2a..3b3bd780 100644 Binary files a/docs/assets/images/docs/add-tool-button.png and b/docs/assets/images/docs/add-tool-button.png differ diff --git a/docs/assets/images/docs/github-app-logs-button.png b/docs/assets/images/docs/github-app-logs-button.png index 147d3bd4..9b6265bf 100644 Binary files a/docs/assets/images/docs/github-app-logs-button.png and b/docs/assets/images/docs/github-app-logs-button.png differ diff --git a/docs/assets/images/docs/github-repo-settings.png b/docs/assets/images/docs/github-repo-settings.png index f66613cb..18e8bb19 100644 Binary files a/docs/assets/images/docs/github-repo-settings.png and b/docs/assets/images/docs/github-repo-settings.png differ diff --git a/docs/assets/images/docs/google-colab-authorize-with-github.png b/docs/assets/images/docs/google-colab-authorize-with-github.png new file mode 100644 index 00000000..0f580efd Binary files /dev/null and b/docs/assets/images/docs/google-colab-authorize-with-github.png differ diff --git a/docs/assets/images/docs/google-colab-open-notebook-github.png b/docs/assets/images/docs/google-colab-open-notebook-github.png new file mode 100644 index 00000000..26cbf032 Binary files /dev/null and b/docs/assets/images/docs/google-colab-open-notebook-github.png differ diff --git a/docs/assets/images/docs/google-colab-save-on-github-dialog.png b/docs/assets/images/docs/google-colab-save-on-github-dialog.png new file mode 100644 index 00000000..009c52e4 Binary files /dev/null and b/docs/assets/images/docs/google-colab-save-on-github-dialog.png differ diff --git a/docs/assets/images/docs/google-colab-save-on-github.png b/docs/assets/images/docs/google-colab-save-on-github.png new file mode 100644 index 00000000..bd4261ef Binary files /dev/null and b/docs/assets/images/docs/google-colab-save-on-github.png differ diff --git a/docs/assets/images/docs/launch-with-google-colab.png b/docs/assets/images/docs/launch-with-google-colab.png new file mode 100644 index 00000000..ffee6e69 Binary files /dev/null and b/docs/assets/images/docs/launch-with-google-colab.png differ diff --git a/docs/assets/images/docs/list-published-notebooks.png b/docs/assets/images/docs/list-published-notebooks.png new file mode 100644 index 00000000..cbd2a165 Binary files /dev/null and b/docs/assets/images/docs/list-published-notebooks.png differ diff --git a/docs/assets/images/docs/manage-gh-app-installation.png b/docs/assets/images/docs/manage-gh-app-installation.png index 7b57d3d3..f17d4664 100644 Binary files a/docs/assets/images/docs/manage-gh-app-installation.png and b/docs/assets/images/docs/manage-gh-app-installation.png differ diff --git a/docs/assets/images/docs/my-dashboard-sidebar.png b/docs/assets/images/docs/my-dashboard-sidebar.png new file mode 100644 index 00000000..095ab5ac Binary files /dev/null and b/docs/assets/images/docs/my-dashboard-sidebar.png differ diff --git a/docs/assets/images/docs/my-services-filled.png b/docs/assets/images/docs/my-services-filled.png index e34831e0..8f7e729c 100644 Binary files a/docs/assets/images/docs/my-services-filled.png and b/docs/assets/images/docs/my-services-filled.png differ diff --git a/docs/assets/images/docs/my-tools.png b/docs/assets/images/docs/my-tools.png deleted file mode 100644 index 15b9790c..00000000 Binary files a/docs/assets/images/docs/my-tools.png and /dev/null differ diff --git a/docs/assets/images/docs/notebooks-public-page.png b/docs/assets/images/docs/notebooks-public-page.png new file mode 100644 index 00000000..2d701186 Binary files /dev/null and b/docs/assets/images/docs/notebooks-public-page.png differ diff --git a/docs/assets/images/docs/notebooks-warning-banner.png b/docs/assets/images/docs/notebooks-warning-banner.png new file mode 100644 index 00000000..ab14b23c Binary files /dev/null and b/docs/assets/images/docs/notebooks-warning-banner.png differ diff --git a/docs/assets/images/docs/register-tool-github-apps.png b/docs/assets/images/docs/register-tool-github-apps.png index 2a314505..b960c75d 100644 Binary files a/docs/assets/images/docs/register-tool-github-apps.png and b/docs/assets/images/docs/register-tool-github-apps.png differ diff --git a/docs/assets/images/docs/submit-metrics/aggregated-executions-example.png b/docs/assets/images/docs/submit-metrics/aggregated-executions-example.png new file mode 100644 index 00000000..b7cb0cb7 Binary files /dev/null and b/docs/assets/images/docs/submit-metrics/aggregated-executions-example.png differ diff --git a/docs/assets/images/docs/submit-metrics/aggregated-metrics-schema.png b/docs/assets/images/docs/submit-metrics/aggregated-metrics-schema.png new file mode 100644 index 00000000..5c565b65 Binary files /dev/null and b/docs/assets/images/docs/submit-metrics/aggregated-metrics-schema.png differ diff --git a/docs/assets/images/docs/submit-metrics/individual-executions-example.png b/docs/assets/images/docs/submit-metrics/individual-executions-example.png new file mode 100644 index 00000000..73b22d2d Binary files /dev/null and b/docs/assets/images/docs/submit-metrics/individual-executions-example.png differ diff --git a/docs/assets/images/docs/submit-metrics/metrics-tab.png b/docs/assets/images/docs/submit-metrics/metrics-tab.png new file mode 100644 index 00000000..2a6990c0 Binary files /dev/null and b/docs/assets/images/docs/submit-metrics/metrics-tab.png differ diff --git a/docs/assets/images/docs/submit-metrics/query-parameters.png b/docs/assets/images/docs/submit-metrics/query-parameters.png new file mode 100644 index 00000000..078c0fed Binary files /dev/null and b/docs/assets/images/docs/submit-metrics/query-parameters.png differ diff --git a/docs/assets/images/docs/submit-metrics/request-body-schema.png b/docs/assets/images/docs/submit-metrics/request-body-schema.png new file mode 100644 index 00000000..e48d92db Binary files /dev/null and b/docs/assets/images/docs/submit-metrics/request-body-schema.png differ diff --git a/docs/assets/images/docs/submit-metrics/run-executions-schema.png b/docs/assets/images/docs/submit-metrics/run-executions-schema.png new file mode 100644 index 00000000..2f6c9791 Binary files /dev/null and b/docs/assets/images/docs/submit-metrics/run-executions-schema.png differ diff --git a/docs/assets/images/docs/submit-metrics/validation-executions-schema.png b/docs/assets/images/docs/submit-metrics/validation-executions-schema.png new file mode 100644 index 00000000..0db8f3a9 Binary files /dev/null and b/docs/assets/images/docs/submit-metrics/validation-executions-schema.png differ diff --git a/docs/assets/images/docs/submit-metrics/workflow-launch-with.png b/docs/assets/images/docs/submit-metrics/workflow-launch-with.png new file mode 100644 index 00000000..88af579b Binary files /dev/null and b/docs/assets/images/docs/submit-metrics/workflow-launch-with.png differ diff --git a/docs/assets/templates/notebooks/example-1-name-default-format-language.yml b/docs/assets/templates/notebooks/example-1-name-default-format-language.yml new file mode 100644 index 00000000..55f263b2 --- /dev/null +++ b/docs/assets/templates/notebooks/example-1-name-default-format-language.yml @@ -0,0 +1,5 @@ +version: 1.2 +notebooks: + - name: simple + path: /notebook.ipynb + publish: true diff --git a/docs/assets/templates/notebooks/notebooks.rst b/docs/assets/templates/notebooks/notebooks.rst new file mode 100644 index 00000000..76cc9c6e --- /dev/null +++ b/docs/assets/templates/notebooks/notebooks.rst @@ -0,0 +1,22 @@ +.dockstore.yml for Notebooks Templates (version 1.2) +==================================================== +.. note:: Support for notebooks is in preview mode. + +Two templates are provided here, one for you to quickly copy-paste into your repository, and one with a complete explanation of the possible fields and values you can use. + +Simple template +--------------- +.. include:: /assets/templates/notebooks/template-small.dockstore.yml + :code: + +Full template with explanation of all available fields +------------------------------------------------------ +.. include:: /assets/templates/notebooks/template-big.dockstore.yml + :code: + +See Also +-------- + +* :doc:`.dockstore.yml templates for registering tools ` +* :doc:`.dockstore.yml templates for registering workflows ` +* :doc:`Other documentation regarding the GitHub App ` diff --git a/docs/assets/templates/notebooks/template-big.dockstore.yml b/docs/assets/templates/notebooks/template-big.dockstore.yml new file mode 100644 index 00000000..cd92d129 --- /dev/null +++ b/docs/assets/templates/notebooks/template-big.dockstore.yml @@ -0,0 +1,66 @@ +# The first line refers to the version 1.2 of the .dockstore.yml schema +version: 1.2 + +# An array of notebooks. Each element corresponds to a notebook on Dockstore. +notebooks: + + # The optional notebook name for a notebook, which may only consist of alphanumerics + # and internal underscores and hyphens, but no spaces or other characters. Names may not exceed 256 characters. + # If using a .dockstore.yml with multiple notebooks, this field is required + # to uniquely identify notebooks in the repository. + # + # It should be noted that having the name come first is an arbitrary decision. + # You could use subclass instead, for instance. Provided arrays are not broken + # up, the order of fields within a .dockstore.yml is not important. + - name: + + # The format used for the notebook. This cannot be changed once the notebook is registered. + # Currently, only Jupyter is supported. + # If not provided, the format will default to Jupyter. + format: + + # The programming language of the code contained in the notebook. This cannot be changed once the notebook is registered. + # If not provided, the language will default to Python. + language: + + # The absolute path to the notebook file in the Git repository. + # - For Jupyter notebooks, the notebook is a .ipynb file. + path: + + # Notebook-wide setting that will affect ALL branches/tags; only set this as needed in a main branch. + # Set to true to publish an unpublished notebook, or false to unpublish a published notebook. + # Omitting the publish setting leaves the publish-state unchanged (recommended for all non-primary branches). + publish: + + # An optional array of absolute paths to other files in the Git repository that should be read and included with the notebook. + otherFiles: + + # An optional path to a notebook-specific readme in the Git repository. If not provided, Dockstore will show + # the readme.md present at the root of the Git repository if it is present. + # If you have multiple notebooks in a single Git repository, it is recommend to give each one a readme. + readMePath: + + # An optional array of authorship information. + # Note that if orcid is present, then all other fields will be ignored, as information will be taken from orcid. + # If orcid is not present, make sure to at a minimum include the name field for each author. + authors: + - orcid: + - name: + email: + role: + affiliation: + + # A boolean that will change the default version to be displayed on Dockstore. Default: False. + # A value of true will automatically display the latest tag updated as default. + # A value of false will retain the default version that has been specified via the Dockstore UI. + latestTagAsDefault: + + # The optional filters section allow specifying sets of Git branches and tags to include for the notebook. + # If no filters are given, all branches and tags are included. + # Branches and tags are arrays of pattern-strings. + # Pattern-strings use Unix-style Glob syntax by default (Ex: `develop`, `mynotebook/**`) + # https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/nio/file/FileSystem.html#getPathMatcher(java.lang.String) + # or RegEx when the string is surrounded by / (Ex: `/develop/`, `/mynotebook\/.*/`). + filters: + branches: + tags: diff --git a/docs/assets/templates/notebooks/template-small.dockstore.yml b/docs/assets/templates/notebooks/template-small.dockstore.yml new file mode 100644 index 00000000..ae3ea994 --- /dev/null +++ b/docs/assets/templates/notebooks/template-small.dockstore.yml @@ -0,0 +1,14 @@ +version: 1.2 +notebooks: + - format: + language: + path: + kernel: + otherFiles: + testParameterFiles: + readMePath: + authors: + - name: + email: + # Publish must be set to true in order for you to view your notebook on Dockstore in preview mode + publish: true diff --git a/docs/assets/templates/services/services.rst b/docs/assets/templates/services/services.rst index 2e874e18..8df1ef14 100644 --- a/docs/assets/templates/services/services.rst +++ b/docs/assets/templates/services/services.rst @@ -1,6 +1,6 @@ .dockstore.yml for Services Templates (version 1.2) =================================================== -.. note:: Support for services is in beta. +.. note:: Support for services is in preview mode. Two templates are provided here, one for you to quickly copy-paste into your repository, and one with a complete explanation of the possible fields and values you can use. diff --git a/docs/assets/templates/services/template-big.dockstore.yml b/docs/assets/templates/services/template-big.dockstore.yml index 4113be85..0528b860 100644 --- a/docs/assets/templates/services/template-big.dockstore.yml +++ b/docs/assets/templates/services/template-big.dockstore.yml @@ -12,11 +12,11 @@ service: # they will be retrieved from their ORCID record. If included, they will be ignored. # If an author's ORCID iD is not provided, include at least the name of the author. authors: + - orcid: - name: email: role: affiliation: - - orcid: # An optional, but recommended description of the service description: diff --git a/docs/assets/templates/template.rst b/docs/assets/templates/template.rst index e06b8cc9..b5b2c330 100644 --- a/docs/assets/templates/template.rst +++ b/docs/assets/templates/template.rst @@ -4,16 +4,18 @@ Templates * :doc:`.dockstore.yml templates for registering services ` * :doc:`.dockstore.yml templates for registering tools ` * :doc:`.dockstore.yml templates for registering workflows ` +* :doc:`.dockstore.yml templates for registering notebooks ` Not sure if you are working with a workflow, a tool, or a service? :doc:`Check out our introduction on the main differences between them. ` Tips and tricks --------------- -* This form of registration is only available to workflows, tools, and services that are hosted on GitHub +* This form of registration is only available to workflows, tools, services, and notebooks that are hosted on GitHub * Make sure your file is saved as ``.dockstore.yml``, not ``dockstore.yml`` or ``.dockstore.yaml`` * Put the .dockstore.yml file in the top of your repo or inside ``.github/`` -* The first line of a .dockstore.yml file references the version of .dockstore.yml syntax being used, not the version/tag of the workflow/tool/service it describes -* You can use a single .dockstore.yml file to register multiple tools and workflows, provided they are all in the same repo as the .dockstore.yml file +* If you have multiple workflows/tools/services/notebooks in a single GitHub repository, each one needs a unique ``name``, and it's a good idea (but not strictly required) to give each one a unique ``readMePath`` too. +* The first line of a .dockstore.yml file references the version of .dockstore.yml syntax being used, not the version/tag of the workflow/tool/service/notebook it describes +* You can use a single .dockstore.yml file to register multiple tools, workflows, services, and notebooks, provided they are all in the same repo as the .dockstore.yml file * A **workflow** registered with via a .dockstore.yml file is not fundamentally different than a workflow registered in another method, other than the fact that the .dockstore.yml version will be kept up-to-date automatically -- but a .dockstore.yml registered **tool** is different from tools registered in other methods (:ref:`see this table for more information `) Examples of the filters field @@ -44,6 +46,7 @@ Examples of the filters field .dockstore.yml templates for registering tools .dockstore.yml templates for registering workflows .dockstore.yml templates for registering services + .dockstore.yml templates for registering notebooks .. discourse:: :topic_identifier: 6490 diff --git a/docs/assets/templates/tools/template-big.dockstore.yml b/docs/assets/templates/tools/template-big.dockstore.yml index 25b8e7c1..72189be4 100644 --- a/docs/assets/templates/tools/template-big.dockstore.yml +++ b/docs/assets/templates/tools/template-big.dockstore.yml @@ -39,8 +39,8 @@ tools: # Note that if orcid is present, then all other fields will be ignored, as information will be taken from orcid. # If orcid is not present, make sure to at a minimum include the name field for each author. authors: + - orcid: - name: - orcid: email: role: affiliation: diff --git a/docs/assets/templates/workflows/example-3-multiworkflow-multiauthor.dockstore.yml b/docs/assets/templates/workflows/example-3-multiworkflow-multiauthor.dockstore.yml index 684d2766..234b911d 100644 --- a/docs/assets/templates/workflows/example-3-multiworkflow-multiauthor.dockstore.yml +++ b/docs/assets/templates/workflows/example-3-multiworkflow-multiauthor.dockstore.yml @@ -2,6 +2,7 @@ version: 1.2 workflows: - subclass: WDL primaryDescriptorPath: /assoc-aggregate/assoc-aggregate.wdl + readMePath: /assoc-aggregate/readme.md testParameterFiles: - /assoc-aggregate/assoc-aggregate-terra-allele.json - /assoc-aggregate/assoc-aggregate-terra-position.json @@ -11,6 +12,7 @@ workflows: - orcid: 0000-0003-4896-1858 - subclass: WDL primaryDescriptorPath: /pc-air/pc-air.wdl + readMePath: /pc-air/README.md testParameterFiles: - /pc-air/pc-air-local.json - /pc-air/pc-air-terra.json diff --git a/docs/assets/templates/workflows/template-big.dockstore.yml b/docs/assets/templates/workflows/template-big.dockstore.yml index 8725639f..68b77833 100644 --- a/docs/assets/templates/workflows/template-big.dockstore.yml +++ b/docs/assets/templates/workflows/template-big.dockstore.yml @@ -37,12 +37,17 @@ workflows: # - /null-model/null-model-binary.json testParameterFiles: + # An optional path to a workflow-specific readme in the Git repository. If not provided, Dockstore will show + # the readme.md present at the root of the Git repository if it is present. + # If you have multiple workflows in a single Git repository, it is recommend to give each one a readme. + readMePath: + # An optional array of authorship information. # Note that if orcid is present, then all other fields will be ignored, as information will be taken from orcid. # If orcid is not present, make sure to at a minimum include the name field for each author. authors: + - orcid: - name: - orcid: email: role: affiliation: diff --git a/docs/assets/templates/workflows/template-small.dockstore.yml b/docs/assets/templates/workflows/template-small.dockstore.yml index d591b548..c1c959fe 100644 --- a/docs/assets/templates/workflows/template-small.dockstore.yml +++ b/docs/assets/templates/workflows/template-small.dockstore.yml @@ -3,6 +3,7 @@ workflows: - subclass: primaryDescriptorPath: testParameterFiles: + readMePath: authors: - name: email: diff --git a/docs/assets/templates/workflows/workflows.rst b/docs/assets/templates/workflows/workflows.rst index 71b13c8c..3dba3b88 100644 --- a/docs/assets/templates/workflows/workflows.rst +++ b/docs/assets/templates/workflows/workflows.rst @@ -10,11 +10,11 @@ Simple generic template for a workflow .. include:: /assets/templates/workflows/template-small.dockstore.yml :code: -Always use :ref:`absolute paths ` to specify the :ref:`primary descriptor ` and :ref:`test parameter ` files. +Always use :ref:`absolute paths ` to specify the :ref:`primary descriptor `, :ref:`test parameter `, and readMePath files. Filled-out example of a single workflow without a name ------------------------------------------------------ -In this example, the workflow author is identified with an orcid. When an orcid is specified, there is no need to specify an author's name and email as that information will be pulled from the orcid. There are also three test parameter files given for the workflow. +In this example, the workflow author is identified with an ORCID iD. When an ORCID iD is specified, there is no need to specify an author's name and email as that information will be pulled from the ORCID API. There are also three test parameter files given for the workflow. Since no readMePath is specified, Dockstore will show the top-level readme (if one is present), eg, ``/readme.md`` .. include:: /assets/templates/workflows/example-1-noname.yml :code: @@ -28,9 +28,9 @@ This example is identical to the one above, but the workflow in question now is Filled-out example of multiple workflows in the same repository --------------------------------------------------------------- -First, you will notice that we swapped the order of the ``name`` and ``author`` fields for association-aggregate-wdl compared to the examples above, to demonstrate that the order is arbitrary. Next, we added a new section for pc-air-wdl, which has a different author and its own test parameter files and descriptor file. +First, you will notice that we swapped the order of the ``name`` and ``author`` fields for assoc-aggregate-wdl compared to the examples above, to demonstrate that the order is arbitrary. Next, we added a new section for pc-air-wdl, which has a different author and its own test parameter files and descriptor file. We have also added the optional ``readMePath`` value to these workflows so that each entry gets its own workflow-specific readme. -This .dockstore.yml will result in the creation of two entries on Dockstore -- one for association-aggregate-wdl, and one for pc-air-wdl. +This .dockstore.yml will result in the creation of two entries on Dockstore -- one for assoc-aggregate-wdl, and one for pc-air-wdl. assoc-aggregate-wdl's entry will show the readme located at ``/assoc-aggreate/readme.md``, while pc-air-wdl's entry will show the readme located at ``/pc-air/README.md``. Although ``readMePath`` is optional, if we did not add it to these entries, both entries would instead show the same top-level readme.md (or README.md) of the git repo. .. include:: /assets/templates/workflows/example-3-multiworkflow-multiauthor.dockstore.yml :code: diff --git a/docs/conf.py b/docs/conf.py index 5f997e60..f5e6f8fa 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -60,7 +60,9 @@ 'launch-with/galaxy-launch-with.html', 'launch-with/nextflow-tower-launch-with.html', 'launch-with/terra-launch-with.html', 'https://doi.org/*', # Confusing, linkcheck and curl report expired cert, but cert date is correct in browser - 'https://support.orcid.org/hc/en-us/articles/360006971593-Do-you-have-more-than-one-account' + 'https://support.orcid.org/hc/en-us/articles/360006971593-Do-you-have-more-than-one-account', + # elwazi cert expired, remove when renewed + 'https://elwazi.org/' ] diff --git a/docs/end-user-topics/language-support.rst b/docs/end-user-topics/language-support.rst index 2476c90c..f955a615 100644 --- a/docs/end-user-topics/language-support.rst +++ b/docs/end-user-topics/language-support.rst @@ -39,6 +39,10 @@ Limitations ` for limitations. [10] Use the Dockstore CLI optional parameter --wdl-output-target which allows you to specify a remote path to provision output files to ex: s3://oicr.temp/testing-launcher/ +[11] For workflows and :ref:`GitHub App Tools `, not for :ref:`Legacy Tools `. + +[12] For workflows only, not tools. + .. _converting-file-path-based-imports-to-public-http-s-based-imports-for-wdl: diff --git a/docs/faq.rst b/docs/faq.rst index 329e58c9..6fca2b71 100644 --- a/docs/faq.rst +++ b/docs/faq.rst @@ -1,3 +1,5 @@ +.. _topFAQ: + FAQ === @@ -22,6 +24,8 @@ achieved success with much lower system requirements. However, launching tools will have higher system requirements dependent on the specific tool. Consult a tool's README or CWL/WDL description when in doubt. +:ref:`(back to top) ` + .. _what-is-a-verified-tool-or-workflow: @@ -32,6 +36,8 @@ A verified tool/workflow means that at least one version has been verified to be See :doc:`/advanced-topics/verification` for full details on this feature. +:ref:`(back to top) ` + What is a default version of a tool or workflow? ------------------------------------------------ @@ -64,14 +70,32 @@ not limited to): would be equivalent to ``$ dockstore tool launch --entry quay.io/cancercollaboratory/dockstore-tool-bedgraph-bigwig --json Dockstore.json`` -4. The docker pull command in the tools search reflects the defaultversion +4. The docker pull command in the tools search reflects the default version +:ref:`(back to top) ` + +.. _what-is-an-open-tool-or-workflow: + +What is an Open Data tool or workflow version? +---------------------------------------------- + +On Dockstore, an Open Data tool or workflow version includes everything needed to run the version, without any +additional data access. This usually means there is a test parameter file whose file input parameter values +only refer to open data -- data that is not restricted access nor costs any money to download. + +Note that tool and workflow versions that do not require any file input parameters are also considered Open Data. + +You can find Open Data workflows and tools by selecting the Open Data facet on the `search `_ page. +When viewing an individual tool or workflow, you can tell which versions are Open Data on the Versions tab via the Open column. + +:ref:`(back to top) ` How should I register my work in Dockstore? ------------------------------------------- .. include:: /getting-started/how-to-register-work.rst - + +:ref:`(back to top) ` How do I send private messages to administrators or report security vulnerabilities? ------------------------------------------------------------------------------------ @@ -91,6 +115,7 @@ The following steps can be taken to create a helpdesk ticket (also shown `here < Entering 5 topics and viewing 30 posts over a minimum of 10 minutes will raise your privileges. \ You will be notified of any privilege changes to your account via the mailbox. +:ref:`(back to top) ` How do I cite Dockstore? ------------------------ @@ -102,6 +127,8 @@ For citing the actual code, we recommend looking at our Zenodo entry. You will find a variety of citation styles and ways to export it at |DOI|. +:ref:`(back to top) ` + .. _faq-header-github-integration: Integration with GitHub @@ -120,6 +147,7 @@ only be associated with one account at a time. You will need to link with a different account for each login method or delete your account if you want to assign them to a new Dockstore account. +:ref:`(back to top) ` What happens if I rename my GitHub repository? @@ -139,6 +167,8 @@ Please note the GitHub warning: If you create a new repository under your account in the future, do not reuse the original name of the renamed repository. If you do, redirects to the renamed repository will break. +:ref:`(back to top) ` + .. _faq-header-permissions: @@ -157,11 +187,13 @@ the "Discover Existing Dockstore Workflows" button in the "My Workflows" menu so If the workflow was added by manually registering it, click Refresh Organization. For participants of the :doc:`limited sharing -beta `, you can enter the email +preview `, you can enter the email addresses of the users you wish to share with to give them permissions to your workflow. This is only available for hosted workflows and users with Google accounts linked to Terra. +:ref:`(back to top) ` + Why are my workflows from an organization I belong to not visible? ------------------------------------------------------------------ @@ -179,6 +211,8 @@ itself. GitHub provides a `tutorial `__ for approving third party apps access to your organization. +:ref:`(back to top) ` + Why do I get an error when I try to request a DOI? -------------------------------------------------- @@ -204,6 +238,8 @@ The workflow version DOI can be found on the Versions tab. .. figure:: /assets/images/docs/workflow-version-doi.png :alt: Workflow version DOI +:ref:`(back to top) ` + .. _faq-header-other: Other @@ -218,6 +254,8 @@ example `_ If you have the right permissions, you can delete some and then refresh a tool on Dockstore to clean-up. +:ref:`(back to top) ` + How do I get more space inside my CWL tool running in a container? ------------------------------------------------------------------ @@ -273,6 +311,8 @@ Also be aware that some tools will use space from your root filesystem. For example, Docker's storage driver and data volumes will by default install to and use space on your root filesystem. +:ref:`(back to top) ` + Do you have tips on creating Dockerfiles? ----------------------------------------- @@ -286,6 +326,8 @@ Do you have tips on creating Dockerfiles? will interfere with this - try to keep your Docker images small +:ref:`(back to top) ` + Do you have tips on creating CWL files? --------------------------------------- @@ -320,6 +362,7 @@ Additionally: your container. Make sure your host running Docker has sufficient scratch space for processing your genomics data. +:ref:`(back to top) ` .. _why-would-i-want-to-add-a-specific-version-of-a-workflow-to-a-collection: @@ -334,6 +377,8 @@ Note that the version of a workflow can be especially important when working wit In summary: you can pin either a specific version of a workflow or a workflow in general depending on what relationship you wish to express. We recommend explaining further for others in the accompanying Markdown description. +:ref:`(back to top) ` + Any last tips on using Dockstore? --------------------------------- @@ -347,6 +392,8 @@ Any last tips on using Dockstore? - related to the two above, you can use non-standard file paths if you customize your registrations in the Version tab of Dockstore +:ref:`(back to top) ` + What happens if I link to a different ORCID account? ---------------------------------------------------- @@ -354,6 +401,8 @@ What happens if I link to a different ORCID account? If you run into this situation, please use the `Help Desk` link in the https://dockstore.org footer to contact the Dockstore team. +:ref:`(back to top) ` + .. |DOI| image:: https://zenodo.org/badge/DOI/10.5281/zenodo.321679.svg :target: https://zenodo.org/record/321679 diff --git a/docs/getting-started/dockstore-tools.rst b/docs/getting-started/dockstore-tools.rst index a74fc78a..fe901dac 100644 --- a/docs/getting-started/dockstore-tools.rst +++ b/docs/getting-started/dockstore-tools.rst @@ -36,7 +36,7 @@ keeping Dockstore synced with GitHub automatically. You can learn more :doc:`in .. include:: /getting-started/github-apps/snippet--installation.rst -Once GitHub app is installed on a repository or organization, you will need to add a .dockstore.yml file to +Once the GitHub app is installed on a repository in an organization or your personal account, you will need to add a .dockstore.yml file to the root directory of a branch of the repository that contains your tool. This file contains information like tool path, test parameter file, tool name, etc. When a push is made or a tag is created on GitHub with a .dockstore.yml, Dockstore will add that branch to the corresponding tool on Dockstore. If the diff --git a/docs/getting-started/dockstore-workflows.rst b/docs/getting-started/dockstore-workflows.rst index 34b37d98..71250081 100644 --- a/docs/getting-started/dockstore-workflows.rst +++ b/docs/getting-started/dockstore-workflows.rst @@ -41,7 +41,7 @@ with GitHub automatically. .. include:: /getting-started/github-apps/snippet--installation.rst -Once you've installed our GitHub app on a repository or organization, you'll need to add a dockstore.yml file to +Once you've installed our GitHub app on a repository in an organization or your personal account, you'll need to add a dockstore.yml file to the root directory of a branch of the repository that contains your workflow. This file contains information like workflow path, test parameter file, workflow name, etc. When a push is made on GitHub to a branch with a .dockstore.yml, Dockstore will add that branch to the corresponding workflow on Dockstore. If the diff --git a/docs/getting-started/getting-started-with-notebooks.rst b/docs/getting-started/getting-started-with-notebooks.rst new file mode 100644 index 00000000..37745991 --- /dev/null +++ b/docs/getting-started/getting-started-with-notebooks.rst @@ -0,0 +1,162 @@ +Notebooks +========= + +.. contents:: + :local: + :depth: 2 + +.. note:: Dockstore Notebooks is currently in preview mode. Also note that the screenshots below were taken on our staging site. + +Tutorial Goals +-------------- + +- Get familiarized with the concept of notebooks on Dockstore +- Learn the basics of creating a notebook using a .dockstore.yml file +- Learn how to register a notebook +- Publish your notebook to Dockstore + +Notebooks are documents that can include code, text, images, and other media resources. They can be executed in notebook environments like Google Colab and JupyterLab. +This document will outline what is needed to register, update, and publish a notebook onto Dockstore. Please keep in mind that notebook functionality is currently in preview mode. + +Overview +-------- + +In this tutorial, you will register a notebook using the Dockstore GitHub App, which automatically syncs your notebook from GitHub to Dockstore. More information about the Dockstore GitHub App can be found :doc:`here `. + +Dockstore Notebooks takes advantage of Google Colab's integration with GitHub. The integration allows you to develop your notebook in Google Colab and push your changes to GitHub from Google Colab. With the Dockstore GitHub App installed on your repository, any edits saved to your notebook from Google Colab will automatically be synced to Dockstore. + +Setting up the Google Colab and GitHub integration +-------------------------------------------------- + +To get started, you will need a Google Colab account and a GitHub account. + +In your Google Colab account, link your Google Colab account to your GitHub account. Click on the Settings gear icon in the upper right corner, then select GitHub in the dialog box. Click on Authorise with GitHub and authorize Google Colab to access your GitHub account. + +.. figure:: /assets/images/docs/google-colab-authorize-with-github.png + :width: 60 % + +Next, we will set up a notebook on Google Colab so that it's also stored in one of your GitHub repositories. If you already have a notebook in Google Colab that's saved to a GitHub repository, you may proceed to :ref:`create the .dockstore.yml file for your repository`. + +We will cover two scenarios: + +1. :ref:`You don't have a notebook in GitHub`. You may or may not have a notebook in Google Colab. +2. :ref:`You have a notebook in GitHub, but it's not in Google Colab` + +.. _no-notebook-in-github: + +You don't have a notebook in GitHub +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If you don't have a notebook in GitHub *nor* Google Colab, navigate to Google Colab and create a new notebook by clicking on File > New notebook. Edit your new notebook if you like. + +If you already have a notebook in Google Colab, proceed to the next step. + +Click on File > Save a copy in GitHub to save your notebook to a GitHub repository. You may want to create a new repository for your notebook if you don't already have a repository that you want to save the notebook to. + +.. figure:: /assets/images/docs/google-colab-save-on-github.png + :width: 35 % + +Configure where you want to save the notebook to on GitHub then click OK. Google Colab will push a commit to your repository with the notebook. + +.. figure:: /assets/images/docs/google-colab-save-on-github-dialog.png + :width: 60 % + +.. _no-notebook-in-google-colab: + +You have a notebook in GitHub, but it's not in Google Colab +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Click on File > Open notebook then select the GitHub tab in the dialog box. Select the notebook that you want to import from GitHub. This will open the notebook in Google Colab. + +.. figure:: /assets/images/docs/google-colab-open-notebook-github.png + :width: 60 % + +.. _create-dockstore-yml: + +Create Your .dockstore.yml File +------------------------------- + +The first step is to create a file named ``.dockstore.yml`` which acts as a configuration file used to describe your notebook. This file should be created in the GitHub repository containing your notebook. We will cover an example .dockstore.yml written for a simple hello world Jupyter notebook written in Python. You can also view template .dockstore.yml files :doc:`here ` that describe the possible fields and values you can use to configure your notebook. + +.. include:: /assets/templates/notebooks/example-1-name-default-format-language.yml + :code: + +Line 1 specifies the ``.dockstore.yml`` version. You should use the latest version, which is version 1.2. + +Next is a required key named ``notebooks`` where your notebooks will be configured. + +Within this, you can to specify the ``format`` and ``language`` of the notebook. These are optional keys and if left out, like in the example above, default to ``Jupyter`` and ``Python`` respectively. Currently, we only support ``Jupyter`` notebooks. For a list of all the languages supported, view the :doc:`notebook template files `. + +Next, you need to specify the absolute path to the notebook file in the Git repository using the ``path`` key. + +Finally, you must set ``publish: true`` to publish your notebook in order for you to be able to view your notebook in preview mode. In the future, we will implement a My Notebooks page where you can view your unpublished notebooks. + +Registering Your Notebook +------------------------- +.. include:: /getting-started/github-apps/note--registration.rst + +To register your Notebook on Dockstore, you need to install our Dockstore GitHub application on the repository containing your notebook and .dockstore.yml file. By doing so, Dockstore will automatically register the notebooks and update your notebooks whenever your repository is updated on GitHub. + +.. include:: /getting-started/github-apps/note--vocabulary.rst + +Install the Dockstore GitHub App +-------------------------------- +.. include:: /getting-started/github-apps/snippet--installation.rst + +Publishing Your Notebook +------------------------ +In this release, you are only able to publish your notebook using the ``publish`` key in the .dockstore.yml file, and your notebook must be published in order for you to view it on Dockstore. + +Publishing will create a public page that is very similar to the ones we have for tools and workflows. + +.. figure:: /assets/images/docs/notebooks-public-page.png + +Now other users can view and :doc:`star <../end-user-topics/starring>` your notebook. You will also have the option to add published notebooks to a :doc:`collection <../advanced-topics/organizations-and-collections>`. + +Saving Changes to Your Notebook on Google Colab to GitHub +--------------------------------------------------------- + +At this point, your Google Colab and GitHub integration should be set up, you have installed the Dockstore GitHub App to your GitHub repository containing your notebook, and the published notebook appears on Dockstore. + +You may wish to continue developing your notebook on Google Colab by modifying and re-running it. In order for the changes to your notebook to appear on Dockstore, you must click on File > Save a copy in GitHub to save your notebook to your GitHub repository. + +.. figure:: /assets/images/docs/google-colab-save-on-github.png + :width: 35 % + +Configure the GitHub repository and branch that you want to save the notebook to. This should be the repository and branch that contains your .dockstore.yml describing the notebook you have on Dockstore. Once configured, click OK. Google Colab will push a commit to your repository with the notebook. + +.. figure:: /assets/images/docs/google-colab-save-on-github-dialog.png + :width: 60 % + +Dockstore will detect the new commit and automatically sync your notebook from GitHub to Dockstore. + +Launch With Google Colab +------------------------ + +When viewing a public notebook on Dockstore, you can launch the notebook into Google Colab by clicking on the Google Colab link on the Launch With panel located on the right side of the page. This will open the notebook in Google Colab. + +.. figure:: /assets/images/docs/launch-with-google-colab.png + +Using the Notebooks Feature Flag +-------------------------------- + +There is a ``notebooks`` feature flag that can be used to explore Dockstore Notebooks features that are in development. + +.. warning:: The features hidden behind this flag may not be fully functional, but using the flag offers you a preview of the features to come. + +To use the ``notebooks`` feature flag, append ``notebooks`` to the Dockstore URL as a query parameter. You only need to do this once, unless you refresh/close your browser. + +For example, if you're on the https://dockstore.org page, append ``notebooks`` such that it looks like this: https://dockstore.org?notebooks. + +If you're on a page that already contains query parameters, indicated by the presence of a question mark, append ``notebooks`` to the URL using an ampersand. For example, if you're on the https://dockstore.org/starred?tab=workflows page, append ``notebooks`` such that it looks like https://dockstore.org/starred?tab=workflows¬ebooks. + +After applying the notebooks feature flag, Dockstore Notebooks features will appear throughout the site, marked with a warning banner. + +.. figure:: /assets/images/docs/notebooks-warning-banner.png + +Features that are activated with the feature flag: + +- Notebooks component is displayed on the My Dashboard page for a logged-in user +- Notebooks tab is displayed on the My Starred page for a logged in user +- Notebooks tab is displayed on the Search page +- Notebooks link is displayed on the Sitemap diff --git a/docs/getting-started/getting-started-with-services.rst b/docs/getting-started/getting-started-with-services.rst index 6c00322c..f8a44488 100644 --- a/docs/getting-started/getting-started-with-services.rst +++ b/docs/getting-started/getting-started-with-services.rst @@ -3,7 +3,7 @@ Services Services are containerized programs that are meant to be long running processes, usually web services or interactive applications, and that can be launched by a user into different environments. -.. note:: Dockstore Services is currently in beta mode. Also note that the screenshots below were taken on our staging site. +.. note:: Dockstore Services is currently in preview mode. Also note that the screenshots below were taken on our staging site. Tutorial Goals -------------- @@ -16,7 +16,7 @@ Tutorial Goals .. David will add context here. Services are meant to be long running processes, usually web services or interactive applications, that can be launched by a user. The entire process for creating a service that can launch successfully is complex and dependent on the service or application you are trying to describe; so it is beyond the scope of this tutorial. -However, this document will outline what is needed to register, update, and publish a service onto Dockstore. Also, please keep in mind that service functionality is currently in beta mode. +However, this document will outline what is needed to register, update, and publish a service onto Dockstore. Also, please keep in mind that service functionality is currently in preview mode. Create Your Service ------------------- diff --git a/docs/getting-started/github-apps/github-apps-landing-page.rst b/docs/getting-started/github-apps/github-apps-landing-page.rst index 860aeb9f..9d6b67bd 100644 --- a/docs/getting-started/github-apps/github-apps-landing-page.rst +++ b/docs/getting-started/github-apps/github-apps-landing-page.rst @@ -3,9 +3,9 @@ Dockstore GitHub App ====================================================== GitHub apps are a GitHub feature used to improve the interaction between external applications and GitHub. -Users can grant a GitHub app specific permissions on the repos and/or organizations of their choosing. +Users can grant a GitHub app access to repositories in an organization and/or personal account of their choosing. -The Dockstore GitHub App adds support for registering and automatically syncing your files between Dockstore and GitHub. It is our preferred method for registration, so you can find a breakdown of that process on the respective pages for :doc:`tool registration `, :doc:`workflow registration `, and :doc:`service registration `. +The Dockstore GitHub App adds support for registering and automatically syncing your files between Dockstore and GitHub. It is our preferred method for registration, so you can find a breakdown of that process on the respective pages for :doc:`tool registration `, :doc:`workflow registration `, :doc:`service registration `, and :doc:`notebook registration `. Check out the following guides to learn more: diff --git a/docs/getting-started/github-apps/github-apps-troubleshooting-tips.rst b/docs/getting-started/github-apps/github-apps-troubleshooting-tips.rst index fe7f9565..284c90bb 100644 --- a/docs/getting-started/github-apps/github-apps-troubleshooting-tips.rst +++ b/docs/getting-started/github-apps/github-apps-troubleshooting-tips.rst @@ -1,3 +1,5 @@ +.. _topGHAFAQ: + ================================= GitHub App Troubleshooting & FAQs ================================= @@ -11,14 +13,16 @@ GitHub App Troubleshooting & FAQs Why should I migrate my existing workflows to use GitHub Apps and a .dockstore.yml? ---------------------------------------------------------------------------------------- -Installing our Dockstore GitHub App onto your GitHub repository or organization will automatically sync your workflow/tool on Dockstore whenever code is pushed to GitHub. +Installing our Dockstore GitHub App onto repositories in a GitHub organization or your GitHub personal account will automatically sync your workflow/tool on Dockstore whenever code is pushed to GitHub. This means less manual work for workflow developers, and less waiting for content to update. This requires the addition of a .dockstore.yml file to your repository on GitHub. This file contains information that Dockstore will use to setup the corresponding workflow/tool entry on Dockstore. Branches that do not have a .dockstore.yml file, or that are filtered out via the filter feature, will not be synchronized. -You can read more about it at :doc:`/getting-started/github-apps/github-apps`. +You can read more about it at :doc:`/getting-started/github-apps/github-apps`. + +:ref:`(back to top) ` How does this change my development flow? ------------------------------------------- @@ -29,21 +33,27 @@ Therefore, as long as your workflow is already registered on Dockstore and your You can use filters in a .dockstore.yml to avoid generating a corresponding workflow-version on Dockstore. -*Note:* If you want to edit version information, such as workflow path, you will have to update the .dockstore.yml file directly on the corresponding GitHub branch. For example, if develop has a .dockstore.yml that points to my_workflow.wdl, but my_workflow.wdl is moved to another path on the branch develop-but-better, then the .dockstore.yml on develop-but-better will need to point to the new location of my_workflow.wdl. +*Note:* If you want to edit version information, such as workflow path, you will have to update the .dockstore.yml file directly on the corresponding GitHub branch. For example, if develop has a .dockstore.yml that points to my_workflow.wdl, but my_workflow.wdl is moved to another path on the branch develop-but-better, then the .dockstore.yml on develop-but-better will need to point to the new location of my_workflow.wdl. + +:ref:`(back to top) ` .. _Check GitHub App installation on repository: How do I check if the Dockstore GitHub App was installed on an individual repository? -------------------------------------------------------------------------------------- -Go to your repo on GitHub, click the Settings tab, click Integrations on the left and verify our app is installed and configured correctly +Go to your repo on GitHub and click the Settings tab. On the left hand side under the Integrations section, click on GitHub Apps and verify our app is installed and configured correctly. .. image:: /assets/images/docs/github-repo-settings.png -You may not have access to this page if you are not an administrator of your GitHub organization, :ref:`but you may still be able to see if it installed. ` +If you're checking if the GitHub App was installed on a repository of an organization, you may not have access to this page if you are not an administrator of your GitHub organization, :ref:`but you may still be able to see if it installed. ` + +:ref:`(back to top) ` How do I configure the GitHub App across multiple repositories? ------------------------------------------------------------------ -GitHub Apps can be installed on either an a user level, or an organizational level. If you installed the app for your own repos that are not in an organization, you will be able to verify the Dockstore GitHub App is installed by clicking "Applications" in the left menu in your GitHub settings. Our app, along with any others you may have installed, will be there. Clicking the "configure" button will allow you to adjust which repos the app has access to. +GitHub Apps can be installed on either an a user level, or an organizational level. If you installed the app for your own repositories that are not in an organization, you will be able to verify the Dockstore GitHub App is installed by clicking "Applications" in the left menu in your GitHub settings. Our app, along with any others you may have installed, will be there. Clicking the "configure" button will allow you to adjust which repos the app has access to. + +:ref:`(back to top) ` .. _view github app permissions hack: @@ -55,6 +65,8 @@ A GitHub organization's admins can configure an app directly in the organization .. image:: /assets/images/docs/reinstall-app-to-cheese-org-settings.png :width: 50% +:ref:`(back to top) ` + .. _GitHub App permissions FAQ: Can I use the GitHub App if I am not the admin of the organization and/or repo I am trying to use it with? @@ -80,13 +92,15 @@ You may also still run into scenarios where your changes appear to not get saved If it seems your GitHub App access just won't "stick" or you are having other permissions issues, consider asking the administrator of your organization to install the app. If they set it up to have access to all repositories on the organization, this will only need to be done once. +:ref:`(back to top) ` + The changes made to my GitHub repo aren't appearing on Dockstore, but I've already installed the GitHub app and made the .dockstore.yml file. How can I figure out what's going wrong? ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ The general troubleshooting flow we recommend is the following: - Try waiting a couple of minutes and refreshing the browser on the My Workflows/My Tools page again. Sometimes, GitHub takes a few minutes to send Dockstore the changes made to a repository. -- Verify that the GitHub app was given access to the right repository or organization. -- If access was given to the wrong organization or repository, or this is your first time installing the app, you'll need to push another commit after adding the correct repository to activate the sync to Dockstore. +- Verify that the GitHub app was given access to the right repository in an organization or your personal account. +- If access was given to the wrong repository or organization/personal account, or this is your first time installing the app, you may need to push another commit after adding the correct repository to activate the sync to Dockstore. - Double check the .dockstore.yml file. - Is it in the root directory? @@ -97,12 +111,20 @@ The general troubleshooting flow we recommend is the following: If you've already tried these, you can view GitHub App logs through Dockstore to see if there have been any errors (see next section). +:ref:`(back to top) ` + +.. _GitHub App logs FAQ: + How do I view the GitHub App logs? ---------------------------------- -Navigate to the ``/my-workflows`` (or ``/my-tools``) page and expand the GitHub Organization that the repository belongs to on the left hand side. Then click on the bottom where it says ``See GitHub App Logs``. +Navigate to the ``/my-workflows`` (or ``/my-tools``) page. + +.. note:: If you're trying to view the GitHub App logs for a notebook, which doesn't have its own ``/my-notebooks`` page yet, navigate to ``/my-workflows`` then continue following the steps. + +Expand the GitHub account that the repository belongs to on the left hand side. Then click on the bottom where it says ``Apps Logs``. .. image:: /assets/images/docs/github-app-logs-button.png - :width: 40 % + :width: 30 % Once loaded, the following window will be displayed. @@ -128,6 +150,8 @@ It is saying that the workflow name ``single workflow`` is invalid. The workflow If you're having trouble finding the relevant logs, try searching for the name of your repository by using the filter on the upper left. You can also sort the rows by clicking on a column heading. For example, if you click the ``Success`` column heading once, it will list all the events that failed first. +:ref:`(back to top) ` + Why was a new workflow registered instead of migrating my existing one? -------------------------------------------------------------------------- .. @@ -144,6 +168,7 @@ A new separate workflow/tool will be registered if the original name isn't inclu :alt: Tool to Migrate :width: 55 % +:ref:`(back to top) ` How can I convert my entire existing workflow/tool at once? ------------------------------------------------------------- @@ -153,6 +178,8 @@ automatically detect and sync changes with the corresponding version on Dockstor If you have a .dockstore.yml file in your master or develop branches on GitHub, any new branches you create from these as your template will have a .dockstore.yml. +:ref:`(back to top) ` + Why are only some branches appearing on my workflow/tool? ---------------------------------------------------------- @@ -160,6 +187,8 @@ The Dockstore GitHub App is currently unable to parse branches that use special However, even if you have branches with unsupported names, other branches with names like `main` and `develop` will continue to update on Dockstore as normal. The public view of your published entry will not show any errors -- it will simply not show the branches with unsupported names. +:ref:`(back to top) ` + .. discourse:: :topic_identifier: 6485 diff --git a/docs/getting-started/github-apps/github-apps.rst b/docs/getting-started/github-apps/github-apps.rst index 94f6fa9a..60e32a87 100644 --- a/docs/getting-started/github-apps/github-apps.rst +++ b/docs/getting-started/github-apps/github-apps.rst @@ -19,8 +19,7 @@ What are GitHub Apps? `GitHub apps `_ are a GitHub feature used to improve the interaction between external applications and GitHub. Users can -grant a GitHub app specific permissions on the repos and/or -organizations of their choosing. When events occur on the GitHub repos, e.g., +grant a GitHub app access to repositories in organizations and/or their personal account. When events occur on the GitHub repositories, e.g., creating a new release, the GitHub app issues notifications. Why have a Dockstore GitHub App? @@ -41,6 +40,8 @@ the Refresh button or making an API call to the Dockstore API. Due to the manual nature of this process, it is easy for Dockstore to get out of sync with the linked GitHub repository if the Dockstore GitHub App is not being used. +The Dockstore GitHub App also makes it easy to register multiple workflows in the same repository quickly, as well as set up their metadata (test parameter files, authors, readmes, etc). You can even publish a workflow or set a default branch from the .dockstore.yml which can be useful for both automated systems and users registering workflows in bulk. + How the Dockstore GitHub App works ---------------------------------- @@ -102,6 +103,7 @@ See Also - :doc:`Getting Started with Services ` - :doc:`Getting Started with Workflows ` - :doc:`Getting Started with Tools ` +- :doc:`Getting Started with Notebooks ` - :doc:`Other docs involving the Dockstore GitHub App ` .. discourse:: diff --git a/docs/getting-started/github-apps/migrating-workflows-to-github-apps.rst b/docs/getting-started/github-apps/migrating-workflows-to-github-apps.rst index a0612d94..125bc63e 100644 --- a/docs/getting-started/github-apps/migrating-workflows-to-github-apps.rst +++ b/docs/getting-started/github-apps/migrating-workflows-to-github-apps.rst @@ -6,8 +6,7 @@ Dockstore 1.9.0 provides users with a way to keep their workflows automatically GitHub App Installation ----------------------- -The first step to migrating a workflow is the same as adding a new workflow via GitHub Apps: install our Dockstore GitHub App onto your repository or -organization, if you have not already done so. +The first step to migrating a workflow is the same as adding a new workflow via GitHub Apps: install our Dockstore GitHub App in an organization or your personal account, if you have not already done so. .. include:: /getting-started/github-apps/snippet--installation.rst @@ -16,7 +15,7 @@ organization, if you have not already done so. Creating a .dockstore.yml File ------------------------------- -Once the GitHub app is installed on the correct repo, the next step is to create a .dockstore.yml file. We'll cover a very straightforward example +Once the GitHub app is installed and given access to the correct repository, the next step is to create a .dockstore.yml file. We'll cover a very straightforward example first, but depending on how you configured the workflow during registration and whether your GitHub repository houses multiple workflows published on Dockstore, there will be additional steps to writing your .dockstore.yml file. diff --git a/docs/getting-started/github-apps/note--registration.rst b/docs/getting-started/github-apps/note--registration.rst index d092a548..64d5fde6 100644 --- a/docs/getting-started/github-apps/note--registration.rst +++ b/docs/getting-started/github-apps/note--registration.rst @@ -1 +1 @@ -.. note:: You do not need an account to search for workflows on Dockstore or to launch them with our compute partners. However, to register content on Dockstore, you must :doc:`have an account on Dockstore and link the necessary third-party accounts `. Once this is done you can register workflows from the My Workflows page, tools from the My Tools page, or services from the My Services page. \ No newline at end of file +.. note:: You do not need an account to search for workflows on Dockstore or to launch them with our compute partners. However, to register content on Dockstore, you must :doc:`have an account on Dockstore and link the necessary third-party accounts `. Once this is done you can register workflows from the My Workflows page, tools from the My Tools page, services from the My Services page, or notebooks from the My Workflows page (currently in preview mode). \ No newline at end of file diff --git a/docs/getting-started/github-apps/snippet--installation.rst b/docs/getting-started/github-apps/snippet--installation.rst index f3fafdbe..333e4fdd 100644 --- a/docs/getting-started/github-apps/snippet--installation.rst +++ b/docs/getting-started/github-apps/snippet--installation.rst @@ -1,37 +1,48 @@ -Installing the GitHub App is simple. Navigate to ``/my-tools``, ``/my-workflows``, or ``/my-services`` using the drop down menu in the top right. In these screenshots, we will go via ``/my-tools``, but the process is essentially the same for any of the other options. +Installing the GitHub App is simple. Navigate to ``/my-tools``, ``/my-workflows``, or ``/my-services`` by navigating to My Dashboard then selecting the desired option in the left sidebar. In these screenshots, we will go via ``/my-tools``, but the process is essentially the same for any of the other options. -.. image:: /assets/images/docs/my-tools.png +.. note:: Since Dockstore Notebooks is still in preview mode, it does not have a dedicated ``/my-notebooks`` page yet. To register a notebook, navigate to ``/my-workflows`` in the step above. -Click the ``+`` button on the left hand sidebar. +.. image:: /assets/images/docs/my-dashboard-sidebar.png + +| + +Click the ``Register a Tool`` button on the left hand sidebar. .. image:: /assets/images/docs/add-tool-button.png - :width: 40 % + :width: 30 % + +| A window will appear asking how you would like to register your tool, workflow, or service. Select ``Register using GitHub Apps``. .. image:: /assets/images/docs/register-tool-github-apps.png :width: 50 % -Click ``+ Manage Dockstore Installation on GitHub``. You'll then be redirected to GitHub where you can select which repositories can be accessed by the GitHub app. +Click ``+ Manage Dockstore Installations on GitHub``. .. image:: /assets/images/docs/manage-gh-app-installation.png :width: 50 % -You'll then be redirected to GitHub where you can grant the app access to specific repositories within whatever organization you are installing into. Note that GitHub treats your username as its own "organization." For instance, my GitHub username is aofarrel. If I want to install the GitHub App so it could access aofarrel/mycoolrepo, I would choose the first option here. +You'll then be redirected to GitHub where you can install the app in an organization or your personal account. For example, the username for my personal GitHub account is aofarrel. If I want to install the GitHub App so it could access aofarrel/mycoolrepo, I would choose the first option here. .. figure:: /assets/images/docs/gh-app-install-where.png :width: 65 % - Install our GitHub App on either all current and future repositories in an organization or on specific repositories + Install our GitHub App in an organization or your personal account -After selection of an organization, you can select whether to give access to all current and future repositories or only select ones. If the organization you choose is intended to be just for Dockstore tools/workflows/services, you may want to allow access to all repositories. Otherwise, it is may be more intuitive to select only certain repositories. Click save and you will be taken back to the page you started on in Dockstore -- either ``/my-tools``, ``/my-workflows``, or ``/my-services``, depending where you started. +After selection of an organization or a personal account, you can select whether to give access to all current and future repositories or only select ones. If the organization or personal account you choose is intended to be just for Dockstore tools/workflows/services/notebooks, you may want to allow access to all repositories. Otherwise, it may be more intuitive to select only certain repositories. Click save and you will be taken back to the page you started on in Dockstore -- either ``/my-tools``, ``/my-workflows``, or ``/my-services``, depending on where you started. .. important:: The GitHub user who first adds a workflow onto Dockstore must correspond to a user on Dockstore. -You should now see the organization and the repositories you chose to keep track of in the "unpublished" tab. Here's an example involving ``/my-services``: +On Dockstore, under the GITHUB section, you should see the names of GitHub accounts that you have access to, such as organizations that you belong to and your personal account. If your repositories that you chose to keep track of contained a .dockstore.yml at the time of installing the GitHub App, then you will see the repositories under the GitHub personal/organization account name that it belongs to. Here's an example involving ``/my-services``: .. figure:: /assets/images/docs/my-services-filled.png +.. note:: You will not see unpublished notebooks because there is not a dedicated ``/my-notebooks`` page yet. To view your notebook, which should've been configured to automatically publish on Dockstore via the .dockstore.yml, navigate to ``/notebooks`` + and locate your notebook in the list of published notebooks. + + .. image:: /assets/images/docs/list-published-notebooks.png + A note on permissions when installing the Dockstore GitHub App to a GitHub organization ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -50,5 +61,8 @@ For more information on troubleshooting GitHub App permissions, please see :ref: Ensuring sychronization ~~~~~~~~~~~~~~~~~~~~~~~ -Once the GitHub App is installed and a .dockstore.yml is present, please make sure to push one *additional* commit to your repository. This helps make sure your workflows, tools, and services show up in Dockstore. +Upon installing the GitHub App, Dockstore will find branches in your repository that contain a .dockstore.yml and attempt to register your workflows, tools, services, and notebooks. There may be some cases where Dockstore is unable to find all branches containing a .dockstore.yml, for example, if the GitHub repository has many branches. + +If your workflow, tool, service, or notebook is not showing up on Dockstore after 5 minutes, push one *additional* commit to the branch in your repository that contains the .dockstore.yml that's not being synchronized. Dockstore will synchronize the branch that was updated, which helps make sure that your workflows, tools, services, and notebooks show up in Dockstore. +If your workflow, tool, service, or notebook still doesn't show up, check the :ref:`GitHub App logs ` to see if Dockstore encountered an error while processing your .dockstore.yml. diff --git a/docs/index.rst b/docs/index.rst index 4937b5ab..c748094b 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -44,7 +44,8 @@ Getting Started 5. :doc:`Register a tool on Dockstore ` 6. :doc:`Register a workflow on Dockstore ` 7. :doc:`Hosted tools and workflows ` -* :doc:`Services (beta) ` +* :doc:`Services (preview) ` +* :doc:`Notebooks (preview) ` * :doc:`Registering workflows with the Dockstore GitHub App ` .. toctree:: @@ -64,6 +65,7 @@ Getting Started getting-started/dockstore-workflows getting-started/hosted-tools-and-workflows getting-started/getting-started-with-services + getting-started/getting-started-with-notebooks getting-started/github-apps/github-apps-landing-page Launching tools and workflows from Dockstore @@ -195,6 +197,7 @@ Advanced developer topics * :doc:`GA4GH Write API (intended as a proof-of-concept and for developers with a large number of tools) ` * :doc:`Using batch services (AWS, Azure, Google, and Consonance) ` * :doc:`Verified workflows and tools ` +* :doc:`Workflow metrics ` * :doc:`Organizations and collections ` .. Purposely not including the AWS and Azure batch links as the batch-services one links to them and is pretty short so they are easy to find. I would even argue for removing them from the sidebar or making them a subfolder of batch-services. @@ -223,6 +226,7 @@ Advanced developer topics advanced-topics/azure-batch advanced-topics/posting-zips advanced-topics/verification + advanced-topics/metrics advanced-topics/organizations-and-collections advanced-topics/wes/cli-wes-tutorial