Skip to content

DOCK-2170: Fix tools and DAG generation problem#5232

Merged
svonworl merged 8 commits intodevelopfrom
feature/dock-2170/fix-cwljava-dag-tools-bug
Nov 14, 2022
Merged

DOCK-2170: Fix tools and DAG generation problem#5232
svonworl merged 8 commits intodevelopfrom
feature/dock-2170/fix-cwljava-dag-tools-bug

Conversation

@svonworl
Copy link
Copy Markdown
Contributor

@svonworl svonworl commented Nov 8, 2022

Description
This PR fixes some bugs in the CWL "tool list" and DAG generation code.

While testing the previous PR (#5063), Charles found a workflow for which the list of tools and DAG were generated incorrectly:
https://qa.dockstore.org/workflows/github.com/sevenbridges-openworkflows/Broad-Best-Practice-Data-pre-processing-CWL1.0-workflow-GATK-4.1.0.0/GATK_4_1_0_0_data_pre_processing_workflow:master?tab=info

The problem was that the workflow ID included slashes, which messed up the workflow step ID and dependency parsing algorithms. This PR changes the algorithms to be more robust.

Yes, that one function takes a ridiculous number of arguments, and now, it's even ridiculous-er, because I added an argument.
Alas, there's not an obviously-better alternative, unless we rewrite large chunks of the code (which we might if we ever implement multi-level DAG generation).

Review Instructions
Clone the repo of the above-mentioned workflow, register the cloned workflow, and confirm that the resulting list of tools and DAG are equivalent to those referenced by the original link.

Issue
https://ucsc-cgl.atlassian.net/browse/DOCK-2170
#4906

Please make sure that you've checked the following before submitting your pull request. Thanks!

  • Check that you pass the basic style checks and unit tests by running mvn clean install
  • Ensure that the PR targets the correct branch. Check the milestone or fix version of the ticket.
  • Follow the existing JPA patterns for queries, using named parameters, to avoid SQL injection
  • If you are changing dependencies, check the Snyk status check or the dashboard to ensure you are not introducing new high/critical vulnerabilities
  • Assume that inputs to the API can be malicious, and sanitize and/or check for Denial of Service type values, e.g., massive sizes
  • Do not serve user-uploaded binary images through the Dockstore API
  • Ensure that endpoints that only allow privileged access enforce that with the @RolesAllowed annotation
  • Do not create cookies, although this may change in the future
  • If this PR is for a user-facing feature, create and link a documentation ticket for this feature (usually in the same milestone as the linked issue). Style points if you create a documentation PR directly and link that instead.

@codecov
Copy link
Copy Markdown

codecov bot commented Nov 8, 2022

Codecov Report

Base: 73.04% // Head: 73.00% // Decreases project coverage by -0.04% ⚠️

Coverage data is based on head (311d7d8) compared to base (4db3e99).
Patch coverage: 85.71% of modified lines in pull request are covered.

❗ Current head 311d7d8 differs from pull request most recent head 190ef62. Consider uploading reports for the commit 190ef62 to get more accurate results

Additional details and impacted files
@@              Coverage Diff              @@
##             develop    #5232      +/-   ##
=============================================
- Coverage      73.04%   73.00%   -0.05%     
+ Complexity      4358     4350       -8     
=============================================
  Files            287      287              
  Lines          16707    16704       -3     
  Branches        1835     1836       +1     
=============================================
- Hits           12204    12194      -10     
- Misses          3638     3642       +4     
- Partials         865      868       +3     
Flag Coverage Δ
bitbuckettests 28.28% <0.00%> (+<0.01%) ⬆️
integrationtests 57.70% <64.28%> (-0.02%) ⬇️
languageparsingtests 10.87% <0.00%> (+<0.01%) ⬆️
toolintegrationtests 30.42% <0.00%> (-0.02%) ⬇️
unit-tests_and_non-confidential-tests 26.35% <50.00%> (-0.04%) ⬇️
workflowintegrationtests 40.40% <82.14%> (-0.06%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
.../io/dockstore/webservice/languages/CWLHandler.java 62.45% <85.71%> (-0.61%) ⬇️
...re/webservice/helpers/DockerRegistryAPIHelper.java 69.48% <0.00%> (-2.60%) ⬇️
...store/webservice/resources/DockerRepoResource.java 77.01% <0.00%> (-0.46%) ⬇️
.../java/io/openapi/api/impl/ToolsApiServiceImpl.java 70.28% <0.00%> (-0.45%) ⬇️
...ckstore/webservice/resources/WorkflowResource.java 71.07% <0.00%> (ø)
...main/java/io/swagger/api/impl/ToolsImplCommon.java 86.58% <0.00%> (+0.81%) ⬆️
...bservice/resources/AliasableResourceInterface.java 96.87% <0.00%> (+3.12%) ⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

@svonworl svonworl self-assigned this Nov 8, 2022
@svonworl svonworl requested a review from kathy-t November 8, 2022 22:45
@denis-yuen
Copy link
Copy Markdown
Member

Alas, there's not an obviously-better alternative, unless we rewrite large chunks of the code (which we might if we ever implement multi-level DAG generation).

Can you link to this, I think I'd like to read up on this and wonder if its a reference to a previous ticket/discussion

@svonworl
Copy link
Copy Markdown
Contributor Author

svonworl commented Nov 9, 2022

Alas, there's not an obviously-better alternative, unless we rewrite large chunks of the code (which we might if we ever implement multi-level DAG generation).

Can you link to this, I think I'd like to read up on this and wonder if its a reference to a previous ticket/discussion

I would add a link, but I don't have one. :) The "not an obviously-better alternative" part is my software-engineering opinion, "multi-level DAG" is an idea for a DAG that renders the steps in the subworkflows (rather than represent each subworkflow as a single node, as we have historically).

@denis-yuen
Copy link
Copy Markdown
Member

multi-level DAG generation

was mainly curious what this meant, thanks!

Copy link
Copy Markdown
Collaborator

@coverbeck coverbeck left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be nice to have a test for this so it doesn't break in the future.

@svonworl svonworl merged commit ce7cb2c into develop Nov 14, 2022
@svonworl svonworl deleted the feature/dock-2170/fix-cwljava-dag-tools-bug branch November 14, 2022 20:11
@sonarqubecloud
Copy link
Copy Markdown

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 2 Code Smells

88.1% 88.1% Coverage
0.0% 0.0% Duplication

@denis-yuen denis-yuen restored the feature/dock-2170/fix-cwljava-dag-tools-bug branch November 28, 2022 21:24
@denis-yuen denis-yuen deleted the feature/dock-2170/fix-cwljava-dag-tools-bug branch November 28, 2022 21:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants