Skip to content

Add CSV coverage PR commenter #6028

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Jun 18, 2021

Conversation

tamasvajk
Copy link
Contributor

@tamasvajk tamasvajk commented Jun 7, 2021

This PR adds a github action which runs on PRs. It compares the merge commit of the PR and the head of the base branch. On both commits the CSV reports are generated, and the differences are added as a comment on the PR. The workflow is split into two parts: on: pull_request to generate the reports and on: workflow_run to compare them and add a PR comment.

The PR is based on #6008, so only the last commit(s) should be reviewed.

I've tested the commenting functionality in https://github.com/dsp-testing/codeql-csv-coverage-pr-commenter/pull/1.

@github-actions github-actions bot added the Java label Jun 7, 2021
@tamasvajk tamasvajk force-pushed the tamasvajk/feature/csv-coverage-report-comment branch from e162b91 to 5a8aa33 Compare June 7, 2021 08:42
@tamasvajk tamasvajk removed the Java label Jun 7, 2021
@tamasvajk tamasvajk force-pushed the tamasvajk/feature/csv-coverage-report-comment branch from 5a8aa33 to aa0fa87 Compare June 7, 2021 08:48
@github-actions github-actions bot added the Java label Jun 7, 2021
@tamasvajk tamasvajk removed the Java label Jun 7, 2021
@github-actions github-actions bot added the Java label Jun 7, 2021
@tamasvajk tamasvajk force-pushed the tamasvajk/feature/csv-coverage-report-comment branch 11 times, most recently from 3f13ff7 to 3685194 Compare June 7, 2021 13:19
@github github deleted a comment from github-actions bot Jun 7, 2021
@tamasvajk tamasvajk force-pushed the tamasvajk/feature/csv-coverage-report-comment branch 2 times, most recently from 922e953 to a789d5a Compare June 7, 2021 13:49
@github github deleted a comment from github-actions bot Jun 7, 2021
@tamasvajk tamasvajk force-pushed the tamasvajk/feature/csv-coverage-report-comment branch from a789d5a to 7bb499c Compare June 7, 2021 13:51
@github github deleted a comment from github-actions bot Jun 7, 2021
@github-actions
Copy link
Contributor

github-actions bot commented Jun 7, 2021

⚠️ The head of this PR and the base branch were compared for differences in the CSV coverage reports. The generated reports are available in the artifacts of this workflow run. The following differences were found:

java

Generated files are not matching for java

  • Mismatch in flow-model-coverage-java.rst:
-    `Spring <https://spring.io/>`_,``org.springframework.*``,30,,14,,,,,14,,
+    `Spring <https://spring.io/>`_,``org.springframework.*``,29,,14,,,,,14,,
-    Totals,,85,1233,121,13,6,6,,33,1,2
+    Totals,,84,1233,121,13,6,6,,33,1,2
  • Mismatch in flow-model-coverage-java.csv:
- org.springframework.security.web.savedrequest,,7,,,,,,,,,,,,,7,,
+ org.springframework.security.web.savedrequest,,6,,,,,,,,,,,,,6,,

@tamasvajk tamasvajk removed the Java label Jun 7, 2021
@yo-h
Copy link
Contributor

yo-h commented Jun 9, 2021

Minor changes suggested below (add highlighting, and change wording from "mismatch" to "changes" to avoid giving the impression that there is something wrong).


⚠️ The head of this PR and the base branch were compared for differences in framework coverage. The generated reports are available in the artifacts of this workflow run. The following differences were found:

java

Generated file changes for java

  • Changes to flow-model-coverage-java.rst:
-    `Spring <https://spring.io/>`_,``org.springframework.*``,30,,14,,,,,14,,
+    `Spring <https://spring.io/>`_,``org.springframework.*``,29,,14,,,,,14,,
-    Totals,,85,1233,121,13,6,6,,33,1,2
+    Totals,,84,1233,121,13,6,6,,33,1,2
  • Changes to flow-model-coverage-java.csv:
- org.springframework.security.web.savedrequest,,7,,,,,,,,,,,,,7,,
+ org.springframework.security.web.savedrequest,,6,,,,,,,,,,,,,6,,

@github-actions github-actions bot added the Java label Jun 9, 2021
@github github deleted a comment from github-actions bot Jun 9, 2021
@tamasvajk tamasvajk force-pushed the tamasvajk/feature/csv-coverage-report-comment branch from ab6518b to d65e260 Compare June 9, 2021 08:36
@tamasvajk tamasvajk force-pushed the tamasvajk/feature/csv-coverage-report-comment branch from b0496a0 to 0170004 Compare June 10, 2021 08:06
@github-actions
Copy link
Contributor

⚠️ The head of this PR and the base branch were compared for differences in the framework coverage reports. The generated reports are available in the artifacts of this workflow run. The differences will be picked up by the nightly job after the PR gets merged. The following differences were found:

java

Generated file changes for java

  • Changes to flow-model-coverage-java.rst:
-    `Spring <https://spring.io/>`_,``org.springframework.*``,30,,14,,,,,14,,
+    `Spring <https://spring.io/>`_,``org.springframework.*``,29,,14,,,,,14,,
-    Totals,,85,1233,121,13,6,6,,33,1,2
+    Totals,,84,1233,121,13,6,6,,33,1,2
  • Changes to flow-model-coverage-java.csv:
- org.springframework.security.web.savedrequest,,7,,,,,,,,,,,,,7,,
+ org.springframework.security.web.savedrequest,,6,,,,,,,,,,,,,6,,

@tamasvajk tamasvajk force-pushed the tamasvajk/feature/csv-coverage-report-comment branch 4 times, most recently from f420a41 to aaa07d7 Compare June 10, 2021 13:52
@tamasvajk tamasvajk marked this pull request as ready for review June 10, 2021 14:00
@tamasvajk tamasvajk requested a review from a team as a code owner June 10, 2021 14:00
@tamasvajk tamasvajk requested review from yo-h and adityasharad and removed request for a team June 10, 2021 14:00
Copy link
Collaborator

@adityasharad adityasharad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! Some suggestions on the details of the Actions workflows, but the general setup looks good.

@tamasvajk tamasvajk force-pushed the tamasvajk/feature/csv-coverage-report-comment branch from aaa07d7 to 4d9003b Compare June 15, 2021 10:00
@github-actions github-actions bot added the Java label Jun 15, 2021
@tamasvajk
Copy link
Contributor Author

Thanks @adityasharad for the code review. Here's a PR comment generated after the fixes. Also, here's the latest timeseries run.

# "' match the ones in the codebase.")

# if not all_ok:
# sys.exit(1)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is lots of commented-out code above this line. Should that be removed?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I'll remove these, and some of it will come back as part of the followup work on the automatic PR creation with the changes in main.

adityasharad
adityasharad previously approved these changes Jun 15, 2021
Copy link
Collaborator

@adityasharad adityasharad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actions code looks good, only minor suggestions remaining.

yo-h
yo-h previously approved these changes Jun 16, 2021
tamasvajk and others added 4 commits June 17, 2021 13:04
@tamasvajk tamasvajk force-pushed the tamasvajk/feature/csv-coverage-report-comment branch from 5bab725 to 07b83d5 Compare June 17, 2021 11:05
@tamasvajk tamasvajk merged commit 0545bcf into main Jun 18, 2021
@tamasvajk tamasvajk deleted the tamasvajk/feature/csv-coverage-report-comment branch June 18, 2021 07:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants