Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HDDS-2030. Generate simplifed reports by the dev-support/checks/*.sh scripts #1348

Closed
wants to merge 21 commits into from

Conversation

elek
Copy link
Member

@elek elek commented Aug 24, 2019

hadoop-ozone/dev-support/checks directory contains shell scripts to execute different type of code checks (findbugs, checkstyle, etc.)

Currently the contract is very simple. Every shell script executes one (and only one) check and the shell response code is set according to the result (non-zero code if failed).

To have better reporting in the github pr build, it would be great to improve the scripts to generate simple summary files and save the relevant files for archiving.

See: https://issues.apache.org/jira/browse/HDDS-2030

@elek elek added the ozone label Aug 24, 2019
hadoop-ozone/dev-support/checks/_mvn_unit_report.sh Outdated Show resolved Hide resolved
hadoop-ozone/dev-support/checks/unit.sh Outdated Show resolved Hide resolved
hadoop-ozone/dev-support/checks/acceptance.sh Outdated Show resolved Hide resolved
hadoop-ozone/dev-support/checks/README.md Outdated Show resolved Hide resolved
hadoop-ozone/dev-support/checks/acceptance.sh Outdated Show resolved Hide resolved
@apache apache deleted a comment from hadoop-yetus Aug 27, 2019
@apache apache deleted a comment from hadoop-yetus Aug 29, 2019
@apache apache deleted a comment from hadoop-yetus Aug 29, 2019
@apache apache deleted a comment from hadoop-yetus Aug 29, 2019
@apache apache deleted a comment from hadoop-yetus Aug 29, 2019
@apache apache deleted a comment from hadoop-yetus Aug 29, 2019
@apache apache deleted a comment from hadoop-yetus Aug 29, 2019
@apache apache deleted a comment from hadoop-yetus Aug 29, 2019
@apache apache deleted a comment from hadoop-yetus Aug 29, 2019
@apache apache deleted a comment from hadoop-yetus Aug 29, 2019
@apache apache deleted a comment from hadoop-yetus Aug 29, 2019
@apache apache deleted a comment from hadoop-yetus Aug 29, 2019
@apache apache deleted a comment from hadoop-yetus Aug 29, 2019
@apache apache deleted a comment from hadoop-yetus Aug 29, 2019
@apache apache deleted a comment from hadoop-yetus Aug 29, 2019
@apache apache deleted a comment from hadoop-yetus Aug 29, 2019
@apache apache deleted a comment from hadoop-yetus Aug 29, 2019
@apache apache deleted a comment from hadoop-yetus Aug 29, 2019
@apache apache deleted a comment from hadoop-yetus Aug 29, 2019
@apache apache deleted a comment from hadoop-yetus Aug 29, 2019
@apache apache deleted a comment from hadoop-yetus Aug 29, 2019
Copy link
Contributor

@adoroszlai adoroszlai left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @elek for all the improvements.


for FAILURE in $FAILURES; do
TEST_RESULT_LOCATION="$(realpath --relative-to="$REPORT_DIR" "$TEST_RESULT_FILE")"
TEST_OUTPUT_LOCATION="${TEST_RESULT_LOCATION//.txt/-output.txt/}"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
TEST_OUTPUT_LOCATION="${TEST_RESULT_LOCATION//.txt/-output.txt/}"
TEST_OUTPUT_LOCATION="${TEST_RESULT_LOCATION//.txt/-output.txt}"

find "." -name "*.dump" -exec cp {} "$REPORT_DIR/" \;

## Add the tests where the JVM is crashed
grep -A1 'Crashed tests' ${REPORT_DIR}/output.log \

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shellcheck:26: note: Double quote to prevent globbing and word splitting. [SC2086]

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 41 Docker mode activated.
_ Prechecks _
+1 dupname 0 No case conflicting files found.
0 shelldocs 0 Shelldocs was not available.
0 @author 0 Skipping @author checks as author.sh has been patched.
-1 test4tests 0 The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+1 mvninstall 603 trunk passed
+1 mvnsite 0 trunk passed
+1 shadedclient 823 branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 mvninstall 544 the patch passed
+1 mvnsite 0 the patch passed
-1 shellcheck 0 The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 701 patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 unit 113 hadoop-hdds in the patch passed.
+1 unit 306 hadoop-ozone in the patch passed.
+1 asflicense 50 The patch does not generate ASF License warnings.
3409
Subsystem Report/Notes
Docker Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1348/21/artifact/out/Dockerfile
GITHUB PR #1348
Optional Tests dupname asflicense mvnsite unit shellcheck shelldocs
uname Linux e1e198b89db2 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 915cbc9
shellcheck https://builds.apache.org/job/hadoop-multibranch/job/PR-1348/21/artifact/out/diff-patch-shellcheck.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-1348/21/testReport/
Max. process+thread count 412 (vs. ulimit of 5500)
modules C: hadoop-ozone U: hadoop-ozone
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1348/21/console
versions git=2.7.4 maven=3.3.9 shellcheck=0.4.6
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.


#Copy heap dump and dump leftovers
find "." -name "*.hprof" -exec cp {} "$REPORT_DIR/" \;
find "." -name "*.dump" -exec cp {} "$REPORT_DIR/" \;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please also copy *.dumpstream, which may help finding out the cause of Corrupted STDOUT warning of forked JVM.

Suggested change
find "." -name "*.dump" -exec cp {} "$REPORT_DIR/" \;
find "." -name "*.dump" -exec cp {} "$REPORT_DIR/" \;
find "." -name "*.dumpstream" -exec cp {} "$REPORT_DIR/" \;

http://maven.apache.org/surefire/maven-surefire-plugin/faq.html#dumpfiles

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 125 Docker mode activated.
_ Prechecks _
+1 dupname 0 No case conflicting files found.
0 shelldocs 0 Shelldocs was not available.
0 @author 0 Skipping @author checks as author.sh has been patched.
-1 test4tests 0 The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
-1 mvninstall 45 hadoop-ozone in trunk failed.
+1 mvnsite 0 trunk passed
+1 shadedclient 1065 branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
-1 mvninstall 42 hadoop-ozone in the patch failed.
+1 mvnsite 0 the patch passed
+1 shellcheck 2 There were no new shellcheck issues.
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 877 patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 unit 67 hadoop-hdds in the patch passed.
-1 unit 34 hadoop-ozone in the patch failed.
+1 asflicense 50 The patch does not generate ASF License warnings.
2736
Subsystem Report/Notes
Docker Client=19.03.0 Server=19.03.0 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1348/22/artifact/out/Dockerfile
GITHUB PR #1348
Optional Tests dupname asflicense mvnsite unit shellcheck shelldocs
uname Linux b55e8036b60d 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 85b1c72
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-1348/22/artifact/out/branch-mvninstall-hadoop-ozone.txt
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-1348/22/artifact/out/patch-mvninstall-hadoop-ozone.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1348/22/artifact/out/patch-unit-hadoop-ozone.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-1348/22/testReport/
Max. process+thread count 307 (vs. ulimit of 5500)
modules C: hadoop-ozone U: hadoop-ozone
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1348/22/console
versions git=2.7.4 maven=3.3.9 shellcheck=0.4.6
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@anuengineer
Copy link
Contributor

+1, committed to the trunk. Thanks for the contributions, @elek and @adoroszlai

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
4 participants