Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Decrease Hub API tests verbosity in CI #58

Draft
wants to merge 3 commits into
base: main
Choose a base branch
from
Draft

Decrease Hub API tests verbosity in CI #58

wants to merge 3 commits into from
+2 −1

Conversation

aufi
Copy link
Member

@aufi aufi commented Nov 13, 2023

Hub API tests use binding.RichClient which logs every API call, this could introduce unwanted verbosity in CI test execution.

Updating API test execution script to grep failures only (with some context) to make CI output cleaner and easier to debug. Full API test output is still part of Hub's own API test execution.

Fixes #54

@aufi
Decrease Hub API tests verbosity in CI
9967c5a 
Hub API tests use binding.RichClient which logs every API call, this
could introduce unwanted verbosity in CI test execution.

Updating API test execution script to grep failures only (with some
context) to make CI output cleaner and easier to debug. Full API test
output is still part of Hub's own API test execution.

Fixes konveyor#54

Signed-off-by: Marek Aufart <maufart@redhat.com>
@aufi
Copy link
Member Author

aufi commented Nov 13, 2023

Example output with multiple tests failures:

...
Running Hub API tests (printing FAILs only)..
time=2023-11-13T14:23:46+01:00 level=info msg=[binding] |200|  GET /hub/importsummaries
time=2023-11-13T14:23:46+01:00 level=info msg=[binding] |200|  GET /hub/applications
    api_test.go:45: Mismatch in number of imported Applications: Expected 3, Actual 4
time=2023-11-13T14:23:46+01:00 level=info msg=[binding] |200|  GET /hub/dependencies
time=2023-11-13T14:23:46+01:00 level=info msg=[binding] |200|  GET /hub/importsummaries/1
time=2023-11-13T14:23:46+01:00 level=info msg=[binding] |200|  GET /hub/imports
time=2023-11-13T14:23:46+01:00 level=info msg=[binding] |200|  GET /hub/importsummaries/download
time=2023-11-13T14:23:46+01:00 level=info msg=[binding] |204|  DELETE /hub/importsummaries/1
time=2023-11-13T14:23:46+01:00 level=info msg=[binding] |204|  DELETE /hub/applications/3
time=2023-11-13T14:23:46+01:00 level=info msg=[binding] |204|  DELETE /hub/stakeholders/2
time=2023-11-13T14:23:46+01:00 level=info msg=[binding] |204|  DELETE /hub/applications/4
time=2023-11-13T14:23:46+01:00 level=info msg=[binding] |204|  DELETE /hub/stakeholders/2
time=2023-11-13T14:23:46+01:00 level=info msg=[binding] |204|  DELETE /hub/stakeholders/3
time=2023-11-13T14:23:46+01:00 level=info msg=[binding] |204|  DELETE /hub/applications/5
time=2023-11-13T14:23:46+01:00 level=info msg=[binding] |204|  DELETE /hub/stakeholders/2
time=2023-11-13T14:23:46+01:00 level=info msg=[binding] |204|  DELETE /hub/stakeholders/2
time=2023-11-13T14:23:46+01:00 level=info msg=[binding] |204|  DELETE /hub/stakeholders/3
time=2023-11-13T14:23:46+01:00 level=info msg=[binding] |204|  DELETE /hub/applications/6
time=2023-11-13T14:23:46+01:00 level=info msg=[binding] |204|  DELETE /hub/dependencies/1
time=2023-11-13T14:23:46+01:00 level=info msg=[binding] |204|  DELETE /hub/dependencies/2
--- FAIL: TestImportCSV (1.12s)
    --- FAIL: TestImportCSV/template_application_import.csv (1.12s)
FAIL
FAIL	github.com/konveyor/tackle2-hub/test/api/importcsv	1.162s
--
--- PASS: TestJobFunctionSeed (0.01s)
PASS
ok  	github.com/konveyor/tackle2-hub/test/api/jobfunction	0.557s
time=2023-11-13T14:23:45+01:00 level=info msg=[binding] Hub RichClient created.
time=2023-11-13T14:23:45+01:00 level=info msg=[binding] |201|  POST /hub/auth/login
=== RUN   TestMigrationWaveCRUD
time=2023-11-13T14:23:45+01:00 level=info msg=[binding] |201|  POST /hub/applications
time=2023-11-13T14:23:45+01:00 level=info msg=[binding] |201|  POST /hub/stakeholders
time=2023-11-13T14:23:45+01:00 level=info msg=[binding] |201|  POST /hub/stakeholdergroups
time=2023-11-13T14:23:45+01:00 level=info msg=[binding] |201|  POST /hub/migrationwaves
time=2023-11-13T14:23:45+01:00 level=info msg=[binding] |200|  GET /hub/migrationwaves/1
    api_test.go:193: Expected Wave not found in the list: {1 Sample Stakeholders}
    api_test.go:205: Expected Wave not found in the list: {1 Sample Stakeholders Groups}
time=2023-11-13T14:23:45+01:00 level=info msg=[binding] |204|  PUT /hub/migrationwaves/1
time=2023-11-13T14:23:45+01:00 level=info msg=[binding] |200|  GET /hub/migrationwaves/1
time=2023-11-13T14:23:45+01:00 level=info msg=[binding] |204|  DELETE /hub/migrationwaves/1
time=2023-11-13T14:23:45+01:00 level=info msg=[binding] |204|  DELETE /hub/stakeholdergroups/2
time=2023-11-13T14:23:45+01:00 level=info msg=[binding] |204|  DELETE /hub/stakeholders/2
time=2023-11-13T14:23:45+01:00 level=info msg=[binding] |204|  DELETE /hub/applications/5
time=2023-11-13T14:23:45+01:00 level=info msg=[binding] |404|  GET /hub/migrationwaves/1
--- FAIL: TestMigrationWaveCRUD (0.38s)
=== RUN   TestMigrationWaveList
time=2023-11-13T14:23:45+01:00 level=info msg=[binding] |201|  POST /hub/applications
time=2023-11-13T14:23:45+01:00 level=info msg=[binding] |201|  POST /hub/stakeholders
time=2023-11-13T14:23:45+01:00 level=info msg=[binding] |201|  POST /hub/stakeholdergroups
time=2023-11-13T14:23:45+01:00 level=info msg=[binding] |201|  POST /hub/migrationwaves
time=2023-11-13T14:23:45+01:00 level=info msg=[binding] |200|  GET /hub/migrationwaves
time=2023-11-13T14:23:45+01:00 level=info msg=[binding] |204|  DELETE /hub/migrationwaves/1
time=2023-11-13T14:23:45+01:00 level=info msg=[binding] |204|  DELETE /hub/stakeholdergroups/4
time=2023-11-13T14:23:45+01:00 level=info msg=[binding] |204|  DELETE /hub/stakeholders/2
time=2023-11-13T14:23:45+01:00 level=info msg=[binding] |204|  DELETE /hub/applications/14
--- PASS: TestMigrationWaveList (0.20s)
FAIL
FAIL	github.com/konveyor/tackle2-hub/test/api/migrationwave	0.618s
--
PASS
ok  	github.com/konveyor/tackle2-hub/test/api/ruleset	0.567s
time=2023-11-13T14:23:45+01:00 level=info msg=[binding] Hub RichClient created.
time=2023-11-13T14:23:45+01:00 level=info msg=[binding] |201|  POST /hub/auth/login
=== RUN   TestStakeholderCRUD
=== RUN   TestStakeholderCRUD/Alice
time=2023-11-13T14:23:45+01:00 level=info msg=[binding] |201|  POST /hub/stakeholders
time=2023-11-13T14:23:45+01:00 level=info msg=[binding] |200|  GET /hub/stakeholders/1
time=2023-11-13T14:23:45+01:00 level=info msg=[binding] |204|  PUT /hub/stakeholders/1
time=2023-11-13T14:23:45+01:00 level=info msg=[binding] |200|  GET /hub/stakeholders/1
time=2023-11-13T14:23:45+01:00 level=info msg=[binding] |204|  DELETE /hub/stakeholders/1
time=2023-11-13T14:23:45+01:00 level=info msg=[binding] |200|  GET /hub/stakeholders/1
    api_test.go:50: Resource exits, but should be deleted: {{1 admin.noauth  2023-11-13 13:23:45.39150266 +0000 UTC} Updated Alice alice@acme.local [] [] <nil> [] [] []}
=== RUN   TestStakeholderCRUD/Bob
time=2023-11-13T14:23:45+01:00 level=info msg=[binding] |201|  POST /hub/stakeholders
time=2023-11-13T14:23:45+01:00 level=info msg=[binding] |200|  GET /hub/stakeholders/3
time=2023-11-13T14:23:45+01:00 level=info msg=[binding] |204|  PUT /hub/stakeholders/3
time=2023-11-13T14:23:45+01:00 level=info msg=[binding] |200|  GET /hub/stakeholders/3
time=2023-11-13T14:23:45+01:00 level=info msg=[binding] |204|  DELETE /hub/stakeholders/3
time=2023-11-13T14:23:45+01:00 level=info msg=[binding] |404|  GET /hub/stakeholders/3
--- FAIL: TestStakeholderCRUD (0.40s)
    --- FAIL: TestStakeholderCRUD/Alice (0.21s)
    --- PASS: TestStakeholderCRUD/Bob (0.19s)
=== RUN   TestStakeholderList
time=2023-11-13T14:23:45+01:00 level=info msg=[binding] |201|  POST /hub/stakeholders
time=2023-11-13T14:23:45+01:00 level=info msg=[binding] |201|  POST /hub/stakeholders
time=2023-11-13T14:23:45+01:00 level=info msg=[binding] |200|  GET /hub/stakeholders
time=2023-11-13T14:23:45+01:00 level=info msg=[binding] |204|  DELETE /hub/stakeholders/3
time=2023-11-13T14:23:45+01:00 level=info msg=[binding] |204|  DELETE /hub/stakeholders/4
--- PASS: TestStakeholderList (0.12s)
FAIL
FAIL	github.com/konveyor/tackle2-hub/test/api/stakeholder	0.560s
time=2023-11-13T14:23:45+01:00 level=info msg=[binding] Hub RichClient created.
time=2023-11-13T14:23:45+01:00 level=info msg=[binding] |201|  POST /hub/auth/login
=== RUN   TestStakeholderGroupCRUD
=== RUN   TestStakeholderGroupCRUD/Mgmt
time=2023-11-13T14:23:45+01:00 level=info msg=[binding] |201|  POST /hub/stakeholdergroups
time=2023-11-13T14:23:45+01:00 level=info msg=[binding] |200|  GET /hub/stakeholdergroups/1
time=2023-11-13T14:23:45+01:00 level=info msg=[binding] |204|  PUT /hub/stakeholdergroups/1
time=2023-11-13T14:23:45+01:00 level=info msg=[binding] |200|  GET /hub/stakeholdergroups/1
time=2023-11-13T14:23:45+01:00 level=info msg=[binding] |204|  DELETE /hub/stakeholdergroups/1
time=2023-11-13T14:23:45+01:00 level=info msg=[binding] |200|  GET /hub/stakeholdergroups/1
    api_test.go:50: Resource exits, but should be deleted: {{1 admin.noauth  2023-11-13 13:23:45.391976305 +0000 UTC} Updated Mgmt Management stakeholder group. [] []}
=== RUN   TestStakeholderGroupCRUD/Engineering
time=2023-11-13T14:23:45+01:00 level=info msg=[binding] |201|  POST /hub/stakeholdergroups
time=2023-11-13T14:23:45+01:00 level=info msg=[binding] |200|  GET /hub/stakeholdergroups/3
time=2023-11-13T14:23:45+01:00 level=info msg=[binding] |204|  PUT /hub/stakeholdergroups/3
time=2023-11-13T14:23:45+01:00 level=info msg=[binding] |200|  GET /hub/stakeholdergroups/3
time=2023-11-13T14:23:45+01:00 level=info msg=[binding] |204|  DELETE /hub/stakeholdergroups/3
time=2023-11-13T14:23:45+01:00 level=info msg=[binding] |404|  GET /hub/stakeholdergroups/3
--- FAIL: TestStakeholderGroupCRUD (0.40s)
    --- FAIL: TestStakeholderGroupCRUD/Mgmt (0.22s)
    --- PASS: TestStakeholderGroupCRUD/Engineering (0.19s)
=== RUN   TestStakeholderGroupList
time=2023-11-13T14:23:45+01:00 level=info msg=[binding] |201|  POST /hub/stakeholdergroups
time=2023-11-13T14:23:45+01:00 level=info msg=[binding] |201|  POST /hub/stakeholdergroups
time=2023-11-13T14:23:45+01:00 level=info msg=[binding] |200|  GET /hub/stakeholdergroups
time=2023-11-13T14:23:45+01:00 level=info msg=[binding] |204|  DELETE /hub/stakeholdergroups/2
time=2023-11-13T14:23:45+01:00 level=info msg=[binding] |204|  DELETE /hub/stakeholdergroups/3
--- PASS: TestStakeholderGroupList (0.12s)
FAIL
FAIL	github.com/konveyor/tackle2-hub/test/api/stakeholdergroup	0.558s
--
time=2023-11-13T14:23:46+01:00 level=info msg=[binding] |201|  POST /hub/tasks
time=2023-11-13T14:23:46+01:00 level=info msg=[binding] |200|  GET /hub/tasks
time=2023-11-13T14:23:46+01:00 level=info msg=[binding] |204|  DELETE /hub/tasks/1
--- PASS: TestTaskList (0.00s)
=== RUN   TestTaskBucket
time=2023-11-13T14:23:46+01:00 level=info msg=[binding] |201|  POST /hub/tasks
time=2023-11-13T14:23:46+01:00 level=info msg=[binding] |200|  GET /hub/tasks/1/bucket
time=2023-11-13T14:23:46+01:00 level=info msg=[binding] |204|  DELETE /hub/tasks/1
--- PASS: TestTaskBucket (0.00s)
PASS
ok  	github.com/konveyor/tackle2-hub/test/api/task	0.034s
time=2023-11-13T14:23:46+01:00 level=info msg=[binding] Hub RichClient created.
time=2023-11-13T14:23:46+01:00 level=info msg=[binding] |201|  POST /hub/auth/login
time=2023-11-13T14:23:46+01:00 level=info msg=[binding] Hub RichClient created.
time=2023-11-13T14:23:46+01:00 level=info msg=[binding] |201|  POST /hub/auth/login
=== RUN   TestTicketCRUD
=== RUN   TestTicketCRUD/Ticket_10001_CRUD
time=2023-11-13T14:23:46+01:00 level=info msg=[binding] |201|  POST /hub/applications
time=2023-11-13T14:23:46+01:00 level=info msg=[binding] |409|  POST /hub/identities
    error.go:21: POST /hub/identities failed: 409(Conflict) body: {"error":"UNIQUE constraint failed: Identity.Name"}
--- FAIL: TestTicketCRUD (0.01s)
    --- FAIL: TestTicketCRUD/Ticket_10001_CRUD (0.01s)
=== RUN   TestTicketList
time=2023-11-13T14:23:46+01:00 level=info msg=[binding] |409|  POST /hub/applications
    error.go:21: POST /hub/applications failed: 409(Conflict) body: {"error":"UNIQUE constraint failed: Application.Name"}
--- FAIL: TestTicketList (0.00s)
FAIL
FAIL	github.com/konveyor/tackle2-hub/test/api/ticket	0.029s
--
time=2023-11-13T14:23:46+01:00 level=info msg=[binding] |204|  DELETE /hub/identities/1
time=2023-11-13T14:23:46+01:00 level=info msg=[binding] |404|  GET /hub/trackers/1
--- PASS: TestTrackerCRUD (0.04s)
    --- PASS: TestTrackerCRUD/Tracker_jira-onprem_CRUD (0.01s)
    --- PASS: TestTrackerCRUD/Tracker_jira-onprem_Project (0.01s)
    --- PASS: TestTrackerCRUD/Tracker_jira-cloud_CRUD (0.01s)
    --- PASS: TestTrackerCRUD/Tracker_jira-cloud_Project (0.01s)
=== RUN   TestTrackerList
time=2023-11-13T14:23:46+01:00 level=info msg=[binding] |201|  POST /hub/identities
time=2023-11-13T14:23:46+01:00 level=info msg=[binding] |201|  POST /hub/trackers
time=2023-11-13T14:23:46+01:00 level=info msg=[binding] |201|  POST /hub/identities
time=2023-11-13T14:23:46+01:00 level=info msg=[binding] |201|  POST /hub/trackers
time=2023-11-13T14:23:46+01:00 level=info msg=[binding] |200|  GET /hub/trackers
time=2023-11-13T14:23:46+01:00 level=info msg=[binding] |204|  DELETE /hub/trackers/1
time=2023-11-13T14:23:46+01:00 level=info msg=[binding] |204|  DELETE /hub/identities/1
time=2023-11-13T14:23:46+01:00 level=info msg=[binding] |204|  DELETE /hub/trackers/2
time=2023-11-13T14:23:46+01:00 level=info msg=[binding] |204|  DELETE /hub/identities/2
--- PASS: TestTrackerList (0.01s)
PASS
ok  	github.com/konveyor/tackle2-hub/test/api/tracker	0.064s
FAIL
make[1]: *** [Makefile:138: test-api] Error 1

@aufi aufi marked this pull request as draft November 13, 2023 13:57
@aufi
Copy link
Member Author

aufi commented Nov 13, 2023

Doing further experiments with exit code.

@aufi
Forcing bash to propagate error exit code
2831db6 
Signed-off-by: Marek Aufart <maufart@redhat.com>
@aufi aufi marked this pull request as ready for review November 13, 2023 14:44
@aufi
Revert "Forcing bash to propagate error exit code"
524562f 
This reverts commit 2831db6.

Signed-off-by: Marek Aufart <maufart@redhat.com>
@aufi aufi marked this pull request as draft November 15, 2023 11:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pranavgaikwad pranavgaikwad Awaiting requested review from pranavgaikwad

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Reduce verbosity of logs in CI runs
2 participants
@aufi @pranavgaikwad