Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

as a Dataverse Developer, I'd like the existing automated tests to run without human intervention #4812

Open
pameyer opened this issue Jul 5, 2018 · 10 comments · May be fixed by #6272

Comments

@pameyer
Copy link
Contributor

commented Jul 5, 2018

From discussion with @pdurbin (both in person and in irc); the existing integration tests run on the IQSS jenkins server still require human intervention (aka - push the button). These are composed of three dependent jobs (build, deploy, test).

Specific changes:

  • Primary improvement: Add daily annotation to jenkins job(s) config (or cron w\ jenkins CLI, but from discussions the annotation would be preferable).
  • Slight enhancement: modify the deploy job to check the commit hash from the build job with the api/info/version endpoint to ensure that the correct build has been deployed.
  • Possible enhancement: have the automation report when there are test failures. Some discussion of how (email to dataverse list, create github issue, etc), would need more discussion.

Some additional discussion of larger-scale improvements, to be deferred to later issue.

@djbrooke

This comment has been minimized.

Copy link
Contributor

commented Jul 9, 2018

Thanks @pdurbin and @pameyer! Moved to Backlog column for discussion.

@pdurbin

This comment has been minimized.

Copy link
Member

commented Jul 13, 2018

#4332 is related, a fancier way to raise awareness that tests are failing that what is suggested above.

@djbrooke djbrooke changed the title as a Dataverse contributor, I'd like the existing automated tests to run without human intervention as a Dataverse Developer, I'd like the existing automated tests to run without human intervention Jul 25, 2018
@pameyer

This comment has been minimized.

Copy link
Contributor Author

commented Oct 9, 2018

I'm wondering of doing a local cron job that automatically creates issues might be another way of addressing this.

@djbrooke djbrooke added this to Ready 🙋 in IQSS/dataverse May 8, 2019
@djbrooke djbrooke added the Medium label May 9, 2019
@djbrooke djbrooke removed the Status: Ready label May 22, 2019
@pdurbin pdurbin added this to Needs triage in pdurbin via automation Jun 14, 2019
@pdurbin pdurbin moved this from Needs triage to High priority in pdurbin Jun 14, 2019
@pdurbin pdurbin moved this from Ready 🙋 to Community Dev 💻❤️ in IQSS/dataverse Aug 14, 2019
@pdurbin

This comment has been minimized.

Copy link
Member

commented Aug 14, 2019

@donsizemore @scolapasta and I just met to talk about https://jenkins.dataverse.org and https://github.com/IQSS/dataverse-jenkins

@scolapasta and I decided to use this issue to track getting the API test suite running on that new Jenkins server.

@donsizemore

This comment has been minimized.

Copy link
Contributor

commented Aug 15, 2019

Woo woo! Tried @pdurbin's IQSS-dataverse-api-test-suite-docker-aio-develop job this morning and only one test failed:
https://jenkins.dataverse.org/job/IQSS-dataverse-api-test-suite-docker-aio-develop/14/console

Interestingly, glassfish didn't fall over — I bet I was hitting the DatasetPage bug. Will trigger a build on my old job next.

The failed test, BTW:

DataversesIT.testMoveDataverse:364 JSON path data.total_count doesn't match.
Expected: <1>
  Actual: 0
@donsizemore

This comment has been minimized.

Copy link
Contributor

commented Aug 15, 2019

@pdurbin pdurbin moved this from Community Dev 💻❤️ to IQSS Team Dev 💻 in IQSS/dataverse Aug 20, 2019
@pdurbin

This comment has been minimized.

Copy link
Member

commented Aug 20, 2019

@donsizemore and I are actively troubleshooting (for examples, Dataverse falling over with 500 errors in the middle of an API test suite run: http://irclog.iq.harvard.edu/dataverse/2019-08-19#i_102629 ) so I dragged this to to IQSS Team Dev.

@scolapasta @donsizemore and I talked about spinning up a test Jenkins server using the INSTALL.md writeup I added to https://github.com/IQSS/dataverse-jenkins but I haven't done this yet.

@donsizemore for the record, @djbrooke is fine with us spinning up EC2 instances for API testing rather than using Docker on the Jenkins server itself. I know this costs more but it's worth it. I'm also happy to continue troubleshooting docker-aio but we might want to see if @pameyer has a little time to help us.

@pdurbin pdurbin moved this from High priority for pdurbin to To do list for Don Sizemore in pdurbin Aug 21, 2019
@pdurbin

This comment has been minimized.

Copy link
Member

commented Aug 22, 2019

Good meeting with @donsizemore and @scolapasta today. As promised, I just ran the docker-aio tests on my Mac as of e093c0f which amounts to these two commands from the readme:

  • ./conf/docker-aio/prep_it.bash
  • ./conf/docker-aio/run-test-suite.sh

The full test suite passed:

[INFO] Results:
[INFO] 
[WARNING] Tests run: 97, Failures: 0, Errors: 0, Skipped: 4
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  04:55 min
[INFO] Finished at: 2019-08-22T16:53:14-04:00
[INFO] ------------------------------------------------------------------------

For the record, there's a null pointer exception showing up in server.log about OAI-ORE exports failing but we must not have an integration test for this because this failure doesn't cause the test suite to fail. We're tracking that OAI-ORE issue at #6100.

@djbrooke djbrooke moved this from IQSS Team Dev 💻 to Community Dev 💻❤️ in IQSS/dataverse Sep 18, 2019
pdurbin added a commit that referenced this issue Oct 11, 2019
Also differentiate between the following:

- API Tests on jenkins.dataverse.org
- Unit Tests on Travis
- Unit Test coverage on Coveralls
@pdurbin

This comment has been minimized.

Copy link
Member

commented Oct 11, 2019

As I mentioned at standup yesterday, there has been a major breakthrough thanks to the herculean efforts by @donsizemore in https://github.com/IQSS/dataverse-ansible and https://github.com/IQSS/dataverse-jenkins

I just created pull request #6272 to close this issue and add a button/badge/shield to the README so we can easily see if the API test suite is passing.

@djbrooke

This comment has been minimized.

Copy link
Contributor

commented Oct 11, 2019

Thank you @donsizemore !!!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
IQSS/dataverse
Community Dev 💻❤️
pdurbin
To do list for Don Sizemore
5 participants
You can’t perform that action at this time.