Introduce pytest to help address time issues to run journey tests #142

lipemorais · 2017-09-01T22:17:41Z

What is the purpose of this Pull Request?
The purpose of this PR is use pytest to help address the long time running some journey tests usings --durations, this way we know which test takes more time to run.

What was done to achieve this purpose?
I changed way we run tests to use pytest.

How to test if it really works?
To test it you can just run pytest --verbose --durations=10 --cov-report html tests/unit after install pytest and pytest-cov using pip install pytest pytest-cov.

Who can help reviewing it?
@cuducos @jtemporal @anaschwendler

TODO

Update the new way to run the tests on README
Add a pytest.ini file

Yesterday I runned it on my machine and some tests took too long to run

==================== slowest 10 test durations =====================
4185.53s call     tests/journey/test_chamber_of_deputies_dataset.py::TestChamberOfDeputiesDataset::test_fetch_translate_clean_integration
162.20s call     tests/journey/test_federal_senate_dataset.py::TestJourneyFederalSenateDataset::test_journey_federal_senate_dataset
2.23s call     tests/journey/test_chamber_of_deputies_session_start_times_dataset.py::TestSpeechesDataset::test_fetch
1.93s call     tests/journey/test_chamber_of_deputies_official_missions_dataset.py::TestOfficialMissionsDataset::test_fetch
1.88s call     tests/journey/test_chamber_of_deputies_deputies_dataset.py::TestDeputiesDataset::test_fetch
1.66s call     tests/journey/test_chamber_of_deputies_presences_dataset.py::TestPresencesDataset::test_fetch
0.40s call     tests/journey/test_chamber_of_deputies_speeches_dataset.py::TestSpeechesDataset::test_fetch
0.00s setup    tests/journey/test_chamber_of_deputies_deputies_dataset.py::TestDeputiesDataset::test_fetch
0.00s teardown tests/journey/test_chamber_of_deputies_dataset.py::TestChamberOfDeputiesDataset::test_fetch_translate_clean_integration
0.00s teardown tests/journey/test_federal_senate_dataset.py::TestJourneyFederalSenateDataset::test_journey_federal_senate_dataset
========================= warnings summary =========================

cuducos · 2017-09-02T22:50:10Z

As usual government web API are down on weekends. Gonna re-run tests later (or on Monday)…

cuducos · 2017-09-04T14:02:44Z

@lipemorais Travis CI is still broken with an HTTP error. Meanwhile would you mind pondering on two possibilities:

Maybe document pytest usage in the README.md to assure a similar environment when developing locally and in the CI… what do you think about it?
Is the HTTP error still some external API that is broken?

lipemorais · 2017-09-05T02:24:00Z

Maybe document pytest usage in the README.md to assure a similar environment when developing locally and in the CI… what do you think about it?

I will do it soon. :)

Is the HTTP error still some external API that is broken?
I checked the url that Toolbox tries to fetch and got 404

Here is the url: http://www.camara.leg.br/cotas/Ano-2009.csv.zip

And a print of what I got:

@cuducos I will add t

cuducos · 2017-09-06T20:03:45Z

Travis is green, yay! I think all we need now is a version bump ; )

cuducos · 2017-09-06T20:10:34Z

tests/journey/test_chamber_of_deputies_dataset.py

@@ -20,7 +20,7 @@ def setUp(self):
    def tearDown(self):
        rmtree(self.path, ignore_errors=True)

-
+    @skip('Takes to long to run')


Is this just something to test the CI or is it supposed to go to master?

This is just a test, looks that this test were making build fail using all memory available on Travis.

cuducos · 2017-09-19T19:34:12Z

tests/journey/test_chamber_of_deputies_dataset.py

    def test_fetch_translate_clean_integration(self):
        self.subject.fetch()
-        files = ["Ano-{}.csv".format(n) for n in self.years]
+        files = ["Ano-{}.csv".format(n) for n in [2017]]


What about self.years[-1:]?

This was just a attempt to make this test run without use all the resources available in Travis CI and break the build because of it. Looks that worked because it's breaking for a different reason now.

lipemorais · 2017-09-20T22:35:46Z

There is a test breaking because this dataset is not available. :'(

http://www.senado.gov.br/transparencia/LAI/verba/2008.csv

lipemorais · 2017-09-21T13:48:30Z

Hey! @cuducos @jtemporal @anaschwendler do you have any idea about what can I do to fix this? :(

cuducos · 2017-09-21T15:41:23Z

What about mocking the requests and using tiny fixtures instead?

lipemorais · 2017-09-21T16:54:51Z

@cuducos this is what we do in unit version of this test.

https://github.com/datasciencebr/serenata-toolbox/blob/master/tests/unit/chambers_of_deputies/test_chamber_of_deputies_dataset.py#L42

cuducos · 2017-09-21T17:17:43Z

Yep. An IMHO opinion depending on external services to run the suites tests is a problem per se. If I'm not wrong we adopted the journey test this way (depending on external services) as a temporary fix. As this was not a permanent solution and, given that it is broken now, it might be an interesting opportunity to sor that out.

Not sure about what others @anaschwendler @jtemporal @Irio think of that though…

lipemorais · 2017-09-24T01:16:58Z

Hey, @cuducos @anaschwendler . Looks that it is working now. Could we re run travis?

http://www.senado.gov.br/transparencia/LAI/verba/2008.csv

cuducos · 2017-09-25T14:36:20Z

Sure thing! Just restarted it ; )

cuducos · 2017-09-25T15:15:14Z

Hasn't lasted long enough:

$ curl -I  http://www.camara.leg.br/cotas/Ano-2008.csv.zip
HTTP/1.1 404 Not Found

lipemorais · 2017-09-25T15:22:40Z

Looks now that http://www.camara.leg.br/cotas/Ano-2008.csv.zip is out. :'(

lipemorais · 2017-10-03T15:02:03Z

I just runned it a ALL tests and where all green. :)

GREEN BUILD

Could we rerun it or just merge?

cuducos · 2017-10-03T18:23:15Z

Yay! May we have a version bump?

…ini file

lipemorais · 2017-10-04T01:18:08Z

Yay! May we have a version bump?

Done in a7a2dc2

lipemorais force-pushed the introduce-pytest branch from 7ace0be to 43eee70 Compare September 2, 2017 12:27

cuducos reviewed Sep 6, 2017

View reviewed changes

lipemorais force-pushed the introduce-pytest branch from 9c3ff48 to 57d01c6 Compare September 19, 2017 15:42

cuducos reviewed Sep 19, 2017

View reviewed changes

Felipe Morais and others added 10 commits October 3, 2017 22:15

Introduces pytest to measure unit test time

182e8d0

Introduces pytest to measure journey test time

514daa4

Adds Travis wait time 40

bd5d0be

Adds a pytest.init to avoid repeat pytest parameters

33b3939

Remove unnecessary arguments for pytest because it already in pytest.…

dfc5d82

…ini file

Updates the way to run tests and coverage on readme

49fd99a

Skip a journey that takes too long to run

3bf85e7

Try to make this test faster using just one dataset

8a873ca

Fixes the test because it now recievies one more item in dataframe

05d709e

bumps patch version

a7a2dc2

lipemorais force-pushed the introduce-pytest branch from 3ec5879 to a7a2dc2 Compare October 4, 2017 01:16

cuducos merged commit 4e06c7a into okfn-brasil:master Oct 4, 2017

cuducos mentioned this pull request Oct 7, 2017

Could we use Pytest to help address the problems with total testing time for journey tests? #140

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce pytest to help address time issues to run journey tests #142

Introduce pytest to help address time issues to run journey tests #142

lipemorais commented Sep 1, 2017 •

edited

Loading

cuducos commented Sep 2, 2017

cuducos commented Sep 4, 2017 •

edited

Loading

lipemorais commented Sep 5, 2017

cuducos commented Sep 6, 2017

cuducos Sep 6, 2017

lipemorais Sep 7, 2017

cuducos Sep 19, 2017

lipemorais Sep 19, 2017

lipemorais commented Sep 20, 2017

lipemorais commented Sep 21, 2017

cuducos commented Sep 21, 2017

lipemorais commented Sep 21, 2017

cuducos commented Sep 21, 2017

lipemorais commented Sep 24, 2017

cuducos commented Sep 25, 2017

cuducos commented Sep 25, 2017

lipemorais commented Sep 25, 2017

lipemorais commented Oct 3, 2017

cuducos commented Oct 3, 2017

lipemorais commented Oct 4, 2017

Introduce pytest to help address time issues to run journey tests #142

Introduce pytest to help address time issues to run journey tests #142

Conversation

lipemorais commented Sep 1, 2017 • edited Loading

cuducos commented Sep 2, 2017

cuducos commented Sep 4, 2017 • edited Loading

lipemorais commented Sep 5, 2017

cuducos commented Sep 6, 2017

cuducos Sep 6, 2017

Choose a reason for hiding this comment

lipemorais Sep 7, 2017

Choose a reason for hiding this comment

cuducos Sep 19, 2017

Choose a reason for hiding this comment

lipemorais Sep 19, 2017

Choose a reason for hiding this comment

lipemorais commented Sep 20, 2017

lipemorais commented Sep 21, 2017

cuducos commented Sep 21, 2017

lipemorais commented Sep 21, 2017

cuducos commented Sep 21, 2017

lipemorais commented Sep 24, 2017

cuducos commented Sep 25, 2017

cuducos commented Sep 25, 2017

lipemorais commented Sep 25, 2017

lipemorais commented Oct 3, 2017

cuducos commented Oct 3, 2017

lipemorais commented Oct 4, 2017

lipemorais commented Sep 1, 2017 •

edited

Loading

cuducos commented Sep 4, 2017 •

edited

Loading