[CCI migration] Disable travis tests except for RUN_FLAKE8

[cpuhrsch]

No description provided.

[mthrok]

Looks good to me.

[zhangguanheng66]

Why we want to disable the slow tests? Some datasets are covered only by slow tests.

[cpuhrsch]

@zhangguanheng66 - This is only temporary. Moto is migrating the testing infrastructure over to circleci (which already runs much faster, albeit without slow tests).

Then once it's running on CCI we can enable their distributed caching for all the ~1GB Vector files and datasets and then hopefully get rid of "slow" tests in general.

For more details see this comment on the next steps.

[zhangguanheng66]

We should get them covered by CCI ASAP.

[cpuhrsch]

A lot of the slow tests already weren't working as you can see in this log associated to the PR that removes all slow tags as an example. The issue is that the slow tests pull in multiple GBs of data from the internet and the tests time out.

[zhangguanheng66]

A lot of the slow tests already weren't working as you can see in this log associated to the PR that removes all slow tags as an example. The issue is that the slow tests pull in multiple GBs of data from the internet and the tests time out.

Typically, I expect those slow tests to finish with warnings because some of them take time to run (see bottom here). If the test fails, it means something we need to fix (correct me if I'm wrong).

[cpuhrsch]

@zhangguanheng66 - The only issue with these tests when they time out is that they, depending on the test, seem to never run at all. test_vectors_get_vecs for example seems to never run because it is taking so long. And even more so, if you remove the slow tags from the tests, it's the first one to fail whereas in the log you reference it doesn't show up anywhere except at the bottom together with the tests that all seem to have been skipped. Even though I'm a bit surprised it shows the amount of time it took to run it and otherwise doesn't mention the test.

But if this test never runs we should remove it or change it. I think if we cache the vectors it downloads the test might be able to run in a reasonable time and therefore will always run.

EDIT: When I run all tests (without slow tag) offline it takes about 20minutes the first time around and about 3minutes the second time, because it doesn't need to download the vectors. There is a total of 22GB of .vector_cache and 500MB of .data. If we can keep those around across jobs via cache (or remove the need for them), we can cut down significantly on the amount of time our jobs run to test. Of course there's value in exercising the download paths and such as well. We'll need to see what the balance here is or if there's a different way to test them.