Find where we can speed up our tests

[chenmoneygithub]

Finally good or bad, we are at a stage our test takes longgggg to finish. While we are investigating a new testing infra to split out our nightly testing, we should try our best to identify tests we can speed up!

This is an issue opening to all contributors, I will add more details later.

[Cyber-Machine]

I'm interested.

[kanpuriyanawab]

Hi @chenmoneygithub , I think the first and foremost thing we can try is to reduce the max_sequence_length we use for our tests. I saw some tests that use 128, do you think we can reduce it any further (I think we can, because the purpose the unit tests I saw so far is to make sure the functionality is correct), do we have overfitting tests as well?
The second and obvious thing that comes to mind is reducing the intermediate and hidden dimensions of model instances we use in tests.

[soma2000-lang]

@chenmoneygithub Along with what @shivance said can we also decrease the num_layers?

[TheAthleticCoder]

Not sure how helpful this might be but, we can try using .prefetch(tf.data.AUTOTUNE). Saw this being used in the glue_benchmark example as well. I believe this help in faster batch loading and queueing which could help in making our tests go faster.

[kanpuriyanawab]

Hi @TheAthleticCoder, as far as I know, prefetch shows performance boost when we are dealing with large datasets. If I'm not wrong, it's similar to lazy loading and preprocessing of datasets as per need while training is underway.
Our tests use small hard-coded inputs, and what must be noted is that for some of the test we even hard-coded the input string (which should have been obtained as result of preprocessing). Please correct me if I'm wrong.

[kanpuriyanawab]

@mattdangerw already did most of the great job in #859 !

[mattdangerw]

I opened #867 for porting #859 specifically. Feel free to start doing the same for other models on that issue.

If people have other ideas on speeding up testing that is definitely welcome! But probably worth validating locally where they are large improvements before messaging here. You can see all test durations with pytest --durations=0 or pytest some/path --durations=0. Currently it looks like our biggest time sink is modeling testing, and particularly compilation and saving. #859 covers some of this, but may be more wins to be had here.

[amangupta201]

@chenmoneygithub we can add accelerated support for CPU and GPU

[mattdangerw]

We've recently had a good number of speedups here. I'll close this in favor of #943, which is more specific and probably how we scale our testing better.