Compile speed is slower than 1 month ago (13 minutes -> 22 minutes)

[luotao1]

Following are two teamcity logs:

• #6469 (24 Oct 17 16:19)
  • It costs 27 minutes totally, 22 minutes for compiler.
  • 6 minutes (16:19-16:25) for compilingWITH_GPU=False
  • 16 minutes (16:25-16:41) for compilingWITH_GPU=true
  • 4 minutes for unittest. (Total Test time (real) = 230.80 sec)
• #5019 (30 Sep 17 03:49)
  • It costs 17 minutes totally, 13 minutes for compiler.
  • 4 minutes (3:49-3:53) for compilingWITH_GPU=False
  • 9 minutes (3:53-4:02) for compilingWITH_GPU=true
  • 2 minutes for unittest. (Total Test time (real) = 137.64 sec)

From 13 minutes to 22 minutes, current compile speed is almost 1 times slower than 1 month ago. Maybe the reason is that there are a lot of ops merged now.

Hope somebody can help to make the compiler faster, which will make developing faster as well.

[luotao1]

The reason why teamcity compile both WITH_GPU=False and WITH_GPU=True is in #4476

[lcy-seso]

I felt recent days the compiling time become much slower (but I did not check exactly what happened. I just can clearly feel the changes of compiling speed becomes much slower).

I changed my newly added operator, but many source files (other operators) I did not change re-compiled. Are there any changes in the dependencies (only guess)?

[wangkuiyi]

Is this due to book chapters being added into paddle/framework/tests/test_*?

[luotao1]

Is this due to book chapters being added into paddle/framework/tests/test_*?

I think this is not the main reason, since this issue is created on Oct.24, when book chapters haven't been added in.

[wangkuiyi]

I noticed that in TeamCity log there are

[21:12:28]	251/256 Test #233: test_seq_conv ............................... Passed 19.33 sec
[21:12:30]	252/256 Test #256: test_recommender_system .....................***Failed 8.88 sec
[21:12:30]	Cache file /root/.cache/paddle/dataset/movielens/ml-1m.zip not found, downloading http://files.grouplens.org/datasets/movielens/ml-1m.zip
[21:12:30]	[ ]
[21:12:30]	[= ]
[21:12:30]	[== ]
[21:12:30]	[=== ]
[21:12:30]

which means that our book chapter tests in python/paddle/v2/framework/tests are downloading datasets.

I understand that this is because our python/datasets package, which is used by above chapter tests, cannot find the local cache of these datasets, so it downloads.

This would definitely cause slow CI tests.

[luotao1]

close due to #5712 and #5491