Cache split in TPE for high-dimensional optimization

[nabenabe0928]

Motivation

As TPE significantly slows down for high-dimensional optimization, this PR introduces caching mechanism for the TPE split.

Description of the changes

• Cache the split information in sampler so that we can re-use the result

For n_trials=1000 and dim=10, the runtimes are the following:

This PR	Master
29	47

[not522]

What is the relationship between this PR and #5454? Should we review it after #5454?

[nabenabe0928]

@not522
These two PRs are orthogonal works, so we can separately work!
This PR aims to share the split information in a trial.
The other PR aims to share the information over multiple trials giving the same set of arguments to the hssp solver.

[not522]

@eukaryo @gen740 Could you review this PR?

[gen740]

I believe this change is not relevant to this PR.
This change would be good to include when using features from __future__.

[nabenabe0928]

Done!

[eukaryo]

Sorry, I am temporarily busy because my primary computer is not working, and I suppose @HideakiImamura -san is the appropriate reviewer. @HideakiImamura could you review this PR?

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 89.74%. Comparing base (181d65f) to head (3b61edc).
Report is 158 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #5464      +/-   ##
==========================================
+ Coverage   89.52%   89.74%   +0.22%     
==========================================
  Files         194      195       +1     
  Lines       12626    12592      -34     
==========================================
- Hits        11303    11301       -2     
+ Misses       1323     1291      -32     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[HideakiImamura]

Holding cache on the sampler instance causes the data inconsistency across different processes. Could you give me an opinion that?

[nabenabe0928]

We discussed internally and decided to close this PR.
This issue can be more or less avoided by specifying multivariate=True.

[nabenabe0928]

Just for future reminder, I will leave some comments:

This PR does not cause any issues between processes

As each trial is sampled in a thread and a sampling of a specific trial will not be scattered on multiple processes or threads, we do not have to be concerned about the missing cached data in another process.

However, when using multiple threads, we need to take care of another thread overwriting the cache before one trial completes its sampling.
For this reason, I added a buffer to store the split data up to the latest 64 trials in a thread.

Anyways, missing cache does not cause any issue because we simply need to re-calculate the split, which incurs some more computational burden.