Skip to content
A file caching tool for machine learning and data engineering projects
Python Makefile
Branch: master
Clone or download
Latest commit 07f2556 Dec 5, 2019
Type Name Latest commit message Commit time
Failed to load latest commit information.
hideout Improve logging Nov 18, 2019
sample Improve log Aug 16, 2019
tests Add HIDEOUT_CACHE_DIR Nov 16, 2019
.gitignore Enhance .gitignore Jul 26, 2019
.travis.yml Apply type-check in CI Jul 26, 2019
AUTHORS.rst Init project Jan 18, 2019
LICENSE Init project Jan 18, 2019 Init project Jan 18, 2019
Makefile Add type-check target Jul 26, 2019
README.rst Update README Nov 16, 2019
requirements-dev.txt Add mypy to requirements-dev.txt Jul 26, 2019
requirements.txt Add first test Jul 15, 2019 Tiny improvement Aug 25, 2019
tox.ini Add docstring Jul 26, 2019


Build status Supported Python version


Data processing programs such as machine learning takes sorts of input data. After input data are loaded, program creates intermediate objects using input data. Getting or converting data takes large amount of the time. This problem prevents programmers to refactor data processing programs.


We can install hideout with pip. Run the following command.

$ pip install hideout

Basic Usage

Hideout save and load object with hideout.resume. If the cache file for the object exist, hideout loads it otherwise call specified function to generate expected object.

large_object = hideout.resume_or_generate(
    func_args={"source": ""}

hideout.resume have func_args option which contains the parameters of specified function to generate the expected object. We can specify the prefix of cache file with label option. When we do not specify the label option, resume_or_generate method automatically name the cache file from function name and the arguments.


Enable / Disable Cache

In default, Hideout is not activated and therefore does not save and load cache files. To enable cache we set the provided environment variable HIDOUT_ENABLE_CACHE to True.


Disable Cache for Specified Stages

Hideout provide stage for skipping caches for specified points. Users can add the stage names to the object generation by hideout.resume_or_generate with stage parameter.

large_object = hideout.resume_or_generate(
    func_args={"source": ""}

Specifing stage names with HIDEOUT_SKIP_STAGES, hideout skip the caching. For example, the following command skip caching named preliminaries and integrate.

$ HIDEOUT_SKIP_STAGES=preliminaries,integrate

Specify directory to Store Cache Files

In default, Hideout saves the cache files in caches under the top project directory. If we specify the directory, we specify it with environment variable HIDEOUT_CACHE_DIR.

Inject logger

When you want to apply the logger which you use throughout an application, you can inject the logger with hideout.set_logger() function.

For Developers

We can install the hideout package and upload it to pypi repository.

Install to local environment

$ python install

Upload to pypi

$ python sdist upload





You can’t perform that action at this time.