Skip to content
A file caching tool for machine learning and data engineering projects
Python Makefile
Branch: master
Clone or download
Latest commit c0aa7f6 Nov 18, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
hideout Add HIDEOUT_CACHE_DIR Nov 16, 2019
sample Improve log Aug 16, 2019
tests Add HIDEOUT_CACHE_DIR Nov 16, 2019
.gitignore Enhance .gitignore Jul 26, 2019
.travis.yml Apply type-check in CI Jul 26, 2019
AUTHORS.rst Init project Jan 18, 2019
LICENSE Init project Jan 18, 2019
MANIFEST.in Init project Jan 18, 2019
Makefile Add type-check target Jul 26, 2019
README.rst Update README Nov 16, 2019
requirements-dev.txt Add mypy to requirements-dev.txt Jul 26, 2019
requirements.txt Add first test Jul 15, 2019
setup.py Tiny improvement Aug 25, 2019
tox.ini Add docstring Jul 26, 2019

README.rst

Build status Supported Python version

Hideout

Data processing programs such as machine learning takes sorts of input data. After input data are loaded, program creates intermediate objects using input data. Getting or converting data takes large amount of the time. This problem prevents programmers to refactor data processing programs.

Install

We can install hideout with pip. Run the following command.

$ pip install hideout

Basic Usage

Hideout save and load object with hideout.resume. If the cache file for the object exist, hideout loads it otherwise call specified function to generate expected object.

large_object = hideout.resume_or_generate(
    label="large_object",
    func=generate_large_object,
    func_args={"source": "s3-northeast-8.amazonaws.com/large-dic.txt"}
)

hideout.resume have func_args option which contains the parameters of specified function to generate the expected object. We can specify the prefix of cache file with label option. When we do not specify the label option, resume_or_generate method automatically name the cache file from function name and the arguments.

Usage

Enable / Disable Cache

In default, Hideout is not activated and therefore does not save and load cache files. To enable cache we set the provided environment variable HIDOUT_ENABLE_CACHE to True.

$ HIDEOUT_ENBALE_CACHE=True your_data_engineering_program.py

Disable Cache for Specified Stages

Hideout provide stage for skipping caches for specified points. Users can add the stage names to the object generation by hideout.resume_or_generate with stage parameter.

large_object = hideout.resume_or_generate(
    label="large_object",
    func=generate_large_object,
    func_args={"source": "s3-northeast-8.amazonaws.com/large-dic.txt"}
)

Specifing stage names with HIDEOUT_SKIP_STAGES, hideout skip the caching. For example, the following command skip caching named preliminaries and integrate.

$ HIDEOUT_SKIP_STAGES=preliminaries,integrate your_data_engineering_program.py

Specify directory to Store Cache Files

In default, Hideout saves the cache files in caches under the top project directory. If we specify the directory, we specify it with environment variable HIDEOUT_CACHE_DIR.

Inject logger

When you want to apply the logger which you use throughout an application, you can inject the logger with hideout.set_logger() function.

For Developers

We can install the hideout package and upload it to pypi repository.

Install to local environment

$ python setup.py install

Upload to pypi

$ python setup.py sdist upload

License

MIT

Contribution

See CONTRIBUTING.md.

You can’t perform that action at this time.