Skip to content

Latest commit

 

History

History
186 lines (126 loc) · 4.48 KB

reference.rst

File metadata and controls

186 lines (126 loc) · 4.48 KB
Raw

pfio

API Reference

File System Accessors

pfio.v2.open_url

pfio.v2.from_url

pfio.v2.lazify

pfio.v2.fs.FS

Local file system

pfio.v2.Local

HDFS (Hadoop File System)

pfio.v2.Hdfs

S3 (AWS S3)

pfio.v2.S3

Zip Archive

pfio.v2.Zip

HTTPCachedFS

pfio.v2.HTTPCachedFS

Error

pfio.v2.fs.ForkedError

Pathlib-like API

PFIO v2 API has utility tool that behaves like pathlib in Python's standard library. Paths can be manipulated like this:

from pfio.v2 import from_url
from pfio.v2.pathlib import Path

with from_url('s3://your-bucket') as s3:
  p = Path('foo', fs=s3)
  p2 = p / 'bar'
  with p2.open() as fp:
    # yields s3://your-bucket/foo/bar
    fp.read()

It tries to be compatible with pathlib.Path as much as possible, but several methods are not yet implemented.

pfio.v2.pathlib.Path

Sparse File Cache

Removed at 2.8.

Cache API

pfio.cache

PFIO provides experimental cache API to improve performance of repetitive access to the data collection.

Example

Here let us suppose we have a file that includes a list of paths to images. :

/path/to/image1.jpg
/path/to/image2.jpg
...
/path/to/imageN.jpg

The PyTorch Dataset class with using ~NaiveCache as an example can be implemented as follows. :

from pfio.cache import NaiveCache


class MyDataset(torch.utils.data.Dataset):
    def __init__(self, image_paths):
        self.paths = image_paths
        self.cache = NaiveCache(len(image_paths), do_pickle=True)

    def __len__(self):
        return len(self.paths)

    def _read_image(self, i):
        return cv2.imread(self.paths[i]).transpose(2, 0, 1)

    def __getitem__(self, i):
        x = self.cache.get_and_cache(i, self._read_image)

        # This is equivalent
        # x = self.cache.get(i)
        # if not x:
        #     x = cv2.imread(self.paths[i]).transpose(2, 0, 1)
        #     self.cache.put(i, x)

        return torch.Tensor(x)

By calling get_and_cache of the cache in __getitem__ method, it will check if the data for the specified index is already cached. If there already is, it reads the data from the cache and return, otherwise it calls the actual data loading function, add it to the cache, and return it. Therefore load the data from the storage only when necessary, which is at the first access to each data.

PFIO cache API provides ~NaiveCache, ~FileCache and ~MultiprocessFileCache. They all share the same core idea and interface. The difference is how to manage the cached data.

The ~NaiveCache keeps everything in memory, making it virtually zero overhead. The cache capacity is limited by the memory size, thus it would not be suitable for large-scale datasets.

The ~FileCache and the ~MultiprocessFileCache both store the cached data in a filesystem. The ~FileCache is designed for single-process data load. In case of parallelized data loading, which is relatively common in deep learning workloads, consider using ~MultiprocessFileCache.

Also, these file-based caches support cache data persistency. Once the cache is completely built, we can keep them as files by calling `FileCache.preserve, and we can recover the cache from the preserved files by calling :func:FileCache.preload`. This is useful when we want to reuse the cache already built in a previous workload.

Currently deletion of a data from cache is not supported.

Cache

NaiveCache

FileCache

MultiprocessFileCache

HTTPCache

Toplevel Functions in v1(deprecated)

Note

Toplevel functions will be deprecated in 2.0 and removed in 2.1. Please use V2 API instead.