Skip to content


Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?

Latest commit


Git stats


Failed to load latest commit information.
Latest commit message
Commit time



A parallel file system crawler that manages file and directory metadata in a (postgres) database.



  • Python 3.5

Python Modules:

  • configargparse
  • psycopg2

External Dependencies:

  • a postgres DB and use with create table and drop table rights

Current State

Currently the main program is working to crawl a given path. It is reasonably efficient, but still has some issues with error handling and process clean-up.

Basic Workflow

The main process creates two tables in the DB based on the --tag command line argument. The --tag is turned into a schema, and the tables and report account are created in that schema:

  • *tag*.files - table contains the actual file metadata and full path with filename
  • *tag*.status - a status log of the crawl

Once set up, directory walking processes (walkers), file stating processes (staters) and DB writing processes (dbprocs) are spawned up to the specified number. These processes use queues to pass paths around to each other - a file queue, a directory queue, and a db queue.

Specifically, the walkers pop from the dir queue, and push onto the dir queue and the file queue. The staters pop from the file queue and push onto the db queue, and the db procs pop from the db queue and insert into the DB.

When the dir queue is empty, the walkers are stopped. Then the file queue, and finally the db queue and the crawl is complete.


The basic idea is that by treating each file atomically, the stat process will not get bogged down in a single directory or file location. No guarantees, though!


The *tag*.files table:

             id SERIAL,
             insert_time timestamp with time zone DEFAULT now () NOT NULL,
             path bytea NOT NULL,
             extension bytea NOT NULL,
             st_mode bit(19) NOT NULL,
             st_ino bigint NOT NULL,
             st_dev text NOT NULL,
             st_nlink int NOT NULL,
             st_uid bigint NOT NULL,
             st_gid bigint NOT NULL,
             st_size bigint NOT NULL,
             st_atime int NOT NULL,
             st_mtime int NOT NULL,
             st_ctime int NOT NULL,
             owner text

The *tag*_status table:

             id SERIAL,
             time timestamp with time zone DEFAULT now() NOT NULL,
             status text NOT NULL,
             value int,
             units text


parallel file system crawler that manages file metadata in a postgres database







No releases published


No packages published