This package is intended for the data/document preprocessing stage in the PII Management flow designed by PIISA.
It will contain:
- a Python API and command-line entry points to read a number of file formats and convert them to PII Source Documents, as defined by pii-data
- Utilities for document transformation (to ease PII processing)
The current contents of the package are:
- Classes and an API for reading some file types:
- CSV files (into Table source documents)
- Microsoft Word files (into Sequence or Tree source documents)
- Raw text files (read plain text files into Sequence source documents or, using indentation, into Tree source documents).
- A configurable loader class thar can load formats by dispatching to appropriate subclasses
- Some command-line scripts:
- a generic script that uses the loader class to convert any implemented format to a YAML or plain text file
- scripts for specific formats:
- a script to convert between CSV files and the YAML canonical representation for Source Documents
- a script to convert between plain text files and the YAML canonical representation for Source Documents