Data Import Commands

Data Import Commands

These commands are the first step in any CRISP-T workflow. They bring your raw data—text files, PDFs, or CSVs—into the CRISP-T environment as a "Corpus".

All import commands use the main CLI tool: crisp.

Basic Import

The most common command reads all files from a source directory and saves them as a processed corpus in an output directory.

crisp --source <source_path> --out <output_path>

• --source <path>: The directory containing your raw files (.txt, .pdf, .csv). It can also be a URL.
• --out <path>: The directory where the processed corpus (corpus.json and corpus_df.csv) will be saved. Always specify this.

Example

crisp --source ./raw_interviews --out ./crisp_input

Importing Mixed Data (Text + CSV)

If you have a CSV file that contains free-text columns (e.g., survey responses) alongside numeric data, you must tell CRISP-T which columns should be treated as "unstructured" text documents.

crisp --source <source_path> --unstructured "column_name" --out <output_path>

• --unstructured <col>: The name of the column in your CSV that contains the text you want to analyze as documents. You can use this flag multiple times for multiple columns.

Example

# Import a survey CSV where the "comments" column contains the text
crisp --source ./survey_data --unstructured "comments" --out ./survey_corpus

Special Import Sources

COVID-19 Narratives

Download and import COVID-19 narratives directly from a public Omeka repository.

crisp --covid <url> --source <download_dir>

• --covid <url>: The URL of the Omeka site (e.g., covidstories.omeka.net).
• --source <path>: The directory where the downloaded files will be saved.

Advanced Import Options

Flag	Description	Example
--sources <path>	Import from multiple source directories at once. Use standard --source, --num, --rec limits for each.	crisp --sources ./dir1 --sources ./dir2
--num <int>	Limit the number of text/PDF files to import. Useful for testing on large datasets.	crisp --source ./data --num 10
--rec <int>	Limit the number of CSV rows to import.	crisp --source ./data --rec 50
--ignore <text>	Comma-separated list of words or columns to exclude during import.	crisp --source ./data --ignore "interviewer,id"

Common Questions

Q: Do I need to convert PDFs to text first? A: No, CRISP-T handles PDF extraction automatically during import.

Q: Can I update an existing corpus? A: crisp --source creates a new corpus. To modify an existing one (e.g., add single documents), use the Corpus Manipulation Commands (crispt).