Skip to content

Data Filtering Commands

Jump to bottom
Bell Eapen edited this page Jan 29, 2026 · 1 revision

Data Filtering Commands

Filtering allows you to focus your analysis on specific subsets of your data without permanently deleting unrelated items. You can filter by metadata, keywords, time, or connections.

All filtering is done using the --filters flag in crisp or crispt.

Basic Metadata Filtering

Filter documents or DataFrame rows by exact match on metadata fields.

Syntax: key=value or key:value.

# Only analyze documents where the category is "Health"
crisp --inp ./corpus --filters category=Health --topics

Special Link Filters

These powerful filters allow you to select data based on what it is connected to.

1. Filter Rows by Text (embedding:text, temporal:text)

"Keep only the DataFrame rows that are linked to at least one document."

  • embedding:text: Keep rows linked via embedding similarity.
  • temporal:text: Keep rows linked via time.
# Run regression on only the rows that have relevant text documents linked
crispt --inp ./corpus --filters embedding:text --out ./filtered_corpus

(Legacy shorthand: =embedding, :temporal)

2. Filter Text by Rows (embedding:df, temporal:df)

"Keep only the documents that are linked to at least one DataFrame row."

# Keep only documents that have matching temporal data events
crispt --inp ./corpus --filters temporal:df --out ./filtered_corpus

ID Filtering

Filter down to a specific document or row by its ID.

# Analyze only document with ID 12345
crisp --inp ./corpus --filters id=12345 --nlp

Syncing: To remove orphaned documents or rows (keep only those where IDs exist in both sets):

# Matches IDs across both sets and removes extras
crisp --inp ./corpus --filters id= --out ./synced

Temporal Filtering

Filter the entire corpus to a specific time range.

crispt --inp ./corpus --temporal-filter "start:end" --out ./filtered
  • Format: ISO 8601 timestamps (YYYY-MM-DD or YYYY-MM-DDTHH:MM:SS).
  • Open-ended: Leave one side empty to filter "before" or "after".

Examples

# Filter for year 2025
crispt --inp ./corpus --temporal-filter "2025-01-01:2025-12-31"

# Filter everything after June 2025
crispt --inp ./corpus --temporal-filter "2025-06-01:"

Clone this wiki locally