Skip to content

v1.0.0

Choose a tag to compare

View all tags
@donolu donolu released this 29 Jun 22:44
v1.0.0
0415c1c

First release of jira-ingest under the pipewell organisation.

What it does

An async Python pipeline that fetches data from Jira Cloud and loads it into any SQLAlchemy-compatible database. Supports multiple output sinks including S3, Azure Blob Storage, Google Cloud Storage, and local filesystem.

Features

  • Fetches projects, boards, issues (with custom fields), and issue transitions
  • Async ingestion via aiohttp for high throughput
  • SQLAlchemy-backed loaders with dialect-aware upsert:
    • PostgreSQL and Redshift: INSERT ... ON CONFLICT DO NOTHING
    • SQLite: INSERT OR IGNORE
    • Other dialects: plain INSERT
  • Redshift fast-path: S3 COPY with COMPUPDATE OFF STATUPDATE OFF to avoid post-load analysis hangs
  • fsspec URI abstraction for sink output (S3, Azure, GCS, local)
  • Pydantic v2 settings with JIRA_ env prefix
  • CLI with --database-url, --sink-uri, and --redshift-iam-role options

Usage

pip install pipewell-jira-ingest[database]

jira-ingest fetch-all \
  --jira-url https://yourorg.atlassian.net \
  --database-url postgresql://user:pass@host/db

See the README for full configuration reference and how-to guides.

Assets 2
Loading