Skip to content

feat: prepare snowflake and cron service for snowflake data export [CM-945]#3833

Merged
mbani01 merged 7 commits intomainfrom
feat/snowflake_s3_export_job
Feb 13, 2026
Merged

feat: prepare snowflake and cron service for snowflake data export [CM-945]#3833
mbani01 merged 7 commits intomainfrom
feat/snowflake_s3_export_job

Conversation

@mbani01
Copy link
Copy Markdown
Contributor

@mbani01 mbani01 commented Feb 9, 2026

This pull request introduces a new scheduled job to export data from Snowflake to S3 and significantly refactors the Snowflake client to support programmatic access token authentication in addition to key-pair authentication. It also updates dependencies to use the latest Snowflake SDK. The changes improve flexibility, security, and automation for Snowflake data exports.

Snowflake S3 Export Job:

  • Added a new job (snowflakeS3Export.job.ts) that exports batches of data from Snowflake to S3 as Parquet files, supporting incremental exports based on the last run timestamp, and tracks progress using Redis.

Snowflake Client Refactor:

  • Refactored the SnowflakeClient to support both programmatic access token and key-pair authentication, using a new configuration interface and dynamic connection options. [1] [2]
  • Added a static fromToken method to SnowflakeClient for convenient instantiation using environment variables for token-based authentication.

Dependency Updates:

  • Updated the snowflake-sdk dependency from version ^1.14.0 to ^2.3.3 to ensure compatibility with new features and authentication methods.
  • Added @crowd/snowflake as a dependency to the cron service to enable use of the new client and job.

Note

Medium Risk
Touches data-export infrastructure and Snowflake authentication/connection setup; failures could affect scheduled exports or connectivity, though the changes are localized and additive.

Overview
Introduces a new cron-service job to export Snowflake query results to S3 (Parquet), intended to run incrementally and track progress/state between runs.

Refactors @crowd/snowflake’s SnowflakeClient to support either key-pair JWT auth or programmatic access token auth (new config union + fromToken() helper), and bumps snowflake-sdk to ^2.3.3. The cron service now depends on @crowd/snowflake and adds S3/Parquet-related dependencies needed for the export flow.

Written by Cursor Bugbot for commit 5f98fa2. This will update automatically on new commits. Configure here.

@mbani01 mbani01 self-assigned this Feb 9, 2026
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Feb 9, 2026

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

  • feat: add user authentication (CM-123)
  • feat: add user authentication (IN-123)

Projects:

  • CM: Community Data Platform
  • IN: Insights

Please add a Jira issue key to your PR title.

1 similar comment
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Feb 9, 2026

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

  • feat: add user authentication (CM-123)
  • feat: add user authentication (IN-123)

Projects:

  • CM: Community Data Platform
  • IN: Insights

Please add a Jira issue key to your PR title.

@mbani01 mbani01 changed the title feat: implement cronjob to export data from snowflake to s3 POC feat: implement cronjob to export data from snowflake to s3 POC [CM-945] Feb 9, 2026
@mbani01 mbani01 requested a review from themarolt February 10, 2026 10:59
@mbani01 mbani01 changed the title feat: implement cronjob to export data from snowflake to s3 POC [CM-945] feat: prepare snowflake and cron service for snowflake data export [CM-945] Feb 13, 2026
@mbani01 mbani01 merged commit 37d2a3e into main Feb 13, 2026
17 checks passed
@mbani01 mbani01 deleted the feat/snowflake_s3_export_job branch February 13, 2026 13:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants