fix(ingestion): Add aws_common dependency to cockroachdb and mariadb plugins

[askumar27]

Fix: Add missing dependencies to cockroachdb, mariadb, and sql-queries connectors

Summary

This PR fixes missing dependencies for the cockroachdb, mariadb, and sql-queries connectors, which were causing import errors when these connectors were installed.

Root Cause

The issue was introduced by PR #14899 (feat(ingest/postgres,mysql): Add iam auth support for MySql and Postgresql source), which added AWS RDS IAM authentication support to both postgres.py and mysql.py sources. This change introduced imports from datahub.ingestion.source.aws.aws_common, which requires boto3 to be installed.

Impact Chain

1. Postgres/Mysql sources added IAM auth support → now import aws_common.py → requires boto3
2. CockroachDB source extends PostgresSource → imports postgres.py → transitively requires boto3
3. MariaDB source extends MySQLSource → imports mysql.py → transitively requires boto3

However, the dependency definitions in setup.py were not updated to include aws_common for these derived connectors.

Changes Made

1. Fixed cockroachdb connector dependencies

• File: setup.py (line 461-464)
• Change: Added aws_common to dependencies
• Before: sql_common | postgres_common | {"sqlalchemy-cockroachdb<2.0.0"}
• After: sql_common | postgres_common | aws_common | {"sqlalchemy-cockroachdb<2.0.0"}

2. Fixed mariadb connector dependencies

• File: setup.py (line 546)
• Change: Added aws_common to dependencies (fixing inconsistency with comment)
• Before: sql_common | mysql (comment said "should have same dependency as mysql" but mysql had aws_common)
• After: sql_common | mysql | aws_common

3. Fixed sql-queries connector dependencies

• File: setup.py (line 574)
• Change: Added aws_common and smart-open[s3] to dependencies
• Before: usage_common | sqlglot_lib
• After: usage_common | sqlglot_lib | aws_common | {"smart-open[s3]>=5.2.1"}
• Reason: sql_queries.py directly imports smart_open for S3 file access and imports AwsConnectionConfig from aws_common

Errors Observed

Error 1: cockroachdb/mariadb

When installing cockroachdb or mariadb connectors without this fix:

ModuleNotFoundError: No module named 'boto3'
...
datahub.configuration.common.ConfigurationError: cockroachdb is disabled due to a missing dependency: boto3; try running `pip install 'acryl-datahub[cockroachdb]'`

Error 2: sql-queries

When installing sql-queries connector without this fix:

sql-queries is disabled due to a missing dependency: smart_open; try running `pip install 'acryl-datahub[sql-queries]'`

Testing

The fix ensures that when users install:

• acryl-datahub[cockroachdb] → boto3 is automatically installed
• acryl-datahub[mariadb] → boto3 is automatically installed
• acryl-datahub[sql-queries] → boto3 and smart-open[s3] are automatically installed

Related PRs

• PR feat(ingest/postgres,mysql): Add iam auth support for MySql and Postgresql source #14899: feat(ingest/postgres,mysql): Add iam auth support for MySql and Postgresql source
  • This PR introduced AWS RDS IAM authentication support
  • Added imports from aws_common.py to both postgres.py and mysql.py
  • Required boto3 dependency but only updated direct dependencies, not transitive ones

Verification

Sources importing from postgres.py or mysql.py:

• ✅ cockroachdb - fixed (imports from postgres.py)
• ✅ mariadb - fixed (imports from mysql.py)
• ✅ No other sources affected

Sources with direct imports requiring dependencies:

• ✅ sql-queries - fixed (directly imports smart_open and aws_common)
• ✅ Verified that sql_queries.py uses smart_open for S3 file access (line 284)
• ✅ Verified that sql_queries.py imports AwsConnectionConfig from aws_common (line 44)

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ All tests successful. No failed tests found.

📢 Thoughts on this report? Let us know!