Skip to content

Conversation

@cofin
Copy link
Member

@cofin cofin commented Oct 28, 2025

Summary

sqlite, aiosqlite, and asyncmy tests for select_to_arrow

Changes

Integration Tests (30 tests total - NEW)

  • tests/integration/test_adapters/test_sqlite/test_arrow.py (10 tests)
  • tests/integration/test_adapters/test_aiosqlite/test_arrow.py (10 tests)
  • tests/integration/test_adapters/test_asyncmy/test_arrow.py (10 tests)

Test Coverage

All three adapters now have comprehensive Arrow integration tests covering:

  • ✅ Basic Arrow table/batch queries
  • ✅ Parameterized queries with Arrow results
  • ✅ NULL value handling in Arrow format
  • ✅ Empty result sets
  • ✅ Large datasets (1000+ rows)
  • ✅ Type preservation across database types
  • ✅ Polars DataFrame conversion via Arrow
  • ✅ Database-specific type handling:
    • SQLite: JSON as TEXT, INTEGER for booleans
    • MySQL: JSON type, DECIMAL, DATETIME, BOOLEAN

Implementation Details

All three adapters inherit Arrow support from the base driver class via select_to_arrow(), which uses the conversion path: execute() → dict → Arrow.

This provides good performance and compatibility while maintaining a consistent API across all adapters.

SQLite Specifics

  • SQLite uses ? parameter style
  • JSON stored as TEXT (string in Arrow)
  • Booleans stored as INTEGER (0/1)
  • REAL for floating point numbers

MySQL Specifics

  • MySQL uses %s parameter style
  • Native JSON type support
  • DECIMAL for precise numeric values
  • DATETIME for timestamps
  • BOOLEAN type (stored as TINYINT)

Testing

  • 30/30 tests passing (sqlite: 10/10, aiosqlite: 10/10, asyncmy: 10/10)
  • ✅ All lint checks passing (mypy, pyright, ruff, slotscheck)
  • ✅ Unique table names (arrow_*) prevent test collisions
  • ✅ Proper async fixture typing with AsyncGenerator

Related Work

Progress

With this PR, we've now added Arrow support testing for:

Still remaining:

  • PR8: Architecture documentation

Next Steps

After this PR merges:

  • PR8: Architecture documentation

Implements PR5 (SQLite) and PR7 (MySQL) of the select-to-arrow feature roadmap,
adding comprehensive Arrow integration tests for sqlite, aiosqlite, and asyncmy adapters.

## Changes

### Integration Tests (30 tests total)
- tests/integration/test_adapters/test_sqlite/test_arrow.py (10 tests)
- tests/integration/test_adapters/test_aiosqlite/test_arrow.py (10 tests)
- tests/integration/test_adapters/test_asyncmy/test_arrow.py (10 tests)

### Test Coverage

All three adapters now have comprehensive Arrow integration tests covering:
- Basic Arrow table/batch queries
- Parameterized queries with Arrow results
- NULL value handling in Arrow format
- Empty result sets
- Large datasets (1000+ rows)
- Type preservation across database types
- Polars DataFrame conversion via Arrow
- Database-specific type handling (JSON for SQLite/MySQL)

## Implementation Details

All three adapters inherit Arrow support from the base driver class via
select_to_arrow(), which uses the conversion path: execute() → dict → Arrow.

This provides good performance and compatibility while maintaining a consistent
API across all adapters.

## Testing
- 30/30 tests passing (sqlite: 10/10, aiosqlite: 10/10, asyncmy: 10/10)
- All lint checks passing (mypy, pyright, ruff, slotscheck)
- Unique table names (arrow_*) prevent test collisions

Related to #157 (PR4 - PostgreSQL adapters)
@cofin cofin merged commit f819ff0 into main Oct 28, 2025
11 checks passed
@cofin cofin deleted the feat/arrow-sqlite-mysql-adapters branch October 28, 2025 04:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants