✨ add support for SQLite's JSONB column type
#156
Conversation
WalkthroughAdds SQLite JSONB detection and conversion helpers, integrates JSONB support into the transporter (detection, type mapping, and dynamic SELECT list building using a JSON-conversion expression), and extends tests with a SQLite-version guard, a recording cursor helper, and JSONB-focused test cases. Changes
Sequence DiagramsequenceDiagram
participant T as Transporter.transfer()
participant U as sqlite_jsonb_column_expression()
participant C as check_sqlite_jsonb_support()
participant S as SQLite DB / Cursor
participant M as MySQL Insert
T->>C: check_sqlite_jsonb_support(sqlite_version) -> bool
T-->>T: set self._sqlite_jsonb_support
T->>T: _get_table_info() -> columns metadata
T->>T: _declared_type_is_jsonb() -> identify jsonb_columns
rect rgb(230,245,255)
Note over T,U: Build SELECT list
alt column is JSONB and support true
T->>U: request expression for "col"
U-->>T: Return CASE WHEN "col" IS NULL THEN NULL ELSE json("col") END AS "col"
else non-JSONB column
T->>T: quote identifier -> "col"
end
end
T->>S: Execute SELECT <dynamic select_parts>
S-->>T: Return rows (JSON text / NULL preserved)
rect rgb(245,230,255)
Note over T,M: Insert phase
T->>M: Prepare/execute INSERT(s) with fetched values
M-->>T: Insert results
end
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes
Poem
Pre-merge checks and finishing touches❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #156 +/- ##
==========================================
- Coverage 98.27% 98.05% -0.23%
==========================================
Files 8 8
Lines 1100 1129 +29
==========================================
+ Hits 1081 1107 +26
- Misses 19 22 +3 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
src/sqlite3_to_mysql/transporter.py (1)
473-531: JSONB to JSON translation needs MySQL capability gate in type translatorThe type translator at lines 527–528 returns
"JSON"unconditionally, but this should be gated onself._mysql_json_support. Without this check, the code will emitJSONtype even for MySQL < 5.7.8 or MariaDB < 10.2.3, causing table creation to fail.Although
self._mysql_json_supportis correctly initialised in__init__(line 237), it is not used in the type translator itself. The_translate_type_from_sqlite_to_mysql_legacymethod should fallback toself._mysql_text_typewhen JSON is unsupported:if data_type in {"JSONB"} or data_type.startswith("JSONB"): if self._mysql_json_support: return "JSON" return self._mysql_text_typeAdditionally, tests should verify that the fallback works correctly when
_mysql_json_supportisFalse.
🧹 Nitpick comments (4)
src/sqlite3_to_mysql/sqlite_utils.py (1)
58-60: SQLite JSONB SELECT helper looks correct; minor naming/doc nitThe CASE expression correctly preserves SQL NULL while normalising JSONB blobs via
json(...), and the quoting pattern matches_sqlite_quote_identusage in the caller. To avoid confusion, consider renamingquoted_column_name(or adjusting the docstring) to emphasise that the argument should be an escaped identifier without surrounding quotes, since the function itself adds the quotes.src/sqlite3_to_mysql/transporter.py (2)
308-314: JSONB declared-type detector is simple and robustThe
_declared_type_is_jsonbhelper is defensive aboutNoneand tolerates variants likejsonb(16)viastrip().upper().startswith("JSONB"), which matches SQLite’s flexible type declarations. If JSONB-related checks grow, you might later move this intosqlite_utilsalongside other SQLite feature helpers, but it’s fine as a static method for now.
1336-1363: JSONB-aware SELECT list construction intransfer()looks soundThe new flow — fetching table info once, filtering out hidden=1 columns, detecting JSONB via
_declared_type_is_jsonb, and buildingselect_partsthat:
- optionally prepend
rowid AS "rowid"whentransfer_rowidis true, and- emit
sqlite_jsonb_column_expression(...)for JSONB columns and a quoted identifier for others —preserves identifier escaping, keeps the column order aligned with the table metadata, and ensures a non-empty
SELECTlist when columns exist (falling back to*only in degenerate cases). This integrates cleanly with the existing logic that derives MySQL column names fromself._sqlite_cur.description.It might be worth, in a follow-up, introducing a small SQLite JSONB capability flag (similar to
_sqlite_table_xinfo_support) to short-circuit JSONB-specific handling on engines without JSONB support, and have tests reference that helper rather than duplicating the version check in test code.tests/unit/sqlite3_to_mysql_test.py (1)
506-581: JSONB transfer tests are valuable; consider a tiny lambda tidy-upThe new tests cover:
- selection of JSONB columns via the
json("payload")expression, and- end-to-end conversion of JSONB values to textual JSON while preserving SQL NULLs,
which directly exercise the new helper and transfer path. That’s exactly the sort of regression coverage this change needs.
Minor nit:
instance._sqlite_table_has_rowid = lambda table: Falsetriggers the RuffARG005warning; you can silence it by marking the argument as intentionally unused:- instance._sqlite_table_has_rowid = lambda table: False + instance._sqlite_table_has_rowid = lambda _table: FalseThis keeps linters clean without changing behaviour.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (3)
src/sqlite3_to_mysql/sqlite_utils.py(1 hunks)src/sqlite3_to_mysql/transporter.py(4 hunks)tests/unit/sqlite3_to_mysql_test.py(4 hunks)
🧰 Additional context used
📓 Path-based instructions (8)
src/sqlite3_to_mysql/{mysql_utils.py,sqlite_utils.py}
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
src/sqlite3_to_mysql/{mysql_utils.py,sqlite_utils.py}: Centralize dialect/version feature checks, type adaptation, and identifier safety in mysql_utils.py and sqlite_utils.py; add new MySQL capability gates here
Add new MySQL/SQLite capability checks in mysql_utils.py/sqlite_utils.py and keep them as dedicated helpers
Files:
src/sqlite3_to_mysql/sqlite_utils.py
src/sqlite3_to_mysql/**/*.py
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
src/sqlite3_to_mysql/**/*.py: Always wrap dynamic table/index/column names with safe_identifier_length(...) before emitting SQL
Use the class _logger for logging; do not print directly; respect the quiet flag for progress/INFO while always emitting errors
Files:
src/sqlite3_to_mysql/sqlite_utils.pysrc/sqlite3_to_mysql/transporter.py
src/**/*.py
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
src/**/*.py: Format with Black (line length 120) and isort (profile=black); adhere to Flake8’s 88-column soft cap to avoid long chained expressions on one line
Preserve and add type hints for new public interfaces
Files:
src/sqlite3_to_mysql/sqlite_utils.pysrc/sqlite3_to_mysql/transporter.py
src/sqlite3_to_mysql/transporter.py
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
src/sqlite3_to_mysql/transporter.py: Keep SQLite3toMySQL.transfer() flow: create DB (if missing) → create tables → optionally truncate → bulk insert (chunked or streamed) → then create indices → then add foreign keys
On table creation failures due to expression DEFAULTs, retry without DEFAULTs (two-pass _create_table with skip_default=True)
Initialize MySQL capability booleans (e.g., _mysql_fulltext_support, _allow_expr_defaults) early in init, and gate conditional behavior on them
Extend column type translation only in _translate_type_from_sqlite_to_mysql instead of scattering conversions
Handle DEFAULT normalization in _translate_default_for_mysql; return empty string to suppress invalid defaults
During index creation, handle duplicate names by appending numeric suffixes, unless _ignore_duplicate_keys is set to skip retries
Implement chunked transfer with cursor.fetchmany when --chunk is set; otherwise use fetchall with tqdm progress
Only add foreign keys when no table include/exclude filters are set and --without-foreign-keys is not in effect
Implement insert methods: IGNORE (default), UPDATE using ON DUPLICATE KEY UPDATE (with optional VALUES alias), and DEFAULT (no modifiers)
Expose new capability booleans from init of SQLite3toMySQL for downstream logic
Use prepared cursors (cursor(prepared=True)) and batch inserts via executemany for performance
For large transfers, prefer --chunk and preserve commit granularity in any new bulk path
Files:
src/sqlite3_to_mysql/transporter.py
src/sqlite3_to_mysql/{cli.py,transporter.py}
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
In debug mode (--debug), surface exceptions instead of swallowing them
Files:
src/sqlite3_to_mysql/transporter.py
tests/{unit,func}/**/*.py
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
Place unit tests under tests/unit and functional/CLI tests under tests/func
Files:
tests/unit/sqlite3_to_mysql_test.py
tests/**/*.py
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
Format test code with Black/isort and respect Flake8’s 88-column soft cap
Files:
tests/unit/sqlite3_to_mysql_test.py
tests/unit/**/*.py
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
Add unit tests for new behavior, isolating pure functions
Files:
tests/unit/sqlite3_to_mysql_test.py
🧠 Learnings (15)
📓 Common learnings
Learnt from: CR
Repo: techouse/sqlite3-to-mysql PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-10-23T18:16:08.709Z
Learning: Applies to src/sqlite3_to_mysql/transporter.py : Extend column type translation only in _translate_type_from_sqlite_to_mysql instead of scattering conversions
Learnt from: CR
Repo: techouse/sqlite3-to-mysql PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-10-23T18:16:08.709Z
Learning: Applies to src/sqlite3_to_mysql/{mysql_utils.py,sqlite_utils.py} : Add new MySQL/SQLite capability checks in mysql_utils.py/sqlite_utils.py and keep them as dedicated helpers
Learnt from: CR
Repo: techouse/sqlite3-to-mysql PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-10-23T18:16:08.709Z
Learning: Applies to src/sqlite3_to_mysql/transporter.py : Expose new capability booleans from __init__ of SQLite3toMySQL for downstream logic
Learnt from: CR
Repo: techouse/sqlite3-to-mysql PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-10-23T18:16:08.709Z
Learning: Applies to src/sqlite3_to_mysql/{mysql_utils.py,sqlite_utils.py} : Centralize dialect/version feature checks, type adaptation, and identifier safety in mysql_utils.py and sqlite_utils.py; add new MySQL capability gates here
📚 Learning: 2025-10-23T18:16:08.709Z
Learnt from: CR
Repo: techouse/sqlite3-to-mysql PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-10-23T18:16:08.709Z
Learning: Applies to src/sqlite3_to_mysql/{mysql_utils.py,sqlite_utils.py} : Add new MySQL/SQLite capability checks in mysql_utils.py/sqlite_utils.py and keep them as dedicated helpers
Applied to files:
src/sqlite3_to_mysql/sqlite_utils.pysrc/sqlite3_to_mysql/transporter.pytests/unit/sqlite3_to_mysql_test.py
📚 Learning: 2025-10-23T18:16:08.709Z
Learnt from: CR
Repo: techouse/sqlite3-to-mysql PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-10-23T18:16:08.709Z
Learning: Applies to src/sqlite3_to_mysql/transporter.py : Extend column type translation only in _translate_type_from_sqlite_to_mysql instead of scattering conversions
Applied to files:
src/sqlite3_to_mysql/sqlite_utils.pysrc/sqlite3_to_mysql/transporter.pytests/unit/sqlite3_to_mysql_test.py
📚 Learning: 2025-10-23T18:16:08.709Z
Learnt from: CR
Repo: techouse/sqlite3-to-mysql PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-10-23T18:16:08.709Z
Learning: Applies to src/sqlite3_to_mysql/transporter.py : Expose new capability booleans from __init__ of SQLite3toMySQL for downstream logic
Applied to files:
src/sqlite3_to_mysql/transporter.pytests/unit/sqlite3_to_mysql_test.py
📚 Learning: 2025-10-23T18:16:08.709Z
Learnt from: CR
Repo: techouse/sqlite3-to-mysql PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-10-23T18:16:08.709Z
Learning: Applies to src/sqlite3_to_mysql/{mysql_utils.py,sqlite_utils.py} : Centralize dialect/version feature checks, type adaptation, and identifier safety in mysql_utils.py and sqlite_utils.py; add new MySQL capability gates here
Applied to files:
src/sqlite3_to_mysql/transporter.pytests/unit/sqlite3_to_mysql_test.py
📚 Learning: 2025-10-23T18:16:08.709Z
Learnt from: CR
Repo: techouse/sqlite3-to-mysql PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-10-23T18:16:08.709Z
Learning: Applies to src/sqlite3_to_mysql/transporter.py : Keep SQLite3toMySQL.transfer() flow: create DB (if missing) → create tables → optionally truncate → bulk insert (chunked or streamed) → then create indices → then add foreign keys
Applied to files:
src/sqlite3_to_mysql/transporter.pytests/unit/sqlite3_to_mysql_test.py
📚 Learning: 2025-10-23T18:16:08.709Z
Learnt from: CR
Repo: techouse/sqlite3-to-mysql PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-10-23T18:16:08.709Z
Learning: Applies to src/sqlite3_to_mysql/types.py : Provide typed parameter/attribute structures in types.py for consumption via typing.Unpack in SQLite3toMySQL.__init__
Applied to files:
src/sqlite3_to_mysql/transporter.pytests/unit/sqlite3_to_mysql_test.py
📚 Learning: 2025-10-23T18:16:08.709Z
Learnt from: CR
Repo: techouse/sqlite3-to-mysql PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-10-23T18:16:08.709Z
Learning: Applies to src/sqlite3_to_mysql/transporter.py : Initialize MySQL capability booleans (e.g., _mysql_fulltext_support, _allow_expr_defaults) early in __init__, and gate conditional behavior on them
Applied to files:
src/sqlite3_to_mysql/transporter.pytests/unit/sqlite3_to_mysql_test.py
📚 Learning: 2025-10-23T18:16:08.709Z
Learnt from: CR
Repo: techouse/sqlite3-to-mysql PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-10-23T18:16:08.709Z
Learning: Applies to src/sqlite3_to_mysql/transporter.py : Implement insert methods: IGNORE (default), UPDATE using ON DUPLICATE KEY UPDATE (with optional VALUES alias), and DEFAULT (no modifiers)
Applied to files:
src/sqlite3_to_mysql/transporter.py
📚 Learning: 2025-10-23T18:16:08.709Z
Learnt from: CR
Repo: techouse/sqlite3-to-mysql PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-10-23T18:16:08.709Z
Learning: Applies to src/sqlite3_to_mysql/transporter.py : Use prepared cursors (cursor(prepared=True)) and batch inserts via executemany for performance
Applied to files:
src/sqlite3_to_mysql/transporter.pytests/unit/sqlite3_to_mysql_test.py
📚 Learning: 2025-10-23T18:16:08.709Z
Learnt from: CR
Repo: techouse/sqlite3-to-mysql PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-10-23T18:16:08.709Z
Learning: Applies to src/sqlite3_to_mysql/transporter.py : For large transfers, prefer --chunk and preserve commit granularity in any new bulk path
Applied to files:
src/sqlite3_to_mysql/transporter.py
📚 Learning: 2025-10-23T18:16:08.709Z
Learnt from: CR
Repo: techouse/sqlite3-to-mysql PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-10-23T18:16:08.709Z
Learning: Applies to src/sqlite3_to_mysql/cli.py : Define new CLI flags above cli() with consistent help text and error messages; maintain mutual exclusion patterns
Applied to files:
tests/unit/sqlite3_to_mysql_test.py
📚 Learning: 2025-10-23T18:16:08.709Z
Learnt from: CR
Repo: techouse/sqlite3-to-mysql PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-10-23T18:16:08.709Z
Learning: Applies to src/sqlite3_to_mysql/**/*.py : Always wrap dynamic table/index/column names with safe_identifier_length(...) before emitting SQL
Applied to files:
tests/unit/sqlite3_to_mysql_test.py
📚 Learning: 2025-10-23T18:16:08.709Z
Learnt from: CR
Repo: techouse/sqlite3-to-mysql PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-10-23T18:16:08.709Z
Learning: Applies to src/sqlite3_to_mysql/cli.py : Define and maintain the Click CLI in cli.py, validating mutual exclusions and implied flags for options
Applied to files:
tests/unit/sqlite3_to_mysql_test.py
📚 Learning: 2025-10-23T18:16:08.709Z
Learnt from: CR
Repo: techouse/sqlite3-to-mysql PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-10-23T18:16:08.709Z
Learning: Applies to src/sqlite3_to_mysql/cli.py : Make --sqlite-tables and --exclude-sqlite-tables mutually exclusive; setting either implies --without-foreign-keys
Applied to files:
tests/unit/sqlite3_to_mysql_test.py
🧬 Code graph analysis (2)
src/sqlite3_to_mysql/transporter.py (1)
src/sqlite3_to_mysql/sqlite_utils.py (1)
sqlite_jsonb_column_expression(58-60)
tests/unit/sqlite3_to_mysql_test.py (1)
src/sqlite3_to_mysql/transporter.py (3)
_get_table_info(301-306)transfer(1289-1450)_translate_type_from_sqlite_to_mysql(424-432)
🪛 Ruff (0.14.4)
tests/unit/sqlite3_to_mysql_test.py
572-572: Unused lambda argument: table
(ARG005)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (9)
- GitHub Check: Test (python3.12, mariadb:5.5, 1, true, 3.12)
- GitHub Check: Test (python3.10, mariadb:5.5, 1, true, 3.10)
- GitHub Check: Test (python3.11, mariadb:5.5, 1, true, 3.11)
- GitHub Check: Test (python3.14, mariadb:5.5, 1, true, 3.14)
- GitHub Check: Test (python3.9, mariadb:5.5, 1, true, 3.9)
- GitHub Check: Test (python3.10, mariadb:10.0, 1, true, 3.10)
- GitHub Check: Test (python3.13, mariadb:5.5, 1, true, 3.13)
- GitHub Check: Test (python3.9, mariadb:10.0, 1, true, 3.9)
- GitHub Check: Codacy Static Code Analysis
🔇 Additional comments (6)
src/sqlite3_to_mysql/transporter.py (1)
36-45: Import ofsqlite_jsonb_column_expressionis consistent with new JSONB flowPulling the helper from
sqlite_utilskeeps JSONB-specific SQL construction centralised instead of inlining the CASE/json() expression here, which is in line with the repo’s “centralise helpers” guidance.tests/unit/sqlite3_to_mysql_test.py (5)
27-28: Runtime flag for JSONB support is a good guard for version-dependent testsDeriving
SQLITE_SUPPORTS_JSONBfromsqlite3.sqlite_version_infoand using it to gate JSONB tests keeps the suite robust across environments where JSONB is not yet available, without affecting the main library code.
404-408: Extending_make_transfer_stubwith_get_table_infokeeps tests aligned with new transfer flowStubbing
_get_table_infoto return a single visibleTEXTcolumn ensures that tests drivingtransfer()continue to work now that the method consults table metadata to build the SELECT list. This keeps the stub consistent with the production code’s expectations.
412-430:RecordingMySQLCursorhelper cleanly captures executed SQL and batched rowsThe minimal cursor implementation (recording SQL strings and materialising
rowsintoinserted_batches) is sufficient for the JSONB integration test and keeps MySQL interactions observable without needing a real server. Nice separation of concerns between SQLite and MySQL sides in the test harness.
503-504: Adjusted identifier-escaping expectation matches the new column-wise SELECTThe assertion now expects
SELECT "c1" FROM "tbl""quote", which matches the updatedselect_listconstruction logic (per-column selection using_sqlite_quote_ident). This keeps the test aligned with the new query shape while still validating correct escaping of the SQLite table name.
663-671: JSONB→JSON type translation test directly exercises the new mapping
test_translate_type_from_sqlite_maps_jsonb_to_jsonexplicitly checks bothJSONBandjsonb(16)cases against_translate_type_from_sqlite_to_mysql, which is ideal to guard the new branch in_translate_type_from_sqlite_to_mysql_legacyfrom regressions or future refactors.
There was a problem hiding this comment.
Actionable comments posted: 1
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
src/sqlite3_to_mysql/transporter.py(4 hunks)tests/unit/sqlite3_to_mysql_test.py(4 hunks)
🧰 Additional context used
📓 Path-based instructions (7)
tests/{unit,func}/**/*.py
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
Place unit tests under tests/unit and functional/CLI tests under tests/func
Files:
tests/unit/sqlite3_to_mysql_test.py
tests/**/*.py
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
Format test code with Black/isort and respect Flake8’s 88-column soft cap
Files:
tests/unit/sqlite3_to_mysql_test.py
tests/unit/**/*.py
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
Add unit tests for new behavior, isolating pure functions
Files:
tests/unit/sqlite3_to_mysql_test.py
src/sqlite3_to_mysql/transporter.py
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
src/sqlite3_to_mysql/transporter.py: Keep SQLite3toMySQL.transfer() flow: create DB (if missing) → create tables → optionally truncate → bulk insert (chunked or streamed) → then create indices → then add foreign keys
On table creation failures due to expression DEFAULTs, retry without DEFAULTs (two-pass _create_table with skip_default=True)
Initialize MySQL capability booleans (e.g., _mysql_fulltext_support, _allow_expr_defaults) early in init, and gate conditional behavior on them
Extend column type translation only in _translate_type_from_sqlite_to_mysql instead of scattering conversions
Handle DEFAULT normalization in _translate_default_for_mysql; return empty string to suppress invalid defaults
During index creation, handle duplicate names by appending numeric suffixes, unless _ignore_duplicate_keys is set to skip retries
Implement chunked transfer with cursor.fetchmany when --chunk is set; otherwise use fetchall with tqdm progress
Only add foreign keys when no table include/exclude filters are set and --without-foreign-keys is not in effect
Implement insert methods: IGNORE (default), UPDATE using ON DUPLICATE KEY UPDATE (with optional VALUES alias), and DEFAULT (no modifiers)
Expose new capability booleans from init of SQLite3toMySQL for downstream logic
Use prepared cursors (cursor(prepared=True)) and batch inserts via executemany for performance
For large transfers, prefer --chunk and preserve commit granularity in any new bulk path
Files:
src/sqlite3_to_mysql/transporter.py
src/sqlite3_to_mysql/**/*.py
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
src/sqlite3_to_mysql/**/*.py: Always wrap dynamic table/index/column names with safe_identifier_length(...) before emitting SQL
Use the class _logger for logging; do not print directly; respect the quiet flag for progress/INFO while always emitting errors
Files:
src/sqlite3_to_mysql/transporter.py
src/**/*.py
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
src/**/*.py: Format with Black (line length 120) and isort (profile=black); adhere to Flake8’s 88-column soft cap to avoid long chained expressions on one line
Preserve and add type hints for new public interfaces
Files:
src/sqlite3_to_mysql/transporter.py
src/sqlite3_to_mysql/{cli.py,transporter.py}
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
In debug mode (--debug), surface exceptions instead of swallowing them
Files:
src/sqlite3_to_mysql/transporter.py
🧠 Learnings (15)
📓 Common learnings
Learnt from: CR
Repo: techouse/sqlite3-to-mysql PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-10-23T18:16:08.709Z
Learning: Applies to src/sqlite3_to_mysql/transporter.py : Extend column type translation only in _translate_type_from_sqlite_to_mysql instead of scattering conversions
Learnt from: CR
Repo: techouse/sqlite3-to-mysql PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-10-23T18:16:08.709Z
Learning: Applies to src/sqlite3_to_mysql/{mysql_utils.py,sqlite_utils.py} : Add new MySQL/SQLite capability checks in mysql_utils.py/sqlite_utils.py and keep them as dedicated helpers
Learnt from: CR
Repo: techouse/sqlite3-to-mysql PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-10-23T18:16:08.709Z
Learning: Applies to src/sqlite3_to_mysql/transporter.py : Expose new capability booleans from __init__ of SQLite3toMySQL for downstream logic
Learnt from: CR
Repo: techouse/sqlite3-to-mysql PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-10-23T18:16:08.709Z
Learning: Applies to src/sqlite3_to_mysql/{mysql_utils.py,sqlite_utils.py} : Centralize dialect/version feature checks, type adaptation, and identifier safety in mysql_utils.py and sqlite_utils.py; add new MySQL capability gates here
📚 Learning: 2025-10-23T18:16:08.709Z
Learnt from: CR
Repo: techouse/sqlite3-to-mysql PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-10-23T18:16:08.709Z
Learning: Applies to src/sqlite3_to_mysql/{mysql_utils.py,sqlite_utils.py} : Add new MySQL/SQLite capability checks in mysql_utils.py/sqlite_utils.py and keep them as dedicated helpers
Applied to files:
tests/unit/sqlite3_to_mysql_test.pysrc/sqlite3_to_mysql/transporter.py
📚 Learning: 2025-10-23T18:16:08.709Z
Learnt from: CR
Repo: techouse/sqlite3-to-mysql PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-10-23T18:16:08.709Z
Learning: Applies to src/sqlite3_to_mysql/{mysql_utils.py,sqlite_utils.py} : Centralize dialect/version feature checks, type adaptation, and identifier safety in mysql_utils.py and sqlite_utils.py; add new MySQL capability gates here
Applied to files:
tests/unit/sqlite3_to_mysql_test.pysrc/sqlite3_to_mysql/transporter.py
📚 Learning: 2025-10-23T18:16:08.709Z
Learnt from: CR
Repo: techouse/sqlite3-to-mysql PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-10-23T18:16:08.709Z
Learning: Applies to src/sqlite3_to_mysql/transporter.py : Extend column type translation only in _translate_type_from_sqlite_to_mysql instead of scattering conversions
Applied to files:
tests/unit/sqlite3_to_mysql_test.pysrc/sqlite3_to_mysql/transporter.py
📚 Learning: 2025-10-23T18:16:08.709Z
Learnt from: CR
Repo: techouse/sqlite3-to-mysql PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-10-23T18:16:08.709Z
Learning: Applies to src/sqlite3_to_mysql/transporter.py : Expose new capability booleans from __init__ of SQLite3toMySQL for downstream logic
Applied to files:
tests/unit/sqlite3_to_mysql_test.pysrc/sqlite3_to_mysql/transporter.py
📚 Learning: 2025-10-23T18:16:08.709Z
Learnt from: CR
Repo: techouse/sqlite3-to-mysql PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-10-23T18:16:08.709Z
Learning: Applies to src/sqlite3_to_mysql/transporter.py : Use prepared cursors (cursor(prepared=True)) and batch inserts via executemany for performance
Applied to files:
tests/unit/sqlite3_to_mysql_test.pysrc/sqlite3_to_mysql/transporter.py
📚 Learning: 2025-10-23T18:16:08.709Z
Learnt from: CR
Repo: techouse/sqlite3-to-mysql PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-10-23T18:16:08.709Z
Learning: Applies to src/sqlite3_to_mysql/transporter.py : Initialize MySQL capability booleans (e.g., _mysql_fulltext_support, _allow_expr_defaults) early in __init__, and gate conditional behavior on them
Applied to files:
tests/unit/sqlite3_to_mysql_test.pysrc/sqlite3_to_mysql/transporter.py
📚 Learning: 2025-10-23T18:16:08.709Z
Learnt from: CR
Repo: techouse/sqlite3-to-mysql PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-10-23T18:16:08.709Z
Learning: Applies to src/sqlite3_to_mysql/types.py : Provide typed parameter/attribute structures in types.py for consumption via typing.Unpack in SQLite3toMySQL.__init__
Applied to files:
tests/unit/sqlite3_to_mysql_test.pysrc/sqlite3_to_mysql/transporter.py
📚 Learning: 2025-10-23T18:16:08.709Z
Learnt from: CR
Repo: techouse/sqlite3-to-mysql PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-10-23T18:16:08.709Z
Learning: Applies to src/sqlite3_to_mysql/cli.py : Define new CLI flags above cli() with consistent help text and error messages; maintain mutual exclusion patterns
Applied to files:
tests/unit/sqlite3_to_mysql_test.py
📚 Learning: 2025-10-23T18:16:08.709Z
Learnt from: CR
Repo: techouse/sqlite3-to-mysql PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-10-23T18:16:08.709Z
Learning: Applies to src/sqlite3_to_mysql/transporter.py : Implement chunked transfer with cursor.fetchmany when --chunk is set; otherwise use fetchall with tqdm progress
Applied to files:
tests/unit/sqlite3_to_mysql_test.py
📚 Learning: 2025-10-23T18:16:08.709Z
Learnt from: CR
Repo: techouse/sqlite3-to-mysql PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-10-23T18:16:08.709Z
Learning: Applies to src/sqlite3_to_mysql/transporter.py : Keep SQLite3toMySQL.transfer() flow: create DB (if missing) → create tables → optionally truncate → bulk insert (chunked or streamed) → then create indices → then add foreign keys
Applied to files:
tests/unit/sqlite3_to_mysql_test.pysrc/sqlite3_to_mysql/transporter.py
📚 Learning: 2025-10-23T18:16:08.709Z
Learnt from: CR
Repo: techouse/sqlite3-to-mysql PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-10-23T18:16:08.709Z
Learning: Applies to src/sqlite3_to_mysql/cli.py : Define and maintain the Click CLI in cli.py, validating mutual exclusions and implied flags for options
Applied to files:
tests/unit/sqlite3_to_mysql_test.py
📚 Learning: 2025-10-23T18:16:08.709Z
Learnt from: CR
Repo: techouse/sqlite3-to-mysql PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-10-23T18:16:08.709Z
Learning: Applies to src/sqlite3_to_mysql/cli.py : Make --sqlite-tables and --exclude-sqlite-tables mutually exclusive; setting either implies --without-foreign-keys
Applied to files:
tests/unit/sqlite3_to_mysql_test.py
📚 Learning: 2025-10-23T18:16:08.709Z
Learnt from: CR
Repo: techouse/sqlite3-to-mysql PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-10-23T18:16:08.709Z
Learning: Applies to src/sqlite3_to_mysql/transporter.py : Implement insert methods: IGNORE (default), UPDATE using ON DUPLICATE KEY UPDATE (with optional VALUES alias), and DEFAULT (no modifiers)
Applied to files:
src/sqlite3_to_mysql/transporter.py
📚 Learning: 2025-10-23T18:16:08.709Z
Learnt from: CR
Repo: techouse/sqlite3-to-mysql PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-10-23T18:16:08.709Z
Learning: Applies to src/sqlite3_to_mysql/transporter.py : For large transfers, prefer --chunk and preserve commit granularity in any new bulk path
Applied to files:
src/sqlite3_to_mysql/transporter.py
🧬 Code graph analysis (2)
tests/unit/sqlite3_to_mysql_test.py (1)
src/sqlite3_to_mysql/transporter.py (5)
_get_table_info(301-306)transfer(1289-1450)_create_mysql_view(847-866)_translate_sqlite_view_definition(760-775)_translate_type_from_sqlite_to_mysql(424-432)
src/sqlite3_to_mysql/transporter.py (1)
src/sqlite3_to_mysql/sqlite_utils.py (1)
sqlite_jsonb_column_expression(58-60)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (9)
- GitHub Check: Test (python3.10, mariadb:10.0, 1, true, 3.10)
- GitHub Check: Test (python3.13, mariadb:5.5, 1, true, 3.13)
- GitHub Check: Test (python3.9, mariadb:10.0, 1, true, 3.9)
- GitHub Check: Test (python3.10, mariadb:5.5, 1, true, 3.10)
- GitHub Check: Test (python3.14, mariadb:5.5, 1, true, 3.14)
- GitHub Check: Test (python3.12, mariadb:5.5, 1, true, 3.12)
- GitHub Check: Test (python3.11, mariadb:5.5, 1, true, 3.11)
- GitHub Check: Test (python3.9, mariadb:5.5, 1, true, 3.9)
- GitHub Check: Codacy Static Code Analysis
🔇 Additional comments (10)
src/sqlite3_to_mysql/transporter.py (3)
43-43: LGTM: Import is correctly placed.The import of
sqlite_jsonb_column_expressionfromsqlite_utilsfollows the established pattern of centralising SQLite-specific helpers.Based on learnings
308-313: LGTM: JSONB detection method is sound.The static method correctly identifies JSONB column declarations by checking if the normalised type starts with "JSONB", handling None gracefully.
527-528: LGTM: JSONB type mapping follows established patterns.The translation of SQLite JSONB to MySQL JSON (when supported) or TEXT (fallback) is correctly placed in
_translate_type_from_sqlite_to_mysql_legacyand respects the existing_mysql_json_supportcapability flag.Based on learnings
tests/unit/sqlite3_to_mysql_test.py (7)
27-27: LGTM: Version constant correctly gates JSONB tests.The
SQLITE_SUPPORTS_JSONBconstant appropriately checks for SQLite 3.45+ to conditionally skip tests on older versions.
404-408: LGTM: Mock enhancement supports new transfer logic.Mocking
_get_table_infoprovides the column metadata required by the updated transfer flow that builds dynamic SELECT statements.
412-430: LGTM: Test helper facilitates verification of data conversion.The
RecordingMySQLCursorclass cleanly captures executed SQL and inserted batches, enabling precise assertions on the JSONB-to-JSON conversion behaviour.
503-503: LGTM: Assertion updated to reflect explicit column selection.The change from
SELECT *toSELECT "c1"correctly reflects the new behaviour where visible columns are listed explicitly when building the transfer SELECT statement.
506-530: LGTM: Test verifies JSONB column selection via json() function.The test confirms that columns declared as JSONB are selected using SQLite's
json()function, ensuring correct extraction of JSON data from the binary JSONB format.
532-582: LGTM: Comprehensive end-to-end JSONB conversion test with proper version gating.This test thoroughly verifies the complete JSONB transfer pipeline, including NULL preservation, using a real SQLite database. The
@pytest.mark.skipifdecorator appropriately guards against execution on incompatible SQLite versions.
663-679: LGTM: Parametrised test covers JSONB type translation scenarios.The test correctly verifies that JSONB maps to JSON when MySQL JSON support is enabled and falls back to TEXT otherwise, covering both JSONB and JSONB with parameters (e.g.,
jsonb(16)).
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (1)
src/sqlite3_to_mysql/transporter.py (1)
310-315: JSONB declared‑type detection helper is simple and robust
_declared_type_is_jsonbcorrectly normalises the declared type and treats anyJSONB...prefix as JSONB, which covers common forms likeJSONBandJSONB NOT NULLwithout overcomplicating things. You could optionally re‑use this helper in_translate_type_from_sqlite_to_mysql_legacyto avoid duplicating thestartswith("JSONB")logic, but the current approach is perfectly serviceable.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (3)
src/sqlite3_to_mysql/sqlite_utils.py(1 hunks)src/sqlite3_to_mysql/transporter.py(5 hunks)tests/unit/sqlite3_to_mysql_test.py(4 hunks)
🚧 Files skipped from review as they are similar to previous changes (2)
- tests/unit/sqlite3_to_mysql_test.py
- src/sqlite3_to_mysql/sqlite_utils.py
🧰 Additional context used
📓 Path-based instructions (4)
src/sqlite3_to_mysql/transporter.py
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
src/sqlite3_to_mysql/transporter.py: Keep SQLite3toMySQL.transfer() flow: create DB (if missing) → create tables → optionally truncate → bulk insert (chunked or streamed) → then create indices → then add foreign keys
On table creation failures due to expression DEFAULTs, retry without DEFAULTs (two-pass _create_table with skip_default=True)
Initialize MySQL capability booleans (e.g., _mysql_fulltext_support, _allow_expr_defaults) early in init, and gate conditional behavior on them
Extend column type translation only in _translate_type_from_sqlite_to_mysql instead of scattering conversions
Handle DEFAULT normalization in _translate_default_for_mysql; return empty string to suppress invalid defaults
During index creation, handle duplicate names by appending numeric suffixes, unless _ignore_duplicate_keys is set to skip retries
Implement chunked transfer with cursor.fetchmany when --chunk is set; otherwise use fetchall with tqdm progress
Only add foreign keys when no table include/exclude filters are set and --without-foreign-keys is not in effect
Implement insert methods: IGNORE (default), UPDATE using ON DUPLICATE KEY UPDATE (with optional VALUES alias), and DEFAULT (no modifiers)
Expose new capability booleans from init of SQLite3toMySQL for downstream logic
Use prepared cursors (cursor(prepared=True)) and batch inserts via executemany for performance
For large transfers, prefer --chunk and preserve commit granularity in any new bulk path
Files:
src/sqlite3_to_mysql/transporter.py
src/sqlite3_to_mysql/**/*.py
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
src/sqlite3_to_mysql/**/*.py: Always wrap dynamic table/index/column names with safe_identifier_length(...) before emitting SQL
Use the class _logger for logging; do not print directly; respect the quiet flag for progress/INFO while always emitting errors
Files:
src/sqlite3_to_mysql/transporter.py
src/**/*.py
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
src/**/*.py: Format with Black (line length 120) and isort (profile=black); adhere to Flake8’s 88-column soft cap to avoid long chained expressions on one line
Preserve and add type hints for new public interfaces
Files:
src/sqlite3_to_mysql/transporter.py
src/sqlite3_to_mysql/{cli.py,transporter.py}
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
In debug mode (--debug), surface exceptions instead of swallowing them
Files:
src/sqlite3_to_mysql/transporter.py
🧠 Learnings (10)
📓 Common learnings
Learnt from: CR
Repo: techouse/sqlite3-to-mysql PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-10-23T18:16:08.709Z
Learning: Applies to src/sqlite3_to_mysql/transporter.py : Extend column type translation only in _translate_type_from_sqlite_to_mysql instead of scattering conversions
Learnt from: CR
Repo: techouse/sqlite3-to-mysql PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-10-23T18:16:08.709Z
Learning: Applies to src/sqlite3_to_mysql/{mysql_utils.py,sqlite_utils.py} : Add new MySQL/SQLite capability checks in mysql_utils.py/sqlite_utils.py and keep them as dedicated helpers
Learnt from: CR
Repo: techouse/sqlite3-to-mysql PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-10-23T18:16:08.709Z
Learning: Applies to src/sqlite3_to_mysql/transporter.py : Expose new capability booleans from __init__ of SQLite3toMySQL for downstream logic
Learnt from: CR
Repo: techouse/sqlite3-to-mysql PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-10-23T18:16:08.709Z
Learning: Applies to src/sqlite3_to_mysql/{mysql_utils.py,sqlite_utils.py} : Centralize dialect/version feature checks, type adaptation, and identifier safety in mysql_utils.py and sqlite_utils.py; add new MySQL capability gates here
📚 Learning: 2025-10-23T18:16:08.709Z
Learnt from: CR
Repo: techouse/sqlite3-to-mysql PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-10-23T18:16:08.709Z
Learning: Applies to src/sqlite3_to_mysql/{mysql_utils.py,sqlite_utils.py} : Add new MySQL/SQLite capability checks in mysql_utils.py/sqlite_utils.py and keep them as dedicated helpers
Applied to files:
src/sqlite3_to_mysql/transporter.py
📚 Learning: 2025-10-23T18:16:08.709Z
Learnt from: CR
Repo: techouse/sqlite3-to-mysql PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-10-23T18:16:08.709Z
Learning: Applies to src/sqlite3_to_mysql/transporter.py : Extend column type translation only in _translate_type_from_sqlite_to_mysql instead of scattering conversions
Applied to files:
src/sqlite3_to_mysql/transporter.py
📚 Learning: 2025-10-23T18:16:08.709Z
Learnt from: CR
Repo: techouse/sqlite3-to-mysql PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-10-23T18:16:08.709Z
Learning: Applies to src/sqlite3_to_mysql/transporter.py : Expose new capability booleans from __init__ of SQLite3toMySQL for downstream logic
Applied to files:
src/sqlite3_to_mysql/transporter.py
📚 Learning: 2025-10-23T18:16:08.709Z
Learnt from: CR
Repo: techouse/sqlite3-to-mysql PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-10-23T18:16:08.709Z
Learning: Applies to src/sqlite3_to_mysql/{mysql_utils.py,sqlite_utils.py} : Centralize dialect/version feature checks, type adaptation, and identifier safety in mysql_utils.py and sqlite_utils.py; add new MySQL capability gates here
Applied to files:
src/sqlite3_to_mysql/transporter.py
📚 Learning: 2025-10-23T18:16:08.709Z
Learnt from: CR
Repo: techouse/sqlite3-to-mysql PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-10-23T18:16:08.709Z
Learning: Applies to src/sqlite3_to_mysql/transporter.py : Initialize MySQL capability booleans (e.g., _mysql_fulltext_support, _allow_expr_defaults) early in __init__, and gate conditional behavior on them
Applied to files:
src/sqlite3_to_mysql/transporter.py
📚 Learning: 2025-10-23T18:16:08.709Z
Learnt from: CR
Repo: techouse/sqlite3-to-mysql PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-10-23T18:16:08.709Z
Learning: Applies to src/sqlite3_to_mysql/transporter.py : Keep SQLite3toMySQL.transfer() flow: create DB (if missing) → create tables → optionally truncate → bulk insert (chunked or streamed) → then create indices → then add foreign keys
Applied to files:
src/sqlite3_to_mysql/transporter.py
📚 Learning: 2025-10-23T18:16:08.709Z
Learnt from: CR
Repo: techouse/sqlite3-to-mysql PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-10-23T18:16:08.709Z
Learning: Applies to src/sqlite3_to_mysql/types.py : Provide typed parameter/attribute structures in types.py for consumption via typing.Unpack in SQLite3toMySQL.__init__
Applied to files:
src/sqlite3_to_mysql/transporter.py
📚 Learning: 2025-10-23T18:16:08.709Z
Learnt from: CR
Repo: techouse/sqlite3-to-mysql PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-10-23T18:16:08.709Z
Learning: Applies to src/sqlite3_to_mysql/transporter.py : Implement insert methods: IGNORE (default), UPDATE using ON DUPLICATE KEY UPDATE (with optional VALUES alias), and DEFAULT (no modifiers)
Applied to files:
src/sqlite3_to_mysql/transporter.py
📚 Learning: 2025-10-23T18:16:08.709Z
Learnt from: CR
Repo: techouse/sqlite3-to-mysql PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-10-23T18:16:08.709Z
Learning: Applies to src/sqlite3_to_mysql/**/*.py : Always wrap dynamic table/index/column names with safe_identifier_length(...) before emitting SQL
Applied to files:
src/sqlite3_to_mysql/transporter.py
🧬 Code graph analysis (1)
src/sqlite3_to_mysql/transporter.py (1)
src/sqlite3_to_mysql/sqlite_utils.py (2)
check_sqlite_jsonb_support(58-61)sqlite_jsonb_column_expression(64-66)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (9)
- GitHub Check: Test (python3.9, mariadb:10.0, 1, true, 3.9)
- GitHub Check: Test (python3.10, mariadb:5.5, 1, true, 3.10)
- GitHub Check: Test (python3.11, mariadb:5.5, 1, true, 3.11)
- GitHub Check: Test (python3.10, mariadb:10.0, 1, true, 3.10)
- GitHub Check: Test (python3.12, mariadb:5.5, 1, true, 3.12)
- GitHub Check: Test (python3.13, mariadb:5.5, 1, true, 3.13)
- GitHub Check: Test (python3.14, mariadb:5.5, 1, true, 3.14)
- GitHub Check: Test (python3.9, mariadb:5.5, 1, true, 3.9)
- GitHub Check: Codacy Static Code Analysis
🔇 Additional comments (4)
src/sqlite3_to_mysql/transporter.py (4)
36-46: JSONB helpers imported cleanly and centralisedImporting
check_sqlite_jsonb_supportandsqlite_jsonb_column_expressionfromsqlite_utilskeeps feature detection and SQL expression building centralised, matching the existing pattern for SQLite helpers.As per coding guidelines
191-194: SQLite JSONB capability flag is initialised in the correct placeDeriving
self._sqlite_jsonb_supportfromcheck_sqlite_jsonb_support(self._sqlite_version)alongside_sqlite_table_xinfo_supportkeeps capability flags together and makes downstream gating straightforward.Based on learnings
529-530: JSONB → MySQL JSON/Text mapping is correctly isolated in type translationHandling
JSONBin_translate_type_from_sqlite_to_mysql_legacyand mapping it toJSONwhenself._mysql_json_supportis true (falling back to the configured text type otherwise) aligns with the guidance to keep type translation centralised in this method and gives a sensible degradation path on older MySQL/MariaDB versions.As per coding guidelines
1338-1367: JSONB‑aware SELECT construction looks correct and safely gatedThe new block that:
- pulls
table_column_infovia_get_table_info,- filters to
visible_columns(excluding onlyhidden == 1),- computes
jsonb_columnsonly whenself._sqlite_jsonb_supportis true using_declared_type_is_jsonb, and- builds
select_partswithsqlite_jsonb_column_expressionfor JSONB columns and a quoted identifier otherwise,achieves the desired behaviour of converting JSONB blobs to textual JSON while preserving NULLs, and only when the runtime is known to support JSONB. The use of
_sqlite_quote_identfor column/table names keeps the dynamically constructedSELECTsafe despite Ruff’s generic S608 warning, and therowid AS "rowid"handling integrates cleanly with the existingwith_rowidpath.As per coding guidelines
This pull request adds robust support for SQLite's
JSONBcolumn type during database transfers to MySQL. It ensures thatJSONBcolumns are properly detected, selected, and converted to MySQL'sJSONtype, preserving data integrity and null values. The PR also introduces comprehensive tests to verify the new functionality.Support for SQLite JSONB columns:
sqlite_jsonb_column_expressionto generate SQL expressions that convert SQLiteJSONBblobs to textual JSON while preserving NULLs. (src/sqlite3_to_mysql/sqlite_utils.pysrc/sqlite3_to_mysql/sqlite_utils.pyR56-R60)_declared_type_is_jsonbto detect columns declared asJSONB. (src/sqlite3_to_mysql/transporter.pysrc/sqlite3_to_mysql/transporter.pyR308-R314)JSONBcolumns using thejson()function, ensuring correct data extraction and null preservation. (src/sqlite3_to_mysql/transporter.pysrc/sqlite3_to_mysql/transporter.pyR1336-R1361)Type translation enhancements:
_translate_type_from_sqlite_to_mysql_legacyto map SQLiteJSONBtypes to MySQLJSONtypes. (src/sqlite3_to_mysql/transporter.pysrc/sqlite3_to_mysql/transporter.pyR527-R528)Testing and validation:
JSONBcolumns are selected using thejson()function and transferred as textual JSON, with nulls preserved. (tests/unit/sqlite3_to_mysql_test.pytests/unit/sqlite3_to_mysql_test.pyL475-R581)JSONBtoJSONis correct. (tests/unit/sqlite3_to_mysql_test.pytests/unit/sqlite3_to_mysql_test.pyR663-R672)JSONBcolumns, using a mock MySQL cursor to capture and assert the correct data. (tests/unit/sqlite3_to_mysql_test.py[1] [2]JSONBsupport based on SQLite version. (tests/unit/sqlite3_to_mysql_test.pytests/unit/sqlite3_to_mysql_test.pyR27-R29)These changes ensure seamless migration of
JSONBdata from SQLite to MySQL, with comprehensive test coverage to prevent regressions.Partially fixes #154