✨ transfer MySQL views as native SQLite views #110
Conversation
WalkthroughAdds sqlglot as a dependency; a CLI flag to materialise MySQL views as tables; new types and plumbing to control view handling; sqlglot-based view translation and SQLite view creation with fallbacks and reconnect/error handling; and unit tests covering CLI, transporter, translation and reconnect paths. Changes
Sequence Diagram(s)sequenceDiagram
autonumber
participant CLI
participant Transporter
participant MySQL
participant SQLite
participant Parser as sqlglot
CLI->>Transporter: transfer(views_as_views=True|False)
Transporter->>MySQL: Query TABLE_NAME, TABLE_TYPE
MySQL-->>Transporter: [("users","BASE TABLE"), ("active_users","VIEW")]
alt Base table OR views_as_views=False
Transporter->>Transporter: _create_table(name)
Transporter->>SQLite: CREATE TABLE ...
Transporter->>MySQL: SELECT * FROM users
MySQL-->>Transporter: [rows]
Transporter->>SQLite: INSERT INTO users ...
else View AND views_as_views=True
Transporter->>MySQL: Fetch VIEW_DEFINITION (information_schema or SHOW CREATE VIEW)
MySQL-->>Transporter: view_def_sql
Transporter->>Parser: parse_one(view_def_sql, read="mysql")
alt parse success
Parser-->>Transporter: AST
Transporter->>Parser: to_sql(dialect="sqlite")
Parser-->>Transporter: converted_select_sql
else parse fails
Transporter->>Transporter: fallback wrap SELECT
end
Transporter->>SQLite: CREATE VIEW IF NOT EXISTS ... AS (converted_select_sql)
end
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Suggested reviewers
Poem
Pre-merge checks and finishing touches❌ Failed checks (2 warnings)
✅ Passed checks (1 passed)
✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #110 +/- ##
==========================================
+ Coverage 94.11% 94.43% +0.31%
==========================================
Files 8 8
Lines 680 773 +93
==========================================
+ Hits 640 730 +90
- Misses 40 43 +3 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (10)
requirements_dev.txt (1)
20-20: Pin or cap sqlglot for reproducibility.sqlglot output can change subtly across minor releases and break string-based tests. Prefer an upper cap (e.g., <28) or pin the exact version used by CI.
Apply one of:
-sqlglot>=27.27.0 +sqlglot>=27.27.0,<28or pin exactly in CI. As per coding guidelines.
pyproject.toml (1)
49-49: Guard against sqlglot formatting churn.To avoid unexpected test breakage when sqlglot changes formatting, cap the major range.
- "sqlglot>=27.27.0", + "sqlglot>=27.27.0,<28",Also ensure requirements_dev.txt matches. As per coding guidelines.
tests/unit/test_transporter.py (1)
13-39: Good: verifies view path skips data transfer.Covers the views_as_views=True branch well.
Add a complementary test where views_as_views=False to assert _create_table is called and data transfer occurs. As per coding guidelines.
tests/unit/test_cli_views_flag.py (1)
10-39: Flag wiring tests look solid.Asserts both exit code and propagated kwarg views_as_views=False with --mysql-views-as-tables.
Add a default-path test (no -T) asserting views_as_views=True to lock in the default behaviour. As per coding guidelines.
tests/unit/test_views_sqlglot.py (2)
9-24: Covers schema stripping and SQLite dialect emission.Good assertions; note that exact identifier formatting can vary across sqlglot versions.
Relax brittle string checks to tolerate formatting, e.g. assert no backticks and presence of FROM "users" OR use a regex that's less strict on whitespace/aliasing.
25-42: Tidy unused-args in boom to satisfy linters.Avoid ARG001 by discarding args.
- def boom(*args, **kwargs): + def boom(*_, **__): raise ParseError("boom")src/mysql_to_sqlite3/transporter.py (4)
657-688: Transpile and de-qualify logic looks good.Whitespace normalisation, MySQL read, SQLite write, and schema qualifier stripping via exp.Table are all sensible.
Consider an optional normalisation to quote identifiers consistently (e.g. tree = tree.transform(...) or formatter options) to reduce string-diff noise across sqlglot upgrades. As per coding guidelines.
753-777: Prefer logging.exception for richer diagnostics; executescript optional.When persisting the view, execute is fine for a single statement; if you keep the trailing semicolon, executescript also works. Use logging.exception to include tracebacks.
- self._logger.error( + self._logger.exception( "MySQL failed reading view definition from view %s: %s", view_name, err, ) @@ - self._logger.error("SQLite failed creating view %s: %s", view_name, err) + self._logger.exception("SQLite failed creating view %s: %s", view_name, err)
896-918: Create VIEW vs TABLE branching is correct.Skips table creation when VIEW and views_as_views=True.
Improve the log message to reflect object type.
- self._logger.info( - "%s%sTransferring table %s", + self._logger.info( + "%s%sTransferring %s %s", "[WITHOUT DATA] " if self._without_data else "", "[ONLY DATA] " if self._without_tables else "", - table_name, + "view" if table_type == "VIEW" else "table", + table_name, )
131-137: Use sqlite_version_info tuple for version checks.String comparison can misorder versions (e.g., "3.10.0" vs "3.9.9"). Prefer sqlite_version_info.
- if self._sqlite_strict and sqlite3.sqlite_version < "3.37.0": + if self._sqlite_strict and sqlite3.sqlite_version_info < (3, 37, 0):Add/adjust unit tests to patch sqlite3.sqlite_version_info. As per coding guidelines.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (8)
pyproject.toml(1 hunks)requirements_dev.txt(1 hunks)src/mysql_to_sqlite3/cli.py(3 hunks)src/mysql_to_sqlite3/transporter.py(6 hunks)src/mysql_to_sqlite3/types.py(2 hunks)tests/unit/test_cli_views_flag.py(1 hunks)tests/unit/test_transporter.py(1 hunks)tests/unit/test_views_sqlglot.py(1 hunks)
🧰 Additional context used
📓 Path-based instructions (8)
tests/**
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
Use pytest; keep tests under tests/ mirroring src/mysql_to_sqlite3/ structure
Files:
tests/unit/test_cli_views_flag.pytests/unit/test_transporter.pytests/unit/test_views_sqlglot.py
tests/unit/test_*.py
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
tests/unit/test_*.py: Add unit tests in tests/unit/ for isolated helpers; create targeted files named test_.py
Add at least one unit test covering new flag behavior/validation
Add unit tests for new type mappings in _translate_type_from_mysql_to_sqlite
Add unit tests for new default expression translations
Files:
tests/unit/test_cli_views_flag.pytests/unit/test_transporter.pytests/unit/test_views_sqlglot.py
{src,tests}/**/*.py
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
Adhere to lint suite: Black line length 120 and isort profile=black import ordering for all Python files
Files:
tests/unit/test_cli_views_flag.pysrc/mysql_to_sqlite3/types.pytests/unit/test_transporter.pysrc/mysql_to_sqlite3/cli.pysrc/mysql_to_sqlite3/transporter.pytests/unit/test_views_sqlglot.py
src/mysql_to_sqlite3/types.py
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
src/mysql_to_sqlite3/types.py: Keep type hints exhaustive in types.py and update when MySQLtoSQLite constructor kwargs change
When constructor kwargs change, update types.py accordingly for mypy
Update typing in types.py when new CLI parameters are added to the constructor
Files:
src/mysql_to_sqlite3/types.py
src/**/*.py
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
Keep mypy passing (Python 3.9 baseline); avoid untyped dynamic attributes in source code
Files:
src/mysql_to_sqlite3/types.pysrc/mysql_to_sqlite3/cli.pysrc/mysql_to_sqlite3/transporter.py
src/mysql_to_sqlite3/cli.py
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
src/mysql_to_sqlite3/cli.py: Add new user-facing CLI options only in src/mysql_to_sqlite3/cli.py (Click command entrypoint)
Enforce strict option conflict handling in CLI (e.g., --mysql-tables vs --exclude-mysql-tables; disallow --without-tables with --without-data)
When adding new flags, define @click.option in cli.py and keep mutually exclusive logic consistent
Fail fast on invalid configuration in CLI using click.ClickException or click.UsageError
Files:
src/mysql_to_sqlite3/cli.py
src/mysql_to_sqlite3/{cli.py,transporter.py}
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
src/mysql_to_sqlite3/{cli.py,transporter.py}: Never log raw passwords
Support --skip-ssl to disable MySQL SSL; default to encrypted where possible and do not silently change defaults
Files:
src/mysql_to_sqlite3/cli.pysrc/mysql_to_sqlite3/transporter.py
src/mysql_to_sqlite3/transporter.py
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
_
src/mysql_to_sqlite3/transporter.py: Implement and modify core transfer logic (schema introspection, batching, type/default translation, reconnection handling) in transporter.py
Convert AUTO_INCREMENT single primary keys to INTEGER PRIMARY KEY AUTOINCREMENT only when the translated type is integer; log a warning otherwise
Avoid index naming collisions: if an index name equals its table name or --prefix-indices is set, prefix with
Centralize default value translation in _translate_default_from_mysql_to_sqlite; extend this function for new MySQL constructs
Handle JSON columns: if SQLite has JSON1, map MySQL JSON to JSON; otherwise map to TEXT unless --json-as-text is set
Skip foreign key generation when a table subset restriction is applied
Use _setup_logger for logging; do not instantiate new loggers ad hoc
For large tables, prefer chunked transfer using fetchmany(self._chunk_size) and executemany inserts
On CR_SERVER_LOST during schema or data transfer, attempt a single reconnect; preserve this behavior
Disable PRAGMA foreign_keys during bulk load and re-enable in finally; ensure early returns still re-enable
Use INSERT OR IGNORE to handle potential duplicate inserts
Thread new CLI parameters through the MySQLtoSQLite constructor
Raise ValueError in constructor for invalid configuration
When --debug is not set, swallow and log errors; when --debug is set, re-raise for stack inspection
Extend _translate_type_from_mysql_to_sqlite when adding type mappings and add corresponding tests
Extend _translate_default_from_mysql_to_sqlite to support new default expressions and add tests
Centralize progress/UI changes around tqdm and honor the --quiet flagFiles:
src/mysql_to_sqlite3/transporter.py🧠 Learnings (13)
📓 Common learnings
Learnt from: CR PR: techouse/mysql-to-sqlite3#0 File: .github/copilot-instructions.md:0-0 Timestamp: 2025-10-18T21:08:55.926Z Learning: Applies to src/mysql_to_sqlite3/transporter.py : Extend _translate_type_from_mysql_to_sqlite when adding type mappings and add corresponding testsLearnt from: CR PR: techouse/mysql-to-sqlite3#0 File: .github/copilot-instructions.md:0-0 Timestamp: 2025-10-18T21:08:55.925Z Learning: Applies to src/mysql_to_sqlite3/transporter.py : Implement and modify core transfer logic (schema introspection, batching, type/default translation, reconnection handling) in transporter.py📚 Learning: 2025-10-18T21:08:55.926Z
Learnt from: CR PR: techouse/mysql-to-sqlite3#0 File: .github/copilot-instructions.md:0-0 Timestamp: 2025-10-18T21:08:55.926Z Learning: Applies to tests/unit/test_*.py : Add at least one unit test covering new flag behavior/validationApplied to files:
tests/unit/test_cli_views_flag.py📚 Learning: 2025-10-18T21:08:55.926Z
Learnt from: CR PR: techouse/mysql-to-sqlite3#0 File: .github/copilot-instructions.md:0-0 Timestamp: 2025-10-18T21:08:55.926Z Learning: Applies to src/mysql_to_sqlite3/types.py : Update typing in types.py when new CLI parameters are added to the constructorApplied to files:
src/mysql_to_sqlite3/types.pysrc/mysql_to_sqlite3/cli.py📚 Learning: 2025-10-18T21:08:55.925Z
Learnt from: CR PR: techouse/mysql-to-sqlite3#0 File: .github/copilot-instructions.md:0-0 Timestamp: 2025-10-18T21:08:55.925Z Learning: Applies to src/mysql_to_sqlite3/types.py : Keep type hints exhaustive in types.py and update when MySQLtoSQLite constructor kwargs changeApplied to files:
src/mysql_to_sqlite3/types.py📚 Learning: 2025-10-18T21:08:55.926Z
Learnt from: CR PR: techouse/mysql-to-sqlite3#0 File: .github/copilot-instructions.md:0-0 Timestamp: 2025-10-18T21:08:55.926Z Learning: Applies to src/mysql_to_sqlite3/transporter.py : Extend _translate_type_from_mysql_to_sqlite when adding type mappings and add corresponding testsApplied to files:
tests/unit/test_transporter.pysrc/mysql_to_sqlite3/transporter.py📚 Learning: 2025-10-18T21:08:55.925Z
Learnt from: CR PR: techouse/mysql-to-sqlite3#0 File: .github/copilot-instructions.md:0-0 Timestamp: 2025-10-18T21:08:55.925Z Learning: Applies to src/mysql_to_sqlite3/cli.py : Add new user-facing CLI options only in src/mysql_to_sqlite3/cli.py (Click command entrypoint)Applied to files:
src/mysql_to_sqlite3/cli.py📚 Learning: 2025-10-18T21:08:55.926Z
Learnt from: CR PR: techouse/mysql-to-sqlite3#0 File: .github/copilot-instructions.md:0-0 Timestamp: 2025-10-18T21:08:55.926Z Learning: Applies to src/mysql_to_sqlite3/cli.py : When adding new flags, define click.option in cli.py and keep mutually exclusive logic consistentApplied to files:
src/mysql_to_sqlite3/cli.py📚 Learning: 2025-10-18T21:08:55.926Z
Learnt from: CR PR: techouse/mysql-to-sqlite3#0 File: .github/copilot-instructions.md:0-0 Timestamp: 2025-10-18T21:08:55.926Z Learning: Applies to src/mysql_to_sqlite3/transporter.py : Thread new CLI parameters through the MySQLtoSQLite constructorApplied to files:
src/mysql_to_sqlite3/cli.pysrc/mysql_to_sqlite3/transporter.py📚 Learning: 2025-10-18T21:08:55.925Z
Learnt from: CR PR: techouse/mysql-to-sqlite3#0 File: .github/copilot-instructions.md:0-0 Timestamp: 2025-10-18T21:08:55.925Z Learning: Applies to src/mysql_to_sqlite3/cli.py : Enforce strict option conflict handling in CLI (e.g., --mysql-tables vs --exclude-mysql-tables; disallow --without-tables with --without-data)Applied to files:
src/mysql_to_sqlite3/cli.py📚 Learning: 2025-10-18T21:08:55.925Z
Learnt from: CR PR: techouse/mysql-to-sqlite3#0 File: .github/copilot-instructions.md:0-0 Timestamp: 2025-10-18T21:08:55.925Z Learning: Applies to src/mysql_to_sqlite3/transporter.py : Implement and modify core transfer logic (schema introspection, batching, type/default translation, reconnection handling) in transporter.pyApplied to files:
src/mysql_to_sqlite3/transporter.py📚 Learning: 2025-10-18T21:08:55.926Z
Learnt from: CR PR: techouse/mysql-to-sqlite3#0 File: .github/copilot-instructions.md:0-0 Timestamp: 2025-10-18T21:08:55.926Z Learning: Applies to src/mysql_to_sqlite3/transporter.py : Extend _translate_default_from_mysql_to_sqlite to support new default expressions and add testsApplied to files:
src/mysql_to_sqlite3/transporter.py📚 Learning: 2025-10-18T21:08:55.926Z
Learnt from: CR PR: techouse/mysql-to-sqlite3#0 File: .github/copilot-instructions.md:0-0 Timestamp: 2025-10-18T21:08:55.926Z Learning: Applies to src/mysql_to_sqlite3/transporter.py : Centralize default value translation in _translate_default_from_mysql_to_sqlite; extend this function for new MySQL constructsApplied to files:
src/mysql_to_sqlite3/transporter.py📚 Learning: 2025-10-18T21:08:55.926Z
Learnt from: CR PR: techouse/mysql-to-sqlite3#0 File: .github/copilot-instructions.md:0-0 Timestamp: 2025-10-18T21:08:55.926Z Learning: Applies to tests/unit/test_*.py : Add unit tests for new type mappings in _translate_type_from_mysql_to_sqliteApplied to files:
tests/unit/test_views_sqlglot.py🧬 Code graph analysis (3)
tests/unit/test_cli_views_flag.py (1)
src/mysql_to_sqlite3/transporter.py (1)
transfer(838-972)tests/unit/test_transporter.py (1)
src/mysql_to_sqlite3/transporter.py (4)
_create_view(753-776)_create_table(633-655)_transfer_table_data(778-836)transfer(838-972)tests/unit/test_views_sqlglot.py (1)
src/mysql_to_sqlite3/transporter.py (1)
_mysql_viewdef_to_sqlite(658-687)🪛 Ruff (0.14.0)
src/mysql_to_sqlite3/transporter.py
745-745: Avoid specifying long messages outside the exception class
(TRY003)
768-772: Use
logging.exceptioninstead oflogging.errorReplace with
exception(TRY400)
775-775: Use
logging.exceptioninstead oflogging.errorReplace with
exception(TRY400)
tests/unit/test_views_sqlglot.py
29-29: Unused function argument:
args(ARG001)
29-29: Unused function argument:
kwargs(ARG001)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (9)
- GitHub Check: Test (python3.11, mariadb:10.0, 1, true, 3.11)
- GitHub Check: Test (python3.9, mariadb:10.0, 1, true, 3.9)
- GitHub Check: Test (python3.10, mariadb:10.0, 1, true, 3.10)
- GitHub Check: Test (python3.9, mariadb:5.5, 1, true, 3.9)
- GitHub Check: Test (python3.13, mariadb:5.5, 1, true, 3.13)
- GitHub Check: Test (python3.12, mariadb:5.5, 1, true, 3.12)
- GitHub Check: Test (python3.11, mariadb:5.5, 1, true, 3.11)
- GitHub Check: Test (python3.10, mariadb:5.5, 1, true, 3.10)
- GitHub Check: Codacy Static Code Analysis
🔇 Additional comments (10)
src/mysql_to_sqlite3/types.py (2)
47-48: Types updated for new CLI param.views_as_views added to MySQLtoSQLiteParams — consistent with constructor threading.
85-88: Internal attribute added._views_as_views typed and aligns with transporter usage.
tests/unit/test_cli_views_flag.py (1)
40-68: Short flag coverage is appreciated.Covers -T path equivalently.
src/mysql_to_sqlite3/cli.py (3)
146-150: New flag is well-scoped and clear.Option name/short flag and help text are intuitive.
191-196: CLI signature keeps option hygiene.Param added without disrupting existing ordering; conflicts remain enforced elsewhere.
240-245: Correct inversion to preserve new default.views_as_views=not mysql_views_as_tables matches the intended default.
src/mysql_to_sqlite3/transporter.py (4)
18-20: Scoped sqlglot imports are appropriate.Minimal surface (exp, parse_one, ParseError) keeps dependencies tidy.
125-126: Sane default for views_as_views.Defaults to True; aligns with new CLI behaviour.
846-864: Fetching TABLE_TYPE for subset transfers is correct.Prevents misclassifying VIEWs when using --mysql-tables/--exclude-mysql-tables.
875-891: Robust coercion of rows from information_schema.TABLES.Handles bytes, missing type, and odd row shapes.
…ewlines in view definitions
…icks in view definitions
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
src/mysql_to_sqlite3/transporter.py (1)
453-455: Apply backtick-escaping helper to prevent SQL injection via MySQL identifiers.All four table_name interpolations in MySQL execute statements lack backtick escaping. If a table name contains backticks, this enables SQL injection or statement breakage.
Verified unescaped instances:
- Line 453:
f"SHOW COLUMNS FROM{table_name}"- Lines 929–930:
f"FROM (SELECT * FROM{table_name}LIMIT {self._limit_rows}) AStable"- Line 934:
f"SELECT COUNT(*) AStotal_recordsFROM{table_name}"- Lines 946–947:
.format(table_name=table_name, ...)with{table_name}in stringImplement the helper and apply to all four sites as suggested in the original review. Note: line 723 already demonstrates this pattern with
safe_view_name = view_name.replace("", "``")`, confirming the approach is established in the codebase.
🧹 Nitpick comments (8)
tests/unit/test_indices_prefix_and_uniqueness.py (1)
24-53: Good coverage of prefix-indices path; minor assertion brittleness.The logic exercised is solid. To reduce brittleness, avoid depending on the double space from an empty
{unique}: assert on a substring like 'INDEX IF NOT EXISTS "users_idx_name" ON "users" ("name")' instead.src/mysql_to_sqlite3/transporter.py (4)
421-447: Reduce noise: log index renames at DEBUG, not INFO.Renames may be frequent on large schemas; consider downgrading to DEBUG to keep logs clean.
- self._logger.info( + self._logger.debug( 'Index "%s" renamed to "%s" to ensure uniqueness across the SQLite database.', base_name, candidate, )
658-689: Tighten exception scope around SQL parsing; keep robust fallback.Catching Exception is broad. Prefer catching sqlglot.ParseError and ValueError; let unexpected errors surface.
- except (ParseError, ValueError, Exception): # pylint: disable=W0718 + except (ParseError, ValueError): # Fallback: return a basic CREATE VIEW using the original SELECT return f'CREATE VIEW IF NOT EXISTS "{view_name}" AS\n{cleaned_sql};'
756-781: Prefer logging.exception for richer traces; keep single-retry logic.Use exception() to capture stack traces on failures.
- self._logger.error( + self._logger.exception( "MySQL failed reading view definition from view %s: %s", view_name, err, ) @@ - except sqlite3.Error as err: - self._logger.error("SQLite failed creating view %s: %s", view_name, err) + except sqlite3.Error as err: + self._logger.exception("SQLite failed creating view %s: %s", view_name, err) raise
967-968: Remove no-op broad catch.
except Exception: raiseadds no value; drop it and let exceptions propagate naturally.- except Exception: # pylint: disable=W0706 - raise + # Let exceptions propagatetests/unit/test_views_build_paths_extra.py (1)
49-58: LGTM: error surfaced when definitions are unavailable.Appropriate to raise sqlite3.Error in this scenario.
Please confirm there’s a unit test covering the SHOW CREATE VIEW fallback with an “AS\nSELECT …” multi-line create statement (to exercise the DOTALL regex). If missing, I can add one.
tests/unit/test_create_view_reconnect_and_errors.py (1)
30-49: Use pytest.raises instead of manual try/except.Cleaner and satisfies TRY003 guidance.
- try: - inst._create_view("v") - except sqlite3.Error: - pass - else: - raise AssertionError("Expected sqlite3.Error to be raised") + import pytest + with pytest.raises(sqlite3.Error): + inst._create_view("v")tests/unit/test_create_and_transfer_reconnect.py (1)
46-51: Prefer pytest.raises for exception testing.The manual try/except pattern is less idiomatic than using
pytest.raises. This also addresses the static analysis hint about the long message in the AssertionError.Apply this diff to use the pytest idiom:
+ import pytest + - try: + with pytest.raises(sqlite3.Error): inst._create_table("t") - except sqlite3.Error: - pass - else: - raise AssertionError("Expected sqlite3.Error to be raised") inst._logger.error.assert_called()
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (6)
src/mysql_to_sqlite3/transporter.py(8 hunks)tests/unit/test_create_and_transfer_reconnect.py(1 hunks)tests/unit/test_create_view_reconnect_and_errors.py(1 hunks)tests/unit/test_indices_prefix_and_uniqueness.py(1 hunks)tests/unit/test_types_and_defaults_extra.py(1 hunks)tests/unit/test_views_build_paths_extra.py(1 hunks)
🧰 Additional context used
📓 Path-based instructions (6)
tests/**
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
Use pytest; keep tests under tests/ mirroring src/mysql_to_sqlite3/ structure
Files:
tests/unit/test_types_and_defaults_extra.pytests/unit/test_create_and_transfer_reconnect.pytests/unit/test_create_view_reconnect_and_errors.pytests/unit/test_views_build_paths_extra.pytests/unit/test_indices_prefix_and_uniqueness.py
tests/unit/test_*.py
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
tests/unit/test_*.py: Add unit tests in tests/unit/ for isolated helpers; create targeted files named test_.py
Add at least one unit test covering new flag behavior/validation
Add unit tests for new type mappings in _translate_type_from_mysql_to_sqlite
Add unit tests for new default expression translations
Files:
tests/unit/test_types_and_defaults_extra.pytests/unit/test_create_and_transfer_reconnect.pytests/unit/test_create_view_reconnect_and_errors.pytests/unit/test_views_build_paths_extra.pytests/unit/test_indices_prefix_and_uniqueness.py
{src,tests}/**/*.py
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
Adhere to lint suite: Black line length 120 and isort profile=black import ordering for all Python files
Files:
tests/unit/test_types_and_defaults_extra.pytests/unit/test_create_and_transfer_reconnect.pytests/unit/test_create_view_reconnect_and_errors.pytests/unit/test_views_build_paths_extra.pysrc/mysql_to_sqlite3/transporter.pytests/unit/test_indices_prefix_and_uniqueness.py
src/mysql_to_sqlite3/transporter.py
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
_
src/mysql_to_sqlite3/transporter.py: Implement and modify core transfer logic (schema introspection, batching, type/default translation, reconnection handling) in transporter.py
Convert AUTO_INCREMENT single primary keys to INTEGER PRIMARY KEY AUTOINCREMENT only when the translated type is integer; log a warning otherwise
Avoid index naming collisions: if an index name equals its table name or --prefix-indices is set, prefix with
Centralize default value translation in _translate_default_from_mysql_to_sqlite; extend this function for new MySQL constructs
Handle JSON columns: if SQLite has JSON1, map MySQL JSON to JSON; otherwise map to TEXT unless --json-as-text is set
Skip foreign key generation when a table subset restriction is applied
Use _setup_logger for logging; do not instantiate new loggers ad hoc
For large tables, prefer chunked transfer using fetchmany(self._chunk_size) and executemany inserts
On CR_SERVER_LOST during schema or data transfer, attempt a single reconnect; preserve this behavior
Disable PRAGMA foreign_keys during bulk load and re-enable in finally; ensure early returns still re-enable
Use INSERT OR IGNORE to handle potential duplicate inserts
Thread new CLI parameters through the MySQLtoSQLite constructor
Raise ValueError in constructor for invalid configuration
When --debug is not set, swallow and log errors; when --debug is set, re-raise for stack inspection
Extend _translate_type_from_mysql_to_sqlite when adding type mappings and add corresponding tests
Extend _translate_default_from_mysql_to_sqlite to support new default expressions and add tests
Centralize progress/UI changes around tqdm and honor the --quiet flagFiles:
src/mysql_to_sqlite3/transporter.pysrc/**/*.py
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
Keep mypy passing (Python 3.9 baseline); avoid untyped dynamic attributes in source code
Files:
src/mysql_to_sqlite3/transporter.pysrc/mysql_to_sqlite3/{cli.py,transporter.py}
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
src/mysql_to_sqlite3/{cli.py,transporter.py}: Never log raw passwords
Support --skip-ssl to disable MySQL SSL; default to encrypted where possible and do not silently change defaultsFiles:
src/mysql_to_sqlite3/transporter.py🧠 Learnings (8)
📚 Learning: 2025-10-18T21:08:55.926Z
Learnt from: CR PR: techouse/mysql-to-sqlite3#0 File: .github/copilot-instructions.md:0-0 Timestamp: 2025-10-18T21:08:55.926Z Learning: Applies to src/mysql_to_sqlite3/transporter.py : Extend _translate_type_from_mysql_to_sqlite when adding type mappings and add corresponding testsApplied to files:
tests/unit/test_types_and_defaults_extra.pytests/unit/test_create_and_transfer_reconnect.pytests/unit/test_views_build_paths_extra.pysrc/mysql_to_sqlite3/transporter.py📚 Learning: 2025-10-18T21:08:55.926Z
Learnt from: CR PR: techouse/mysql-to-sqlite3#0 File: .github/copilot-instructions.md:0-0 Timestamp: 2025-10-18T21:08:55.926Z Learning: Applies to src/mysql_to_sqlite3/transporter.py : Extend _translate_default_from_mysql_to_sqlite to support new default expressions and add testsApplied to files:
tests/unit/test_types_and_defaults_extra.pysrc/mysql_to_sqlite3/transporter.py📚 Learning: 2025-10-18T21:08:55.926Z
Learnt from: CR PR: techouse/mysql-to-sqlite3#0 File: .github/copilot-instructions.md:0-0 Timestamp: 2025-10-18T21:08:55.926Z Learning: Applies to tests/unit/test_*.py : Add unit tests for new type mappings in _translate_type_from_mysql_to_sqliteApplied to files:
tests/unit/test_types_and_defaults_extra.pytests/unit/test_create_and_transfer_reconnect.pytests/unit/test_create_view_reconnect_and_errors.pytests/unit/test_views_build_paths_extra.pytests/unit/test_indices_prefix_and_uniqueness.py📚 Learning: 2025-10-18T21:08:55.926Z
Learnt from: CR PR: techouse/mysql-to-sqlite3#0 File: .github/copilot-instructions.md:0-0 Timestamp: 2025-10-18T21:08:55.926Z Learning: Applies to src/mysql_to_sqlite3/transporter.py : Centralize default value translation in _translate_default_from_mysql_to_sqlite; extend this function for new MySQL constructsApplied to files:
tests/unit/test_types_and_defaults_extra.py📚 Learning: 2025-10-18T21:08:55.926Z
Learnt from: CR PR: techouse/mysql-to-sqlite3#0 File: .github/copilot-instructions.md:0-0 Timestamp: 2025-10-18T21:08:55.926Z Learning: Applies to tests/unit/test_*.py : Add unit tests for new default expression translationsApplied to files:
tests/unit/test_types_and_defaults_extra.py📚 Learning: 2025-10-18T21:08:55.926Z
Learnt from: CR PR: techouse/mysql-to-sqlite3#0 File: .github/copilot-instructions.md:0-0 Timestamp: 2025-10-18T21:08:55.926Z Learning: Applies to src/mysql_to_sqlite3/transporter.py : On CR_SERVER_LOST during schema or data transfer, attempt a single reconnect; preserve this behaviorApplied to files:
tests/unit/test_create_and_transfer_reconnect.py📚 Learning: 2025-10-18T21:08:55.925Z
Learnt from: CR PR: techouse/mysql-to-sqlite3#0 File: .github/copilot-instructions.md:0-0 Timestamp: 2025-10-18T21:08:55.925Z Learning: Applies to src/mysql_to_sqlite3/transporter.py : Implement and modify core transfer logic (schema introspection, batching, type/default translation, reconnection handling) in transporter.pyApplied to files:
tests/unit/test_create_and_transfer_reconnect.pysrc/mysql_to_sqlite3/transporter.py📚 Learning: 2025-10-18T21:08:55.926Z
Learnt from: CR PR: techouse/mysql-to-sqlite3#0 File: .github/copilot-instructions.md:0-0 Timestamp: 2025-10-18T21:08:55.926Z Learning: Applies to src/mysql_to_sqlite3/transporter.py : Avoid index naming collisions: if an index name equals its table name or --prefix-indices is set, prefix with <table>_Applied to files:
tests/unit/test_indices_prefix_and_uniqueness.py🧬 Code graph analysis (5)
tests/unit/test_types_and_defaults_extra.py (2)
src/mysql_to_sqlite3/sqlite_utils.py (1)
CollatingSequences(51-56)src/mysql_to_sqlite3/transporter.py (4)
_valid_column_type(216-217)_column_type_length(220-224)_data_type_collation_sequence(398-412)_translate_default_from_mysql_to_sqlite(294-395)tests/unit/test_create_and_transfer_reconnect.py (1)
src/mysql_to_sqlite3/transporter.py (3)
_build_create_table_sql(448-631)_create_table(633-656)_transfer_table_data(782-841)tests/unit/test_create_view_reconnect_and_errors.py (2)
src/mysql_to_sqlite3/transporter.py (2)
_build_create_view_sql(690-754)_create_view(756-780)tests/unit/mysql_to_sqlite3_test.py (1)
commit(410-411)tests/unit/test_views_build_paths_extra.py (1)
src/mysql_to_sqlite3/transporter.py (1)
_build_create_view_sql(690-754)tests/unit/test_indices_prefix_and_uniqueness.py (1)
src/mysql_to_sqlite3/transporter.py (2)
MySQLtoSQLite(44-977)_build_create_table_sql(448-631)🪛 Ruff (0.14.0)
tests/unit/test_types_and_defaults_extra.py
65-65: Unused method argument:
monkeypatch(ARG002)
tests/unit/test_create_and_transfer_reconnect.py
51-51: Avoid specifying long messages outside the exception class
(TRY003)
tests/unit/test_create_view_reconnect_and_errors.py
47-47: Avoid specifying long messages outside the exception class
(TRY003)
src/mysql_to_sqlite3/transporter.py
676-676: Do not catch blind exception:
Exception(BLE001)
748-748: Avoid specifying long messages outside the exception class
(TRY003)
772-776: Use
logging.exceptioninstead oflogging.errorReplace with
exception(TRY400)
779-779: Use
logging.exceptioninstead oflogging.errorReplace with
exception(TRY400)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (9)
- GitHub Check: Test (python3.11, mariadb:5.5, 1, true, 3.11)
- GitHub Check: Test (python3.11, mariadb:10.0, 1, true, 3.11)
- GitHub Check: Test (python3.12, mariadb:5.5, 1, true, 3.12)
- GitHub Check: Test (python3.10, mariadb:10.0, 1, true, 3.10)
- GitHub Check: Test (python3.9, mariadb:10.0, 1, true, 3.9)
- GitHub Check: Test (python3.13, mariadb:5.5, 1, true, 3.13)
- GitHub Check: Test (python3.9, mariadb:5.5, 1, true, 3.9)
- GitHub Check: Test (python3.10, mariadb:5.5, 1, true, 3.10)
- GitHub Check: Codacy Static Code Analysis
🔇 Additional comments (14)
tests/unit/test_indices_prefix_and_uniqueness.py (1)
55-86: LGTM: collision + uniqueness suffix path behaves correctly.The test accurately simulates a table-name collision and verifies prefixed, unique index naming (‘accounts_dup’).
src/mysql_to_sqlite3/transporter.py (4)
125-126: Flag plumbing looks good.new views-as-views flag is read and defaulted to True; matches the CLI intent.
153-156: Good: global index-name tracking to enforce SQLite-wide uniqueness.Internal sets/counters are a pragmatic solution to SQLite’s global namespace for index names.
562-579: Meets collision policy for index names.Prefixing when a table-name collision exists, and when --prefix-indices is set, aligns with the guideline to avoid index naming collisions; uniqueness is then ensured via _get_unique_index_name.
Based on learnings
690-755: Nice: resilient view-definition path with safe backtick escaping and robust AS-regex.The SHOW CREATE VIEW fallback now escapes backticks and matches AS across newlines; this addresses prior fragility.
tests/unit/test_types_and_defaults_extra.py (4)
11-31: LGTM: type token/length and collation applicability tests.These assert the intended parsing and collation behaviour correctly.
32-42: LGTM: default keyword mappings are asserted clearly.The parametrised checks keep this concise and robust.
43-64: LGTM: charset introducer (hex/bin) coverage for str and bytes.Good coverage of DEFAULT_GENERATED paths for BLOB-backed defaults.
70-74: LGTM: BOOLEAN defaults with version gate.Setting sqlite3.sqlite_version for the test is appropriate and verifies TRUE/FALSE forms.
tests/unit/test_views_build_paths_extra.py (1)
18-47: LGTM: exercises bytes decode failure and converter invocation.Good use of a stub converter to validate inputs and output shape.
tests/unit/test_create_view_reconnect_and_errors.py (1)
10-28: LGTM: single-retry reconnect path validated.Asserts reconnect, execute, and commit exactly once; matches implementation.
tests/unit/test_create_and_transfer_reconnect.py (3)
1-8: LGTM!All necessary imports are present and appropriately organised for testing reconnect behaviour.
10-32: LGTM!The test correctly verifies the single reconnect behaviour on CR_SERVER_LOST during table creation. The mock setup with
side_effectproperly simulates the error-then-success scenario, and assertions confirm the expected call sequence.Based on learnings.
56-78: LGTM!The test correctly verifies the single reconnect behaviour on CR_SERVER_LOST during data transfer. The mock setup properly simulates the non-chunked transfer path (_chunk_size=None), and the side_effect configuration correctly tests the error-then-success scenario for fetchall.
Based on learnings.
There was a problem hiding this comment.
Actionable comments posted: 1
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (3)
tests/unit/test_create_and_transfer_reconnect.py(1 hunks)tests/unit/test_create_view_reconnect_and_errors.py(1 hunks)tests/unit/test_types_and_defaults_extra.py(1 hunks)
✅ Files skipped from review due to trivial changes (1)
- tests/unit/test_types_and_defaults_extra.py
🚧 Files skipped from review as they are similar to previous changes (1)
- tests/unit/test_create_view_reconnect_and_errors.py
🧰 Additional context used
📓 Path-based instructions (3)
tests/**
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
Use pytest; keep tests under tests/ mirroring src/mysql_to_sqlite3/ structure
Files:
tests/unit/test_create_and_transfer_reconnect.py
tests/unit/test_*.py
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
tests/unit/test_*.py: Add unit tests in tests/unit/ for isolated helpers; create targeted files named test_.py
Add at least one unit test covering new flag behavior/validation
Add unit tests for new type mappings in _translate_type_from_mysql_to_sqlite
Add unit tests for new default expression translations
Files:
tests/unit/test_create_and_transfer_reconnect.py
{src,tests}/**/*.py
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
Adhere to lint suite: Black line length 120 and isort profile=black import ordering for all Python files
Files:
tests/unit/test_create_and_transfer_reconnect.py
🧠 Learnings (4)
📚 Learning: 2025-10-18T21:08:55.926Z
Learnt from: CR
PR: techouse/mysql-to-sqlite3#0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-10-18T21:08:55.926Z
Learning: Applies to tests/unit/test_*.py : Add unit tests for new type mappings in _translate_type_from_mysql_to_sqlite
Applied to files:
tests/unit/test_create_and_transfer_reconnect.py
📚 Learning: 2025-10-18T21:08:55.926Z
Learnt from: CR
PR: techouse/mysql-to-sqlite3#0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-10-18T21:08:55.926Z
Learning: Applies to src/mysql_to_sqlite3/transporter.py : On CR_SERVER_LOST during schema or data transfer, attempt a single reconnect; preserve this behavior
Applied to files:
tests/unit/test_create_and_transfer_reconnect.py
📚 Learning: 2025-10-18T21:08:55.926Z
Learnt from: CR
PR: techouse/mysql-to-sqlite3#0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-10-18T21:08:55.926Z
Learning: Applies to src/mysql_to_sqlite3/transporter.py : Extend _translate_type_from_mysql_to_sqlite when adding type mappings and add corresponding tests
Applied to files:
tests/unit/test_create_and_transfer_reconnect.py
📚 Learning: 2025-10-18T21:08:55.925Z
Learnt from: CR
PR: techouse/mysql-to-sqlite3#0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-10-18T21:08:55.925Z
Learning: Applies to src/mysql_to_sqlite3/transporter.py : Implement and modify core transfer logic (schema introspection, batching, type/default translation, reconnection handling) in transporter.py
Applied to files:
tests/unit/test_create_and_transfer_reconnect.py
🧬 Code graph analysis (1)
tests/unit/test_create_and_transfer_reconnect.py (2)
src/mysql_to_sqlite3/transporter.py (3)
_build_create_table_sql(448-631)_create_table(633-656)_transfer_table_data(782-841)tests/unit/mysql_to_sqlite3_test.py (2)
commit(410-411)fetchall(560-561)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (9)
- GitHub Check: Test (python3.9, mariadb:10.0, 1, true, 3.9)
- GitHub Check: Test (python3.9, mariadb:5.5, 1, true, 3.9)
- GitHub Check: Test (python3.11, mariadb:10.0, 1, true, 3.11)
- GitHub Check: Test (python3.10, mariadb:10.0, 1, true, 3.10)
- GitHub Check: Test (python3.13, mariadb:5.5, 1, true, 3.13)
- GitHub Check: Test (python3.12, mariadb:5.5, 1, true, 3.12)
- GitHub Check: Test (python3.10, mariadb:5.5, 1, true, 3.10)
- GitHub Check: Test (python3.11, mariadb:5.5, 1, true, 3.11)
- GitHub Check: Codacy Static Code Analysis
🔇 Additional comments (2)
tests/unit/test_create_and_transfer_reconnect.py (2)
11-33: LGTM! Reconnect logic correctly tested.The test accurately verifies that when
_build_create_table_sqlraises aCR_SERVER_LOSTerror, the system attempts a single reconnect and successfully creates the table on retry. The assertions correctly reflect thatexecutescriptis only invoked during the successful second attempt.
35-51: LGTM! SQLite error handling verified.The test correctly confirms that SQLite errors during table creation are both logged and propagated to the caller.
This pull request introduces support for transferring MySQL views as native SQLite views instead of materializing them as tables, controlled by a new CLI flag. It also adds dependency management for SQL transpilation and comprehensive tests for the new functionality. The main changes are grouped below.
New Feature: MySQL Views as SQLite Views
--mysql-views-as-tables(-T) CLI flag to control whether MySQL views are materialized as tables (legacy) or transferred as native SQLite views. The default is now to transfer views as views. [1] [2] [3]transporter.pyto fetch MySQL view definitions, transpile them to SQLite syntax usingsqlglot, and create corresponding SQLite views when the flag is not set. [1] [2] [3] [4] [5]views_as_viewsparameter. [1] [2]Dependency Management
sqlglotlibrary to bothpyproject.tomlandrequirements_dev.txtfor SQL dialect transpilation. [1] [2]Testing and Validation
Supersedes #63