chore(sqlite-out): eradicate SQLite vocabulary from scitex-python (58 -> 0) - #362
Open
ywatanabe1989 wants to merge 3 commits into
Open
chore(sqlite-out): eradicate SQLite vocabulary from scitex-python (58 -> 0)#362ywatanabe1989 wants to merge 3 commits into
ywatanabe1989 wants to merge 3 commits into
Conversation
Fleet-wide directive: SQLite is retired across SciTeX. This removes every
occurrence of the string "sqlite" (any case) from the repo outside
docs/adr/, build artefacts and vendored trees.
Committed runtime artefacts (binary SQLite files) removed:
- .scitex/clew/runtime/db.sqlite (now ignored via .scitex/clew/)
- examples/_legacy/scitex/clew/multi_parent/scitex/clew.db
Retired tutorial removed:
- examples/_legacy/notebooks/19_scitex_db.ipynb was end-to-end a tutorial
for the retired stx.db.SQLite3 engine. The master index no longer links
to it.
Docs and README now name only the surviving PostgreSQL client. The README
example was rewritten against the real stx.db.PostgreSQL API (constructor,
execute, save_array, get_rows, load_array); the previous example's to_df,
db.check_health() and delete_duplicates calls do not exist on / are not
valid for that client.
Code and tests:
- scripts/maintenance/_pypi_packages.py no longer lists sqlite3 as a
stdlib module, so an `import sqlite3` is now classified as a missing
PyPI dependency -- the desired outcome under the directive.
- tests/integration/test_cross_package_imports.py no longer asserts
scitex_io._load_modules._sqlite3 is importable (that module is itself
slated for removal from scitex-io); coverage narrows by one module.
- tests/integration/test_integration.py::test_db_delegates drops the
SQLite3 attribute assertion and keeps PostgreSQL + check_health.
Peer review on PR #362. stx.db.PostgreSQL defaults password to None so the previous call did not raise, but a README example should show the real call: pass password explicitly from the environment. check_health and delete_duplicates are SQLite-only helpers (both are implemented over sqlite3 in scitex-db), so they are no longer advertised as capabilities of the PostgreSQL client, in the README prose or in the docs/05_ADDITIONAL_MODULES.md Key-API cell.
…ective PR #361 (chore/purge-sqlite) swept this repo for the same directive a few hours earlier. Both branches reach 0 on the agreed measurement, so one has to go; this commit takes everything #361 did better so the survivor is a strict superset, and #361 is closed. From #361: - Delete examples/_legacy/scitex/db/_sqlite3/ (12 files). They are EMPTY placeholders, so the content grep never saw them -- only their paths carry the retired engine's name. Path-level traces count too. - Better wording: docs now name the Clew store (the actual SciTeX provenance store, per the 'Provenance Tracking (Clew)' section) instead of a generic 'local database'; the schematic box is a 'verification store'; the CrossRef entries say 'database file'. Kept from this branch, which #361 lacks: - .gitignore gains .scitex/clew/, so the deleted runtime artefact cannot reappear on the next run. Verified with git check-ignore. - The README example passes password from the environment and INSERTs the row that save_array(ids=1) then targets.
This was referenced Aug 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Removes every occurrence of the string
sqlite(any case) from this repo, as part of the standing fleet directive to eradicate SQLite from SciTeX.Measurement command (the agreed instrument):
origin/develop(before)feat/eradicate-sqlite-vocabulary(after)Positive control on the same branch with
storeinstead ofsqlite: 202 hits, so the zero above is a real zero, not an instrument failure.Changes
Committed binary SQLite database files removed (runtime artefacts that should never have been tracked)
.scitex/clew/runtime/db.sqlite— the top-level.gitignorenow carries.scitex/clew/so it cannot come back.examples/_legacy/scitex/clew/multi_parent/scitex/clew.db— already covered by the existing**/scitex/clew.dbignore.Retired tutorial removed
examples/_legacy/notebooks/19_scitex_db.ipynbwas end-to-end a tutorial for the retiredstx.db.SQLite3engine (import sqlite3,sqlite3.connect,SELECT ... FROM sqlite_master).00_SCITEX_MASTER_INDEX.ipynbno longer links to it — both the "Database Operations" section and the module-index entry are gone, so the index has no dangling link..gitignoreboilerplate — the inherited Djangodb.sqlite3/db.sqlite3-journallines (twice in the root file, once in the example project) and**/*.sqlite.Docs / README —
scitex.dbis now described as a PostgreSQL client only. The README example was rewritten against the realstx.db.PostgreSQLAPI rather than a mechanical find-and-replace; see the note below.Code / tests
scripts/maintenance/_pypi_packages.pyno longer listssqlite3in its stdlib-module set. Consequence: animport sqlite3anywhere in SciTeX is now classified as a missing PyPI dependency — under the directive, being flagged is the desired outcome.tests/integration/test_cross_package_imports.pyno longer assertsscitex_io._load_modules._sqlite3is importable (that module is itself slated for removal from scitex-io). This narrows coverage by one module.tests/integration/test_integration.py::test_db_delegatesdrops theSQLite3assertion; it still assertsPostgreSQLandcheck_health, so the test stays meaningful.Note on the README example
The previous example was not merely SQLite-flavoured, it was partly untrue. Verified against
scitex-dbbefore rewriting:to_dfdoes not exist on either client — the real accessor isget_rows(table_name, ...), which returns a DataFrame.check_healthis a module-level function taking a database path (stx.db.check_health(db_path)), not a method on the client.delete_duplicatesis deprecated and SQLite3-specific (it forwards todelete_sqlite3_duplicates), so it has no place in a PostgreSQL example.with db:closes the connection on exit (_BaseConnectionMixin.__exit__callsclose()); it is not a transaction, and the old comment said otherwise.The new example uses only verified surface:
PostgreSQL(dbname=, user=, password=, host=),execute,save_array(table, data, column=, ids=),get_rows(table),load_array(table, column, ids=).passworddefaults toNoneon the exportedPostgreSQLclass (scitex_db/_postgresql/_PostgreSQL.py:42), so omitting it does not raise — but the example passes it from the environment because that is the real call. The prose and thedocs/05_ADDITIONAL_MODULES.mdKey-API cell no longer advertisecheck_health/delete_duplicatesfor the PostgreSQL client: both are SQLite-only helpers in scitex-db.Supersedes #361
PR #361 (
chore/purge-sqlite) swept this repo for the same directive a few hours earlier — a parallel effort neither side knew about. It is now closed in favour of this PR, which absorbed everything #361 did better:examples/_legacy/scitex/db/_sqlite3/(12 files) deleted. chore: purge every trace of the SQLite engine #361 found these and this PR originally did not. They are empty files (alle69de29, the empty blob), which is exactly why the content grep never counted them — only their paths carry the retired engine's name. Path-level traces count too.verification store; the CrossRef entries say "database file".Two things this PR has that #361 lacked, which is why this is the survivor:
.gitignoregains.scitex/clew/. Both PRs delete the committed.scitex/clew/runtime/db.sqlite, but only this one stops it reappearing on the next run. Verified:git check-ignore -vreports.gitignore:927:.scitex/clew/.passwordfrom the environment andINSERTs the row thatsave_array(..., ids=1)targets.Diffing the two branches after the absorption leaves no
D/Adifferences at all — every file #361 removed, this PR removes.Known-red check, not caused by this PR — tracked as #363
pytest-matrix-on-ubuntu-py3.13fails withFatal Python error: Segmentation fault(core dumped), retried three times by the workflow. It fails identically on #361's branch, it is an interpreter crash rather than a test assertion, and thetestsworkflow was already failing ondevelopbefore either PR existed. Filed as #363, which corrects a misreading worth flagging: the three crashes in that job have three DIFFERENT Python stacks but the same top frame,Garbage-collecting.check_optional_depsappears in only one of the three, and only as the caller at the bottom of the stack. The crash site is the GC (199 C extensions loaded), so it is not a SciTeX call-site bug. Every other check passes.Useful datum for whoever picks that up:
test_db_delegatespassed in CI on #361's branch, which confirms CI installs thescitex[db]extra and that dropping theSQLite3assertion is safe. Locally that test fails only becausescitex[db]is not installed in this container — the pre-existinghasattr(stx.db, "SQLite3")raises the identicalImportErrorthere, so the local failure is environmental, not introduced here.Traps checked, nothing suppressed
CHANGELOG.mdcontains nosqliteline, so nothing had to be touched there.examples/_legacy/notebooks/27_scitex_scholar.ipynbopened the third-partyimpact_factorpackage's own bundled database directly (sqlite3.connect(impact_factor.DEFAULT_DB)) in two cells. That is a library reading someone else's file format, not SciTeX storing its own state — but the notebook's own narrative already saysScholar(enrich_by_default=True)performs that enrichment, so the raw-DB blocks were a redundant manual duplicate. They are removed and the notebook now relies on Scholar's enrichment.impact_factoris not installed in this environment, so its public API could not be verified and was deliberately not invented. Both notebooks were re-validated as JSON after editing.