Skip to content

chore(sqlite-out): eradicate SQLite vocabulary from scitex-python (58 -> 0) - #362

Open
ywatanabe1989 wants to merge 3 commits into
developfrom
feat/eradicate-sqlite-vocabulary
Open

chore(sqlite-out): eradicate SQLite vocabulary from scitex-python (58 -> 0)#362
ywatanabe1989 wants to merge 3 commits into
developfrom
feat/eradicate-sqlite-vocabulary

Conversation

@ywatanabe1989

@ywatanabe1989 ywatanabe1989 commented Aug 30, 2026

Copy link
Copy Markdown
Collaborator

What

Removes every occurrence of the string sqlite (any case) from this repo, as part of the standing fleet directive to eradicate SQLite from SciTeX.

Measurement command (the agreed instrument):

git grep -a -i -n sqlite <REF> | grep -a -vE ':(docs/adr/|build/|.*egg-info/|vendor/)'
ref lines
origin/develop (before) 58
feat/eradicate-sqlite-vocabulary (after) 0

Positive control on the same branch with store instead of sqlite: 202 hits, so the zero above is a real zero, not an instrument failure.

Changes

Committed binary SQLite database files removed (runtime artefacts that should never have been tracked)

  • .scitex/clew/runtime/db.sqlite — the top-level .gitignore now carries .scitex/clew/ so it cannot come back.
  • examples/_legacy/scitex/clew/multi_parent/scitex/clew.db — already covered by the existing **/scitex/clew.db ignore.

Retired tutorial removed

  • examples/_legacy/notebooks/19_scitex_db.ipynb was end-to-end a tutorial for the retired stx.db.SQLite3 engine (import sqlite3, sqlite3.connect, SELECT ... FROM sqlite_master). 00_SCITEX_MASTER_INDEX.ipynb no longer links to it — both the "Database Operations" section and the module-index entry are gone, so the index has no dangling link.

.gitignore boilerplate — the inherited Django db.sqlite3 / db.sqlite3-journal lines (twice in the root file, once in the example project) and **/*.sqlite.

Docs / READMEscitex.db is now described as a PostgreSQL client only. The README example was rewritten against the real stx.db.PostgreSQL API rather than a mechanical find-and-replace; see the note below.

Code / tests

  • scripts/maintenance/_pypi_packages.py no longer lists sqlite3 in its stdlib-module set. Consequence: an import sqlite3 anywhere in SciTeX is now classified as a missing PyPI dependency — under the directive, being flagged is the desired outcome.
  • tests/integration/test_cross_package_imports.py no longer asserts scitex_io._load_modules._sqlite3 is importable (that module is itself slated for removal from scitex-io). This narrows coverage by one module.
  • tests/integration/test_integration.py::test_db_delegates drops the SQLite3 assertion; it still asserts PostgreSQL and check_health, so the test stays meaningful.

Note on the README example

The previous example was not merely SQLite-flavoured, it was partly untrue. Verified against scitex-db before rewriting:

  • to_df does not exist on either client — the real accessor is get_rows(table_name, ...), which returns a DataFrame.
  • check_health is a module-level function taking a database path (stx.db.check_health(db_path)), not a method on the client.
  • delete_duplicates is deprecated and SQLite3-specific (it forwards to delete_sqlite3_duplicates), so it has no place in a PostgreSQL example.
  • with db: closes the connection on exit (_BaseConnectionMixin.__exit__ calls close()); it is not a transaction, and the old comment said otherwise.

The new example uses only verified surface: PostgreSQL(dbname=, user=, password=, host=), execute, save_array(table, data, column=, ids=), get_rows(table), load_array(table, column, ids=). password defaults to None on the exported PostgreSQL class (scitex_db/_postgresql/_PostgreSQL.py:42), so omitting it does not raise — but the example passes it from the environment because that is the real call. The prose and the docs/05_ADDITIONAL_MODULES.md Key-API cell no longer advertise check_health / delete_duplicates for the PostgreSQL client: both are SQLite-only helpers in scitex-db.

Supersedes #361

PR #361 (chore/purge-sqlite) swept this repo for the same directive a few hours earlier — a parallel effort neither side knew about. It is now closed in favour of this PR, which absorbed everything #361 did better:

  • examples/_legacy/scitex/db/_sqlite3/ (12 files) deleted. chore: purge every trace of the SQLite engine #361 found these and this PR originally did not. They are empty files (all e69de29, the empty blob), which is exactly why the content grep never counted them — only their paths carry the retired engine's name. Path-level traces count too.
  • Better docs wording adopted: the docs now name the Clew store (the actual SciTeX provenance store — the section is titled "Provenance Tracking (Clew)") rather than a generic "a local database"; the schematic box is a verification store; the CrossRef entries say "database file".

Two things this PR has that #361 lacked, which is why this is the survivor:

  • .gitignore gains .scitex/clew/. Both PRs delete the committed .scitex/clew/runtime/db.sqlite, but only this one stops it reappearing on the next run. Verified: git check-ignore -v reports .gitignore:927:.scitex/clew/.
  • The README example passes password from the environment and INSERTs the row that save_array(..., ids=1) targets.

Diffing the two branches after the absorption leaves no D/A differences at all — every file #361 removed, this PR removes.

Known-red check, not caused by this PR — tracked as #363

pytest-matrix-on-ubuntu-py3.13 fails with Fatal Python error: Segmentation fault (core dumped), retried three times by the workflow. It fails identically on #361's branch, it is an interpreter crash rather than a test assertion, and the tests workflow was already failing on develop before either PR existed. Filed as #363, which corrects a misreading worth flagging: the three crashes in that job have three DIFFERENT Python stacks but the same top frame, Garbage-collecting. check_optional_deps appears in only one of the three, and only as the caller at the bottom of the stack. The crash site is the GC (199 C extensions loaded), so it is not a SciTeX call-site bug. Every other check passes.

Useful datum for whoever picks that up: test_db_delegates passed in CI on #361's branch, which confirms CI installs the scitex[db] extra and that dropping the SQLite3 assertion is safe. Locally that test fails only because scitex[db] is not installed in this container — the pre-existing hasattr(stx.db, "SQLite3") raises the identical ImportError there, so the local failure is environmental, not introduced here.

Traps checked, nothing suppressed

  • No guard was removed. There is no linter rule or test in this repo whose purpose is to ban sqlite (such a guard must name what it forbids). The two test edits remove positive assertions that the retired engine still exists — the opposite of a guard.
  • No history was rewritten. CHANGELOG.md contains no sqlite line, so nothing had to be touched there.
  • The one third-party-format case is called out below.

examples/_legacy/notebooks/27_scitex_scholar.ipynb opened the third-party impact_factor package's own bundled database directly (sqlite3.connect(impact_factor.DEFAULT_DB)) in two cells. That is a library reading someone else's file format, not SciTeX storing its own state — but the notebook's own narrative already says Scholar(enrich_by_default=True) performs that enrichment, so the raw-DB blocks were a redundant manual duplicate. They are removed and the notebook now relies on Scholar's enrichment. impact_factor is not installed in this environment, so its public API could not be verified and was deliberately not invented. Both notebooks were re-validated as JSON after editing.

Fleet-wide directive: SQLite is retired across SciTeX. This removes every
occurrence of the string "sqlite" (any case) from the repo outside
docs/adr/, build artefacts and vendored trees.

Committed runtime artefacts (binary SQLite files) removed:
  - .scitex/clew/runtime/db.sqlite  (now ignored via .scitex/clew/)
  - examples/_legacy/scitex/clew/multi_parent/scitex/clew.db

Retired tutorial removed:
  - examples/_legacy/notebooks/19_scitex_db.ipynb was end-to-end a tutorial
    for the retired stx.db.SQLite3 engine. The master index no longer links
    to it.

Docs and README now name only the surviving PostgreSQL client. The README
example was rewritten against the real stx.db.PostgreSQL API (constructor,
execute, save_array, get_rows, load_array); the previous example's to_df,
db.check_health() and delete_duplicates calls do not exist on / are not
valid for that client.

Code and tests:
  - scripts/maintenance/_pypi_packages.py no longer lists sqlite3 as a
    stdlib module, so an `import sqlite3` is now classified as a missing
    PyPI dependency -- the desired outcome under the directive.
  - tests/integration/test_cross_package_imports.py no longer asserts
    scitex_io._load_modules._sqlite3 is importable (that module is itself
    slated for removal from scitex-io); coverage narrows by one module.
  - tests/integration/test_integration.py::test_db_delegates drops the
    SQLite3 attribute assertion and keeps PostgreSQL + check_health.
Peer review on PR #362. stx.db.PostgreSQL defaults password to None so the
previous call did not raise, but a README example should show the real call:
pass password explicitly from the environment.

check_health and delete_duplicates are SQLite-only helpers (both are
implemented over sqlite3 in scitex-db), so they are no longer advertised as
capabilities of the PostgreSQL client, in the README prose or in the
docs/05_ADDITIONAL_MODULES.md Key-API cell.
…ective

PR #361 (chore/purge-sqlite) swept this repo for the same directive a few
hours earlier. Both branches reach 0 on the agreed measurement, so one has
to go; this commit takes everything #361 did better so the survivor is a
strict superset, and #361 is closed.

From #361:
  - Delete examples/_legacy/scitex/db/_sqlite3/ (12 files). They are EMPTY
    placeholders, so the content grep never saw them -- only their paths
    carry the retired engine's name. Path-level traces count too.
  - Better wording: docs now name the Clew store (the actual SciTeX
    provenance store, per the 'Provenance Tracking (Clew)' section) instead
    of a generic 'local database'; the schematic box is a 'verification
    store'; the CrossRef entries say 'database file'.

Kept from this branch, which #361 lacks:
  - .gitignore gains .scitex/clew/, so the deleted runtime artefact cannot
    reappear on the next run. Verified with git check-ignore.
  - The README example passes password from the environment and INSERTs the
    row that save_array(ids=1) then targets.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant