Repository navigation
0.2.0 - 2026-10-25
Highlights
Upgrading an existing database takes two commands, in this order. Back up
~/kospex/kospex.db first:
kospex upgrade-db -apply
kospex rebuild developer-stats -allMigration 0011 is required: without it every sync fails at its
developer-stats step. The rebuild is required too, or the counts on / and
/summary/ are wrong. Details are under Upgrade notes.
- Some commands and pages now report a failure where they used to return an
empty success.developers,tech-landscapeandhotspotexit 1 for a
-repothat was never synced./dependencies/{id}and/osi/{id}return 400
for an author id,/collab/returns 404 for an unknown repository, and routes
return their real 4xx instead of 500. Scripts that check exit codes, and
alerts on 5xx, will see the difference. kospex scano longer saves by default. Pass-save, which refuses
unless the repository and its HEAD are synced and the worktree is clean.
-scopereplaces-dev.- kweb answers only to
127.0.0.1andlocalhost, and sends no CORS
headers (CWE-346). SetKOSPEX_WEB_ALLOWED_HOSTSto reach it by another
name, for example behind a reverse proxy or from another Compose service.
kweb now exits with code 3 when it can't start. - More of the dependency tree is read.
package-lock.json(v1–v3) and
uv.lockare parsed, and multi-documentpnpm-lock.yamlfiles are read in
full. Lockfile rows record whether a dependency is dev-only, optional, or runs
an install script (migration0009). A manifest kospex can't read is now
reported asparse_error, not as one that declares nothing. krunner osi -next Nscans dependencies in batches suited to cron, least
recently examined first.-max-secondsbounds a run, and per-run
diagnostics go toosi_runs.- Panopticas tags in the web UI:
/tags/and/tag/{tag}.?absent=true
on tag and technology pages lists the repositories that don't have one,
such as every repo without a Dependabot file. kospex rebuildrecomputes derived data from the database without
re-syncing. The first task isdeveloper-stats.- Faster. The summary pages count statuses in 0.19s rather than 1.81s on
1.1M commits. Repeat loads of/and/repos/are cached.krunner osi's
read path drops from 11.5s to 0.4s on 167 repositories. - Removed or disabled: the DuckDB proof of concept (
kgit sync-repoand
kospex init-duckdb;duckdbis no longer a dependency), kweb's package
check page, andkospex advisory-history, pending its redesign. - Dependency minimums raised, including click 8.4.1, fastapi 0.140,
uvicorn 0.52 and requests 2.34. certifi 2026.10.19 or later is now a direct
dependency. Git metadata is printed literally rather than parsed as console
markup (CWE-150).
Upgrade notes
Breaking: an existing database needs two steps, in this order. There are
five schema changes, one required data rebuild and one optional clean-up.
Newly created databases need none of this.
kospex upgrade-db -apply # back up ~/kospex/kospex.db first
kospex rebuild developer-stats -all # seconds; reads the DB only, no clones-
The rebuild is required. kweb's
/and/summary/now count statuses
fromdeveloper_stats(see Changed), with no fallback to the old queries.
Rows written by earlier versions hold author-time dates, so until they are
rebuilt the summary counts are wrong: some developers land in the wrong
status, and a repo with no stats rows is left out.kospex system-status
shows how many repos still need it under "Summary stats coverage". Re-syncing
every repo (kgit pull --all) also rebuilds them, but takes far longer. -
The optional clean-up clears the stored
''described under Fixed for
#258. It is not needed for
kospex to work, anddocs/refreshing-data.mdhas
the SQL and the reason the figures depend onlatest. -
0007adds an index tofile_metadata. Index only: no rows are read,
rewritten or deleted, and a database left un-upgraded keeps working, just
without the speed-up. -
0008adds theosi_runstable and its index, used bykrunner osi -next
for per-run diagnostics. A new empty table: nothing existing is touched, and a
database left un-upgraded keeps working —-nextlogs that it could not record
diagnostics and carries on scanning, because the scan is the product and the
diagnostics are not. -
0009addsis_dev,is_optionalandruns_install_scriptto
dependency_data, for the lockfile install state described under Added
below. Columns only, defaulting to NULL, which means unknown rather than
false: nothing existing is rewritten, and a database left un-upgraded keeps
working — the scanners record name and version as before and simply store no
install state. Re-runningkrunner osiafter the upgrade is what populates
them for lockfiles already scanned. -
0010addsfinished_attoosi_runs, becausestarted_atwas recording
the finish (see Fixed). A column only, defaulting to NULL: nothing existing is
touched, and a database left un-upgraded keeps working —-nextlogs that it
could not record diagnostics and carries on scanning. Rows written before the
upgrade keep astarted_atthat is really a finish time and are deliberately
not backfilled, becauseduration_msexcludes time the machine spent asleep, so
the true start cannot be recovered by subtraction.DELETE FROM osi_runsis the
clean option if you would rather start the diagnostics over. -
0011adds a covering index oncommits, and two columns,
first_authoredandlast_authored, todeveloper_stats. The index is
about 4% of the database size and took 4.6s to build on a 1.1M-commit
database. It makes per-row commit writes during sync about 26% slower. The
columns default to NULL and are filled bykospex rebuild developer-stats
(or each repo's next sync). Unlike the others, this one is required: sync
writes the two columns, so on a database without0011every sync fails at
its developer-stats step.
Otherwise what changes is how kospex responds when it cannot answer a
question. In four places it previously returned something that looked like an
answer; it now says it cannot. Each is a fix, but each can break a caller that
was relying on the old, wrong success.
-
Three CLI commands now exit 1 for a repo that is not in the database.
developers,tech-landscapeandhotspot, when scoped with-repo
(-repo_idis unaffected), previously printed an empty result and exited 0
for a repository kospex had never synced — indistinguishable from one that is
synced and genuinely dormant. They now print what is wrong and exit 1.
A script that ran these and checked the exit code will now see a failure
where it previously saw success. That is the point — the old success was
reporting "no developers" for repositories with recent commits — but a
pipeline that treats any non-zero as fatal will stop where it used to
continue.tech-landscapeis the sharpest change: it previously printed an
empty table with no message at all. -
/dependencies/{id}and/osi/{id}return 400 for an author email.
A base64-encoded author email in the path is a scope these pages cannot
apply. They previously ignored it and rendered every row in the table —
852 rows on a 111-repo estate, identical to no filter at all. Any bookmark,
link or script hitting those URLs with an author id now gets a 400 instead of
a page full of unfiltered data. -
/collab/{repo_id}returns 404 for an unknown or malformed id.
It previously rendered an empty table, which reads as "this repository has no
collaborators". Both an unparseable id and a well-formed id for a repository
not in the database now 404, with different messages. -
Routes now return their real status code instead of 500. A route raising
a deliberate 404 or 400 had it converted to a 500 by its own error handler.
If you alert on 5xx, expect that signal to drop — some of what was
reported as kospex failing was a malformed request./file-collab/without a
file_pathis the clearest example: it returned 500 and now returns 400. -
kospex scano longer saves by default, and its output and flags changed
(#237). A script calling
kospex sca FILEthat relied on it writingdependency_datamust now pass
-save— and-saverefuses (exit 1) unless the repository and its HEAD
commit are synced and no tracked file is modified.-devis gone, replaced
by-scope(defaultdirect, so dev dependencies are no longer shown by
default). The-outCSV, which previously wrote nothing, now writes the
table's columns: noauthorsormalwarecolumn, anddays_agois a
whole number.-malwareis accepted but ignored. Details under Changed. -
kwebexits with code 3 when it can't start, for example when its port
is already in use. uvicorn 0.50 made 3 its exit code for every startup
failure, and kospex now requires uvicorn 0.52 or later. A service manager or
script that checks for a particular non-zero code should accept 3.
Added
-
Panopticas tags are reachable from the web UI:
/tags/and/tag/{tag}
(#293).
file_metadata.tech_typeholds the tags panopticas assigns from a filename —
AI,license,Dependabot,llms.txtand ~60 others — and nothing
rendered or filtered on it./tech/{tech}could not: it filtersLanguage,
and a tag is not a Language. The AI signal in particular is in the filename,
sosccrecordsCLAUDE.mdas Markdown while panopticas tags itAI;
/tech/AIreturned 200 with zero rows, which reads as "no repositories have
AI" when 51 of them do.Deliberately a separate route rather than a fallback inside
/tech/: 9
names are both a Language and a tag and count very different things —Go
is 24,236 files as a Language and 345 as a tag. A route guessing between them,
in either order, would not be a page anyone could trust./tags/lists the vocabulary in three alphabetical columns, so no listed tag
can 404 — it is the same setwhere_tag()validates against. It lists what
panopticas can assign, which is deliberately not "the tags in this
database"; counting the tags actually present needs the per-row split
described in #322. Neither
route is in the header nav orsitemap.xml. -
?absent=truelists the repositories that do not have a tag or
technology, on both/tag/{tag}and/tech/{tech}. The adoption question —
"everyone should have a Dependabot file" — is only answerable from the
complement, so the positive listing reports both sides and links across: 122
repositories with this tag, 252 repositories do not have files with this tag.
The absent page states its denominator and says so in its heading (Tag:
Dependabot is not present).The denominator is repositories with current
file_metadata, not the
repostable. A repository whose file metadata was never collected cannot
be said to be missing a tagged file — "we have not looked" is a different
finding from "it is not there", and conflating them would put false entries in
a compliance list./tech/shares the same denominator, becauseLanguage
andtech_typeare columns of the same table.In the absent view
# Filesis the repository's total current file count, not
a count of tagged files, which would be zero on every row. The page says so,
since a non-zero count beside a repository that has none of the tag otherwise
reads as a contradiction. -
kospex rebuild <task>recomputes derived data without re-syncing. It
takes a scope, exactly one ofREPO_ID,-all,-org ORG_KEYor
-server SERVER, so nothing is rebuilt estate-wide by accident.-dry-run
lists the repos in scope without changing anything. Each repo is rebuilt
independently: a failure is reported and the rest carry on, and the command
exits 1 if any failed. The first task isdeveloper-stats, which reads only
the database, so it needs no clones, network or credentials.kospex rebuild listshows the tasks, the tables each writes and whether it needs a clone.
Further tasks (file metadata, repo status) are planned on the same pattern. -
Lockfile install state is recorded:
is_dev,is_optional,
runs_install_script(migration0009). Lockfiles state per entry whether a
dependency reaches production, whether it installs at all, and whether it
executes code at install time. kospex read the name and version and discarded
the rest.requirements_typecannot carry these — itsdevvalue means
"declared as a devDependency", and a transitive entry has no declaration of its
own, so on one real repository 870 dev-only transitive packages were recorded
identically to the 1169 production ones.runs_install_scriptis the notable
one: it answers "which of these executes during install", which is what
distinguishes a compromised package that ran from one merely present in a tree.NULL means unknown, not false. A manifest declares intent, not install state
—requirements.txtcannot say whether a package runs an install script — and
lockfileVersion: 1package-lock.jsonfiles record no flags at all. Claiming
falsethere would assert something the file does not say. Same distinction
resolutiondraws for the resolve layer andlast_checkedfor staleness.Populated by the pnpm extractor today; named format-neutrally so the parsers in
#229 land in the same columns rather than inventing their own.declared_scopeholds the raw string the file used, verbatim, in that
format's own vocabulary —dev,optional,requiresBuild,
snapshots:optional, and later Maven'sprovidedor a Python extra's name.
The booleans are a projection of it. The column exists because every mapping
error in building them happened at write time, where it is unrecoverable:
pnpm'srequiresBuildis not npm'shasInstallScript(it additionally covers
native modules with no scripts), and an absent key means false in pnpm v6 but
unknown in v9. With the declared string stored, a mapping that turns out wrong
is a query to correct rather than a re-parse of every lockfile in an estate.
docs/dependency-flags.mdis the column's reference page, carrying each
format's own definition, quoted and linked.Install state is read per lockfile version, because where it lives is a
version fact, not a pnpm fact. v5/v6 put all three flags inpackages:and
omit a key when false, so absence there is a genuine no. v9 moved the resolved
graph to a separatesnapshots:section and records none of them in
packages:— so v9 readsoptionalfromsnapshots:and leavesis_devand
runs_install_scriptNULL, dev-ness being derivable only by walking the graph
from each importer'sdevDependenciesand built packages not being tracked in
the lockfile at all. A package counts as optional only if every resolution
path to it is optional: peer-dependency suffixes put onename@versionin
snapshots:several times and the variants can disagree, and one required path
means it installs. -
krunner osi -next N— batched, cron-friendly dependency scanning.osi -all
is a single sweep that takes hours on a large estate, so it cannot run from a
scheduler and is all-or-nothing when it fails.-next Nprocesses the N
repositories least recently examined, so a schedule of small runs converges on
full coverage. Composes with a scope:-next 10 github.com~acme.Three things make it safe to schedule. It exits 0 when nothing is due, where
-allexits 1 on "No results" and would report a failure on every tick once the
estate is current. It takes a run lock and exits 0 if another run holds it,
since overlap is normal under a short interval; a lock older than 6 hours is
taken over, because one crashed run blocking the schedule forever is worse than
a double scan. And it writes no CSV by default —-csvopts in, writing one
OSI-<repo_id>.csvper repository to the assessments directory only, so a
scheduled run cannot overwrite a fullOSI-all.csvexport. -
-max-secondsbounds a batch by time as well as by count. Measured per-repo
cost spans four orders of magnitude — 2ms for a repository with no dependency
files against 109s for a monorepo with 215 of them and 1101 packages — so a
fixedNgives no predictable tick duration. The budget is checked between
repositories: one in flight always finishes, because abandoning it would either
save rows with no outcome recorded or record an outcome for work not done. A tick
can therefore overshoot by the cost of one repository, sometimes greatly (a 10s
budget produced a 109s tick), and the run says which repository did it. -
Extraction outcomes are recorded (#148).
A manifest producing zero dependency rows used to mean several different things
with nothing recording which — most sharplyempty(genuinely declares nothing,
fine) againstparse_error(the parser raised and was silently skipped, a
defect). Six outcomes now distinguish them:extracted,empty,unsupported,
unclassified,not_a_manifest,parse_error, modelled on
dependency_data.resolution, which already does this one layer later. Stored
per repository inobservations, no schema change. Classifying the 12,666
dependency-tagged files on a few-hundred-repo estate gives 85.6%extracted,
9.1%empty, 3.0%not_a_manifest(go.sum,dependabot.yml,Dockerfile,
runtime pins), 2.0%unsupported, 0.2%unclassified(setup.py,setup.cfg
— #137) and 0.06%
parse_error, all eight of which are deliberately malformed test fixtures.
Three formats remainunsupportedafter the parsers added in this release:
yarn.lock,build.gradleandpom.xml. Separatingnot_a_manifestout
makes theunsupportedfigure an honest parser backlog rather than an inflated
one.The distribution was measured before the
uv.lockparser below landed, so the
24uv.lockfiles it counts underunsupportednow extract. -
Per-run diagnostics in a new
osi_runstable (migration0008). One row
per(run_id, repository)with duration, files, packages, deps.dev requests
actually made, and outcome. The log lines cannot answer what only exists across
runs: whatNfits an interval, whether a repository is getting slower, whether
the response cache is working, whether the schedule is completing passes.
observationscannot hold it — its primary key includeslatest, so it keeps
one current row per key.lookupscounts cache misses separately from
packageswritten, which makes cache effectiveness measurable: a first real run
shows 76% of babel's 1101 package lookups served from cache. -
package-lock.jsonis now parsed (#229).
The extractor registry has classified npm lockfiles since July 2026 without
being able to read one, so a repository whose only manifest was a
package-lock.jsonproduced no dependency rows at all. All three lockfile
versions are read: v3 and v2 frompackages:, v1 from its nested
dependencies:tree. A v2 file's duplicatedependencies:tree is ignored,
since it describes the same closure.The four install-state columns from migration
0009are populated —
is_dev,is_optional,runs_install_scriptanddeclared_scope— so npm
rows now answer the same questions as pnpm rows. Three npm-specific points
are worth knowing before querying them:-
devOptionalrecordsis_dev=1withis_optionalNULL. It is npm's
third state rather than the union of the other two: it never co-occurs with
devoroptional, and means "reachable only via a dev path or an
optional path" without saying which. The package does not reach production,
but whether it may fail to install is genuinely unstated, so NULL. -
An npm lockfile kospex cannot read is reported as
parse_error, not
empty(CWE-703).emptymeans "examined, nothing to find", so a file
whose dependencies were never read must not claim it. An unsupported
lockfileVersion, a missing one or malformed JSON now raise, and
lockfileVersionitself is read liberally so"3"works as well as3.
kospex scapreviously called parsers with no guard and would have crashed
on this; it now reports the file and continues, which is what thekrunner osipath has done since #148. Genuinely empty lockfiles still report
empty. -
An aliased dependency is recorded as the package it actually is.
npm i foo@npm:barinstalls bar atnode_modules/foo, and npm names the
real package in anamefield on that entry. Reading the install path
instead would record a package that is not on the registry, so its deps.dev
lookup returns nothing and the row shows no advisories for something that is
genuinely installed. 30 entries on a reference estate are aliases. The alias
is still matched when decidingdirect/dev, since a project declares an
aliased dependency by its alias. v1 encodes the same thing as
version: "npm:bar@1.2.3", which is parsed rather than stored verbatim.Because a lockfile is untrusted input and no field of it corroborates
another (CWE-807), every identity an entry asserts is recorded — the
install path, thename, and the package named in theresolvedURL — with
none treated as authoritative. They almost always agree, so this is one row
per package in practice; across a reference estate it adds 32 rows out of
13,739. A row named for an install-path alias may not be a real registry
package and will carry no advisory data; the row under the real name is the
one that answers the security question. -
A package reached by several install paths is reconciled, not
overwritten.packages:keys are install paths and the primary key has
no path column, so the samename@versionat two depths collapses to one
row.dev/optional/devOptionalare true only when every path
agrees — one production path means the package ships — while
hasInstallScript/peer/inBundleare true if any path says so. -
lockfileVersion: 1rows are alltransitive. v1's top level is the
hoisted install set rather than the declared set, so a direct dependency
cannot be told from a transitive lifted to the root.is_devis still
populated;runs_install_scriptis NULL, because v1 has no
hasInstallScriptfield.
Expect a larger first scan. On a reference estate this adds ~13,700 rows from
~7,400 distinct packages, around 5,900 of which have never been looked up, so
the first sweep does that many deps.dev calls.docs/dependency-flags.mdcarries the measured per-key inventory and the
per-version detail behind all of this. -
-
uv.lockis now parsed (#229).
Registered since July 2026 but unreadable, so everyuv.lockreported
unsupported— and it reaches more repositories than any other unread format,
at one file per project. The full locked closure is read: 2,406 rows from 24
files, being 2,434[[package]]entries less the 28 that are the project
itself and its workspace members.is_dev,is_optional,declared_scopeandrequirements_typeare populated
by walking the dependency graph from the project, becauseuv.lockstates
install state nowhere per package — it is a graph, not a flag list. The flags
were validated against uv's own output, keyed by(name, version): all 2,406
rows agree with what uv's runtime,--all-extrasand--all-groupsexports
imply (388 runtime, 430 optional, 1,446 dev, 142 both).- The format gates were measured against uv 0.11.11, by running
uv export --frozenover perturbed copies of a real lockfile rather than by
reading a spec. uv refusesversion≠ 1, a non-integerrevision, an empty
file and UTF-16, and so does the parser. uv readsrevision = 99, a
missingrevisionand a UTF-8 BOM, and so does the parser. This
deliberately departs from thepackage-lock.jsonprecedent of refusing an
unknown version field: here, refusing would be a false positive against the
tool that owns the format. - An unreadable
uv.lockraises rather than returning[](CWE-703), so
it reportsparse_errorand notempty.[]is reserved for a lockfile
that genuinely locks nothing but the project itself. - First-party entries produce no record.
source = { editable }and
{ virtual }are the project and its workspace members, matching how pnpm
treats local workspace directories. All 13 version-less entries on the
estate are first-party, so no version fallback is needed. declared_scopeover-lists on 15 of 2,406 rows. The walk ignores
environment markers, so it lists a group whose path uv prunes as impossible —
a Python-version contradiction (13 rows) or aconflictsdeclaration (2). It
never under-lists and never changes a flag. Documented rather than fixed:
marker-aware walking needs an environment enumeration that could under-list,
which is the worse failure.
Git-sourced entries are recorded, since deps.dev answers by name and version
rather than by source — a git fork resolves against the registry's release of
the same version, so its advisories are upstream's. The source itself is not
stored.docs/dependency-details.mdcarries the per-decision detail and the drift watch
for what is deliberately not read. - The format gates were measured against uv 0.11.11, by running
Changed
-
An unknown tag or technology is a 404, not an empty table.
/tag/{tag}
rejects a tag panopticas does not define, and/tech/{tech}a technology no
current file has — on the listing, the absent view and the file drill-down
alike. A 200 with no rows is the failure mode these routes exist to remove: it
reads as "no repositories have this". The message names what was asked for
("Fortran is an unknown Technology or Language."), because the address bar
shows the percent-encoded form rather than what was meant.Two consequences worth knowing.
Language = ?has always been
case-sensitive, so/tech/pythonis now a 404 where it was a 200 with an
empty table — both say "wrong URL"; the 404 says it legibly. And a tag
panopticas has retired 404s alongside a typo, becauseget_tags()reports
only the current vocabulary
(#317). -
kweb's package check is disabled:
/package-check/and
/package-check/uploadare commented out and now return 404. The page let
you drop a single dependency file to analyse it, but nothing linked to it and
it was not in use. It will be refactored before it returns. The route code,
templates/package_check.htmland its status classifier remain in the source,
and the route is removed from the page list indocs/kweb/index.md. To check a
file, sync the repo that contains it and use/dependencies/. -
The DuckDB proof of concept is parked:
kgit sync-repoand
kospex init-duckdbare commented out, andduckdbis no longer a
dependency. Both commands were labelled experimental, and nothing in
reporting or the web UI read what they wrote; kospex stores git history in
SQLite (kospex.db). Installing kospex no longer pulls in the duckdb native
library. TheGitDuckDBandGitIngestmodules remain while DuckDB is
researched further for feasibility and usefulness; to use them, install
duckdbyourself. An existing~/kospex/kospex-git.duckdbis left in place. -
kgit statusandkgit pull --checkprint rich tables instead of
PrettyTable, and the--checkcolumnageis renamedfetch_age. The
column always measured days sincelast_fetch— when kgit last cloned or
pulled the repo — not since the last sync or commit, and the new name says
so. A repo kgit has never cloned or pulled still showsnever, even if it is
synced. Piped or redirected output is printed at the table's full width, so
longrepo_ids are not cut off with…, as they had never been with
PrettyTable.kgit statusalso shows directory sizes in MB to three decimal
places (they were KB) and the execution time to four. -
kospex key-personprints a rich table instead of PrettyTable, and no
longer dumps the raw rows above it. Columns and alignment are unchanged.
Piped or redirected output is printed at the table's full width, so long
author emails are not cut off with…. -
kospex key-person -repo_id SERVER~ORG~REPOruns the analysis for a repo
that need not be cloned on disk: the command only ever read the kospex
database, and the directory was used just to work out therepo_id. A
repo_idthe database has never seen, or one with no commits recorded, now
exits 1 and says which, with the command to fix it, instead of printing an
empty table that looks like a dormant repo. With neither a directory nor
-repo_idit prints its help and exits 2, rather than silently using the
current directory: pass.for that. Run on a directory, it now ends
with a note that the directory only supplies therepo_idand the figures are
from the database's last sync of it, with that sync's date: commits made since
are not counted until the repo is synced again. -
kospex advisory-historyis disabled until its redesign
(#282). It now exits 1 with a
notice on stderr instead of running. It crashed withIndexErroron the
first scoped npm or Maven version (@babel/core,
org.apache.logging.log4j:log4j-core), fetched every version rather than the
"previous 10" its help promised, in deps.dev's alphabetical order rather than
release order, and printed each version's raw deps.dev response. Its options
are kept, so existing invocations get the notice rather than a usage error.
The redesigned command — a window of recent versions ordered by publish date,
a table, and-json— is planned for Q4 2026. -
Dependency minimums raised. Each minimum is now one of the three newest
releases, or a release from the last three months: click 8.4.1, packaging
26.1, python-dotenv 1.2.2 (clears GHSA-mf9w-mj56-hr94), requests 2.34.0,
rich 14.3.3, fastapi 0.140.0 and uvicorn 0.52.0. click, requests, uvicorn,
prettytable and PyYAML were exact pins and are now ranges. Tested at both
the minimum and the newest version allowed. -
certifi is now a direct dependency, at 2026.10.19 or later. It came in
through requests before. Listing it means upgrading kospex also upgrades an
older certifi that is already installed. 2026.10.19 removes emSign root CAs
whose intermediate CA keys were compromised (GHSA-6q48-6c7w-9gw2). -
kospex scaoutput, scope and saving reworked
(#237). sca is an analyst's look
at a dependency file, not a way to record work in progress, so it now behaves
like one:- Output. The file path, one tab-separated line per package (name, declared
version, release date), then a rich table, then one line naming the scope
shown and the other scopes the file holds with their counts. The Python
dict dump of every enriched record — printed twice for a file inside a git
repository — is gone. Missing values show as-, notNone. -scopereplaces-dev, which could not be turned off. A comma-separated
list ofdirect,dev,peer,optional,transitive,unknown(no scope
recorded) orall; defaultdirect. It filters what is shown — the table,
-jsonand-out— and never what is extracted or saved.-jsonprints the table rows as JSON on stdout and nothing else;
informational lines go to stderr, sokospex sca -json FILE > sca.jsonworks.-saveis opt-in, and guarded. It was declared as an option with a value
defaulting to on, so it could not be switched off. It is now a flag, and it
refuses before any deps.dev lookup when the file is not in a git repository,
the repository or its HEAD commit is not in the kospex DB, the worktree has
modified or staged files, or the file itself is untracked. A saved row is
keyed on the HEAD hash, so it must describe a commit kospex knows, read from a
worktree that matches it. Untracked files elsewhere do not count.- Columns.
authorsis dropped: it counted authors of the package's source
repo only when that repo happened to be synced locally, so it read 0 or-
for nearly everything. Thekospex deps -filetable shares these columns and
loses it too;depsstill computes the value for its author ratio.
malwareis dropped: the lookup behind it is a deprecated service, and
-malwarenow prints a notice and is ignored.days_agois the whole
number of days since the release of the version looked up — the floor of a
range — and blank when there is none; it was a two-decimal float. -outwrites a CSV of the table. It was read into a variable that nothing
used, so it never wrote a file.
- Output. The file path, one tab-separated line per package (name, declared
-
The external response cache TTL is a day, not an hour, and is now
configurable. Every deps.dev and PyPI lookup goes through
url_request_with_status(), whose TTL defaulted to 3600s; seven call sites rely
on that default and none overrode it. An hour is enough for a single
full-estate sweep, which finishes inside the window and so reuses any repeated
lookups — but wrong for a batched sweep run from cron over
hours or days, where every tick starts cold and re-fetches the same packages. A
167-repo estate issues 6,387 lookups for 4,864 distinct package+version pairs,
so 24% are repeats a cache can serve. That duplication is mostly within
repositories rather than across them — 96% of distinct package+versions appear in
exactly one repo, and the most widely shared appears in ten — so the saving comes
from monorepos repeating declarations, not from an estate converging on common
packages. The default is now 86400 (one day), overridable with
KOSPEX_URL_CACHE_SECONDS. A day rather than a week because a cache hit means
deps.dev was not contacted, sodependency_data.last_checkedattests
"evaluated" rather than "fetched" and the honest bound on advisory age is
last_checked + TTL. An explicitcache=argument still wins, so a caller
needing fresher data can ask. -
kospex_metaremoved from the schema definitions. It was an abandoned
first draft ofobservations, added 2024-07-27 and never executable: its
PRIMARY KEYnamed columns the table did not declare, so running it raised
no such column: hash. In 2.2 years it was never read, never written and never
created in any database —connect_or_create_kospex_db()always omitted it.
No action needed: no database has the table, and nothing referenced the
constant. The practical effect is thatDB_CREATE_STATEMENTSis now safe to
iterate; all 18 statements execute. -
krunner osiqueries the database directly instead of copying the estate
into RAM.load_dependency_memory_db()copiedcommit_files,
file_metadata,reposandurl_cacheinto an in-memory database, and it ran
before osi scoped to the requested repos — sokrunner osi <repo_id>paid a
whole-estate copy to look at one repo. Only two of the four tables were ever
read: tracing the SQL gives 5 statements tofile_metadata, 1 torepos, and
zero tocommit_filesorurl_cache. The copy carried no indexes either
(CREATE TABLE AS SELECTdoes not reproduce the primary key), so every lookup
was a full scan of RAM rather than a seek. On a 167-repo estate the read path
goes 11.51s → 5.24s from this change alone, and ~840MB of resident memory
is no longer allocated. The other fivekrunnercommands that build in-memory
databases are untouched — they readcommits/commit_filesand genuinely use
them. -
file_metadatais now indexed on(_repo_id, latest)(migration0007).
Its only index was the primary key(Provider, hash, _repo_id), led by the
filename, so every repo-scoped query scanned the whole table — including the
UPDATE file_metadata SET LATEST = 0 WHERE _repo_id = ?that runs on every
sync. For a median-sized repo,file_metadata(repo_id)goes 25.11ms → 0.19ms
at 131k rows and 8,506ms → 1.63ms at 8.05M rows; the sync reset goes 294ms
→ 2.4ms at 2.04M rows. Combined with the change above, the osi read path over
167 repos goes 11.51s → 0.40s, returning the same 1081 dependency files.
Cost: roughly 9% on bulk insert and 12.6% on thefile_metadatatable (8.5%
on the real 2.0GB dev database). Estate-wide queries —tech_landscape()with
no repo scope,repos_with_tech()— never constrain_repo_idand cannot use
this index. -
kweb's
/and/repos/serve repeat loads from cache./goes from
~1.7s to ~0.01s, and a repeat/repos/load from ~7s to ~0.09s, on a
1.1M-commit database./repos/now caches itsrepos(),developers()and
active_devs()queries, and shares entries with/. The first load after a
sync, or after the hour-long expiry, still runs the full queries.
Seechanges/202610-kweb-summary-cache-and-threadpool.md. -
kweb route handlers run in FastAPI's threadpool. They are plain
def
rather thanasync def, so a slow page no longer blocks other requests behind
it:/healthanswers in ~2ms while a cold/repos/load is running.
GraphServiceopens its SQLite connection per use rather than at import,
because sqlite3 refuses a connection created on another thread. -
kweb --debugwatches only kweb's ownsrc/directory for reloads.
Started from the repo root, it was checking about 4,900.pyfiles every
0.25s,.venvand any worktrees included. That held the reloader at 36–67%
CPU, and a pip upgrade or an edit in another worktree restarted kweb and
emptied its cache. Templates still update without a restart. -
kweb and
kospex_querytimings and debug output go to the log files, not
stdout.@timerresults go tokospex_query.log, and kweb's page timings
tokweb2.log. 22 leftover prints on request paths became debug or warning
log lines, or were removed. Among the removed ones,/orphans/printed every
repo's committer emails on each request.krunner'sStart date:and
Executing query:lines now appear only at DEBUG level. -
Repo and active-developer summaries read from an index instead of the
commitstable (migration0011). On a 1.1M-commit database,repos()
goes from 1.66s to 0.49s,repos(org)from 0.92s to 0.09s, and
active_devs()from 1.02s to 0.08s. These back kweb's/,/summary/and
/repos/pages, and the developer-stats rebuild at the end of every sync.
Seechanges/202610-sync-time-summary-stats.md. -
Developer tenure is calculated about 2.8x faster.
days_between_datetimes()
parsed every timestamp with dateutil's general-purpose parser. It now tries
datetime.fromisoformatfirst, and falls back to dateutil only for values that
aren't ISO 8601.developers()on a 72,649-developer database goes from 3.96s
to 1.43s, which speeds up every page and command that shows tenure. Results are
unchanged: all 1,131,765 distinct stored timestamps parse identically both
ways, including the two malformed shapes found in real data.
Seechanges/202610-sync-time-summary-stats.md. -
developer_stats.first_commit/last_commitare committer time, and
author time moves tofirst_authored/last_authored. They held author
time whilecommits_last_90_days, and every web page, used committer time.
Rebases, cherry-picks and applied patches keep the original author time, so
about a quarter of (repo, author) pairs disagreed.kospex statstherefore
shows committer dates from a repo's next sync onward, and rebased work shows
later dates. Existing rows are rebuilt bykospex rebuild developer-stats -all
(see Upgrade notes), andlast_authored IS NULLmarks a row that hasn't
been rebuilt yet.kospex key-personis unchanged: it calculates tenure live
fromcommits, by author time. -
kweb's
/and/summary/count statuses fromdeveloper_statsinstead
of scanningcommits. For the whole estate that's 0.19s instead of 1.81s
warm (~5s cold), on 1.1M commits, with identical counts for every scope
measured. There is no fallback: on a database upgraded from an earlier
version, runkospex rebuild developer-stats -allfirst (see Upgrade
notes).kospex system-statushas a new "Summary stats coverage" section
that reports any repo whose stats still need rebuilding. -
Rebuilding a repo's developer stats is one transaction, and 17x faster.
sqlite-utils commits each write statement on its own unless a transaction is
open, soupdate_developer_stats()committed, and synced to disk, once per
developer. A failure part-way also left the repo with only some of its rows,
because the old rows were deleted and committed first. It now deletes and
re-inserts a repo's rows in one transaction. Rebuilding all 374 repos of a
1.1M-commit database went from 63s to 3.0s (DefinitelyTyped: 11.9s to 0.3s),
and every sync runs this step. -
A tag is now checked against
panopticas.get_tags()rather than a
character pattern. With the value bound (see Fixed), the check is no longer
what keeps the tag out of the SQL text, so it checks something useful instead:
that the tag exists. The comparison is case-insensitive, because SQLite's
LIKEis itself ASCII case-insensitive — a case-sensitive gate would reject
spellings that would have matched anyway — and because panopticas recases tags
between releases (eslintbecameESLintin 0.0.21, while 66 stored files
keep the old spelling). The caller's spelling is passed through unchanged. The
vocabulary is read per call rather than cached; it is a comprehension over
panopticas' rule tables, measured at 0.03 ms.One known gap:
get_tags()reports only the tags panopticas can currently
emit, so a tag it has renamed is rejected like a typo. On live data that is
Claude Code(7 files), now brand-levelClaude, which reaches the same rows.
Tracked as #317, pending
get_deprecated_tags()upstream.
Fixed
-
/tech/{tech}lost its entire DataTable to an out-of-range sort index.
repos-tech.htmlcarriedorder: [[4, "asc"]]against a two-column table.
DataTables 1.13.11's sort-flatten readsaoColumns[i].sTypewith no bounds
check, so an index past the last column dereferencesundefinedand throws
during init — costing the whole table: no search, no paging, no sorting, just
every row dumped into the page. The listing now has Server / Org / Repo /Files columns and orders on the count, which the query already returned and
nothing rendered. A test renders the page and asserts the configured order
index is within the table's column count rather than hard-coding it. -
The 404 page discarded every route's own message.
404.htmlhas had an
error callout since it was written and nothing ever filled it: the handler
droppedexc.detail, so a bad tag, an unknown repository and a mistyped path
all rendered the identical "Oops! Page not found." The detail is now shown,
which also revives the messages several routes already wrote ("Repository not
in the kospex database: …", "Organisation not found"). A detail that only
restates the heading — starlette'sNot Found, the catch-all route'sPage not found— is suppressed rather than repeated in a warning box below it. -
/summary/<org_key>showed the estate-wide developer count next to the
org's repo count (e.g. 3 repos and 72,649 developers). kweb cached the
developer list under one fixed key, so whichever scope loaded first was served
to every scope. The scope is now part of the cache key. Cached entries also
expire after an hour, because statuses and "days ago" are relative to now.
last_syncvalues are compared as times rather than strings, so a sync
recorded in a different UTC offset still invalidates the cache. -
/summary/<base64 email>returned a 500. The summary route decodes a
base64 id to a developer's email, but its queries take no email, so the page
always failed. Nothing links to it, and a summary is per repo, org or server,
so it now returns a 404 that points to/developers/. -
Key person
tenurewas about 3% too high. It divided the days between an
author's first and last commit by 354 rather than 365, so a year of commits
showed as1.031. Every other years figure in kospex already used 365. It
affectskospex key-personand the web key person page. Nothing is stored, so
the next run shows the corrected value. -
The package check page returned a 500 for a file it can't read.
Uploading a file whose name kospex doesn't recognise (req.txt,
dev-requirements.txt,Pipfile), or one its parser can't read (invalid
JSON), failed with'NoneType' object is not iterable. An unrecognised name
is now a 400 and an unreadable file a 422, each with a message saying which.
The page also showed every failure asHTTP 500: Internal Server Error
because its error handler discarded the server's message; it now shows it.
An unexpected error no longer sends its exception text to the browser
(CWE-209). -
kospex sca -jsonprinted invalid JSON for a repo with an unparseable
remote (#277). When
originwas something the remote-URL parser can't read, such as a clone of a
local path,KospexGit.set_remote_url()printed
WARNING: Failed specific parsing of remote URLto stdout, ahead of the
JSON array, sojson.load/jqfailed even with stderr discarded. The
warning now goes to stderr and thekospex_gitlog, and names the URL with
any embeddeduser:password@removed, so a token in a remote is not written
to either (CWE-532). -
An un-enriched dependency row counted as vulnerable
(#258).
dependency_data.advisoriesandversions_behindareINTEGERcolumns, and
every record template seeded them with''. SQLite's type ordering sorts TEXT
above every INTEGER regardless of content, so'' > 0is true: a package
parsed from a manifest but never enriched matchedWHERE advisories > 0— the
predicate meaning "has a security advisory" — while being in none of> 0,
= 0orIS NULL, the three buckets a reader would treat as exhaustive.The two layers then disagreed about the same row. SQL called it vulnerable,
while the web UI'sitem.get("advisories") or 0coalesced the falsy''to 0
and rendered it clean. Neither was right; the honest answer is unknown.save_dependencies()already carried a guard whose comment stated the intent
exactly — "None, not 0 — 'we did not ask' is not 'we asked and found none'" —
and it could never fire, becausesetdefaultonly fills a missing key and
the templates always supplied one. All six copies of the record template now
useNone, and the save path coerces''away for any future caller that
reintroduces it. A measured0still means "we asked and found none" and is
left alone.Existing rows do not self-heal. A re-save supersedes rather than deletes,
so most affected rows are atlatest = 0, where no reporting query looks: on a
several-hundred-repository estate 1,968 rows carried''and 1,962 of them
were superseded, leaving 6 of 3,537 current reported-vulnerable rows wrong. The
same estate reports 35% inflation if you drop thelatest = 1filter, because
that counts superseded history. Clearing them is a hand-run data change — see
Clearing unknown advisory counts indocs/refreshing-data.md. -
osi_runs.started_atrecorded when a repository finished, not when it
started (#259). The column was
declared "UTC, when this repository began" and held the opposite:
record_repo_run()defaulted it to now, and its only caller runs after
extraction, enrichment, save and the outcome record are all done. On a
whole-estate run the earlieststarted_atwas 14m34s after the run's own
"starting" log line, and that row'sduration_mswas 874,159ms — exactly
14m34s. Every row's start was its finish._process_reponow stamps a wall-clock start alongside the existing monotonic
one and passes it through, andstarted_atno longer defaults to now — a
caller that does not know the start writes NULL, meaning unknown, rather than a
confident wrong value. That default was the whole defect.Migration
0010addsfinished_at, because neither end of the window can be
derived from the other:duration_mscomes fromtime.monotonic(), which does
not advance while the machine is asleep, sofinished_at - duration_msis not
the start andstarted_at + duration_msis not the finish. On the run above,
1h51m of a 5h34m wall clock was system sleep. One end plus a duration measures
cost; two ends measure occupancy, and their difference is time the process was
not running — otherwise recoverable only from operating-system power logs that
rotate away.ORDER BY started_atalso returned completion order, which is
identical for a serial run and wrong as soon as anything is concurrent. -
kospex deps -filecrashed on any manifest with an unresolved package
(#264). Since #106 an unresolved
deps.dev lookup — an unpinned requirement, a range, an unknown package — has
versions_behind = None, meaning "could not tell", and the summary compared it
with> 2. Most realrequirements.txtfiles hit it. The summary now reports
ResolvedandUnresolvedcounts, and takes theCompliant/Non Compliant
percentages over resolved packages only, labelled(of resolved), omitting
them when nothing resolved. Previously an unresolved package could not be
counted at all; counting it as compliant would have reported a file full of
unpinned packages as nearly 100% compliant. What compliance should mean is
left to #265. An unsupported or unparseable file also crashed the summary
(len(None)); it now exits 1 afterassess()has said why. -
A package removed from a manifest stayed current after
kospex sca -save
(#151).assess()upserted
rows atlatest=1without demoting the ones they superseded, so a removed or
renamed package was still reported as a current dependency. It now writes
throughsave_dependencies(), the writerkrunner osiuses, which demotes the
whole file in the same transaction and stampssource(kospex sca). A
whole-file demote is safe because extraction is always complete —-scope
filters only what is displayed. -
A file inside a linked git worktree or a submodule was attributed to the
enclosing repository.find_git_base()recognised only a.git
directory; in a worktree or submodule.gitis a file, so the walk carried
on up to the outer checkout andassess()stamped rows with that checkout's
repo id and HEAD. It now accepts either. -
A
package.jsonorpackage-lock.jsonbeginning with a UTF-8 byte order
mark was reported as unreadable, though npm reads it. A BOM is what a
Windows editor writes and is never content: verified against npm 11,
npm pkg get namereturns the name from a BOM-prefixedpackage.jsonand
npm ls --package-lock-onlyresolves the locked version from a BOM-prefixed
lockfile. Both are now read withutf-8-sig, which consumes a BOM when
present and is identical toutf-8when it is not.Found by listing every file that reported
parse_errorafter the fail-open
fix below and reading them one at a time — eight of the nine were test
fixtures the upstream projects deliberately committed as malformed, and the
ninth was this. The contract did not cause the bug; it made it visible,
because the file previously failed open asempty.Two formats keep refusing a BOM, because their own tooling does and the file
genuinely cannot be built:pyproject.toml, wherepip install .dies at
tomllib.loads, andgo.mod, whose parser has no BOM handling and whose
ModulePathtests amoduleprefix that a BOM defeats. Both now name the
BOM in the error. Thego.modcheck runs before themodulegate, which
had been reporting "nomoduledirective, so not a go.mod" about a file
whose first directive ismodule— the right outcome with a message that
sent the reader after the wrong defect.pnpm-lock.yamland*.csproj
already tolerated a BOM, as the YAML and XML specs permit one, and
requirements.txthas since the UTF-16 fix; all three now have regression
tests. UTF-16 stays refused, as npm refuses it too. -
A multi-document
pnpm-lock.yamlread as zero dependencies. When a
project pins pnpm itself throughpackageManagerDependencies, pnpm writes
the closure of the pnpm binary as one YAML document and the project's own
closure as a second.yaml.safe_loadaccepts exactly one document and
raisesComposerErroron the stream, so the whole file produced nothing —
and, before the fail-open fix below, was reported asempty. On a
development estate six lockfiles across three repositories held 7,337
package entries between them, all invisible; one was a 3,652-entry closure
for a single repository.Read with
safe_load_alland extracted per document, because
classification is a per-document fact: pooling the stream's importers would
resolve a direct dependency of the second document against the first
document's importers, which do not mention it. Every document is validated
before any is extracted, so an unusable second document cannot return the
first one's packages as if the file had been read in full. Verified through
the production extraction loop: one affected repository went from 27 rows to
700, another from 6,466 to 11,022.Re-run
krunner osito pick these up — the rows do not exist until a
scan writes them. -
A manifest kospex could not read was recorded as one declaring nothing
(CWE-392).classify_outcomeendsreturn EXTRACTED if row_count else EMPTY, andparse_erroris reachable only when the parser raises —
so raising is the only error channel an extractor has. Four parsers caught
their own exceptions and returned[], which arrives asrow_count == 0
with no error and is stored as the outcomeempty, whose documented meaning
is "parser ran, produced nothing, genuinely declares none". An unreadable
pnpm-lock.yaml,go.mod,.csprojorpyproject.tomltherefore made its
repository look clean. Manifests come from cloned third-party repositories,
so the input is attacker-influenced.Sharpest case: the
.csprojDOCTYPE guard (CWE-776, uncontrolled entity
expansion) refused a hostile file correctly and then returned[], so the
security control's own outcome was indistinguishable from a clean scan.All four now raise
UnreadableManifest, and both scan paths already turn
that into aparse_erroragainst the one file.[]is kept only for a file
that was read and genuinely declares nothing — a validgo.modwith no
require(42 of 188 on a real estate), an empty<Project></Project>(4 of
9), a pnpm lockfile whose only importer is empty. An unrecognised pnpm
lockfileVersionnow raises rather than reportingempty, which matters
because v10 is released and the parser reads 5, 6 and 9.Two gates were added where a format had none, each verified against every
such file on a real estate before being added: ago.modmust declare
module(188 of 188 do), and a.csprojmust have a<Project>root (9 of
9 do)..csprojPackageReferencelookup is now namespace-agnostic, so the
root gate does not open a fresh silent-empty path for a legal
namespace-qualified project file.The contract is now driven from the extractor registry — a test resolves
every registered parser and asserts none returns a value for a path it
cannot read. That test is what found thepyproject.tomlcase, which no
one had looked for. Of 357 real manifests parsed before and after,
349 are byte-identical; no schema change and no stored data rewritten.The
pyproject.tomlchange reverses
#97, where the swallow was
added because a malformed TOML fixture aborted an entirekrunner osirun.
Both scan paths have since gained broad per-file guards, so the parser no
longer has to swallow anything for the batch to survive; the extraction loop
was re-run over that same repository to confirm it completes, with the
malformed file as the oneparse_error. #97's other half — skipping a
single invalid requirement string inside a file that was read — is
unchanged. -
peerDependencieswere recorded asdirect, and extras as nothing at all.
Two scan paths writepackage_use, and each kept its own copy of the
requirements_type -> package_usemapping. The copies agreed on every value
they both mapped and diverged on the one neither did:parse_package_json
emitspeerandoptionalandparse_pyproject_fileemitsoptional, none
of which had an entry, so the fallback decided what was stored --
directon thescaside,""on theosiside, for the same file. On a
populated database 497 rows carriedpackage_use = ''(185 from
package.json, 312 frompyproject.toml), andkospex scarecorded peer
dependencies asdirect, which asserts the opposite of what a
peerDependencydeclares -- that the consumer supplies it, not this package.
PACKAGE_USE_PEERandPACKAGE_USE_OPTIONALexisted inPACKAGE_USE_VALUES
and were unreachable.There is now one mapping,
KospexSchema.REQ_TO_PACKAGE_USE, imported by
both paths as the same object rather than copied -- the old comment asking the
two tables to "mirror" each other was a request, not a mechanism. An unmapped
value now stores NULL, following migration0009's rule that NULL means
unknown:directclaimed the manifest said something it did not, and""
is indistinguishable from a parser that produced no value, which is how this
stayed invisible. A contract test reads the parser sources and fails if any
parser emits arequirements_typewith no mapping, because the original
defect was a missing mapping and nothing failed while it was missing.Existing rows keep their old values until the estate is re-scanned.
package_useis currently written by both scan paths and read by none, so
this corrects stored data rather than anything displayed. -
A UTF-16
requirements.txtlost its first dependency.
detect_encodinginspects the leading bytes for a byte-order mark and returns
an encoding. For UTF-8 it returnedutf-8-sig, which consumes the BOM; for
UTF-16 it returnedutf-16-le/utf-16-be, which do not, so the mark
survived decoding and the first declaration became
"\ufeffrequests==2.31.0"— not a valid PEP 508 requirement, and dropped.
The function existed to handle BOMs and got two of its three cases wrong. Now
returns bareutf-16, which reads the mark, picks the byte order from it, and
consumes it. Only the first line was affected, which is why it was quiet. -
A malformed
pyproject.tomlinvented dependencies that do not exist.
PEP 621 declaresdependenciesand eachoptional-dependenciesgroup as
arrays. A string is also iterable, sodependencies = "requests"was iterated
character by character and produced eight rows namedr,e,q,u,
e,s,t,s— written todependency_dataand looked up on deps.dev.Not a silent miss but fabrication:
e,q,u,s,tandpare all
real PyPI packages, so a malformed manifest made kospex report a repository as
depending on packages it has nothing to do with, with those packages' real
advisories attached.package_nameis part of the primary key, so the rows
persisted. A TOML table is now rejected as well — iterating one yields its
keys, which happened to give the right answer and hid the shape of the file.
Both now raise, which the scan paths record as aparse_error. -
A
package.jsonwhose top level was not a JSON object reported no
dependencies. A file containing a JSON array or string returned zero records
rather than raising, because"dependencies" in ["x"]is a legitimate
membership test that answersFalse; a file containing a number raised
TypeError. All three are valid JSON, so the load could not catch them, and
npm rejects such a file outright — so zero records read as "this repository
declares no npm dependencies" for a file kospex could not parse. It now raises,
which both scan paths record as aparse_erroroutcome. -
pyproject.tomlextras group names were read and discarded.
for group, deps in optional.items()never usedgroup, so every entry under
[project.optional-dependencies]produced an identical row -- 129 distinct
group names across 432 declared dependencies in one estate collapsed to one
value. Both manifest parsers now populatedeclared_scope(migration0009)
with the raw section name in the file format's own vocabulary:
dependencies/devDependencies/peerDependencies/
optionalDependenciesfor npm, and the extras group name for Python.A
devextras group is still recorded asoptional, deliberately. PEP 621
extras mean "installed only when requested", which is not npm's
devDependencies, andcheck/cover/typewould be guesses. The name
is stored as evidence so the classification is a later query, not a re-scan. -
kospex scarecorded no advisory data for pnpm transitive dependencies. One
condition in_enrich_dependency_recordsskipped deps.dev entirely when a
pnpm-lock.yamlentry was neitherdirectnordev. On a real repository that
left 1991 of 2079 packages (96%) with noadvisoriesand no
versions_behind— concentrated in the layer where supply-chain compromise
actually lands, since transitives are where malware hides precisely because
nobody inspects them. It was also anosi/scadivergence:krunner osinever
skipped, so the same manifest produced different advisory data depending on which
tool read it. The skip is removed; every entry is looked up, and transitives stay
labelledtransitiveso a consumer can still tell a declared dependency from one
pulled in beneath it. The cost that motivated the skip is handled where it
belongs rather than by not looking —depsdev_recordalready short-circuits a
version it cannot resolve, so aworkspace:*entry still costs no round trip. -
Stale advisory counts survived an un-enriched re-save, under a fresh
last_checked.save_dependencies()never calls deps.dev, so it resets
resolved_versionto""for a caller that did not enrich — otherwise a
resolved version from a lookup that did not happen this time would sit under a
freshly advanced timestamp.advisoriesandversions_behindwere not reset
the same way, so the previous run's counts carried over instead, which a
reader cannot tell apart from counts confirmed a moment ago. Not reachable
throughkrunner osi, whose only call site enriches immediately before
saving; it becomes reachable the moment extraction and enrichment are
decoupled, such as a batched run that parses manifests estate-wide but
enriches only the current batch. Both columns are now reset toNULL— not
0, since "we did not ask" is not "we asked and found none". -
A repo kospex has never synced reported as having no developers.
kospex developers -repo .on an unsynced repository printed "No active
developers found in the kospex DB" and exited 0 — byte-identical to a synced
repository that is genuinely dormant. The reported case had 137 commits, the
most recent that week. For a tool whose purpose is identifying unmaintained
code, a false "unmaintained" is the most damaging answer available, and the
wrong reading is the natural one.developers,tech-landscapeandhotspot
now say which situation it is and exit 1, so scripted callers can branch.
tech-landscapepreviously rendered an empty table with no message at all.
Three states are now distinguished: not in the database, in the database with
no commits recorded (an incomplete sync), and genuinely dormant. -
Sync suggestions named a command that does not exist, and then the wrong
one.hotspot's out-of-sync message said to runkospex sync <path>— that
command is commented out and parked (#123), so following it produced a usage
error. The suggestion now matches the situation, since the three real
commands do different things:kospex sync-directory <path>syncs what is on
disk with no network,kgit sync <url>clones and syncs, andkgit pull <repo_id>refreshes a clone kospex already knows about. A repo absent
from the database is told tosync-directory(kgit pullcannot help — it
only refreshes known repos); a repo that is known but stale is told tokgit pull, withsync-directoryoffered as the no-network alternative. -
/dependencies/and/osi/returned every row when the URL segment was a
base64 author email. Both hand-rolled the scope cascade, and the fallback
branch printed an error then fell through with noWHEREclause, so the query
ran unscoped — 852 rows on a 111-repo database, identical to no scope at all.
A scope the query cannot honour is now refused, and the routes return 400
rather than rendering the whole table. "No scope requested" still means all
scope, whichkrunnerrelies on. -
/collab/rendered an empty table for an id that was never valid. The id
was passed straight to the query with no validation, so an unparseable id and
a repository with no collaborators looked identical. It now returns 404,
with different messages for "not a repo_id" and "not in the kospex database". -
Web routes converted their own
4xxinto a500. A route raising
HTTPExceptioninside atrywhoseexcept Exceptionmaps everything to a
500 reported its own client error as a server error —HTTPExceptionis an
Exception./file-collab/was doing this in production: its deliberate
"file_path parameter is required" 400 reached clients as a 500. All 42
handlers that convert to a 500 now re-raiseHTTPExceptionfirst. Two
handlers that recover rather than convert — the 404 template fallback and
the supply-chain search form — are deliberately left alone, since re-raising
there would turn a handled error into a propagated one. -
pom.xmlwas reported asunclassifiedrather thanunsupported
(#221). It had no extractor
registry entry, so a Maven manifest looked like a file kospex had never heard
of instead of one it recognises but cannot yet read. It is now registered as
amavenpackage manifest with no parser, alongsidebuild.gradle. Nothing
is extracted from it yet — this only moves it into the parser backlog that
unsupportedis meant to count. -
7 of the tags in use could not be queried
(#291).where_tag()
validated the tag against^[a-zA-Z-]+$and raisedValueErrorfor anything
else, so.NET,ASP.NET,Azure DevOps,C#,Claude Code,
Visual Studioandllms.txt— 7 of the 63 distinct tags in a live database
— were rejected on a., a#or a space.krunner devs-by-tag -tag "C#"
raised rather than returning its developers. The tag is now a bind
parameter, so every tag in the data is queryable. The pattern escapes%,
_and\and the clause declaresESCAPE, so a tag containing a LIKE
wildcard matches itself and not its neighbours — a tagA_Bwould otherwise
have matchedAxB.
Security
-
kweb's web security setup is simplified for running on localhost. kweb
listens on 127.0.0.1 and its pages only call kweb itself, so it no longer
sends CORS headers. The previousAccess-Control-Allow-Origin: *allowed
other websites to read kweb's responses, and kweb never needed it.kweb also now checks the
Hostheader and answers only to127.0.0.1and
localhost(any port), returning400 Invalid host headerotherwise
(CWE-346). Binding to 127.0.0.1 limits who can connect, and this adds a
check on the name a request uses. That stops a website reaching kweb by
pointing its own hostname at 127.0.0.1 (DNS rebinding).To reach kweb by another name or IP, list them in
KOSPEX_WEB_ALLOWED_HOSTS
as comma-separated values, set in the environment or inkospex.env. This
applies from another machine, through a reverse proxy that forwards the
originalHost, or from another container by its Compose service name.
Wildcards take the form*.example.com. A malformed pattern is skipped with
a warning, and*turns the check off and logs that it has. In Docker,
browsing to a published port onlocalhostneeds no setting. Publish it as
-p 127.0.0.1:8000:8000, because a plain-p 8000:8000exposes kweb on
every interface of the host.http://[::1]:8000is refused; uselocalhost. -
Values from git metadata are printed literally, not parsed as console
markup (CWE-150). The CLI's rich consoles parsed every string they printed,
including table cells and titles, as markup. That covered author emails,
remote URLs andrepo_ids, which whoever commits to or names a repo
controls, and git accepts brackets in all of them. An email such as
a[/]@example.comaborted the command withMarkupError;[link=...]
rendered a terminal hyperlink to a different target;[conceal]hid the
value. Thekgit,kospexandkrunnerconsoles, andprint_table()'s
piped output, now have markup disabled. Messages that were styled with markup
keep their styling. -
where_tag()formatted the tag into the SQL text rather than binding it
(CWE-89). The clause was built asf"tech_type LIKE '%|{tag}|%'", placing a
caller-supplied value inside a quoted SQL literal, with an^[a-zA-Z-]+$
input filter on the line above as the only control. That filter held — letters
and a dash cannot close the quote — and the two call sites are both local
krunnerarguments, with no web route reaching the function, so no released
version was exploitable. It is fixed anyway, because an input filter should
not be the thing standing between a value and the statement: the tag is now
bound and the format check is a plain non-empty-string check. See Fixed
above for the behaviour change this also resolves.