Deleted records still show up

[simonw]

As illustrated by: https://open-data.datasette.io/socrata/resources?_sort=id&domain__isblank=1

It looks to me like deleted records stay listed in the database, maybe due to the code that restores their old stored statistics.

[simonw]

Relevant code:

scrape-open-data/build_socrata_db.py

Lines 22 to 31 in 6cbeb68

	for jsonl in files:
	if ".stats." in jsonl.name:
	db["resources"].upsert_all(_stats(jsonl), pk="id", alter=True)
	else:
	db["resources"].upsert_all(
	_docs(jsonl),
	pk="id",
	alter=True,
	column_order=("id", "domain", "link", "name", "description"),
	)

[simonw]

I'm going to look for link is null and delete those records at the end of the script: https://open-data.datasette.io/socrata/resources?_sort=id&link__isnull=1

[simonw]

https://open-data.datasette.io/socrata/resources now lists 10,484 records, none of which have a blank link field.