shmistorical-art-data-server

This repo contains the code for the backend Flask program(s) that shm used / continues to use to build up their art history "knowledgebase" and serve it to the frontend browser hosted on data.snailbunny.site and applications like Artographer and ArtiFactor--- this repo runs as a cloud server (via Akamai Linode)and supports similarity-searching and other fun functionalities (e.g. voronoi similarity map stuff for artographer) via a growing list of Flask templates and API routes...

This github repo doesn't contain the data itself, but you can download chunks of it in JSON form on data.snailbunny.site and

About this knowledgebase

Where does the data come from?

The historical artworks and art terms are the most substantial part of this corpus ( 26,886 artworks, 1,643 text entries including artist names and art / aesthetic / cultural terms). We assembled it by combining public-domain art and metadata from WikiArt with labels from The Artsy Genome Project, an art-classification system with ~1,000 terms for describing artworks. We have a text entry for every term and artist, plus links that connect related terms, artists, and artworks across the database. We also have image (ResNet50), text (MiniLM), and multimodal (CLIP) embeddings for every entry.

The comics are public-domain comics from Comic Book Plus. This dataset is still being built and organized; planned work includes OCRing, describing, and embedding every comic page.

The poetry dataset is Allison Parrish's Gutenberg Poetry Corpus.

How did you make this, and what can it do?

When there is no already-assembled dataset (or API) for a corpus of knowledge, I turn to tools like BeautifulSoup and Playwright to scrape data from open sources. I've built several tools that support the data curation / staging / organization process (ex: data cleaner, staging review, map check, etc) as well as API routes for querying this server and using the data in applications like Artographer.

I'm working on making these tools, workflows, and APIs more available and accessible for public use, which includes more documentation and public-facing design work, plus creating policies and licenses about how this work should be used.

If you'd like to support this effort, please feel invited to reach out or send a tip.

How to rebuild the database(s) on your end if you wanted to do that

The live databases and images aren't in this repo (they're large + licensing-sensitive, and mounted at runtime). But you can rebuild a small, browsable local copy from public JSON exports:

1. Visit data.snailbunny.site and, for each dataset (art / comics / poetry) and table, click Download as JSON.
2. Drop the files into app/LOCALDB/JSONs/ — a sample set (50–100 rows per table) is already included, so you can skip straight to step 3.
3. Run the converter (stdlib only, nothing to install):

   cd app/LOCALDB && python convert_json_to_db.py

This writes knowledgebase.db / comics.db / poetry.db and downloads images into app/LOCALDB/, exactly where app/config.py looks when running locally. It rebuilds the browsable row tables only — similarity search and maps also need embeddings, which means running the models in app/LOCALDB/build_scripts/. Full details in app/LOCALDB/README.md.

list of stuff it currently supports

• Similarity search over an art knowledgebase using precomputed embeddings stored in SQLite via sqlite-vec:
  • MiniLM (sentence-transformers) for text
  • ResNet50 (torchvision) for images
  • CLIP for multimodal "baseline" search Only the incoming query is embedded at request time; the database embeddings are precomputed.
• Map generation — projects embeddings to 2D (UMAP) and builds Voronoi region maps (including a hierarchical variant), served synchronously or via an async job queue.
• Comics browse API.

api endpoints (overview)

• Similarity: POST /keyword_check, POST /lookup_text, POST /image, POST /lookup_entry
• Direct reads: GET /text/<id>, GET /artwork/<id>, POST /database_request
• Maps: GET /generate_initial_map, GET /generate_voronoi_map, GET /generate_hierarchical_voronoi_map, POST /merge_voronoi_regions, async POST /submit_map_job → GET /job_status/<id> → GET /get_result/<key>, GET /demo_maps

See docs/MAP_API.md for map request/response shapes.

more nitty gritty details that claude thought u should know

Gunicorn serves the Flask app (app/index.py + blueprints under app/templates/, helpers under app/helper_functions/). A background worker (app/jobs/worker.py) runs long-running map jobs, with the queue and results in a local SQLite (jobs/jobs.db). Both processes start from app/bootstrap.sh.

The databases and images are large and licensing-sensitive, so they are mounted at runtime, never committed:

• /app/LOCALDB — knowledgebase.db (+ images/, comics.db)
• /app/generated_maps — the canonical + demo maps that get served, and the job cache

Quickstart (Docker)

docker build -t shmistorical-art-data-server:local .

# point these at your mounted data, then run:
LOCALDB_PATH=/path/to/LOCALDB \
GENERATED_MAPS=/path/to/generated_maps \
./docker_run.sh
# serves on http://localhost:8080  (models download + cache on first run)

docker_run.sh mounts the data, the model caches, sets --restart unless-stopped, and replaces any existing container.

Configuration (env vars)

var	purpose
FINAL_SQL_ADMIN_PASSWORD	gates the admin SQL endpoints (unset → disabled)
STAGING_ADMIN_PASSWORD	gates the data-cleaner UI (unset → disabled, fail closed)
RUNNING_IN_DOCKER	set to true inside the container

See .env.example.

License

MIT — see LICENSE.