Skip to content

History / Recipe CKAN Integration

Revisions

  • wiki: Phase C batch 2 - 4 intermediate cookbook recipes Recipe-Geographic-Enrichment: reverse-geocode NYC 311 lat/lon to city + borough + county + FIPS codes with %dyncols; add high-resolution NTA via Luau lookup tables with fallbacks; ~360k records/sec. Variations: forward geocode, IP lookup, WKT geometry export to GeoJSON, custom Geonames city sets. Recipe-CKAN-Integration: expanded from legacy snippets. Pull metadata from CKAN via ckanapi+jq+qsv jsonl, push CSVs via safenames+applydp+to postgres for DataPusher+, generate Data Packages with embedded stats, describegpt for CKAN data dictionaries with controlled tag vocabularies. Recipe-JSON-Schema-Validate: 6-step workflow on NYC 311 (1M sample): pre-populate stats cache, generate schema, hand-edit to tighten rules, validate at ~780k rec/sec. Deep-dive on three custom keywords: currency, dynamicEnum (file/http/dathere/ckan schemes), uniqueCombinedWith. Variations: --fancy-regex, Polars schema, CI gate, CKAN-hosted enum. Recipe-Multi-Table-Joins: three scenarios in one recipe - lookup join (wcp + country_continent with inner/left/anti/semi), asof time-series join (NYC 311 + NOAA GHCN daily weather with --strategy backward), repeated-column wide merge (original cookbook scenario, updated for current command syntax). Plus non-equi salary bands, cross join, pre-join filtering. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

    @jqnatividad jqnatividad committed May 13, 2026
  • wiki: add stubs for Phase B/C/D/E pages so sidebar links resolve Adds 39 placeholder pages so every sidebar entry resolves to real content rather than a 404. Each stub declares its tier, the phase it will be filled in, and a one-paragraph preview of what's coming. They link back to Home / Getting-Started / Command-Reference / Cookbook for navigation. Pages added: - Phase B (Command Reference, 13): Command-Reference, Selection-and- Inspection, Transform-and-Reshape, Aggregation-and-Statistics, Joins- and-Set-Ops, SQL-and-Polars, Validation-and-Schema, Conversion-and-IO, Geospatial, HTTP-and-Web, Scripting-Luau-Python, Indexing-Compression- Diff, AI-and-Documentation - Phase C (Cookbook recipes, 12): Recipe-Inspect-Unknown-CSV, Recipe- Clean-and-Normalize, Recipe-Geographic-Enrichment, Recipe-Date- Enrichment, Recipe-CKAN-Integration, Recipe-JSON-Schema-Validate, Recipe-Build-a-Data-Pipeline, Recipe-Stats-to-Insights, Recipe-Fetch- and-Cache, Recipe-Larger-than-RAM, Recipe-Diff-and-Audit, Recipe-Multi- Table-Joins - Phase D (Tuning + ecosystem, 8): Performance-Tuning, Environment- Variables, Stats-Cache-and-Caching, Lookup-Tables, Claude-Cowork-Plugin, MCP-Server, qsv-pro-Spotlight, Integrations - Phase E (Polish, 6): Troubleshooting, FAQ, Comparison, Glossary, External-Resources, Contributing-to-the-Wiki Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

    @jqnatividad jqnatividad committed May 13, 2026