Skip to content

Releases

MapSmith 0.12.1

Choose a tag to compare

@mapsmith-ai mapsmith-ai released this 02 Nov 20:11

Records continue to declare spec_version 1.0.0-draft.10; no manifest key is new or renamed in this release.

Fixed

  • Compare every declared geometry CRS semantically in calculate_field. Primary and secondary GeoParquet CRS declarations, including native GeoArrow declarations, must match the input. Equivalent PROJJSON serializations remain accepted, while an actual CRS change fails verification and retains the manifest.
  • Correct the public showcases. Include 0.12.x in the draft.10 range, describe Argleton's current multi-step scope without calling every trap single-step, and correct the current trap count.

MapSmith 0.12.0

Choose a tag to compare

@mapsmith-ai mapsmith-ai released this 02 Nov 16:46

Records continue to declare spec_version 1.0.0-draft.10; no manifest key is new or renamed in this release.

Added

  • Restricted, deterministic field calculation. calculate_field creates or replaces one numeric or boolean attribute through run_operation, with arithmetic, comparisons, boolean logic, CASE, COALESCE and NULLIF. DuckDB's parsed expression tree is validated fail-closed; geometries, all other attributes and CRS stay unchanged, and non-finite calculated values are refused with the provenance record retained.

Fixed

  • Declare the workspace's SELECT-only behavior in the run_sql catalog summary.
  • Correct the multi-step method-choice benchmark. Expose output cardinality and geometry constraints, permit independent repeated-operation branches, separate unsupported requests from method-choice scoring, and accept minimal defensible sequences. Unsupported outcomes now have exhaustive, explicitly disclosed classes.
  • Clarify preview and unsupported planning contracts. The preview_map summary declares EPSG:4326 without changing its discovery terms, and the planner refuses read-only approximations of writes, per-feature iteration and multi-engine requests.

MapSmith 0.11.0

Choose a tag to compare

@mapsmith-ai mapsmith-ai released this 29 Oct 16:15

Records continue to declare spec_version 1.0.0-draft.10; no manifest key is
new or renamed in this release.

Added

  • An executable agentic pilot and a measured fine-tuning decision.
    examples/agentic_pilot.py demonstrates typed planning, a validated brief,
    QGIS execution, overlays and a closed-form 1,300 m² answer with its plan
    manifest. The current mixed method-choice baseline does not justify
    proprietary-model fine-tuning; revisit only after a complete baseline shows a
    real advantage and explicit cost.
  • QGIS Processing through a GPL-isolated subprocess, starting with
    native:buffer.
    qgis_processing exchanges JSON with qgis_process, never
    imports QGIS, pins buffer semantics and returns a MapSmith-written,
    MapSmith-verified output with the actual QGIS version in the manifest.
    Distances remain explicitly in the input CRS's units. ArcGIS Pro and QGIS
    integrations are exercised locally, with engine identity recorded in provenance.
  • Vector/raster conversion chosen from measured gaps in public training
    exercises.
    rasterize_layer burns geometries or a numeric field onto a
    reference raster's exact grid, refusing CRS mismatches and invalid field
    values. polygonize_raster excludes nodata, supports declared 4/8
    connectivity, bounds feature counts, emits stable row order and records
    mechanical geometry repairs. Both are available through run_operation, with
    catalog contracts, provenance and closed-form round-trip tests.
  • Bounded, protocol-level agent repair for verification failures (issue #26).
    execute_plan returns structured checks with stable names, arguments,
    observed/expected values and hints. An agent may submit a corrected plan with
    the previous plan manifest, up to three attempts total; every attempt is
    linked on disk, reusing an earlier output is refused to preserve provenance,
    and a fourth attempt is refused before the engine runs. MapSmith does not
    choose repairs or execute arbitrary repair scripts.
  • Deterministic project briefs derived only from validated plans (D-063).
    brief_plan refuses invalid plans before producing project text; every brief
    statement carries an operation, parameter, validation, engine or installation
    source. Sixteen executable plan fixtures verify rendering, execution,
    plan-level manifests and closed-form results on real vector and raster engines.
  • A multi-step method-choice benchmark with 20 public cases and no change to
    Argleton's meaning.
    benchmarks/method_choice.py measures missing and
    reordered operations, wrong tools, output contracts, required assumptions,
    planning-silent errors and reproducibility, with defensible alternatives and
    separate calibration/evaluation splits (10 cases each). Cases come from
    paraphrased exercises in the QGIS Training Manual (GNU FDL 1.3,
    docs.qgis.org); repeated-tool exercises that need SQL arguments are outside
    this first method-only bench. API failures, invalid responses, stale prompts
    and unattempted cases have separate counts rather than becoming wrong plans.
  • Machine-checkable argument contracts in the catalog and in plan validation
    (issue #27).
    Choices, numeric bounds, cardinality, extensions and
    cross-parameter rules are declared once in catalog.py, exposed by
    list_operations(detail=true) and enforced before execution with
    INVALID_ARGUMENT_VALUE messages naming the rule and the offending value.
    resample_raster also accepts warp-only methods that its runtime already
    used, convert_format refuses unsupported extensions statically, and
    spatial_join refuses unknown engine names before routing.

Security

  • PROJ resource resolution is confined at startup, and SQL replacement scans
    cannot hide dataset paths through CTE scopes.
    pyproj, rasterio and pyogrio
    receive explicit absolute search paths without discarding grid-only
    directories; the process working directory then moves inside the workspace so
    DuckDB's PROJ cannot use the old cwd for explicit or EPSG-selected grids.
    DuckDB has no search-path setter, so run_sql also refuses explicitly named
    external grids. SQL path analysis now follows CTE visibility by scope and by
    recursive branch, including replacement scans without a file extension.
  • SQLite SRS catalogues are bounded and read case-insensitively. A catalogue
    is limited to 20,000 rows, 512 referenced systems, 64 KiB per definition and
    16 MiB overall. Reference views disable selective parsing, numeric text ids
    follow SQLite affinity, and every row gets the textual path check. Legacy
    SpatiaLite srs_wkt is read alongside srtext and proj4text.
  • JSON, text, XML, sidecar and ZIP-member parsing is bounded before whole-file
    reads.
    Inputs over 32 MiB are refused under a workspace instead of being
    materialised and parsed to find a coordinate system; a safe-looking prefix is
    never accepted. .prj, .aux.xml and SRSName text is bounded to 64 KiB,
    with oversized declarations refused in both modes.
  • Only CRS refusals, never permissions, are cached by file metadata. An
    allowed input is checked again, so replacing its bytes while preserving size
    and modification time cannot reuse a stale permission. Per-text CRS verdicts
    remain cached.
  • Audit sidecars stay inside the workspace and cannot replace one another.
    Provenance, failure and plan manifests reject external symbolic links and are
    replaced atomically. Plan validation and bounded repair reserve every output
    together with its three audit sidecars before an engine runs, so a step cannot
    overwrite another step's dataset or audit trail. Lineage reads use the same
    containment rule.

Fixed

  • An ordinary SpatiaLite EPSG catalogue no longer receives thousands of full
    PROJ parses per input.
    Only coordinate systems referenced by layers are
    parsed fully; an ordinary 5,804-row catalogue needs one in the regression
    fixture. Per-text CRS verdicts and refusal caching remain in place.
  • A native Parquet CRS no longer becomes OGC:CRS84 merely because geo.crs
    is absent.
    Readers and CRS probes now use a native declaration when one
    exists; an unresolved declaration stays unknown with its reason, while no
    declaration at all keeps the GeoParquet default. Explicit geo.crs values,
    including null, retain their meaning on primary and secondary columns alike,
    and multi-geometry files arbitrate by primary_column rather than the first
    geometry in schema order.
  • The agent-facing discovery guidance no longer quotes a stale 51-operation
    table.
    list_operations now explains the delivery mechanism without
    duplicating measurements maintained in the published benchmark. Unused CRS
    and scalar-transform helpers were removed after moving their remaining test
    references onto the real public paths.
  • Rasterizing disjoint or empty geometries no longer returns silent success.
    The background raster is still written, but a non-critical warning in the
    result and provenance says that no cells contain an input geometry.
  • Malformed repair manifests and unknown engines fail structurally. Invalid
    previous-plan sections produce INVALID_REPAIR_CHAIN instead of Python
    attribute errors, and an unknown spatial-join engine no longer generates an
    install command for an invented package extra.

0.10.1

Choose a tag to compare

@mapsmith-ai mapsmith-ai released this 26 Oct 11:34

Update if you run MapSmith under MAPSMITH_WORKSPACE. A dataset in a
format whose coordinate system MapSmith did not read itself -- GeoJSON with a
crs member, GML, FlatGeobuf and others -- could make PROJ open a file outside
the workspace, or a network share, before MapSmith refused it
(GHSA-36xg-cr27-6r4m).
This is the part 0.10.0's SECURITY.md and GHSA-3wvg-2qh8-8gg6 listed as still
open. Under a workspace, netCDF, HDF, JPEG 2000, ERDAS IMG and GRIB rasters are
now refused until MapSmith reads their coordinate system itself; without a
workspace nothing changes. Records still declare spec_version
1.0.0-draft.10, no key is new or renamed, and the answers are those of 0.10.0.

Security

  • Under a workspace, the coordinate system of every dataset format MapSmith
    accepts is read before GDAL opens the file, and a format it does not
    recognise is refused.
    0.10.0 read the declared SRS of GeoPackage,
    shapefile and GeoParquet only; for the other formats GDAL read it first, and
    PROJ opened any grid file it named, before MapSmith refused the file
    (SECURITY.md listed this as open). The path gate now reads it from the
    bytes for GeoJSON and Esri JSON (with a JSON parser, so an escaped key is
    seen), GML and its .gfs, FlatGeobuf, GeoTIFF's GeoKey text, file
    geodatabases, folders of shapefiles, zips of these, and a .prj beside any
    file. Under MAPSMITH_WORKSPACE a format it does not recognise -- an ERDAS
    .img, a netCDF, a JPEG 2000 -- is refused with the list of the ones it
    does; without a workspace it is opened as before. run_sql's ST_Read
    accepts every recognised format, where 0.10.0 accepted three. The verdict is
    cached by file name, size and modification time, so one input is no longer
    parsed by the gate, the probe and the reader in turn.

0.10.0

Choose a tag to compare

@mapsmith-ai mapsmith-ai released this 25 Oct 15:48

Update if you run MapSmith under MAPSMITH_WORKSPACE. A coordinate
reference system -- a tool argument, or the one a file declares -- could make
PROJ read a file outside the workspace or open a network share, and in
run_sql DuckDB's own PROJ could do the same, in every release since the
workspace and the refusal of network shares arrived (0.2.0). Both are closed
here (Security below, and SECURITY.md), and both are in one advisory,
GHSA-3wvg-2qh8-8gg6.

Records still declare spec_version 1.0.0-draft.10, and no key is renamed.
Two keys of MapSmith's own are new inside crs_decisions.transformation:
x-mapsmith:shift_from_crs_definition, when the datum shift is written into
the CRS itself (+towgs84), and x-mapsmith:default_was_ballpark, when the
engine's own choice would have applied no shift. Results move where an
operation that brings a vector input onto another CRS now chooses the datum
transformation as reproject_layer does (Monte Mario from Rome: up to about
80 m); nearest_neighbour_index refuses a feature with no position. Under a
workspace, run_sql runs only SELECT and takes coordinate systems and
dataset paths as literals.

Security

  • A coordinate reference system that reaches MapSmith's own code can no
    longer make PROJ read a file outside the workspace or contact a host.
    +nadgrids=, +geoidgrids= and the WKT
    and PROJJSON spellings of a parameter file name grid files by path, +init=
    names a file to parse, and a CRS reaches PROJ as a tool argument
    (target_crs) and from the metadata of the files an agent reads. Under a
    workspace a path outside it was read and its content moved the output; a
    //host/share grid was opened over SMB; with PROJ networking on, an
    http:// grid made requests to any host (measured 2026-10-23, present since
    reproject_layer exists). A CRS policy now refuses, wherever a CRS enters --
    tool arguments, the CRS a file declares, every reader, the datum module, and
    the admission of every input path, before GDAL probes the file -- any grid,
    init or model name that is not a bare name from PROJ's data directories
    (@null allowed), any transformation written as a PROJ string, and any
    declared SRS PROJ cannot read; the declared SRS is read as text from the
    file's own catalogue (GeoPackage by its bytes, shapefile .prj, GeoParquet
    geo), and the server switches PROJ networking off under a workspace at
    startup, for pyproj's PROJ and GDAL's. SECURITY.md records it, and names
    what stays open: datasets in other GDAL formats, which GDAL opens first.
  • run_sql under a workspace can no longer make DuckDB's own PROJ read a
    file outside it.
    DuckDB's spatial extension carries a PROJ of its own,
    which opens files natively and never met the connection sandbox:
    ST_Transform(geom, 'EPSG:4326', '+init=<a file outside>:1') read that file
    -- the output moved by the false easting written in it -- and so did
    ST_SetCRS followed by a two-argument ST_Transform, while read_text on
    the same path was refused (measured 2026-10-25; every release with the
    sandboxed connection, 0.2.0 to 0.9.0). Under a workspace the statement is
    now read with DuckDB's own parser (json_serialize_sql) before it runs:
    every CRS in ST_Transform, ST_SetCRS or a GEOMETRY('…') type must be a
    literal the CRS policy accepts, every path given to ST_Read,
    ST_Read_Meta, read_parquet or named in FROM must be a literal the path
    gate accepts (globs are expanded by the sandboxed connection and each match
    is checked), what GDAL would open must be a GeoPackage, a shapefile's .shp
    or a GeoParquet, recognised by its bytes -- FlatGeobuf, GeoJSON and the other
    formats whose SRS GDAL reads before MapSmith can are refused in run_sql --
    and the rest is refused with how to write it: statements other than
    SELECT (pass output_path to keep a result), SQL held in a string
    (query()), GDAL open options, and a CRS or a path computed in SQL. Without
    a workspace nothing changes: file access there is unconfined by design.
  • The path gate reads the CRS a Parquet file declares in the column's
    native GEOMETRY/GEOGRAPHY type, not only in the geo key.
    A file
    with the CRS there and no geo key at all passed the gate with nothing to
    check, and DuckDB read that CRS and handed it to its PROJ: through
    run_sql, a grid outside the workspace was read (found by the audit of the
    change above, before release). The projjson:<key> and srid:<n> forms
    are followed too. Every tool's input admission goes through the same gate.

Added

  • 103 phrasings on 36 operations, taken from how the QGIS Training Manual
    poses its exercises.
    Each is a paraphrase of a problem a lesson sets ("strip
    along both banks of a stream", "distance along the roads, not as the crow
    flies"), written from half of the 220 exercises extracted from that manual
    (GNU FDL 1.3, docs.qgis.org) and measured on the other half, which the
    phrasings never saw: BM25 found@3 on words alone 47% -> 59%, with the
    caller's facts 40% -> 51%. On the 118 independent requests of the published
    table every level moved up (words alone 29% -> 30%, fullest declaration
    60% -> 62%; the embedding engine 18% -> 19% and 56% -> 62%), and the README,
    docs/catalog-entry-spec.md, the site and the tool instructions an agent
    reads say the new figures. The ranking code did not change; the order
    list_operations returns for a given request can.

Fixed

  • run_sql can no longer crash the server with ST_Read of a layer that
    is not there.
    DuckDB's spatial extension (1.5.5) ends the process with a
    segmentation fault on ST_Read('roads.gpkg', layer := 'nope'), and the
    MCP server with it: one line of agent-written SQL was a denial of service.
    The defect is upstream (reproduced on a bare connection); MapSmith now
    checks the layer, in either spelling (:= or =), against the names
    ST_Read_Meta lists and refuses a missing one with the list, in every
    mode. A statement other than SELECT cannot be checked that way, so one
    that names ST_Read and a layer is refused with the advice to write the
    SELECT and pass output_path; that matters without a workspace, where
    such statements run (CREATE TABLE … AS still crashed after the first
    version of this fix, found by the pre-release review). Found by the audit
    of the run_sql change above.
  • A datum shift written into the CRS itself is no longer recorded as a
    ballpark.
    For a CRS defined with +towgs84 (or WKT TOWGS84[]) PROJ
    builds a Helmert from those parameters and applies it in full, but reports
    accuracy -1 for it -- the value it also reports for the ballpark -- and
    every crs_decisions.transformation read that value as "no datum shift
    was applied": reproject_layer's since 0.3.0, every other operation's
    since 0.5.0. Measured on Monte Mario with the seven IGM
    parameters: the pipeline carried +proj=helmert and moved the data 79 m,
    and the record said is_ballpark: true. The verdict now comes from PROJ's
    own name for the operation it used ("Ballpark geographic offset" is its
    word, and a concatenation carries every step's) and, failing a name, from
    the pipeline it ran: a shift step is a shift. The name of a shift a CRS
    declares itself is written by whoever wrote the file, so it is cut out of
    the description before the word is looked for: a Helmert called "Ballpark
    geocentric translation" is still a shift (security review, same day). A CRS that carries its own
    shift is also no longer rebuilt from a registry code on the way in, which
    dropped the +towgs84. Such a record says is_ballpark: false, accuracy_m: null (nobody has
    stated one) and carries x-mapsmith:shift_from_crs_definition: true to say
    why the two go together; docs/manifest-vocabulary.md has the entry. Data
    and coordinates are unchanged -- only the record was wrong, in the direction
    that hides a shift. Found by the pre-release review of 0.9.0.
  • Every operation that brings a vector input onto another CRS chooses the
    datum transformation the way reproject_layer does.
    reproject_layer has chosen its transformation since 0.5.0 (datum.best_operation:
    a published operation that covers the data, or a datum's Greenwich twin), but
    the eight vector operations that bring a second input onto the first one's CRS
    -- clip_layer, overlay_layers, nearest_join, merge_layers, spatial_join,
    count_in_polygons, summarize_points_in_polygons, apportion_by_area -- let
    PROJ pick, through to_crs. On a Monte Mario (Rome, EPSG:4806) layer PROJ's
    pick is the ballpark, so a parcel sat 79.5 m apart in clip_layer and in
    reproject_layer, each record true of its own run. The eight now go through
    the same chooser and apply it themselves, and crs_decisions.transformation
    describes what was chosen, with x-mapsmith:default_was_ballpark where the
    engine's own choice would have applied no shift. The twin route also serves a
    target counted from another meridian (a WGS 84 mask onto a Rome layer), which
    it did not. Operations that compute in an estimated UTM zone and come back are
    unchanged: the way out and the way back largely cancel. The coordinates of the eight
    change only where PROJ's default is the ballpark and a covering operation or a
    twin exists; everywhere else the chosen transformation is PROJ's own. The
    same chooser now serves every operation that brings a vector input onto
    another CRS: snap_layer, line_intersections, transform_by_control_points,
    compare_layers, nearest_neighbour_index, the points of
    sample_raster_at_points, elevation_profile and least_cost_path, the
    zones and mask of zonal_statistics and clip_raster, and the pour points
    and stations of watershed and viewshed. What still lets ...
Read more

MapSmith 0.9.0

Choose a tag to compare

@mapsmith-ai mapsmith-ai released this 08 Oct 20:56

Records still declare spec_version 1.0.0-draft.10, and no key is renamed.
Three keys of MapSmith's own are new, each declared with the reason it is not
a synonym of anything the specification has: x-mapsmith:areas_measured_on in
crs_decisions, from apportion_by_area, and x-mapsmith:ballpark_share and
x-mapsmith:outside_area_of_use inside crs_decisions.transformation. Three
check names are new, all from apportion_by_area:
x-mapsmith:extensive_total_preserved, x-mapsmith:every_source_value_placed
and x-mapsmith:no_zone_gives_more_than_it_holds. What
reproject_layer writes changes in two cases
, both described under Fixed:
coordinates outside every published operation's area are no longer moved by
an operation from elsewhere, and a geographic CRS counted from another
meridian whose operations are published only for its Greenwich twin
(EPSG:4806, Monte Mario from Rome) goes through that twin. Shapely 2.1 and
pyarrow 23.0.1 or later are required.

Added

  • apportion_by_area: move counts and rates from the zones they are reported
    in to zones that cross them.
    Cases by municipality to health districts,
    census-tract population to school catchments. Each column is declared
    extensive (a count, split by share of area) or intensive (a rate, averaged by
    area); a call declaring neither is refused, because splitting a rate or
    averaging a count gives a plausible wrong number. Each count comes with
    _min and _max that hold however it is distributed inside the source
    zones; rate bounds rest on the same uniformity the estimate assumes, and the
    record says which is which. Zones are cut in the source zones' own CRS,
    where a boundary both layers share is the same line, and a zone counts as
    wholly inside a target by a topological test, not by a ratio of areas.
    Shares are ratios of areas measured on an equal-area surface -- the source
    CRS when it is one, otherwise a plane centred on the zones on their own
    datum, named in the record -- so the pieces of each zone add up to it.
    Whether targets overlap is decided zone by zone, so a few square metres of
    overlapping boundaries are said in the record where they put a zone's value
    in two targets, and a check fails if a zone with no overlap gives out more
    than it holds. Invalid zones are repaired and the repair recorded; a zone
    the repair leaves with no area or with a coordinate that is not a number,
    zones spread over more than a hemisphere, layers writing longitude in two
    conventions, and a target that the source CRS would tear at one of its
    seams (the 180th meridian, a pole: caught by area and by position, whatever
    the units) are refused; targets keep their own CRS; and the counts written
    are checked against the pieces they came from.

Fixed

  • crs_decisions.transformation now describes what PROJ did to the data's
    own coordinates.
    Which operation PROJ selects depends on where a
    coordinate is, and the record asked at one point chosen from the source CRS
    alone -- for EPSG:4326, the middle of the world, in the Gulf of Guinea. It
    was wrong in both directions: WGS 84 data in Europe moved to EPSG:3035, or in
    England to EPSG:27700, was recorded as a ballpark (is_ballpark: true) where
    PROJ applied a published 1 m or 2 m shift; and NAD27 coordinates in Italy were
    recorded as a 7 m shift where PROJ applied none -- the error a ballpark record
    exists to report, hidden by it. Every operation that moves an input now hands
    over the moved layer, PROJ is asked what it did at a spread of the data's own
    points, inside and outside the areas of use of the operations it has -- outside
    them it applies no shift for some pairs (NAD27) and stretches a published
    operation for others (DHDN, the Swiss grid), so the answer is PROJ's and not
    inferred. A layer straddling the edge of a datum grid is recorded as a
    ballpark with x-mapsmith:ballpark_share saying how many of its vertices got
    no shift; x-mapsmith:outside_area_of_use says how many lie outside the area
    of every operation published for the pair, whatever PROJ then did with them;
    a move of no coordinates at all records that nothing was transformed; and a
    check with no data to hand asks where the two CRSs' areas of use overlap.
  • reproject_layer no longer applies an operation outside its area of use,
    and its output changes where it did.
    When PROJ's own choice at its probe
    point was a ballpark it substituted the first published operation for the
    pair, wherever the data was: WGS 84 data in Madrid moved into EPSG:3003 by
    the Italian operation came out 84.5 m from PROJ's own result, and WGS 84
    data in Italy moved into EPSG:27700 by the British one, 248 m. Now it
    substitutes only an operation whose area of use contains every vertex, and
    otherwise leaves each point where PROJ does. A geographic CRS counted from
    another meridian whose operations are published only for its Greenwich twin
    goes through that twin -- EPSG:4806 (Monte Mario, Rome) and EPSG:4819 (Nord
    Sahara 1959, Paris) among the 23 tried; NTF Paris, EPSG:4807, has an
    operation of its own and does not change. A station in Piedmont on EPSG:4806
    now lands on the published 4 m operation's answer, where it was 6.3 m off.
  • better_available_m names only an operation that covers the data and is
    not installed.
    It reported any published operation for the pair: EPSG:4806
    data in Piedmont was told a 44 m operation needed its grid installed, when
    that operation needs none and is valid in Sardinia only.
  • A record written on a failing path no longer claims an earlier run's
    output.
    With a dataset already at the output path, a run refused by its
    preconditions, or one that crashed before writing, recorded the digest of
    the earlier bytes as its own output -- and a lineage walk by digest then
    attributed them to the inputs of the run that failed -- and its record
    replaced theirs, leaving them with none. Such a record now omits output
    and says why, and where a record on disk still describes the bytes at the
    path it is left in place and the failing run's record is written beside it
    as <output>.failed.provenance.json, which the error names. Bytes the
    failing run did write keep their digest, including when a check of the
    output crashes after the write.

Changed

  • Shapely 2.1 or later is required (was 2.0): apportion_by_area uses its
    maximum inscribed circle to check that a reprojected target still holds its
    own interior.
  • pyarrow 23.0.1 or later is required (was 21), the release fixing
    CVE-2026-25087, a use-after-free reading Arrow IPC files; MapSmith reads none,
    and the floor keeps an install at the minimum from carrying it. pytest 9.0.3
    or later for the test extra.

MapSmith 0.8.0

Choose a tag to compare

@mapsmith-ai mapsmith-ai released this 05 Oct 10:24

Records declare spec_version 1.0.0-draft.10. draft.9 defines the digest
of a dataset made of several files, which is what the first fix below
implements; draft.10 gives each input a role and a configuration of its own.
One key moves: a second input's own georeferencing facts,
inputs[].x-mapsmith:environment on 0.7.x, are now inputs[].environment.

Added

  • cluster_points_by_distance: group points chained together within a
    distance.
    "Group everything within half a kilometre" is single linkage, not
    thinning: two points share a cluster when a chain of points joins them with
    every link at most the distance long, so chains merge, and every point is kept
    with its cluster_id and cluster_size. thin_points keeps one point per
    crowd, and its count is close to the number of groups but not equal to it. A
    check re-reads the written file and re-derives the chains by a second
    algorithm. A geographic CRS is refused, and so are lines, polygons and
    multipart points, which a point cluster has no single position for.

  • list_operations takes a geometry: what the caller's layer holds, as
    point, line or polygon, or a list of them for a mixed layer. The
    catalogue now has 31 operations taking one vector layer to another, one more
    than the size it hands over whole, so a caller declaring everything it knew
    got a shortlist back for the most common request there is; with the geometry
    declared, the right operation is in the answer for 116 of the 118 benchmark
    requests again. An entry declares the families it accepts only where its code
    refuses the others (fifteen do), an entry without a declaration is kept for
    every family, and a test calls each declared entry with every family it
    leaves out and requires the refusal.

  • benchmarks/choice_at_scale.py, with every choice it scores in
    benchmarks/choice_at_scale.json: each of the 118 requests handed to Claude
    Opus 5.5 in sets of 30, 40 and 50 candidates and compared with itself. From
    40 it chose right 6.9 points less often than from 30, from 50 5.1 points
    less, so the size list_operations hands over whole stays 30, and a test
    holds it at or below the largest size measured without loss.

  • Each input of an operation that reads more than one dataset says which
    argument read it
    (inputs[].argument, spec draft.10). The order of
    inputs is not significant, so until now a weighted zonal statistic with
    values and weights swapped produced a record that differed only in file
    names. Eighteen operations name their inputs (twenty code paths:
    spatial_join has three engines), with the names the input checks
    already use in verification[].argument, so the two can be joined; a
    single-input record is unchanged.

  • get_lineage takes a layer. The history of one layer of a file
    geodatabase now starts from the geodatabase itself: its record sits beside
    the container (spec section 3.1), and the walk matches the layer at every hop
    into a container. A layer given for a path that is not a container is
    refused rather than ignored.

Changed

  • get_lineage steps carry inputs[].argument, inputs[].environment and
    output.layer
    when the records hold them.
  • A shapefile output whose member files are ambiguous (out.dbf beside
    out.DBF, possible only on a case-sensitive filesystem) is written with a
    manifest that has no output digest and a note saying why, instead of a
    digest that would choose one of the two.
  • inputs[].x-mapsmith:environment is inputs[].environment, the field
    spec draft.10 defines for the configuration that changed how one input was
    read. Same content; only the name moves, and a reader of 0.7.x records reads
    the old one.
  • With more than one input, no file's facts sit on the record's
    environment, the first input's included.
    A sidecar beside the first
    raster was recorded there, which the format reads as true of the whole run,
    since the order of inputs means nothing: in a clip_raster it read as true
    of the mask. It is now on that input's own environment, and only process
    settings stay on the record. Seven operations can meet it (clip_raster,
    sample_raster_at_points, elevation_profile, least_cost_path, viewshed,
    watershed, zonal_statistics), and only when a sidecar is present. A
    single-input record is unchanged.

Fixed

  • cluster_points_by_distance is bounded. The pairs of points within the
    distance grow with the square of the points when the distance is large, and
    6,000 points at a distance of 1e9 took two minutes and 2.3 GB. The pairs are
    now found in blocks and the call is refused once they pass
    MAPSMITH_MAX_SAMPLES, the operator's limit, before memory is spent.
    hot_spots with weights="distance_band" uses the same query: its weights
    were a double loop over every pair of features in Python, quadratic at any
    band, and a band that is not a finite positive number is now refused.
  • get_lineage no longer hashes every shapefile in the workspace. Since
    the draft.9 digest rule it computed, while indexing, the other rule's digest
    of every shapefile record's output, whatever the chain asked about. A
    shapefile is now hashed only when a hop misses on it.
  • A Windows junction inside a .gdb is not followed. The digest of a
    directory container walked into junctions, which is_symlink() does not
    report, so files outside the container could enter its digest. Section 3.3 of
    the specification says links are neither members nor followed. The size used
    for the re-hash cap now counts the same files as the digest.
  • A single-layer .gdb read by any operation records its layer. Every
    operation now reads a single-layer file geodatabase, not only
    extract_layer, and the input records the layer it read: without it, a
    lineage walk meeting the record of that layer stopped with a false "records
    claim this container's digest, but for other layers".
  • get_lineage refuses a layer holding : or a control character. On
    Windows : in a file name opens an alternate data stream of another file.
  • thin_points refuses a distance that is not a finite positive number,
    like cluster_points_by_distance; and least_cost_path accepts a start or
    end given as a single-part MultiPoint, which it used to count as no point.
  • zonal_statistics refuses zones that are not polygons. Line zones were
    accepted and returned the line's length in pixels as the count, with a
    mean over the cells it crossed: a plausible table, and not a zonal
    statistic. count_in_polygons and summarize_points_in_polygons refuse a
    polygon layer holding anything else the same way. Found by the test that
    holds the new geometry declarations to the code.
  • least_cost_path is found by a caller holding points. It reads a cost
    raster and two point layers but declared its input as raster only, so a
    caller who said, correctly, that it held a vector layer never saw it. A start
    or end layer holding a line or a polygon beside its point is now refused; it
    used to pass with the extra feature ignored.
  • snap_layer snaps to a polygon or multipart reference. Both crashed with
    a raw NotImplementedError from shapely; every vertex of every part, holes
    included, is now a target.
  • A feature with no geometry is refused, with a count, by thin_points,
    sample_raster_at_points, points_along_lines and line_intersections.

    They crashed on it with an AttributeError, "cannot convert float NaN to
    integer" and a failed sort.
  • A shapefile's digest covers its attributes and its CRS. An input or
    output shapefile was digested as its .shp alone, so a record kept matching
    a dataset whose .dbf (attributes) or .prj (CRS) had been rewritten, while
    it said sha256 is of the bytes. The digest is now the listing of the
    .shp, .shx, .dbf, .prj and .cpg files, as section 3.3 of the
    specification defines it since draft.9; a file geodatabase input is digested
    the same way, lock files excluded. get_lineage recomputes digests with the
    same function, so a chain written by this release still resolves. A record
    written by an earlier release for a shapefile carries the old digest:
    get_lineage compares it under the rule it was written with, so an
    unchanged shapefile is not reported as edited, and a chain that crosses
    from a record written by 0.7.1 to one written now still resolves. A record
    written before this release can only vouch for the .shp: an edit to the
    attributes or the CRS made after it was written stays invisible to it, so
    such a hop is reported as reverified_shp_only, not reverified, and a
    history that contains one is not verified.
  • extract_layer works on a file geodatabase. The catalog listed .gdb
    among its containers, and the input digest opened the path as a file: a
    geodatabase is a directory, so the operation died before writing anything.
  • get_lineage matches the layer when a digest belongs to a container.
    Two records for two layers of one geodatabase share the container's digest
    (spec section 6, draft.9); a hop into a container now resolves to the record
    for the layer it names, and stops with other_layer rather than attribute
    one layer to the operation that wrote another. It re-checks a layer's record
    against the container it sits beside (<container>.<layer>.provenance.json,
    spec section 3.1), and two layers of one container reached on two branches
    are two histories.
  • A URI in a manifest path keeps its //, so its credentials are masked.
    The path normaliser collapsed https://user:pw@host/x to https:/user:pw@…,
    and the redaction of URI credentials looks for ://: the password would have
    stayed in the record. Not reachable while remote paths are refused, which is
    the default; found in the 0.7.0 audit. The weights raster of
    `zonal_statistic...
Read more

0.7.1

Choose a tag to compare

@mapsmith-ai mapsmith-ai released this 26 Sep 16:17

Two fixes from the pre-release audit of 0.7.0, both present in earlier releases,
and the Argleton run scored against trap 024's corrected truth, which arrived
after the 0.7.0 tag: that release's PyPI page still quotes the recount of the
old run. No answer 0.7.0 got right changes, and no manifest key moves: records
declare spec_version 1.0.0-draft.8, as in 0.7.0.

Fixed

  • An output that is one of the inputs, or has an extension its writer cannot
    honour, is refused before anything is written

    (#31). zonal_statistics
    with output_path equal to its weights raster replaced the raster with a
    directory of shapefiles — a vector writer given .tif falls back to the
    Shapefile driver — and failed before writing a manifest. The dedicated tools
    now refuse both at the boundary, as run_operation already refused the
    collision; and the plan validator turns an extension no writer can honour from
    a warning into an error (OUTPUT_EXTENSION_REFUSED), keeping the warning for
    writable but non-canonical ones such as .geojson.

  • A spacing could ask for any number of points. elevation_profile and
    points_along_lines materialised every point before looking at how many
    there were: a spacing of 1e-6 on a 90 m line is ninety million. Both now
    count the points from the line lengths and refuse above
    MAPSMITH_MAX_SAMPLES (default 1 000 000), an environment variable of the
    server that no tool argument can move; SECURITY.md says what is and is not
    limited. points_along_lines also refuses a NaN or infinite spacing, which
    spacing <= 0 let through.

0.7.0

Choose a tag to compare

@mapsmith-ai mapsmith-ai released this 26 Sep 14:20

This release changes answers earlier releases gave, and that comes first.
From 0.4.0 to 0.6.1 every position MapSmith read from a point-registered raster
(AREA_OR_POINT=Point — the USGS elevation products and the Copernicus DEM)
was half a cell north-west of the sample: GDAL already corrects for the tag, and
MapSmith corrected a second time. The same premise sat in the correctness suite
MapSmith is graded by, which scored the wrong answer as a pass; it has published
an erratum.
Terrain operations now accept point-registered DEMs, which they refused
outright. Two capabilities arrive on existing tools — weighted zonal statistics
and the gradient along a line — with a Copernicus notebook. No tool was added.
Two records change shape, both below: elevation_profile across two CRSs uses
the specification's source_crs/target_crs, and terrain operations record
the raster registration in crs_decisions.

Records declare spec_version 1.0.0-draft.8, as in 0.6.1.

Added

  • A Copernicus notebook. examples/04_copernicus_terrain_vegetation.ipynb,
    on Monte Baldo: a Copernicus DEM in degrees refused for slope and reprojected,
    a Sentinel-2 NDVI, and the treeline read off both. Two clips ship in
    examples/fixtures/ under their own licences, with the script that rebuilds
    them. It found both fixes above, and one fact about the data: for the item it
    uses, the Sentinel-2 catalogue declares a reflectance offset of -0.1 that the
    pixels already contain — deep water would read -7% — so the clip declares the
    scale alone, and the script re-checks it every time it runs.

  • Weighted zonal statistics. zonal_statistics takes weights_path, a
    raster of weights, and weighted_mean, weighted_sum, weighted_stdev,
    weighted_variance and weighted_frac: the mean heat of a district weighted
    by the population living in each cell, which used to take three operations
    and a division and is a different number from the plain mean wherever people
    are not spread evenly. Both rules were measured on the engine first.

    One is in the record: a cell with a value and no weight -- nodata or NaN --
    counts with weight 0, because exactextract's own default turns the zone's
    weighted statistics into NaN for a single missing weight. The record carries
    weight_of_a_cell_with_no_weight: 0.0 and the weights by name
    (weights_path) as well as in inputs[], and a non-critical check,
    x-mapsmith:every_valued_cell_has_a_weight, names the zones where it
    happened.

    The other is a refusal, so a record exists only when it held: the weights
    must sit on the value raster's own grid -- same CRS, cells (compared relative
    to the cell size) and registration -- because exactextract accepts a coarser
    grid by repeating each coarse cell whole in every fine one, which inflates a
    weighted sum of counts, and never compares the two coordinate systems. A
    weights raster off the grid, with two bands, or with a negative weight inside
    a zone is refused with how to align it; a negative weight outside every zone
    changes no number and is not.

    With two rasters in one record, each one's georeferencing facts stay with
    it: the top-level environment describes inputs[0], and the weights' are
    on their own entry under x-mapsmith:environment. Records of operations
    that read one raster are unchanged. Reached through the same tool; the tool
    count is unchanged.

  • The gradient along a line. elevation_profile takes grade_base_length
    and, optionally, grade_threshold_percent: every point carries
    grade_percent, the rise over the window of that length starting there, and
    the result gives per line the steepest window and where it starts, the total
    ascent and descent, and the stretches above the threshold. It closes four
    railway requests that each took a plan -- "the steepest section in percent",
    "the maximum gradient over any 100 m", "does any part exceed 2.5%" -- and
    three traps are decided rather than defaulted: this is not slope, which is
    the ground's steepest direction and on a track crossing a hillside at 45
    degrees says 2% where the track climbs 1.41%; the window slides by the
    spacing, so it answers "any 100 m stretch" and not "every 100 m from the
    start"; and the base length has no default and must be a whole number of
    steps. A window never crosses the gap between two parts of a multi-part line
    that do not touch (each point carries run_index), nor a point the raster
    could not read; a stretch is one direction of travel, so a climb and the
    descent after it are two. A line whose unit is not the metre needs
    grade_height_unit, because the rise comes from the raster's values, which
    state no unit, and metres over US survey feet is 3.28 times too steep. A check
    recomputes every stored grade from the profile as written and fails on a
    grade missing where a window fits, with the count in its detail.
    The new search phrasings moved discovery figures the right way: words alone
    under BM25 at 76 entries from 50% to 55%.

Fixed

  • Every position read from a point-registered raster was half a cell off,
    since 0.4.0.
    On a file declaring AREA_OR_POINT=Point MapSmith moved each
    sample half a cell north-west of where GDAL puts it: sampling,
    elevation_profile, line_of_sight, locate_extreme_cell, least_cost_path,
    and the zones of zonal_statistics, which were shifted for the coverage
    computation. The premise was that GDAL reports the tag and leaves the
    geotransform alone. It does the opposite — since RFC 33 the GTiff driver moves
    a PixelIsPoint tie point half a cell on write and on read, so its geotransform
    already centres each cell on its sample — and the correction was made twice.
    Found by a Copernicus DEM, whose samples fall on whole arc-seconds by its own
    documentation: 10.8 by GDAL, 10.79986 by MapSmith. Positions now come from
    GDAL's geotransform whatever the tag says; the tag is still recorded, because
    it says what a value represents. The same premise was in Argleton trap 024,
    which scored MapSmith's wrong answer as correct for 26 days; the suite has
    published an erratum.
    A weights raster is no longer refused for a registration different from the
    values': on the same geotransform the samples are in the same places. And the
    fix no longer depends on the environment: GTIFF_POINT_GEO_IGNORE=TRUE, which
    reverts GDAL to its pre-2010 reading, moved every answer half a cell back with
    nothing in the record to say why, so MapSmith now pins it to FALSE on import,
    whether or not remote paths are allowed — which also overrides an operator's
    setting in a process that imports mapsmith as a library, deliberately.

  • The terrain operations refused every point-registered DEM, the Copernicus
    DEM included.
    The engine reads the stored tie point as a cell corner, and
    the guard that compares its grid with GDAL's caught the half cell and refused,
    so the numbers were never wrong and twelve operations were unusable on the
    most used global DEM (the eleven that write a raster, and contour_lines). The engine now gets a copy declared area-registered on
    GDAL's own geotransform — the same pattern as the TIFF predictor, disclosed in
    the manifest — and the output gets the input's registration back. Their
    manifests record the registration in crs_decisions, as the raster writers
    already did, and say when MapSmith retagged an output after the engine wrote
    it, since those bytes are no longer the engine's alone.
    contour_lines loses a branch for point-registered DEMs that corrected half a
    cell the wrong way and could not be reached.

  • elevation_profile measured the spacing in the DEM's units, and on a DEM
    in degrees a 1000 m line came back as one point.
    The line was reprojected
    to the raster's CRS before its positions were computed, so spacing=100
    meant 100 degrees, and the only trace was a GeoPandas warning on the console.
    Positions are now measured along the line in its own projected CRS -- a
    geographic line was already refused -- and only the sample points go to the
    raster's CRS to be read. The output is now in the line's CRS: when the two
    differed it used to be in the raster's, and a plan step reading it must now
    expect the line's (the plan validator does). The record says what was
    transformed in the specification's own keys: source_crs the line's,
    target_crs the raster's, and the transformation the sample points took to
    be read, while output.crs says where the result is. It used to list the line
    under x-mapsmith:inputs_reprojected from its own CRS, which that key's
    definition makes a contradiction. When line and DEM share a CRS nothing
    changes. Found while building a gradient on top of it.

  • elevation_profile stopped short of the far end of a line. A line whose
    length is not a whole number of steps ended on the last whole step: 15 m at a
    step of 4 ended at 12, and a summit in the last 3 m was not in the profile,
    while the docstring and the name of the test said both ends were included --
    and the test asserted the opposite. The far end is now a last, shorter step,
    and the stepping check requires it.

MapSmith 0.6.1

Choose a tag to compare

@mapsmith-ai mapsmith-ai released this 25 Sep 12:35

A small release on the day of 0.6.0. It moves no manifest key and changes no
answer that 0.6.0 got right: it stops a refusal 0.6.0 introduced from firing on
a layer written in 0..360 longitudes, gives two refusals the right way out, and
declares the specification's current draft.

Changed

  • Records declare spec_version 1.0.0-draft.8. Two drafts in one day,
    both narrowings of where a producer may add a key: draft.7 requires every
    key of crs_decisions the specification does not define to be named
    x-<producer>:<name>, which it had left to one reading of an ambiguous
    sentence, and draft.8 carries the rule inside transformation,
    round_trip and repairs[]. MapSmith already named every one of them that
    way, so no key moves: a reader keying on the label sees draft.8, a reader
    keying on fields sees nothing change. The vendored schema and validator are
    draft.8's, and the conformance sweep validates every writer's record
    against them.

Fixed

  • A layer written in 0..360 longitudes had its seam in the wrong place.
    The antimeridian refusal of 0.6.0 took the edge of the plane to be -180/180,
    so the world written as box(0, -90, 360, 90) was refused, the way 0.6.0's
    own review had found Antarctica refused in -180..180. For such a layer the
    edge is 0/360, and the ring that really is drawn naively crosses the prime
    meridian: the refusal now says so and tells its author to split there, where
    it used to send them to 180. A layer is read as 0..360 when it has no
    negative longitude and some beyond 180 -- not whenever one vertex passes
    180, which the first version of this fix did, and which refused the world
    beside a polygon overshooting 180 by 1e-7. Edges along either pair of seams
    are exempt in every layer.
  • "No UTM zone" said the data was centred on the antimeridian, whatever the
    data, and offered no way out.
    The realistic cause is latitude — UTM stops
    at 84N and 80S — and the message now says that and names a polar CRS to use
    (UPS, or a polar stereographic), for data that crosses 180 and for data that
    does not.