Skip to content

MapSmith 0.8.0

Choose a tag to compare

@mapsmith-ai mapsmith-ai released this 05 Oct 10:24
· 98 commits to main since this release

Records declare spec_version 1.0.0-draft.10. draft.9 defines the digest
of a dataset made of several files, which is what the first fix below
implements; draft.10 gives each input a role and a configuration of its own.
One key moves: a second input's own georeferencing facts,
inputs[].x-mapsmith:environment on 0.7.x, are now inputs[].environment.

Added

  • cluster_points_by_distance: group points chained together within a
    distance.
    "Group everything within half a kilometre" is single linkage, not
    thinning: two points share a cluster when a chain of points joins them with
    every link at most the distance long, so chains merge, and every point is kept
    with its cluster_id and cluster_size. thin_points keeps one point per
    crowd, and its count is close to the number of groups but not equal to it. A
    check re-reads the written file and re-derives the chains by a second
    algorithm. A geographic CRS is refused, and so are lines, polygons and
    multipart points, which a point cluster has no single position for.

  • list_operations takes a geometry: what the caller's layer holds, as
    point, line or polygon, or a list of them for a mixed layer. The
    catalogue now has 31 operations taking one vector layer to another, one more
    than the size it hands over whole, so a caller declaring everything it knew
    got a shortlist back for the most common request there is; with the geometry
    declared, the right operation is in the answer for 116 of the 118 benchmark
    requests again. An entry declares the families it accepts only where its code
    refuses the others (fifteen do), an entry without a declaration is kept for
    every family, and a test calls each declared entry with every family it
    leaves out and requires the refusal.

  • benchmarks/choice_at_scale.py, with every choice it scores in
    benchmarks/choice_at_scale.json: each of the 118 requests handed to Claude
    Opus 5.5 in sets of 30, 40 and 50 candidates and compared with itself. From
    40 it chose right 6.9 points less often than from 30, from 50 5.1 points
    less, so the size list_operations hands over whole stays 30, and a test
    holds it at or below the largest size measured without loss.

  • Each input of an operation that reads more than one dataset says which
    argument read it
    (inputs[].argument, spec draft.10). The order of
    inputs is not significant, so until now a weighted zonal statistic with
    values and weights swapped produced a record that differed only in file
    names. Eighteen operations name their inputs (twenty code paths:
    spatial_join has three engines), with the names the input checks
    already use in verification[].argument, so the two can be joined; a
    single-input record is unchanged.

  • get_lineage takes a layer. The history of one layer of a file
    geodatabase now starts from the geodatabase itself: its record sits beside
    the container (spec section 3.1), and the walk matches the layer at every hop
    into a container. A layer given for a path that is not a container is
    refused rather than ignored.

Changed

  • get_lineage steps carry inputs[].argument, inputs[].environment and
    output.layer
    when the records hold them.
  • A shapefile output whose member files are ambiguous (out.dbf beside
    out.DBF, possible only on a case-sensitive filesystem) is written with a
    manifest that has no output digest and a note saying why, instead of a
    digest that would choose one of the two.
  • inputs[].x-mapsmith:environment is inputs[].environment, the field
    spec draft.10 defines for the configuration that changed how one input was
    read. Same content; only the name moves, and a reader of 0.7.x records reads
    the old one.
  • With more than one input, no file's facts sit on the record's
    environment, the first input's included.
    A sidecar beside the first
    raster was recorded there, which the format reads as true of the whole run,
    since the order of inputs means nothing: in a clip_raster it read as true
    of the mask. It is now on that input's own environment, and only process
    settings stay on the record. Seven operations can meet it (clip_raster,
    sample_raster_at_points, elevation_profile, least_cost_path, viewshed,
    watershed, zonal_statistics), and only when a sidecar is present. A
    single-input record is unchanged.

Fixed

  • cluster_points_by_distance is bounded. The pairs of points within the
    distance grow with the square of the points when the distance is large, and
    6,000 points at a distance of 1e9 took two minutes and 2.3 GB. The pairs are
    now found in blocks and the call is refused once they pass
    MAPSMITH_MAX_SAMPLES, the operator's limit, before memory is spent.
    hot_spots with weights="distance_band" uses the same query: its weights
    were a double loop over every pair of features in Python, quadratic at any
    band, and a band that is not a finite positive number is now refused.
  • get_lineage no longer hashes every shapefile in the workspace. Since
    the draft.9 digest rule it computed, while indexing, the other rule's digest
    of every shapefile record's output, whatever the chain asked about. A
    shapefile is now hashed only when a hop misses on it.
  • A Windows junction inside a .gdb is not followed. The digest of a
    directory container walked into junctions, which is_symlink() does not
    report, so files outside the container could enter its digest. Section 3.3 of
    the specification says links are neither members nor followed. The size used
    for the re-hash cap now counts the same files as the digest.
  • A single-layer .gdb read by any operation records its layer. Every
    operation now reads a single-layer file geodatabase, not only
    extract_layer, and the input records the layer it read: without it, a
    lineage walk meeting the record of that layer stopped with a false "records
    claim this container's digest, but for other layers".
  • get_lineage refuses a layer holding : or a control character. On
    Windows : in a file name opens an alternate data stream of another file.
  • thin_points refuses a distance that is not a finite positive number,
    like cluster_points_by_distance; and least_cost_path accepts a start or
    end given as a single-part MultiPoint, which it used to count as no point.
  • zonal_statistics refuses zones that are not polygons. Line zones were
    accepted and returned the line's length in pixels as the count, with a
    mean over the cells it crossed: a plausible table, and not a zonal
    statistic. count_in_polygons and summarize_points_in_polygons refuse a
    polygon layer holding anything else the same way. Found by the test that
    holds the new geometry declarations to the code.
  • least_cost_path is found by a caller holding points. It reads a cost
    raster and two point layers but declared its input as raster only, so a
    caller who said, correctly, that it held a vector layer never saw it. A start
    or end layer holding a line or a polygon beside its point is now refused; it
    used to pass with the extra feature ignored.
  • snap_layer snaps to a polygon or multipart reference. Both crashed with
    a raw NotImplementedError from shapely; every vertex of every part, holes
    included, is now a target.
  • A feature with no geometry is refused, with a count, by thin_points,
    sample_raster_at_points, points_along_lines and line_intersections.

    They crashed on it with an AttributeError, "cannot convert float NaN to
    integer" and a failed sort.
  • A shapefile's digest covers its attributes and its CRS. An input or
    output shapefile was digested as its .shp alone, so a record kept matching
    a dataset whose .dbf (attributes) or .prj (CRS) had been rewritten, while
    it said sha256 is of the bytes. The digest is now the listing of the
    .shp, .shx, .dbf, .prj and .cpg files, as section 3.3 of the
    specification defines it since draft.9; a file geodatabase input is digested
    the same way, lock files excluded. get_lineage recomputes digests with the
    same function, so a chain written by this release still resolves. A record
    written by an earlier release for a shapefile carries the old digest:
    get_lineage compares it under the rule it was written with, so an
    unchanged shapefile is not reported as edited, and a chain that crosses
    from a record written by 0.7.1 to one written now still resolves. A record
    written before this release can only vouch for the .shp: an edit to the
    attributes or the CRS made after it was written stays invisible to it, so
    such a hop is reported as reverified_shp_only, not reverified, and a
    history that contains one is not verified.
  • extract_layer works on a file geodatabase. The catalog listed .gdb
    among its containers, and the input digest opened the path as a file: a
    geodatabase is a directory, so the operation died before writing anything.
  • get_lineage matches the layer when a digest belongs to a container.
    Two records for two layers of one geodatabase share the container's digest
    (spec section 6, draft.9); a hop into a container now resolves to the record
    for the layer it names, and stops with other_layer rather than attribute
    one layer to the operation that wrote another. It re-checks a layer's record
    against the container it sits beside (<container>.<layer>.provenance.json,
    spec section 3.1), and two layers of one container reached on two branches
    are two histories.
  • A URI in a manifest path keeps its //, so its credentials are masked.
    The path normaliser collapsed https://user:pw@host/x to https:/user:pw@…,
    and the redaction of URI credentials looks for ://: the password would have
    stayed in the record. Not reachable while remote paths are refused, which is
    the default; found in the 0.7.0 audit. The weights raster of
    zonal_statistics is recorded through the same normaliser now.
  • A redaction in the environment of a secondary input is declared. Masked
    values there did not raise parameters_redacted, so the record changed a
    value without saying so.
  • The package metadata names the Copernicus NOTICE beside the AGPL licence,
    since the source distribution ships the two Copernicus clips the notebook
    reads.
  • The links on the PyPI page resolve. PyPI renders the README with nothing
    to resolve a relative link against, so every link to a document of the repo
    and the map-panel screenshot answered 404 there. The build now rewrites them
    to GitHub URLs pinned to the release tag; on GitHub the README is unchanged.

Install: uvx mapsmith or docker run ... ghcr.io/mapsmith-ai/mapsmith:0.8.0. Full changelog: https://github.com/mapsmith-ai/MapSmith/blob/v0.8.0/CHANGELOG.md