Repository navigation
MapSmith 0.8.0
Records declare spec_version 1.0.0-draft.10. draft.9 defines the digest
of a dataset made of several files, which is what the first fix below
implements; draft.10 gives each input a role and a configuration of its own.
One key moves: a second input's own georeferencing facts,
inputs[].x-mapsmith:environment on 0.7.x, are now inputs[].environment.
Added
-
cluster_points_by_distance: group points chained together within a
distance. "Group everything within half a kilometre" is single linkage, not
thinning: two points share a cluster when a chain of points joins them with
every link at most the distance long, so chains merge, and every point is kept
with itscluster_idandcluster_size.thin_pointskeeps one point per
crowd, and its count is close to the number of groups but not equal to it. A
check re-reads the written file and re-derives the chains by a second
algorithm. A geographic CRS is refused, and so are lines, polygons and
multipart points, which a point cluster has no single position for. -
list_operationstakes ageometry: what the caller's layer holds, as
point,lineorpolygon, or a list of them for a mixed layer. The
catalogue now has 31 operations taking one vector layer to another, one more
than the size it hands over whole, so a caller declaring everything it knew
got a shortlist back for the most common request there is; with the geometry
declared, the right operation is in the answer for 116 of the 118 benchmark
requests again. An entry declares the families it accepts only where its code
refuses the others (fifteen do), an entry without a declaration is kept for
every family, and a test calls each declared entry with every family it
leaves out and requires the refusal. -
benchmarks/choice_at_scale.py, with every choice it scores in
benchmarks/choice_at_scale.json: each of the 118 requests handed to Claude
Opus 5.5 in sets of 30, 40 and 50 candidates and compared with itself. From
40 it chose right 6.9 points less often than from 30, from 50 5.1 points
less, so the sizelist_operationshands over whole stays 30, and a test
holds it at or below the largest size measured without loss. -
Each input of an operation that reads more than one dataset says which
argument read it (inputs[].argument, specdraft.10). The order of
inputsis not significant, so until now a weighted zonal statistic with
values and weights swapped produced a record that differed only in file
names. Eighteen operations name their inputs (twenty code paths:
spatial_joinhas three engines), with the names the input checks
already use inverification[].argument, so the two can be joined; a
single-input record is unchanged. -
get_lineagetakes alayer. The history of one layer of a file
geodatabase now starts from the geodatabase itself: its record sits beside
the container (spec section 3.1), and the walk matches the layer at every hop
into a container. A layer given for a path that is not a container is
refused rather than ignored.
Changed
get_lineagesteps carryinputs[].argument,inputs[].environmentand
output.layerwhen the records hold them.- A shapefile output whose member files are ambiguous (
out.dbfbeside
out.DBF, possible only on a case-sensitive filesystem) is written with a
manifest that has nooutputdigest and a note saying why, instead of a
digest that would choose one of the two. inputs[].x-mapsmith:environmentisinputs[].environment, the field
specdraft.10defines for the configuration that changed how one input was
read. Same content; only the name moves, and a reader of 0.7.x records reads
the old one.- With more than one input, no file's facts sit on the record's
environment, the first input's included. A sidecar beside the first
raster was recorded there, which the format reads as true of the whole run,
since the order ofinputsmeans nothing: in aclip_rasterit read as true
of the mask. It is now on that input's ownenvironment, and only process
settings stay on the record. Seven operations can meet it (clip_raster,
sample_raster_at_points, elevation_profile, least_cost_path, viewshed,
watershed, zonal_statistics), and only when a sidecar is present. A
single-input record is unchanged.
Fixed
cluster_points_by_distanceis bounded. The pairs of points within the
distance grow with the square of the points when the distance is large, and
6,000 points at a distance of 1e9 took two minutes and 2.3 GB. The pairs are
now found in blocks and the call is refused once they pass
MAPSMITH_MAX_SAMPLES, the operator's limit, before memory is spent.
hot_spotswithweights="distance_band"uses the same query: its weights
were a double loop over every pair of features in Python, quadratic at any
band, and a band that is not a finite positive number is now refused.get_lineageno longer hashes every shapefile in the workspace. Since
the draft.9 digest rule it computed, while indexing, the other rule's digest
of every shapefile record's output, whatever the chain asked about. A
shapefile is now hashed only when a hop misses on it.- A Windows junction inside a
.gdbis not followed. The digest of a
directory container walked into junctions, whichis_symlink()does not
report, so files outside the container could enter its digest. Section 3.3 of
the specification says links are neither members nor followed. The size used
for the re-hash cap now counts the same files as the digest. - A single-layer
.gdbread by any operation records its layer. Every
operation now reads a single-layer file geodatabase, not only
extract_layer, and the input records the layer it read: without it, a
lineage walk meeting the record of that layer stopped with a false "records
claim this container's digest, but for other layers". get_lineagerefuses alayerholding:or a control character. On
Windows:in a file name opens an alternate data stream of another file.thin_pointsrefuses a distance that is not a finite positive number,
likecluster_points_by_distance; andleast_cost_pathaccepts a start or
end given as a single-part MultiPoint, which it used to count as no point.zonal_statisticsrefuses zones that are not polygons. Line zones were
accepted and returned the line's length in pixels as thecount, with a
mean over the cells it crossed: a plausible table, and not a zonal
statistic.count_in_polygonsandsummarize_points_in_polygonsrefuse a
polygon layer holding anything else the same way. Found by the test that
holds the new geometry declarations to the code.least_cost_pathis found by a caller holding points. It reads a cost
raster and two point layers but declared its input as raster only, so a
caller who said, correctly, that it held a vector layer never saw it. A start
or end layer holding a line or a polygon beside its point is now refused; it
used to pass with the extra feature ignored.snap_layersnaps to a polygon or multipart reference. Both crashed with
a rawNotImplementedErrorfrom shapely; every vertex of every part, holes
included, is now a target.- A feature with no geometry is refused, with a count, by
thin_points,
sample_raster_at_points,points_along_linesandline_intersections.
They crashed on it with anAttributeError, "cannot convert float NaN to
integer" and a failed sort. - A shapefile's digest covers its attributes and its CRS. An input or
output shapefile was digested as its.shpalone, so a record kept matching
a dataset whose.dbf(attributes) or.prj(CRS) had been rewritten, while
it saidsha256is of the bytes. The digest is now the listing of the
.shp,.shx,.dbf,.prjand.cpgfiles, as section 3.3 of the
specification defines it since draft.9; a file geodatabase input is digested
the same way, lock files excluded.get_lineagerecomputes digests with the
same function, so a chain written by this release still resolves. A record
written by an earlier release for a shapefile carries the old digest:
get_lineagecompares it under the rule it was written with, so an
unchanged shapefile is not reported as edited, and a chain that crosses
from a record written by 0.7.1 to one written now still resolves. A record
written before this release can only vouch for the.shp: an edit to the
attributes or the CRS made after it was written stays invisible to it, so
such a hop is reported asreverified_shp_only, notreverified, and a
history that contains one is notverified. extract_layerworks on a file geodatabase. The catalog listed.gdb
among its containers, and the input digest opened the path as a file: a
geodatabase is a directory, so the operation died before writing anything.get_lineagematches the layer when a digest belongs to a container.
Two records for two layers of one geodatabase share the container's digest
(spec section 6, draft.9); a hop into a container now resolves to the record
for the layer it names, and stops withother_layerrather than attribute
one layer to the operation that wrote another. It re-checks a layer's record
against the container it sits beside (<container>.<layer>.provenance.json,
spec section 3.1), and two layers of one container reached on two branches
are two histories.- A URI in a manifest path keeps its
//, so its credentials are masked.
The path normaliser collapsedhttps://user:pw@host/xtohttps:/user:pw@…,
and the redaction of URI credentials looks for://: the password would have
stayed in the record. Not reachable while remote paths are refused, which is
the default; found in the 0.7.0 audit. The weights raster of
zonal_statisticsis recorded through the same normaliser now. - A redaction in the environment of a secondary input is declared. Masked
values there did not raiseparameters_redacted, so the record changed a
value without saying so. - The package metadata names the Copernicus NOTICE beside the AGPL licence,
since the source distribution ships the two Copernicus clips the notebook
reads. - The links on the PyPI page resolve. PyPI renders the README with nothing
to resolve a relative link against, so every link to a document of the repo
and the map-panel screenshot answered 404 there. The build now rewrites them
to GitHub URLs pinned to the release tag; on GitHub the README is unchanged.
Install: uvx mapsmith or docker run ... ghcr.io/mapsmith-ai/mapsmith:0.8.0. Full changelog: https://github.com/mapsmith-ai/MapSmith/blob/v0.8.0/CHANGELOG.md