Add a target-size option to -simplify

[christian-weiss]

If you work with GeoJSON from OpenStreetmap Admin Boundaries you may not need the high precision provided. To get faster download, less JavaScript parse-time and less render-time you may want to reduce the details in such a shape file.

A target-file-size (e.g. target-size=6KB) option to -simplify in a batch process (build process; command line tool) would be better than trying to find a value for percentage (e.g. percentage=10%) combined with some try-and-error to bring data down to desired file size.

Please implement a target-file-size option. The result does not have to be exactly 6KB (6144 bytes) - it would be ok, to bring file size below this threshold - so 5840 bytes and similar would be perfect.

Related question:
What command line options do you suggest when someone wants to process e.g. >70 shapes which "touch" each other and if you want to apply the same "simplification" to adjacent shapes - so the result is not getting gaps nor overlapping. (and also remove existing overlapping and existing gaps)? Is there a -sync-shapes feature and a multiple-files feature?

Example for illustration:

[mbloch]

To prevent simplification from introducing gaps between files, you will have to import the files together, so mapshaper can identify shared boundaries. You may need to use the mapshaper-xl script to allocate more heap memory if the files are large. If the total size of the files is too large, even this may not work.

Import the files using the combine-files flag. The following example assumes you have GeoJSON files in a directory named data:

mapshaper-xl data/*.json combine-files -simplify interval=100m -o output/

To apply a consistent amount of simplification across multiple files, you can use the interval= option and specify the amount of simplification as a distance.

I don't support adding a target-size option, for several reasons. One is, mapshaper can export to several file types, each with different size characteristics. Also, for cartographic simplification, the appearance of shapes is of primary importance, and file size is not as directly related to appearance as the other options that are already available (percentage, interval and resolution).

Cheers,
mbloch :)

[christian-weiss]

Thanks for the statement.

It simplifies the shapes but it do not remove overlapping and gaps. Shared borders are not optimized in sync with relevant neighbour-shapes.

Original shapes:

Combine-files:

[christian-weiss]

Above screenshots are from this section:

Page:
https://firmware.freifunk-muensterland.de/md-fw-dl/

[mbloch]

I see... the suggestion I gave you would only work if the shapes fit together perfectly to begin with. It looks like the boundaries of the source polygons might not match exactly.

An alternative method is to merge all the unsimplified shapes into a single layer, remove any gaps or overlaps using the -clean command, simplify the unified layer, and then optionally split the polygons apart again (if you want individual files).

edited to add combine-files. this flag is essential... without it input files are processed separately
mapshaper data/*.json combine-files -merge-layers -clean -simplify interval=100m -split -o output/

The -clean command has a min-gap-area= option, in case the default value is too small to fill all the gaps.

[mbloch]

This issue has been responded-to... closing.

[christian-weiss]

Thank you very, very mutch, to point me to the right direction.

Mapshapers billions of command line options are in the end pretty cool, but for first-time-users it is confusing.

I changed the command to:

mapshaper input/*.geojson combine-files -merge-layers -clean -simplify interval=100m -split -o output/ prettify

But i get the following error:

[merge-layers] Error: Fields [id, localname, timestamp, SRID, admin_level, rpath, tags, ID, NAME, LOCALNAME, BOUNDARY, ADMIN_LEVE, CITY_KEY, REGION_KEY, NOTE, FLAG, CURRENCY, ISO3166-1:, WIKIDATA, WIKIPEDIA, OFFICIAL_N, @id, TMC_cid_58, TMC_cid__1, TMC_cid__2, admin_cent, admin_leve, boundary, de_amtlich, de_place, de_regiona, name_prefi, population, type, wikidata, wikipedia, name:prefix, FID, ISO3166-1_, srid, official_name, note, alltags, ref, area, perimeter, name] are missing from one or more layers
Run mapshaper -h to view help

To fix that is was forcing mapshaper to add missing fields:

mapshaper input/*.geojson combine-files -merge-layers force -clean -simplify interval=100m -split -o output/ prettify

But still get an error:

[merge-layers] Warning: Fields [id, localname, timestamp, SRID, admin_level, rpath, tags, ID, NAME, LOCALNAME, BOUNDARY, ADMIN_LEVE, CITY_KEY, REGION_KEY, NOTE, FLAG, CURRENCY, ISO3166-1:, WIKIDATA, WIKIPEDIA, OFFICIAL_N, @id, TMC_cid_58, TMC_cid__1, TMC_cid__2, admin_cent, admin_leve, boundary, de_amtlich, de_place, de_regiona, name_prefi, population, type, wikidata, wikipedia, name:prefix, FID, ISO3166-1_, srid, official_name, note, alltags, ref, area, perimeter, name] are missing from one or more layers
[merge-layers] Error: Inconsistent data types in "id" field: number, string
Run mapshaper -h to view help

All input files are valid GeoJSON files, but the structure is different:
FeatureCollection->containsExactlyOneElementOf(Feature)->Polygon
FeatureCollection->containsExactlyOneElementOf(Feature)->MultiPolygon
GeometryCollection->containsExactlyOneElementOf(Polygon)
GeometryCollection->containsExactlyOneElementOf(MultiPolygon)

Properties of each Polygon / MultiPolygon are not important to me, i only need the shapes for my application. But for debugging (team of developers) it would be interesting to keep the properties. There is no need to merge the values in my case. Keep the original properties for each splitted output would be fine. Dropping them from output would be ok.

Sad thing is, that -filter-fields is not doing what i want it to do:

mapshaper input/*.geojson combine-files -filter-fields -merge-layers force -clean -simplify interval=100m -split -o output/ prettify
#or
mapshaper input/*.geojson combine-files -filter-fields "" -merge-layers force -clean -simplify interval=100m -split -o output/ prettify
#or
mapshaper input/*.geojson combine-files -filter-fields keepNothing -merge-layers force -clean -simplify interval=100m -split -o output/ prettify

failing. keepNothing is a field name that do not exist in input files (by intention), i tried to use that in the command line options to trick mapshaper, to drop all others.

Is there an option to extract the first Polygon / MultiPolygon from each input file?
Is there an option to drop the Properties from Polygon / MultiPolygon?

By the way i really love the "precision" option!

[mbloch]

It looks like merge-layers is failing because the value in the id field is sometimes a string and sometimes a number. ([merge-layers] Error: Inconsistent data types in "id" field: number, string).

For compatibility with Shapefiles and other reasons, mapshaper applies some restrictions that are not present in the GeoJSON spec.

You can resolve the error by deleting the id field (-drop fields=id1) or by converting all the ids to strings (-each 'id = String(id)').

[christian-weiss]

@mbloch Is there a build-in way to keep the original filenames after split? Or do i have to prepare the input files by adding a .feature[].properties.originalFilename and then rename the output files based on that field?