Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

minimal PMTiles v3 output #10

Closed
bdon opened this issue Sep 16, 2022 · 3 comments
Closed

minimal PMTiles v3 output #10

bdon opened this issue Sep 16, 2022 · 3 comments

Comments

@bdon
Copy link
Contributor

bdon commented Sep 16, 2022

Add PMTiles aș a first class output option using -o output.pmtiles

Add a basic implementation that supports non-clustered archives with a root directory only and no deduplication.

(migrating issue from https://github.com/protomaps/tippecanoe/issues/1)
@bdon
Copy link
Contributor Author

bdon commented Sep 16, 2022 •
edited
Loading

Notes on eventual deduplication:

From protomaps/PMTiles#20 the probability of a 64-bit hash collision becomes unsuitable at ~10 million tiles.

If we must deduplicate tiles based on opaque blobs, we should use FNV-128a, which we would have to grab from somewhere.

A better-alternative is to limit deduplication to tiles that have only vertical and horizontal edges. This would cover the vast majority, if not all, duplicated tiles - squares, straight roads, etc. One way to do this is: when tippecanoe builds a tile internally, have a boolean isAxisAligned that starts out true and tracks all edges being added to the tile, becoming false once a non-axis-aligned geometry is added. This would then need to be passed down through serialization and to the writer implementation. If we're bothering to do all that we might as well also compute a 64-bit hash of the square tile at that step.

bdon added a commit to bdon/tippecanoe that referenced this issue Oct 12, 2022
@bdon
Add PMTiles v3 output stub [felt#10]
bd1938d 
* This does not actually write any directories yet

* Write tile data on-the fly to a tempfile

* Finalizes output by copying tempfile and compressed JSON metadata to output stream

* detects output format using .pmtiles suffix

* Expose mbtiles tilestats function in public header
bdon added a commit to bdon/tippecanoe that referenced this issue Nov 28, 2022
@bdon
Add PMTiles v3 output stub [felt#10]
a0318e5 
* This does not actually write any directories yet

* Write tile data on-the fly to a tempfile

* Finalizes output by copying tempfile and compressed JSON metadata to output stream

* detects output format using .pmtiles suffix

* Expose mbtiles tilestats function in public header
@bdon
Copy link
Contributor Author

bdon commented Nov 28, 2022 •
edited
Loading

Tasks related to other tippecanoe tools:

  • tile-join needs to take PMTiles input and output
  • tippecanoe-enumerate should take PMTiles input (it is MBTiles only right now)
  • tippecanoe-decode should take PMTiles input
  • tippecanoe-json-tool: not affected

@bdon bdon mentioned this issue Dec 1, 2022
Closed
bdon added a commit to bdon/tippecanoe that referenced this issue Dec 13, 2022
@bdon
Add PMTiles output as a postprocessing step on MBTiles. [felt#10]
bc7c248 
* Does not yet write correct metadata.
* decode reads pmtiles input (also without metadata).
* add make pmtiles-test

decode reads pmtiles input without metadata; add make pmtiles-test
bdon added a commit to bdon/tippecanoe that referenced this issue Dec 13, 2022
@bdon
pass metadata to pmtiles_file [felt#10]
c593c8d
bdon added a commit to bdon/tippecanoe that referenced this issue Dec 13, 2022
@bdon
finish writing of metadata to pmtiles [felt#10]
b437594
bdon added a commit to bdon/tippecanoe that referenced this issue Dec 13, 2022
@bdon
pmtiles write progress and checking for existing file [felt#10]
05074e7
bdon added a commit to bdon/tippecanoe that referenced this issue Dec 13, 2022
@bdon
pmtiles test case: support uncompressed tiles [felt#10]
8275b07
bdon added a commit to bdon/tippecanoe that referenced this issue Dec 15, 2022
@bdon
fix pmtiles directory parsing run lengths [felt#10]
25e2060
bdon added a commit to bdon/tippecanoe that referenced this issue Dec 15, 2022
@bdon
tile-join can read pmtiles data but not metadata yet [felt#10]
6498197
bdon added a commit to bdon/tippecanoe that referenced this issue Dec 15, 2022
@bdon
read pmtiles metadata into tmp sqlite dir [felt#10]
7f87abe
bdon added a commit to bdon/tippecanoe that referenced this issue Dec 15, 2022
@bdon
use sqlite formatting for numbers to avoid roundtrip precision proble…
186c28f 
…ms [felt#10]
bdon added a commit to bdon/tippecanoe that referenced this issue Dec 15, 2022
@bdon
add pmtiles-test to test suite [felt#10]
574b416
bdon added a commit to bdon/tippecanoe that referenced this issue Dec 15, 2022
@bdon
fix pmtiles read performance by using deque instead of vector [felt#10]
53e09c3
bdon added a commit to bdon/tippecanoe that referenced this issue Dec 16, 2022
@bdon
finish pmtiles metadata impl [felt#10]
d305b37
bdon added a commit to bdon/tippecanoe that referenced this issue Dec 21, 2022
@bdon
Add PMTiles output as a postprocessing step on MBTiles. [felt#10]
2b680c5 
* Does not yet write correct metadata.
* decode reads pmtiles input (also without metadata).
* add make pmtiles-test

decode reads pmtiles input without metadata; add make pmtiles-test
bdon added a commit to bdon/tippecanoe that referenced this issue Dec 21, 2022
@bdon
pass metadata to pmtiles_file [felt#10]
b17fbf0
bdon added a commit to bdon/tippecanoe that referenced this issue Dec 21, 2022
@bdon
finish writing of metadata to pmtiles [felt#10]
2904878
bdon added a commit to bdon/tippecanoe that referenced this issue Dec 21, 2022
@bdon
pmtiles write progress and checking for existing file [felt#10]
ec37ea4
bdon added a commit to bdon/tippecanoe that referenced this issue Dec 21, 2022
@bdon
pmtiles test case: support uncompressed tiles [felt#10]
4202656
bdon added a commit to bdon/tippecanoe that referenced this issue Dec 21, 2022
@bdon
fix pmtiles directory parsing run lengths [felt#10]
8db09b6
bdon added a commit to bdon/tippecanoe that referenced this issue Dec 21, 2022
@bdon
tile-join can read pmtiles data but not metadata yet [felt#10]
e5ca5f9
bdon added a commit to bdon/tippecanoe that referenced this issue Dec 21, 2022
@bdon
read pmtiles metadata into tmp sqlite dir [felt#10]
b8da83b
bdon added a commit to bdon/tippecanoe that referenced this issue Dec 21, 2022
@bdon
use sqlite formatting for numbers to avoid roundtrip precision proble…
39a0e99 
…ms [felt#10]
bdon added a commit to bdon/tippecanoe that referenced this issue Dec 21, 2022
@bdon
add pmtiles-test to test suite [felt#10]
ebf85fa
bdon added a commit to bdon/tippecanoe that referenced this issue Dec 21, 2022
@bdon
fix pmtiles read performance by using deque instead of vector [felt#10]
9613d2a
bdon added a commit to bdon/tippecanoe that referenced this issue Dec 21, 2022
@bdon
finish pmtiles metadata impl [felt#10]
c974ddb
bdon added a commit to bdon/tippecanoe that referenced this issue Dec 21, 2022
@bdon
add pmtiles.hpp from github.com/protomaps/PMTiles [felt#10]
f6332f3
bdon added a commit to bdon/tippecanoe that referenced this issue Dec 21, 2022
@bdon
tippecanoe main writes pmtiles output. [felt#10]
fedebec 
* detect output format using suffix
* after mbtiles is done writing, replace with pmtiles based on map/image tables.
* add method to write_json for writing json sub-object.
bdon added a commit to bdon/tippecanoe that referenced this issue Dec 21, 2022
@bdon
tippecanoe-decode reads pmtiles input. [felt#10]
64a5a68
bdon added a commit to bdon/tippecanoe that referenced this issue Dec 21, 2022
@bdon
tile-join reads and writes pmtiles. [felt#10]
e958f1a
bdon added a commit to bdon/tippecanoe that referenced this issue Dec 21, 2022
@bdon
pmtiles test suite for decode and tile-join [felt#10]
f1c0bf5
e-n-f added a commit that referenced this issue Dec 29, 2022
@e-n-f @bdon
Add support for pmtiles output format (#55)
3b2599f 
* add pmtiles.hpp from github.com/protomaps/PMTiles [#10]

* tippecanoe main writes pmtiles output. [#10]

* detect output format using suffix
* after mbtiles is done writing, replace with pmtiles based on map/image tables.
* add method to write_json for writing json sub-object.

* tippecanoe-decode reads pmtiles input. [#10]

* tile-join reads and writes pmtiles. [#10]

* pmtiles test suite for decode and tile-join [#10]

* add base GitHub CI action for compiling and test suite.

* update pmtiles.hpp with z>15 fix

* Fix some ordering problems with pmtiles decode

* Pmtiles should also pass the raw tiles tests

* Eradicate spaces from tileset metadata JSON fields

* Eradicate spaces from more test fixtures

* Update more tests

* Pmtiles tests pass now too

* Remove unnecessary sort (and make indent)

* Update changelog

* The allow-existing test for pmtiles needs -o, not -e

* Declare --allow-existing to be unsupported for pmtiles.

It was always a bad idea even for mbtiles.

Co-authored-by: Brandon Liu <bdon@bdon.org>
@bdon
Copy link
Contributor Author

bdon commented Dec 30, 2022

Implemented in #55, released in 2.17

@bdon bdon closed this as completed Dec 30, 2022
@cldellow cldellow mentioned this issue Jan 28, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@bdon