MBTiles DataSource #1015

Merged
merged 7 commits into from Feb 8, 2017

Conversation

Projects
None yet
@hjanetzek
Member

hjanetzek commented Oct 14, 2016

No description provided.

@hjanetzek hjanetzek referenced this pull request Oct 17, 2016

Closed

MBTiles Support Sprint #982

@hjanetzek

This comment has been minimized.

Show comment
Hide comment
@hjanetzek

hjanetzek Oct 18, 2016

Member

Testing with OSM2Vectortiles http://osm2vectortiles.org/downloads/
shot-2016-10-18_13-19-20

Member

hjanetzek commented Oct 18, 2016

Testing with OSM2Vectortiles http://osm2vectortiles.org/downloads/
shot-2016-10-18_13-19-20

@hallahan

This comment has been minimized.

Show comment
Hide comment
@hallahan

hallahan Oct 18, 2016

Contributor

This is awesome! I will be looking through this (and trying out with osm2vectortiles).

Contributor

hallahan commented Oct 18, 2016

This is awesome! I will be looking through this (and trying out with osm2vectortiles).

@hallahan

Overall, I'm happy to see the direction you are bringing MBTiles support. I would have been hesitant to do such a refactor on my own, since I didn't want to make such drastic changes to the overall architecture of Tangram ES. Very cool!

core/src/data/dataSource.h
/* vector of raster sources (as raster samplers) referenced by this datasource */
std::vector<std::shared_ptr<DataSource>> m_rasterSources;
+
+ std::unique_ptr<RawDataSource> m_sources;

This comment has been minimized.

@hallahan

hallahan Oct 18, 2016

Contributor

I see what you're doing with adding MBTiles and Network data sources, but semantically, this is confusing. Data source, and all of the things derived from it, like MVT, GeoJSON, TopoJSON, are different in nature from MBTilesDataSource, NetworkCacheDataSource, etc.

I would say doing this makes the code overall less clear, however, coming up with a different name for RawDataSource and DataSource and the respective subclasses will likely add clarity.

@hallahan

hallahan Oct 18, 2016

Contributor

I see what you're doing with adding MBTiles and Network data sources, but semantically, this is confusing. Data source, and all of the things derived from it, like MVT, GeoJSON, TopoJSON, are different in nature from MBTilesDataSource, NetworkCacheDataSource, etc.

I would say doing this makes the code overall less clear, however, coming up with a different name for RawDataSource and DataSource and the respective subclasses will likely add clarity.

This comment has been minimized.

@hjanetzek

hjanetzek Oct 19, 2016

Member

Yes, I'm also not pleased with the RawDataSource and -DataSource naming. I thought of renaming DataSource to TileDataSource since the result of DataSource::parse is a TileData object. Still open for suggestions and thinking about further refactoring of DataSource

@hjanetzek

hjanetzek Oct 19, 2016

Member

Yes, I'm also not pleased with the RawDataSource and -DataSource naming. I thought of renaming DataSource to TileDataSource since the result of DataSource::parse is a TileData object. Still open for suggestions and thinking about further refactoring of DataSource

This comment has been minimized.

@hallahan

hallahan Oct 19, 2016

Contributor

TileSource sounds good.

@hallahan

hallahan Oct 19, 2016

Contributor

TileSource sounds good.

core/src/data/geoJsonSource.h
@@ -6,15 +6,16 @@ namespace Tangram {
class GeoJsonSource: public DataSource {
+public:
+ using DataSource::DataSource;

This comment has been minimized.

@hallahan

hallahan Oct 18, 2016

Contributor

👍

@hallahan

hallahan Oct 18, 2016

Contributor

👍

core/src/data/dataSource.h
@@ -27,19 +46,23 @@ class DataSource : public std::enable_shared_from_this<DataSource> {
* each of '{x}', '{y}', and '{z}' which will be replaced by the x index, y index,
* and zoom level of tiles to produce their URL.
*/
- DataSource(const std::string& _name, const std::string& _urlTemplate,
+ DataSource(const std::string& _name, std::unique_ptr<RawDataSource> _sources,

This comment has been minimized.

@hallahan

hallahan Oct 18, 2016

Contributor

What if you refactored DataSource and its subclasses to just be Data? MBTilesDataSource indeed describes the data source. MVTSource and the like describes the data that is retrieved from the source.

Also, should _sources be plural?

@hallahan

hallahan Oct 18, 2016

Contributor

What if you refactored DataSource and its subclasses to just be Data? MBTilesDataSource indeed describes the data source. MVTSource and the like describes the data that is retrieved from the source.

Also, should _sources be plural?

core/src/data/mbtilesDataSource.cpp
+ *
+ * https://github.com/mapbox/node-mbtiles/blob/4bbfaf991969ce01c31b95184c4f6d5485f717c3/lib/schema.sql
+ */
+const char* SCHEMA = R"SQL_ESC(BEGIN;

This comment has been minimized.

@hallahan

hallahan Oct 18, 2016

Contributor

Do we want SCHEMA in the Tangram namespace globally?

@hallahan

hallahan Oct 18, 2016

Contributor

Do we want SCHEMA in the Tangram namespace globally?

This comment has been minimized.

@hjanetzek

hjanetzek Oct 19, 2016

Member

no, just missing a static..

@hjanetzek

hjanetzek Oct 19, 2016

Member

no, just missing a static..

core/src/data/mbtilesDataSource.cpp
+
+ MBTilesQueries(SQLite::Database& _db, bool _offline)
+ : getTileData(_db, "SELECT tile_data FROM tiles WHERE zoom_level = ? AND tile_column = ? AND tile_row = ?;"),
+ putMap(_db, _offline ? "REPLACE INTO map (zoom_level, tile_column, tile_row, tile_id) VALUES (?, ?, ?, ?);" : "SELECT 1;" ),

This comment has been minimized.

@hallahan

hallahan Oct 18, 2016

Contributor

What is the purpose of SELECT 1;?

@hallahan

hallahan Oct 18, 2016

Contributor

What is the purpose of SELECT 1;?

This comment has been minimized.

@hallahan

hallahan Oct 18, 2016

Contributor

Do we want to do a noop at the SQL level?

@hallahan

hallahan Oct 18, 2016

Contributor

Do we want to do a noop at the SQL level?

This comment has been minimized.

@hjanetzek

hjanetzek Oct 19, 2016

Member

This is a quick workaround I put in yesterday to test with our Mapzen tilepacks which do not have a map table - needs cleanup.

@hjanetzek

hjanetzek Oct 19, 2016

Member

This is a quick workaround I put in yesterday to test with our Mapzen tilepacks which do not have a map table - needs cleanup.

This comment has been minimized.

@hallahan

hallahan Oct 19, 2016

Contributor

What are tilepacks? No map table?

@hallahan

hallahan Oct 19, 2016

Contributor

What are tilepacks? No map table?

This comment has been minimized.

@hjanetzek

hjanetzek Oct 19, 2016

Member

The map table is not required by the spec - As you wrote somewhere, it's an optimization to store duplicate tile data only once. I'm currently working on initMBTiles to support all MBTiles following the spec.

Tilepacks is our working name for MBTile bundles.

@hjanetzek

hjanetzek Oct 19, 2016

Member

The map table is not required by the spec - As you wrote somewhere, it's an optimization to store duplicate tile data only once. I'm currently working on initMBTiles to support all MBTiles following the spec.

Tilepacks is our working name for MBTile bundles.

scenes/scene.yaml
@@ -80,10 +88,11 @@ styles:
sources:
osm:
type: MVT
- url: https://vector.mapzen.com/osm/all/{z}/{x}/{y}.mvt
+ #url: https://vector.mapzen.com/osm/all/{z}/{x}/{y}.json
+ mbtiles: nyc.mbtiles

This comment has been minimized.

@hallahan

hallahan Oct 18, 2016

Contributor

When you uncomment url, we're getting tiles from Mapzen and populating an MBTiles with data from osm2vectortiles. Since the data comes from different places, there's going to be many inconsistencies. We should somehow check for this.

@hallahan

hallahan Oct 18, 2016

Contributor

When you uncomment url, we're getting tiles from Mapzen and populating an MBTiles with data from osm2vectortiles. Since the data comes from different places, there's going to be many inconsistencies. We should somehow check for this.

This comment has been minimized.

@hjanetzek

hjanetzek Oct 19, 2016

Member

yes, I think we should only write to mbtiles that were created by Tangram. How about adding a metadata entry for tangram_tile_cache and check for it when opening the db?

@hjanetzek

hjanetzek Oct 19, 2016

Member

yes, I think we should only write to mbtiles that were created by Tangram. How about adding a metadata entry for tangram_tile_cache and check for it when opening the db?

This comment has been minimized.

@hallahan

hallahan Oct 19, 2016

Contributor

Well, we might want to cache non-Mapzen sources too. What if I had a web service with vector tiles I wanted cached?

What if we had some sort of field in the metadata saying what the origin is? @pnorman might have some insight.

@hallahan

hallahan Oct 19, 2016

Contributor

Well, we might want to cache non-Mapzen sources too. What if I had a web service with vector tiles I wanted cached?

What if we had some sort of field in the metadata saying what the origin is? @pnorman might have some insight.

This comment has been minimized.

@hallahan

hallahan Oct 19, 2016

Contributor

Disable cross-origin tile sharing 😉

@hallahan

hallahan Oct 19, 2016

Contributor

Disable cross-origin tile sharing 😉

This comment has been minimized.

@hjanetzek

hjanetzek Oct 19, 2016

Member

Having the tile url in the metadata would be an option to allow updating of externally created files. At the moment I'm adding the check to only write to the db when it contains description:MBTiles tile container created by Tangram ES. :)

@hjanetzek

hjanetzek Oct 19, 2016

Member

Having the tile url in the metadata would be an option to allow updating of externally created files. At the moment I'm adding the check to only write to the db when it contains description:MBTiles tile container created by Tangram ES. :)

This comment has been minimized.

@pnorman

pnorman Nov 17, 2016

Contributor

Yes, you'll need to track that a mbtiles file has the same source if you want to avoid writing garbage to some other mbtiles file.

@pnorman

pnorman Nov 17, 2016

Contributor

Yes, you'll need to track that a mbtiles file has the same source if you want to avoid writing garbage to some other mbtiles file.

core/src/data/mbtilesDataSource.cpp
+ if (m_offlineMode) {
+ // Need to explicitly open a SQLite DB with OPEN_READWRITE
+ // and OPEN_CREATE flags to make a file and write.
+ mode = SQLite::OPEN_READWRITE | SQLite::OPEN_CREATE;

This comment has been minimized.

@hallahan

hallahan Oct 18, 2016

Contributor

Perhaps we want to have a m_cacheMode. That way we permit creating and / or writing to a database allowed in this circumstance. Otherwise, we should be read only.

@hallahan

hallahan Oct 18, 2016

Contributor

Perhaps we want to have a m_cacheMode. That way we permit creating and / or writing to a database allowed in this circumstance. Otherwise, we should be read only.

This comment has been minimized.

@hjanetzek

hjanetzek Oct 19, 2016

Member

yeah, there are a few modes to consider.

  1. only use existing tiles
  2. try existing tiles, download and save
  3. try download, fallback to existing tiles
  4. try download and save to mbtiles, fallback to existing tiles

So far I intended to recreate the logic of the core-mbtiles branch.

oh - in this case I just wanted to make sure not to overwrite mbtiles packs :)

@hjanetzek

hjanetzek Oct 19, 2016

Member

yeah, there are a few modes to consider.

  1. only use existing tiles
  2. try existing tiles, download and save
  3. try download, fallback to existing tiles
  4. try download and save to mbtiles, fallback to existing tiles

So far I intended to recreate the logic of the core-mbtiles branch.

oh - in this case I just wanted to make sure not to overwrite mbtiles packs :)

core/src/data/mbtilesDataSource.cpp
+
+ // OSM2Vectiles does not contain a 'compression' field
+ // Try deflate by default..
+ if (inflate(blob, length, _data) != 0) {

This comment has been minimized.

@hallahan

hallahan Oct 18, 2016

Contributor

👍

@hallahan

hallahan Oct 18, 2016

Contributor

👍

This comment has been minimized.

@hallahan

hallahan Oct 18, 2016

Contributor

Maybe when we open the database we should look and see if there is a compression field. If there is no compression field, I agree to try deflate by default and fall back to identity.

@hallahan

hallahan Oct 18, 2016

Contributor

Maybe when we open the database we should look and see if there is a compression field. If there is no compression field, I agree to try deflate by default and fall back to identity.

This comment has been minimized.

@hjanetzek

hjanetzek Oct 19, 2016

Member

Sounds good!

@hjanetzek

hjanetzek Oct 19, 2016

Member

Sounds good!

core/src/util/inflateBuffer.h
+
+namespace Tangram {
+
+int inflate(const char* _data, size_t _size, std::vector<char>& dst);

This comment has been minimized.

@hallahan

hallahan Oct 18, 2016

Contributor

This is awesome! Perhaps we should have a deflate function as well for when we populate the cache? As I was developing, I noticed that the cache got large pretty quickly.

@hallahan

hallahan Oct 18, 2016

Contributor

This is awesome! Perhaps we should have a deflate function as well for when we populate the cache? As I was developing, I noticed that the cache got large pretty quickly.

This comment has been minimized.

@hallahan

hallahan Oct 18, 2016

Contributor

Also, should inflate be in the Tangram global namespace?

@hallahan

hallahan Oct 18, 2016

Contributor

Also, should inflate be in the Tangram global namespace?

This comment has been minimized.

@hjanetzek

hjanetzek Oct 19, 2016

Member

👍, also moving to a separate namespace

@hjanetzek

hjanetzek Oct 19, 2016

Member

👍, also moving to a separate namespace

int32_t minDisplayZoom = -1;
int32_t maxDisplayZoom = -1;
int32_t maxZoom = 18;
+ std::string mime;
+
+ if (type == "GeoJSON") {

This comment has been minimized.

@hallahan

hallahan Oct 18, 2016

Contributor

Perhaps we can discern the mime type automatically from the RawDataSoure by just returning the mime type of the DataSource? We will know the mime type of the DataSource based on the implementation of the DataSource. Then, you can put the logic of figuring out the mime type for raster sources in RasterSource instead of in here?

@hallahan

hallahan Oct 18, 2016

Contributor

Perhaps we can discern the mime type automatically from the RawDataSoure by just returning the mime type of the DataSource? We will know the mime type of the DataSource based on the implementation of the DataSource. Then, you can put the logic of figuring out the mime type for raster sources in RasterSource instead of in here?

This comment has been minimized.

@hjanetzek

hjanetzek Oct 19, 2016

Member

yes, this should be improved - I was not sure which way to go: when passing mime to MBTilesDataSource constructor *Source::getMime(string url) must be static.

@hjanetzek

hjanetzek Oct 19, 2016

Member

yes, this should be improved - I was not sure which way to go: when passing mime to MBTilesDataSource constructor *Source::getMime(string url) must be static.

This comment has been minimized.

@hallahan

hallahan Oct 19, 2016

Contributor

Mime type for a tile source should be standard and well-known. Perhaps just have *Source::getMime(string url) return a constant string?

@hallahan

hallahan Oct 19, 2016

Contributor

Mime type for a tile source should be standard and well-known. Perhaps just have *Source::getMime(string url) return a constant string?

@lygstate

This comment has been minimized.

Show comment
Hide comment
@lygstate

lygstate Oct 23, 2016

Wonderful.

Wonderful.

@lygstate

This comment has been minimized.

Show comment
Hide comment
@lygstate

lygstate Oct 23, 2016

We do not need add the mbtiles attribute,
just support for a new URL schema,
url: mbtiles://file_path;
That's would reduce much code effort.
And also suggest merge all the UrlWorker into a single place.
Do not implement them in platform code.
but in tangram-es/core directly.
I''ve done that in
#836

We do not need add the mbtiles attribute,
just support for a new URL schema,
url: mbtiles://file_path;
That's would reduce much code effort.
And also suggest merge all the UrlWorker into a single place.
Do not implement them in platform code.
but in tangram-es/core directly.
I''ve done that in
#836

@matteblair

This comment has been minimized.

Show comment
Hide comment
@matteblair

matteblair Oct 24, 2016

Member

Hi @lygstate

I don't think we want to invent a new non-standardized URL scheme. An 'mbtiles' scheme wouldn't be usable with existing libraries the way that 'http' and 'file' are. A separate 'mbtiles' entry also allows us to add optional caching in a backwards-compatible way.

The design of UrlWorker isn't strictly related to this PR, but I'd gladly review those changes separately. (By the way, #836 hasn't been forgotten - thanks for staying active in the project!)

Member

matteblair commented Oct 24, 2016

Hi @lygstate

I don't think we want to invent a new non-standardized URL scheme. An 'mbtiles' scheme wouldn't be usable with existing libraries the way that 'http' and 'file' are. A separate 'mbtiles' entry also allows us to add optional caching in a backwards-compatible way.

The design of UrlWorker isn't strictly related to this PR, but I'd gladly review those changes separately. (By the way, #836 hasn't been forgotten - thanks for staying active in the project!)

@hallahan

This comment has been minimized.

Show comment
Hide comment
@hallahan

hallahan Nov 15, 2016

Contributor

Hi @hjanetzek - how is progress on this PR coming?

Contributor

hallahan commented Nov 15, 2016

Hi @hjanetzek - how is progress on this PR coming?

@hjanetzek

This comment has been minimized.

Show comment
Hide comment
@hjanetzek

hjanetzek Nov 16, 2016

Member

Hey @hallahan, I think the branch is in a good shape. DataSource refactoring needs to be reviewed first though and everybody is busy with things we need for the mobile SDKs - I hope we will get back to this soon.

Member

hjanetzek commented Nov 16, 2016

Hey @hallahan, I think the branch is in a good shape. DataSource refactoring needs to be reviewed first though and everybody is busy with things we need for the mobile SDKs - I hope we will get back to this soon.

@pnorman

This comment has been minimized.

Show comment
Hide comment
@pnorman

pnorman Nov 17, 2016

Contributor

This includes code for md5 sums. Is there something already included in one of the other libraries required that could be used, like sha256?

How can we make sure we only write to mbtiles files which are written in a way we understand? There is no guarantee that we can write to an arbitrary mbtiles file, even if it is storing vector tiles from the same source, because they could implement the mbtiles tables interface in a different way.

Contributor

pnorman commented Nov 17, 2016

This includes code for md5 sums. Is there something already included in one of the other libraries required that could be used, like sha256?

How can we make sure we only write to mbtiles files which are written in a way we understand? There is no guarantee that we can write to an arbitrary mbtiles file, even if it is storing vector tiles from the same source, because they could implement the mbtiles tables interface in a different way.

@bcamper

This comment has been minimized.

Show comment
Hide comment
@bcamper

bcamper Nov 17, 2016

Member

Good questions. @blair1618 will make a proposal as to Tangram syntax for handling the different read/write/fallback scenarios.

Member

bcamper commented Nov 17, 2016

Good questions. @blair1618 will make a proposal as to Tangram syntax for handling the different read/write/fallback scenarios.

@hallahan hallahan referenced this pull request Nov 30, 2016

Closed

Offline Maps #1115

@hallahan

This comment has been minimized.

Show comment
Hide comment
@hallahan

hallahan Jan 5, 2017

Contributor

Hey, how are things going? Any word on this PR and MBTiles support?

Contributor

hallahan commented Jan 5, 2017

Hey, how are things going? Any word on this PR and MBTiles support?

@tallytalwar

This comment has been minimized.

Show comment
Hide comment
@tallytalwar

tallytalwar Jan 5, 2017

Member

Yes, @hallahan. We were busy with a project which we finished in december. Will be giving this a good shot now, expect more action on this now :D.

Thanks for the patience and sorry for the delay on this.

Member

tallytalwar commented Jan 5, 2017

Yes, @hallahan. We were busy with a project which we finished in december. Will be giving this a good shot now, expect more action on this now :D.

Thanks for the patience and sorry for the delay on this.

@tallytalwar tallytalwar self-requested a review Jan 17, 2017

@matteblair

This comment has been minimized.

Show comment
Hide comment
@matteblair

matteblair Jan 18, 2017

Member

I have not reviewed all of this yet, but so far it's looking mostly solid.

One problem I've noticed is that SQLiteCpp/sqlite3 seems to have its own implementation for opening and reading a file path. This has two results for us:

  • We need to resolve the mbtiles path from a scene file into an absolute path before we give it to sqlite (paths in the scene file might be absolute, but more often they are relative to the scene file's location).
  • Android applications currently can't bundle an mbtiles file. Android application bundles are effectively zip archives, and so there is no file path that identifies any individual file within the bundle. One could work around this by sneakily copying a bundled mbtiles database to a location on disk at run-time (given suitable permissions and with a start-up delay).
Member

matteblair commented Jan 18, 2017

I have not reviewed all of this yet, but so far it's looking mostly solid.

One problem I've noticed is that SQLiteCpp/sqlite3 seems to have its own implementation for opening and reading a file path. This has two results for us:

  • We need to resolve the mbtiles path from a scene file into an absolute path before we give it to sqlite (paths in the scene file might be absolute, but more often they are relative to the scene file's location).
  • Android applications currently can't bundle an mbtiles file. Android application bundles are effectively zip archives, and so there is no file path that identifies any individual file within the bundle. One could work around this by sneakily copying a bundled mbtiles database to a location on disk at run-time (given suitable permissions and with a start-up delay).
@matteblair

This comment has been minimized.

Show comment
Hide comment
@matteblair

matteblair Jan 19, 2017

Member

Configuration seems to still be an open question here. I see that suggestions have been made for new parameters in the scene file, which is the typical way that we introduce new features, but in this instance I feel that some of these configuration options go beyond "styling a scene" and enter the territory of application implementation. For example, the location of an mbtiles file will probably need to be different on Android and iOS because of their storage mechanisms (as in my previous comment). Even more generally, I don't know whether we would expect to ever support mbtiles files as a data source in the JS engine, though @bcamper may have thoughts here.

With these considerations in mind, I would suggest that an interface for configuring MBTiles data sources/caches should be part of the tangram-es API and not part of the scene file.

Member

matteblair commented Jan 19, 2017

Configuration seems to still be an open question here. I see that suggestions have been made for new parameters in the scene file, which is the typical way that we introduce new features, but in this instance I feel that some of these configuration options go beyond "styling a scene" and enter the territory of application implementation. For example, the location of an mbtiles file will probably need to be different on Android and iOS because of their storage mechanisms (as in my previous comment). Even more generally, I don't know whether we would expect to ever support mbtiles files as a data source in the JS engine, though @bcamper may have thoughts here.

With these considerations in mind, I would suggest that an interface for configuring MBTiles data sources/caches should be part of the tangram-es API and not part of the scene file.

@nvkelso

This comment has been minimized.

Show comment
Hide comment
@nvkelso

nvkelso Jan 19, 2017

Member
Member

nvkelso commented Jan 19, 2017

@tallytalwar

This comment has been minimized.

Show comment
Hide comment
@tallytalwar

tallytalwar Jan 19, 2017

Member

We do have an tangram ES level API to add an MBTile DS in this PR. But yes it utilizes updating the scene ultimately adds a DS via scene yaml load!

If JS has a use case of having offline MBTile Datasource, I am fine keeping it in the scene file description only.

Member

tallytalwar commented Jan 19, 2017

We do have an tangram ES level API to add an MBTile DS in this PR. But yes it utilizes updating the scene ultimately adds a DS via scene yaml load!

If JS has a use case of having offline MBTile Datasource, I am fine keeping it in the scene file description only.

@matteblair

This comment has been minimized.

Show comment
Hide comment
@matteblair

matteblair Jan 19, 2017

Member

@nvkelso I can imagine that an offline data cache of some kind would be useful for tangram-js, but it is unclear to me whether a SQLite database file is a good fit for this purpose. It seems undesirable to add a SQL library to the JS engine for just this use. And even if tangram-js did support reading MBTiles files, the issue would remain that it would likely use a different configuration (file location, timeouts, etc) than, say, an Android application.

Member

matteblair commented Jan 19, 2017

@nvkelso I can imagine that an offline data cache of some kind would be useful for tangram-js, but it is unclear to me whether a SQLite database file is a good fit for this purpose. It seems undesirable to add a SQL library to the JS engine for just this use. And even if tangram-js did support reading MBTiles files, the issue would remain that it would likely use a different configuration (file location, timeouts, etc) than, say, an Android application.

@nvkelso

This comment has been minimized.

Show comment
Hide comment
@nvkelso

nvkelso Jan 19, 2017

Member

@matteblair I recall @bcamper's opinion being similar. But it remains a valid use case that I think we should either support out of the box or provide documentation around the suggested workaround (e.g. running a local tileserver reading from the SQLite DB and masquerading it as a local tile endpoint).

Member

nvkelso commented Jan 19, 2017

@matteblair I recall @bcamper's opinion being similar. But it remains a valid use case that I think we should either support out of the box or provide documentation around the suggested workaround (e.g. running a local tileserver reading from the SQLite DB and masquerading it as a local tile endpoint).

@matteblair

This comment has been minimized.

Show comment
Hide comment
@matteblair

matteblair Jan 19, 2017

Member

@nvkelso Totally! It is a valid use case. I think there was also talk of using zip archives for offline use - tangram-js already has a zip reader so that might get us to a solution faster.

@pnorman 's concern about different access modes to an MBTiles file seems sensible to me - we will want our interface to encourage/enforce safe usage patterns.

I can currently see two use patterns for MBTiles files:

  1. Pre-downloaded data set, read only.

  2. Data cache created by tangram, read+write.

As long as we keep these roles distinct, we can be confident that we won't accidentally write incompatible entries to a database. The files used for these roles could be specified in clearly different ways.

Are there other use patterns that I'm not thinking of?

Member

matteblair commented Jan 19, 2017

@nvkelso Totally! It is a valid use case. I think there was also talk of using zip archives for offline use - tangram-js already has a zip reader so that might get us to a solution faster.

@pnorman 's concern about different access modes to an MBTiles file seems sensible to me - we will want our interface to encourage/enforce safe usage patterns.

I can currently see two use patterns for MBTiles files:

  1. Pre-downloaded data set, read only.

  2. Data cache created by tangram, read+write.

As long as we keep these roles distinct, we can be confident that we won't accidentally write incompatible entries to a database. The files used for these roles could be specified in clearly different ways.

Are there other use patterns that I'm not thinking of?

@matteblair

This comment has been minimized.

Show comment
Hide comment
@matteblair

matteblair Jan 19, 2017

Member

As I'm thinking through using an MBTiles file as a cache mechanism, it occurs to me that we can only store one set of tiles in a single database (or else I misunderstand the MBTiles format). If that's true, then if we wanted to cache the data from a scene like Walkabout (which uses a vector tile set and a raster tile set) then we would need two separate MBTiles files to store the combined data.

In general, it seems that we'll need one database file per network data source to effectively cache map data, which is a minor bummer.

Member

matteblair commented Jan 19, 2017

As I'm thinking through using an MBTiles file as a cache mechanism, it occurs to me that we can only store one set of tiles in a single database (or else I misunderstand the MBTiles format). If that's true, then if we wanted to cache the data from a scene like Walkabout (which uses a vector tile set and a raster tile set) then we would need two separate MBTiles files to store the combined data.

In general, it seems that we'll need one database file per network data source to effectively cache map data, which is a minor bummer.

- if (sourcePtr) {
- _scene->tileSources().push_back(sourcePtr);
- }
+ _scene->tileSources().push_back(sourcePtr);

This comment has been minimized.

@tallytalwar

tallytalwar Jan 19, 2017

Member

We should still check the validity of the sourcePtr here, should not add any sourcePtr if scene file has bad source type.

@tallytalwar

tallytalwar Jan 19, 2017

Member

We should still check the validity of the sourcePtr here, should not add any sourcePtr if scene file has bad source type.

This comment has been minimized.

@hjanetzek

hjanetzek Jan 20, 2017

Member

The source type is now checked at the top of the function.

@hjanetzek

hjanetzek Jan 20, 2017

Member

The source type is now checked at the top of the function.

+ if (m_cacheMode) {
+ // Need to explicitly open a SQLite DB with OPEN_READWRITE
+ // and OPEN_CREATE flags to make a file and write.
+ mode = SQLite::OPEN_READWRITE | SQLite::OPEN_CREATE;

This comment has been minimized.

@tallytalwar

tallytalwar Jan 19, 2017

Member

"- only write to MBTiles if the db was created by tangram" from the commit comments.. I should read it as "only write to MBTiles if the db was created by tangram with cacheMode true"?

@tallytalwar

tallytalwar Jan 19, 2017

Member

"- only write to MBTiles if the db was created by tangram" from the commit comments.. I should read it as "only write to MBTiles if the db was created by tangram with cacheMode true"?

This comment has been minimized.

@matteblair

matteblair Jan 19, 2017

Member

Can you explain your question more?

@matteblair

matteblair Jan 19, 2017

Member

Can you explain your question more?

This comment has been minimized.

@tallytalwar

tallytalwar Jan 19, 2017

Member

I just wanted to clarify my understanding of ""- only write to MBTiles if the db was created by tangram".

This is only when MBTileDataSource is created with cacheMode enabled.

@tallytalwar

tallytalwar Jan 19, 2017

Member

I just wanted to clarify my understanding of ""- only write to MBTiles if the db was created by tangram".

This is only when MBTileDataSource is created with cacheMode enabled.

+ auto& task = static_cast<BinaryTileTask&>(*_task);
+ task.rawTileData = std::make_shared<std::vector<char>>();
+
+ getTileData(tileId, *task.rawTileData);

This comment has been minimized.

@tallytalwar

tallytalwar Jan 19, 2017

Member

Probably make sense to put this under an if block and check for validity.

@tallytalwar

tallytalwar Jan 19, 2017

Member

Probably make sense to put this under an if block and check for validity.

This comment has been minimized.

@matteblair

matteblair Jan 19, 2017

Member

If we can't allocate an empty vector of bytes here then crashing seems like an appropriate response.

@matteblair

matteblair Jan 19, 2017

Member

If we can't allocate an empty vector of bytes here then crashing seems like an appropriate response.

This comment has been minimized.

@tallytalwar

tallytalwar Jan 19, 2017

Member

nopes, in that case (issues with file read from MBTiles most likely) one would want to fallback on the network next datasource.

@tallytalwar

tallytalwar Jan 19, 2017

Member

nopes, in that case (issues with file read from MBTiles most likely) one would want to fallback on the network next datasource.

This comment has been minimized.

@matteblair

matteblair Jan 19, 2017

Member

Mmmm I think that if creating this object fails then we're out of memory and should do nothing but exit the process.

@matteblair

matteblair Jan 19, 2017

Member

Mmmm I think that if creating this object fails then we're out of memory and should do nothing but exit the process.

This comment has been minimized.

@matteblair

matteblair Jan 19, 2017

Member

Or I don't know what you want to check the validity of?

@matteblair

matteblair Jan 19, 2017

Member

Or I don't know what you want to check the validity of?

+ auto& task = static_cast<BinaryTileTask&>(*_task);
+ task.rawTileData = std::make_shared<std::vector<char>>();
+
+ getTileData(_task->tileId(), *task.rawTileData);

This comment has been minimized.

@tallytalwar

tallytalwar Jan 19, 2017

Member

Probably make sense to put this under an if block and check for validity.

@tallytalwar

tallytalwar Jan 19, 2017

Member

Probably make sense to put this under an if block and check for validity.

+ } else if (next) {
+
+ // Don't try this source again
+ _task->rawSource = next->level;

This comment has been minimized.

@tallytalwar

tallytalwar Jan 19, 2017

Member

Hmm... by this it means the mbtile sounce will not be tried again, however if cacheMode is enabled, mbtiles database will get relevant tile data to be used next time? Or you want to fallback on NetworkDataSource's cache at that point?

@tallytalwar

tallytalwar Jan 19, 2017

Member

Hmm... by this it means the mbtile sounce will not be tried again, however if cacheMode is enabled, mbtiles database will get relevant tile data to be used next time? Or you want to fallback on NetworkDataSource's cache at that point?

This comment has been minimized.

@tallytalwar

tallytalwar Jan 19, 2017

Member

Ignore this, makes sense now.

@tallytalwar

tallytalwar Jan 19, 2017

Member

Ignore this, makes sense now.

@tallytalwar

This comment has been minimized.

Show comment
Hide comment
@tallytalwar

tallytalwar Jan 19, 2017

Member

@matteblair you are correct in saying that we will need one database file per network datasource for caching with mbtiles. It looks like its exactly what is implemented in this PR with when caching mode is enabled for an mbtiles datasource.

Way I see it being used for caching with all network datasources is by defining an MBTiles datasource for each network source and enabling offline and caching mode for the corresponding MBTiles datasource.

Member

tallytalwar commented Jan 19, 2017

@matteblair you are correct in saying that we will need one database file per network datasource for caching with mbtiles. It looks like its exactly what is implemented in this PR with when caching mode is enabled for an mbtiles datasource.

Way I see it being used for caching with all network datasources is by defining an MBTiles datasource for each network source and enabling offline and caching mode for the corresponding MBTiles datasource.

@tallytalwar

This comment has been minimized.

Show comment
Hide comment
@tallytalwar

tallytalwar Jan 19, 2017

Member

I can currently see two use patterns for MBTiles files:

  1. Pre-downloaded data set, read only.

  2. Data cache created by tangram, read+write.

As long as we keep these roles distinct, we can be confident that we won't accidentally write incompatible entries to a database. The files used for these roles could be specified in clearly different ways.

So I think currently the core MbTileDatasource code handles this gracefully, and won't accidentally write incompatible entries to a db.
Currently, how it is implemented via SceneLoader all MBTileDataSource will have read+write support. We will have to aptly exposed via an API, and let the user define the behavior when setting a MBTile Datasource.

Member

tallytalwar commented Jan 19, 2017

I can currently see two use patterns for MBTiles files:

  1. Pre-downloaded data set, read only.

  2. Data cache created by tangram, read+write.

As long as we keep these roles distinct, we can be confident that we won't accidentally write incompatible entries to a database. The files used for these roles could be specified in clearly different ways.

So I think currently the core MbTileDatasource code handles this gracefully, and won't accidentally write incompatible entries to a db.
Currently, how it is implemented via SceneLoader all MBTileDataSource will have read+write support. We will have to aptly exposed via an API, and let the user define the behavior when setting a MBTile Datasource.

@matteblair

This comment has been minimized.

Show comment
Hide comment
@matteblair

matteblair Jan 19, 2017

Member

I think MBTilesDataSource is able to follow the policy we want, but it's important that the user interface makes it easy to set up both offline sources and caches correctly, rather than relying on run-time checks.

Member

matteblair commented Jan 19, 2017

I think MBTilesDataSource is able to follow the policy we want, but it's important that the user interface makes it easy to set up both offline sources and caches correctly, rather than relying on run-time checks.

@tallytalwar

This comment has been minimized.

Show comment
Hide comment
@tallytalwar

tallytalwar Jan 19, 2017

Member

Exactly my point :D

Member

tallytalwar commented Jan 19, 2017

Exactly my point :D

core/src/data/rasterSource.h
@@ -36,7 +36,7 @@ class RasterSource : public TileSource {
// TODO Is this always PNG or can it also be JPEG?
virtual const char* mimeType() override { return "image/png"; };
- void loadTileData(std::shared_ptr<TileTask> _task, TileTaskCb _cb);
+ virtual void loadTileData(std::shared_ptr<TileTask> _task, TileTaskCb _cb);

This comment has been minimized.

@hjanetzek

hjanetzek Jan 20, 2017

Member

Since c++11 one can use override to have the compiler check that a function actually overrides a virtual function of the base class. I think this is much better than using virtual in the derived class just as a hint for the programmer.

Admittedly we should make this more consistent one or the other way.

@hjanetzek

hjanetzek Jan 20, 2017

Member

Since c++11 one can use override to have the compiler check that a function actually overrides a virtual function of the base class. I think this is much better than using virtual in the derived class just as a hint for the programmer.

Admittedly we should make this more consistent one or the other way.

This comment has been minimized.

@tallytalwar

tallytalwar Jan 20, 2017

Member

Removed the commit, and lets try to make it consistent as and when one touches that appropriate code.

@tallytalwar

tallytalwar Jan 20, 2017

Member

Removed the commit, and lets try to make it consistent as and when one touches that appropriate code.

@matteblair

This comment has been minimized.

Show comment
Hide comment
@matteblair

matteblair Jan 23, 2017

Member

If we were to use the scene file to configure MBTiles sources, the most sensible approach I can think of is to allow a new type of sources entry where the url field contains the location of a file with the .mbtiles extension and the type field is not used (tile format is deduced from the MBTiles metadata). e.g.

sources:
  _my_source:
    url: /storage/emulated/0/mapdata/newyork.mbtiles
    max_zoom: 16

For the caching use pattern, we can use a new field for sources entries, tentatively called cache, that specifies the location of a file with the .mbtiles extension. The cache file is permitted to not yet exist, but the path to the file must exist. e.g.

sources:
  _my_source:
    type: MVT
    url: https://tile.mapzen.com/mapzen/vector/v1/all/{z}/{x}/{y}.mvt
    url_params:
      api_key: xxxxxxx
    cache: /storage/emulated/0/mapdata/cache.mbtiles

Some things to note about this approach:

  • The paths in these examples are typical for local file storage on Android. I used these paths because unlike most file paths we specify in scene files, these cannot be relative paths from the scene file (at least on Android). The caching use pattern will require (different) absolute file paths on both Android and iOS. The application developer will also need to request any needed permissions to write a file to disk. This illustrates my concerns that configurations using this approach are not portable.
  • The read/write access to MBTiles files should be evident with this syntax. The file in url is always read-only and the file in cache: is always read-write.
  • There is much more that could be specified about caching behavior, e.g. the timeout duration on the primary data source, the expiration of cached data, etc. Many of these parameters are application-dependent, they would differ between a mobile and desktop application. This makes me hesitate to put them in the scene file, which aims to be as platform-agnostic as possible.
Member

matteblair commented Jan 23, 2017

If we were to use the scene file to configure MBTiles sources, the most sensible approach I can think of is to allow a new type of sources entry where the url field contains the location of a file with the .mbtiles extension and the type field is not used (tile format is deduced from the MBTiles metadata). e.g.

sources:
  _my_source:
    url: /storage/emulated/0/mapdata/newyork.mbtiles
    max_zoom: 16

For the caching use pattern, we can use a new field for sources entries, tentatively called cache, that specifies the location of a file with the .mbtiles extension. The cache file is permitted to not yet exist, but the path to the file must exist. e.g.

sources:
  _my_source:
    type: MVT
    url: https://tile.mapzen.com/mapzen/vector/v1/all/{z}/{x}/{y}.mvt
    url_params:
      api_key: xxxxxxx
    cache: /storage/emulated/0/mapdata/cache.mbtiles

Some things to note about this approach:

  • The paths in these examples are typical for local file storage on Android. I used these paths because unlike most file paths we specify in scene files, these cannot be relative paths from the scene file (at least on Android). The caching use pattern will require (different) absolute file paths on both Android and iOS. The application developer will also need to request any needed permissions to write a file to disk. This illustrates my concerns that configurations using this approach are not portable.
  • The read/write access to MBTiles files should be evident with this syntax. The file in url is always read-only and the file in cache: is always read-write.
  • There is much more that could be specified about caching behavior, e.g. the timeout duration on the primary data source, the expiration of cached data, etc. Many of these parameters are application-dependent, they would differ between a mobile and desktop application. This makes me hesitate to put them in the scene file, which aims to be as platform-agnostic as possible.
@vicb

This comment has been minimized.

Show comment
Hide comment
@vicb

vicb Jan 23, 2017

I'm really interested in using this feature so I'll add some comments - I don't know a lot about this project and some might not be relevant:

Using .mbtiles files

tile format is deduced from the MBTiles metadata

May be it could be good to add a type: mbtiles anyway for extensibility purpose ?

cache

sources:
  _my_source:
    cache: /storage/emulated/0/mapdata/cache.mbtiles

App devs probably care about a fs backed cache, do they really care about where the cache will be stored, ie should the path be optional ?
Should you also provision a type: mbtile entry here for extensibility - it could probably be the default for now.

Anyway kudos for this projectand can't wait for the offline support to be merged !

vicb commented Jan 23, 2017

I'm really interested in using this feature so I'll add some comments - I don't know a lot about this project and some might not be relevant:

Using .mbtiles files

tile format is deduced from the MBTiles metadata

May be it could be good to add a type: mbtiles anyway for extensibility purpose ?

cache

sources:
  _my_source:
    cache: /storage/emulated/0/mapdata/cache.mbtiles

App devs probably care about a fs backed cache, do they really care about where the cache will be stored, ie should the path be optional ?
Should you also provision a type: mbtile entry here for extensibility - it could probably be the default for now.

Anyway kudos for this projectand can't wait for the offline support to be merged !

@matteblair

This comment has been minimized.

Show comment
Hide comment
@matteblair

matteblair Jan 23, 2017

Member

Hi @vicb, thanks for the rapid feedback!

May be it could be good to add a type: mbtiles anyway for extensibility purpose ?

I was thinking about this too. The reason I didn't suggest it is that the other options for type are vector tile data formats, whereas MBTiles is a database that can contain tiles in any of the other formats, so it's not really consistent.

App devs probably care about a fs backed cache, do they really care about where the cache will be stored, ie should the path be optional ?

Fair point. The path will be important for some cases, e.g. if an app wants to use a temporary folder for the cache or be able to pull the file off the device later. For other cases it might not matter. I believe that we can access a reasonable "default" location for saving files on all of our supported platforms. To configure this we could make the cache field a mapping, allowing for more options later:

cache: 
  url: /storage/emulated/0/mapdata/cache.mbtiles # optional!

or we could use a Boolean true to represent "unspecified caching":

cache: true

Again, I see both of these as going beyond the intended purpose of the scene file :\

Member

matteblair commented Jan 23, 2017

Hi @vicb, thanks for the rapid feedback!

May be it could be good to add a type: mbtiles anyway for extensibility purpose ?

I was thinking about this too. The reason I didn't suggest it is that the other options for type are vector tile data formats, whereas MBTiles is a database that can contain tiles in any of the other formats, so it's not really consistent.

App devs probably care about a fs backed cache, do they really care about where the cache will be stored, ie should the path be optional ?

Fair point. The path will be important for some cases, e.g. if an app wants to use a temporary folder for the cache or be able to pull the file off the device later. For other cases it might not matter. I believe that we can access a reasonable "default" location for saving files on all of our supported platforms. To configure this we could make the cache field a mapping, allowing for more options later:

cache: 
  url: /storage/emulated/0/mapdata/cache.mbtiles # optional!

or we could use a Boolean true to represent "unspecified caching":

cache: true

Again, I see both of these as going beyond the intended purpose of the scene file :\

@hjanetzek

This comment has been minimized.

Show comment
Hide comment
@hjanetzek

hjanetzek Jan 23, 2017

Member

@matteblair sounds all pretty much good to me. Could you add these changes?
I think it will generally makes sense to pass mbtiles options as SceneUpdate on scene loading (left to the application developer or Mapzen SDK) - No need to consider portability of mbtiles paths imo right now.
Another thing to keep in mind/take care of: When mbtiles are used for raster sources they have to be registered via scene files since the initial tile sources are checked to build the shaders.

Member

hjanetzek commented Jan 23, 2017

@matteblair sounds all pretty much good to me. Could you add these changes?
I think it will generally makes sense to pass mbtiles options as SceneUpdate on scene loading (left to the application developer or Mapzen SDK) - No need to consider portability of mbtiles paths imo right now.
Another thing to keep in mind/take care of: When mbtiles are used for raster sources they have to be registered via scene files since the initial tile sources are checked to build the shaders.

@matteblair

This comment has been minimized.

Show comment
Hide comment
@matteblair

matteblair Jan 23, 2017

Member

@hjanetzek Yep, I will start implementing these suggestions, beginning with the read-only usage since that seems the most clear.

Member

matteblair commented Jan 23, 2017

@hjanetzek Yep, I will start implementing these suggestions, beginning with the read-only usage since that seems the most clear.

@vicb

This comment has been minimized.

Show comment
Hide comment
@vicb

vicb Jan 24, 2017

@matteblair

sources:
  _my_source:
    url: /storage/emulated/0/mapdata/newyork.mbtiles
    max_zoom: 16

From the docs the type is required w/o default.
I think that omitting the type here would harm extensibility - how would you implement a mztiles format for example ?
I understand that the tile themselves could be in different format but still you need to know how to read the container, don't you ?

vicb commented Jan 24, 2017

@matteblair

sources:
  _my_source:
    url: /storage/emulated/0/mapdata/newyork.mbtiles
    max_zoom: 16

From the docs the type is required w/o default.
I think that omitting the type here would harm extensibility - how would you implement a mztiles format for example ?
I understand that the tile themselves could be in different format but still you need to know how to read the container, don't you ?

@matteblair

This comment has been minimized.

Show comment
Hide comment
@matteblair

matteblair Jan 24, 2017

Member

@vicb Yep, type is currently required for all data sources but that could change with the addition of MBTiles sources. What I was proposing was that we use the file extension of the URL to signify that we are using an MBTiles source (or, in the future, .mztiles for a hypothetical MZTiles source).

The drawback I see to mandating type: MBTiles is that it is redundant information and it could invite errors where a type is given that conflicts with the URL (e.g. someone mistakenly uses the vector tile type for an MBTiles source). That said, with MVT tile sources we have a similar tradeoff, since MVT could be inferred from the file extension, and we still require a type for those sources.

I now agree that it would be more consistent to require that MBTiles sources specify type: MBTiles.

Member

matteblair commented Jan 24, 2017

@vicb Yep, type is currently required for all data sources but that could change with the addition of MBTiles sources. What I was proposing was that we use the file extension of the URL to signify that we are using an MBTiles source (or, in the future, .mztiles for a hypothetical MZTiles source).

The drawback I see to mandating type: MBTiles is that it is redundant information and it could invite errors where a type is given that conflicts with the URL (e.g. someone mistakenly uses the vector tile type for an MBTiles source). That said, with MVT tile sources we have a similar tradeoff, since MVT could be inferred from the file extension, and we still require a type for those sources.

I now agree that it would be more consistent to require that MBTiles sources specify type: MBTiles.

@matteblair

This comment has been minimized.

Show comment
Hide comment
@matteblair

matteblair Jan 31, 2017

Member

I'm still working on making the user experience here make sense, these are my current thoughts.

When I'm using an existing MBTiles database as a data source, there should be no need to specify the format of the tiles in the database because the database itself holds that information as metadata. If I download an MBTiles file without knowing exactly what it contains, I am not sure how I would determine the format of its contents (short of running SQL queries on it). I wouldn't know what to put in the type parameter in this case.

This led us to deciding on the type: MBTiles option, which lets tangram-es deduce the tile format from the MBTiles file itself.

There are still cases where the user must know something about the tile format in the database. When I use an MBTiles file as a raster tile source, I must know that the contents are raster tiles so that I can set the rasters parameter of other data sources that use it. When tangram-es is initializing data sources it must also be aware that a source with type: MBTiles might be a raster source, but also might not be, which is a little weird.

In code terms, the type: MBTiles approach also implies that the "type" of a tile source isn't known until after it is initialized, which is difficult with our current inheritance-based approach to tile sources. I'm currently working on resolving this by refactoring for composition instead of inheritance.

Another issue in the code is that the current tile source + data source architecture doesn't easily facilitate a data source telling a tile source what format it should use - which is essentially what we need to do here.

Member

matteblair commented Jan 31, 2017

I'm still working on making the user experience here make sense, these are my current thoughts.

When I'm using an existing MBTiles database as a data source, there should be no need to specify the format of the tiles in the database because the database itself holds that information as metadata. If I download an MBTiles file without knowing exactly what it contains, I am not sure how I would determine the format of its contents (short of running SQL queries on it). I wouldn't know what to put in the type parameter in this case.

This led us to deciding on the type: MBTiles option, which lets tangram-es deduce the tile format from the MBTiles file itself.

There are still cases where the user must know something about the tile format in the database. When I use an MBTiles file as a raster tile source, I must know that the contents are raster tiles so that I can set the rasters parameter of other data sources that use it. When tangram-es is initializing data sources it must also be aware that a source with type: MBTiles might be a raster source, but also might not be, which is a little weird.

In code terms, the type: MBTiles approach also implies that the "type" of a tile source isn't known until after it is initialized, which is difficult with our current inheritance-based approach to tile sources. I'm currently working on resolving this by refactoring for composition instead of inheritance.

Another issue in the code is that the current tile source + data source architecture doesn't easily facilitate a data source telling a tile source what format it should use - which is essentially what we need to do here.

@placz

This comment has been minimized.

Show comment
Hide comment
@placz

placz Feb 6, 2017

Just a couple thoughts since I'm interested in seeing this feature.

It sounds like while technically the more specific type can be inferred, there are some cases in the yaml where it would be an error to have a mismatch, such as the rasters example. Perhaps the type for mbtiles can be used as an assertion that the data source is this type rather than choosing a parser. Thus, specifying 'type:MBTiles-raster' would assert that it's a raster and can be used as a raster in other sources. 'type:MBTiles' would not be able to be used as a raster in other sources.

placz commented Feb 6, 2017

Just a couple thoughts since I'm interested in seeing this feature.

It sounds like while technically the more specific type can be inferred, there are some cases in the yaml where it would be an error to have a mismatch, such as the rasters example. Perhaps the type for mbtiles can be used as an assertion that the data source is this type rather than choosing a parser. Thus, specifying 'type:MBTiles-raster' would assert that it's a raster and can be used as a raster in other sources. 'type:MBTiles' would not be able to be used as a raster in other sources.

@matteblair

This comment has been minimized.

Show comment
Hide comment
@matteblair

matteblair Feb 8, 2017

Member

Alright - here's where I've landed on this.

The scene file now supports only read-only MBTiles sources using existing files. When the url of a sources entry ends with the .mbtiles extension, it is recognized as an MBTiles source. The type field remains unchanged, that is it should be set to the type of vector or raster tile expected from the data source. For example:

sources:
  _savedTiles: #This is an example MBTiles data source.
    type: TopoJSON #We expect TopoJSON data from the source.
    url: /path/to/my/file.mbtiles #Only absolute file paths supported for now.
    max_zoom: 16 #Other optional parameters are applied as usual.

I think we should merge this now to provide the core functionality and then add caching and additional configuration later on as we figure out usage.

Member

matteblair commented Feb 8, 2017

Alright - here's where I've landed on this.

The scene file now supports only read-only MBTiles sources using existing files. When the url of a sources entry ends with the .mbtiles extension, it is recognized as an MBTiles source. The type field remains unchanged, that is it should be set to the type of vector or raster tile expected from the data source. For example:

sources:
  _savedTiles: #This is an example MBTiles data source.
    type: TopoJSON #We expect TopoJSON data from the source.
    url: /path/to/my/file.mbtiles #Only absolute file paths supported for now.
    max_zoom: 16 #Other optional parameters are applied as usual.

I think we should merge this now to provide the core functionality and then add caching and additional configuration later on as we figure out usage.

@matteblair matteblair changed the title from [wip] MBTiles DataSource to MBTiles DataSource Feb 8, 2017

@tallytalwar

This comment has been minimized.

Show comment
Hide comment
@tallytalwar

tallytalwar Feb 8, 2017

Member

Will give the updates a look in a bit and merge if all good.

Member

tallytalwar commented Feb 8, 2017

Will give the updates a look in a bit and merge if all good.

@bcamper

This comment has been minimized.

Show comment
Hide comment
@bcamper

bcamper Feb 8, 2017

Member

Agreed this is the right initial scope, thanks!

Member

bcamper commented Feb 8, 2017

Agreed this is the right initial scope, thanks!

@tallytalwar

There is a small bug in here. See the review comment below. Good otherwise.

Can you also add something in the tangram-docs (at least ES only for the time being) to make sure our users are aware of the current implementation.

Thanks

core/src/data/mbtilesDataSource.cpp
+
+};
+
+ MBTilesDataSource::MBTilesDataSource(std::shared_ptr<Platform> _platform, std::string _name,

This comment has been minimized.

@tallytalwar

tallytalwar Feb 8, 2017

Member

nitpick here ... can we get rid of the indent here?

@tallytalwar

tallytalwar Feb 8, 2017

Member

nitpick here ... can we get rid of the indent here?

This comment has been minimized.

@matteblair

matteblair Feb 8, 2017

Member

Fixed. That's what I get for editing in Xcode haha

@matteblair

matteblair Feb 8, 2017

Member

Fixed. That's what I get for editing in Xcode haha

core/src/scene/sceneLoader.cpp
+ const char* extStr = ".mbtiles";
+ const size_t extLength = strlen(extStr);
+ const size_t urlLength = url.length();
+ isMBTilesFile = urlLength > extLength && (url.compare(urlLength - extLength, urlLength, extStr) == 0);

This comment has been minimized.

@tallytalwar

tallytalwar Feb 8, 2017

Member

Hmm I think this should be:
url.compare(urlLength - extLength, extLength, extStr) == 0

To check if substring [arg1, arg1+arg2) in url is same as arg3!

@tallytalwar

tallytalwar Feb 8, 2017

Member

Hmm I think this should be:
url.compare(urlLength - extLength, extLength, extStr) == 0

To check if substring [arg1, arg1+arg2) in url is same as arg3!

This comment has been minimized.

@tallytalwar

tallytalwar Feb 8, 2017

Member

Also probably make sense for tiled check here. If a url defines a tiled datasource (specified x,y,z) it can not be a mbtiles datasource.

@tallytalwar

tallytalwar Feb 8, 2017

Member

Also probably make sense for tiled check here. If a url defines a tiled datasource (specified x,y,z) it can not be a mbtiles datasource.

This comment has been minimized.

@matteblair

matteblair Feb 8, 2017

Member

You are correct, fixed!

@matteblair

matteblair Feb 8, 2017

Member

You are correct, fixed!

This comment has been minimized.

@matteblair

matteblair Feb 8, 2017

Member

Right - your comment about tiled. In the case of something like url: /w/{x}/{y}/{z}.mbtiles I would lean towards treating it as a path to an MBTiles file - curly braces are valid in paths and are somewhat common on Windows. I would not make it a recommended practice though ;)

@matteblair

matteblair Feb 8, 2017

Member

Right - your comment about tiled. In the case of something like url: /w/{x}/{y}/{z}.mbtiles I would lean towards treating it as a path to an MBTiles file - curly braces are valid in paths and are somewhat common on Windows. I would not make it a recommended practice though ;)

This comment has been minimized.

@tallytalwar

tallytalwar Feb 8, 2017

Member

Sure, again good to document.

@tallytalwar

tallytalwar Feb 8, 2017

Member

Sure, again good to document.

@tallytalwar

This comment has been minimized.

Show comment
Hide comment
@tallytalwar

tallytalwar Feb 8, 2017

Member

Will be merging this when CI builds are done.

Member

tallytalwar commented Feb 8, 2017

Will be merging this when CI builds are done.

@matteblair

This comment has been minimized.

Show comment
Hide comment
@matteblair

matteblair Feb 8, 2017

Member

Rebasing - hang tight

Update: all systems go

Member

matteblair commented Feb 8, 2017

Rebasing - hang tight

Update: all systems go

hallahan and others added some commits Sep 3, 2016

MBTiles support
rebased+squashed mbtiles branch from Nicholas Hallahan

Added SQLiteCpp

Added mbtiles parameter to data sources and scene.yam tangrams/tangram-es#960

Created MBTilesTileTask. tangrams/tangram-es#960

Basic MBTiles functionality working! tangrams/tangram-es#960

No need to make 2 vectors. Writing SQLite blob directly into rawTileData tangrams/tangram-es#960

Database and Statements are class members.

Now caching network tiles in MBTiles. tangrams/tangram-es#960

Now creating fresh MBTiles caches. Using schema from node-mbtiles. tangrams/tangram-es#960

Working on dynamically setting MBTiles from Tangram::Map interface. tangrams/tangram-es#960

Remove setMBTiles functions. Going with Scene Updates. Also isOfflineOnly is now hasNoURL. tangrams/tangram-es#960

MBTiles Scene Update working on Android when commenting out fonts. tangrams/tangram-es#978 tangrams/tangram-es#960

Create Map::setMBTiles and exposing it through JNI tangrams/tangram-es#960

Added MD5 hash for join key on MBTiles tile and images tables tangrams/tangram-es#960

Write to metadata table when setting up MBTiles tangrams/tangram-es#960

ClientGeoJsonSource will never have MBTiles tangrams/tangram-es#960

fixup

disable failing cpplint.py checks

Implemented mimeType method in TestDataSource

compression: identity when there is no compression

linux target_link_libraries includes dl in core

Cleaner way of setting target_link_libraries to include dl for Linux.
wip: add MBTilesDataSource
squashed:
- wip: async loading for MBTiles
- implement MBTiles offline fallback mode
- wip: support compressed MBTiles
  - testing with osm2vectortiles planet
  - try deflate by default
- support general MBTiles schema
  - check metadata compression option
  - differentiate caching and offline-fallback mode
  - only write to MBTiles if the db was created by tangram
Recognize MBTiles sources from .mbtiles extension in URL
This removes the 'mbtiles' field from data sources and effectively hides the MBTiles caching interface until it can be refined further.

@matteblair matteblair merged commit 08090f0 into master Feb 8, 2017

2 checks passed

ci/circleci Your tests passed on CircleCI!
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details

@matteblair matteblair deleted the core-mbtiles-datasource branch Feb 8, 2017

@tallytalwar tallytalwar referenced this pull request Feb 9, 2017

Closed

offline map android #1281

@karimnaaji karimnaaji referenced this pull request in tangrams/ios-framework Feb 20, 2017

Merged

TangramMap v0.5.0 #20

@matteblair matteblair referenced this pull request in tangrams/tangram-docs Feb 23, 2017

Closed

Document MBTiles sources for tangram-es #192

@tallytalwar tallytalwar referenced this pull request Mar 2, 2017

Closed

Android cache #1309

@rwrx

This comment has been minimized.

Show comment
Hide comment
@rwrx

rwrx Jun 9, 2017

Is this MBTiles sources support usable for offline maps on Android? Have you tested it? I am just asking, I didn't tried nor looked into it. Thank you.

rwrx commented Jun 9, 2017

Is this MBTiles sources support usable for offline maps on Android? Have you tested it? I am just asking, I didn't tried nor looked into it. Thank you.

@tallytalwar

This comment has been minimized.

Show comment
Hide comment
@tallytalwar

tallytalwar Jun 9, 2017

Member

Hi @rwrx

Using MBTiles as an offline source of map tiles is working on all platforms we support. There are other usecases of MBTiles discussed in the (merged) PR which are not fully supported.

To use a MBTiles set in your scene you can refer to this comment above.

Member

tallytalwar commented Jun 9, 2017

Hi @rwrx

Using MBTiles as an offline source of map tiles is working on all platforms we support. There are other usecases of MBTiles discussed in the (merged) PR which are not fully supported.

To use a MBTiles set in your scene you can refer to this comment above.

@rwrx

This comment has been minimized.

Show comment
Hide comment
@rwrx

rwrx Jun 10, 2017

It is also possible to change that path to mbtiles file at runtime? I mean not only hardcode it into yaml scene file. And it is possible to have multiple mbtiles paths defined? I mean a use case where user has for example downloaded multiple mbtiles for countries and I need to show them on the map.

rwrx commented Jun 10, 2017

It is also possible to change that path to mbtiles file at runtime? I mean not only hardcode it into yaml scene file. And it is possible to have multiple mbtiles paths defined? I mean a use case where user has for example downloaded multiple mbtiles for countries and I need to show them on the map.

@rwrx

This comment has been minimized.

Show comment
Hide comment
@rwrx

rwrx Jun 22, 2017

I had look at Tangram-ES code and now I understand as mbtiles file usage is meant there. So I have an answer to my question now. Thank you.

rwrx commented Jun 22, 2017

I had look at Tangram-ES code and now I understand as mbtiles file usage is meant there. So I have an answer to my question now. Thank you.

@kueda

This comment has been minimized.

Show comment
Hide comment
@kueda

kueda Sep 11, 2017

Has this functionality been documented anywhere for use in Android, or better yet implemented in a demo? Would be great to have a working example.

kueda commented Sep 11, 2017

Has this functionality been documented anywhere for use in Android, or better yet implemented in a demo? Would be great to have a working example.

@matteblair

This comment has been minimized.

Show comment
Hide comment
@matteblair

matteblair Sep 12, 2017

Member

@kueda There's an open issue to add this in our docs repo: tangrams/tangram-docs#192, thanks for the reminder to add this!

Member

matteblair commented Sep 12, 2017

@kueda There's an open issue to add this in our docs repo: tangrams/tangram-docs#192, thanks for the reminder to add this!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment