Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add SQLite Backend #148

Merged
merged 15 commits into from Jan 15, 2021
Merged
5 changes: 5 additions & 0 deletions CHANGES.md
@@ -1,5 +1,10 @@
# Release Notes

## 3.0.0b2 (TBD)

* add `SQLite` backend (https://github.com/developmentseed/cogeo-mosaic/pull/148)
* fix cached responsed after updating a mosaic (https://github.com/developmentseed/cogeo-mosaic/pull/148/files#r557020660)

## 3.0.0b1 (2020-12-18)

* remove `overview` command (https://github.com/developmentseed/cogeo-mosaic/issues/71#issuecomment-748265645)
Expand Down
6 changes: 4 additions & 2 deletions README.md
Expand Up @@ -35,7 +35,7 @@

**Read the official announcement https://medium.com/devseed/cog-talk-part-2-mosaics-bbbf474e66df**

## Install (python >=3)
## Install
```bash
$ pip install pip -U
$ pip install cogeo-mosaic --pre
Expand All @@ -52,7 +52,7 @@ $ pip install git+http://github.com/developmentseed/cogeo-mosaic
- **pygeos** hosted on pypi migth not compile on certain machine. This has been fixed in the master branch and can be installed with `pip install git+https://github.com/pygeos/pygeos.git`


# See it in actions
## See it in actions

- [**TiTiler**](http://github.com/developmentseed/titiler): A lightweight Cloud Optimized GeoTIFF dynamic tile server (COG, STAC and MosaicJSON).

Expand All @@ -68,6 +68,8 @@ See [LICENSE](https://github.com/developmentseed/cogeo-mosaic/blob/master/LICENS

Created by [Development Seed](<http://developmentseed.org>)

See [contributors](https://github.com/developmentseed/cogeo-mosaic/graphs/contributors) for a listing of individual contributors.

## Changes

See [CHANGES.md](https://github.com/developmentseed/cogeo-mosaic/blob/master/CHANGES.md).
4 changes: 4 additions & 0 deletions cogeo_mosaic/backends/__init__.py
Expand Up @@ -7,6 +7,7 @@
from cogeo_mosaic.backends.dynamodb import DynamoDBBackend
from cogeo_mosaic.backends.file import FileBackend
from cogeo_mosaic.backends.s3 import S3Backend
from cogeo_mosaic.backends.sqlite import SQLiteBackend
from cogeo_mosaic.backends.stac import STACBackend
from cogeo_mosaic.backends.web import HttpBackend

Expand All @@ -25,6 +26,9 @@ def MosaicBackend(url: str, *args: Any, **kwargs: Any) -> BaseBackend:
if parsed.scheme == "dynamodb":
return DynamoDBBackend(url, *args, **kwargs)

if parsed.scheme == "sqlite":
return SQLiteBackend(url, *args, **kwargs)

if parsed.scheme in ["https", "http"]:
return HttpBackend(url, *args, **kwargs)

Expand Down
4 changes: 2 additions & 2 deletions cogeo_mosaic/backends/base.py
Expand Up @@ -101,7 +101,7 @@ def assets_for_point(self, lng: float, lat: float) -> List[str]:

@cached(
TTLCache(maxsize=cache_config.maxsize, ttl=cache_config.ttl),
key=lambda self, x, y, z: hashkey(self.path, x, y, z),
key=lambda self, x, y, z: hashkey(self.path, x, y, z, self.mosaicid),
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unrelated to the SQLite backend, but while writing the tests I realized that we were caching this even when the mosaic changed... to fix it we can use the mosaicid (built from the metadata, e.g mosaic version) to make sure we call the function when the mosaic has been updated.

)
def get_assets(self, x: int, y: int, z: int) -> List[str]:
"""Find assets."""
Expand Down Expand Up @@ -196,7 +196,7 @@ def update(
**kwargs,
):
"""Update existing MosaicJSON on backend."""
new_mosaic = self.mosaic_def.from_features(
new_mosaic = MosaicJSON.from_features(
features,
self.mosaic_def.minzoom,
self.mosaic_def.maxzoom,
Expand Down
7 changes: 4 additions & 3 deletions cogeo_mosaic/backends/dynamodb.py
Expand Up @@ -13,7 +13,6 @@
import attr
import click
import mercantile
from botocore.exceptions import ClientError
from cachetools import TTLCache, cached
from cachetools.keys import hashkey

Expand All @@ -33,9 +32,11 @@
try:
import boto3
from boto3.dynamodb.conditions import Key
from botocore.exceptions import ClientError
except ImportError: # pragma: nocover
boto3 = None # type: ignore
Key = None # type: ignore
ClientError = None # type: ignore


@attr.s
Expand Down Expand Up @@ -151,7 +152,7 @@ def update(
"""Update existing MosaicJSON on backend."""
logger.debug(f"Updating {self.mosaic_name}...")

new_mosaic = self.mosaic_def.from_features(
new_mosaic = MosaicJSON.from_features(
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think using self.mosaic_def.from_features( was a bit misleading

features,
self.mosaic_def.minzoom,
self.mosaic_def.maxzoom,
Expand Down Expand Up @@ -284,7 +285,7 @@ def _read(self) -> MosaicJSON: # type: ignore

@cached(
TTLCache(maxsize=cache_config.maxsize, ttl=cache_config.ttl),
key=lambda self, x, y, z: hashkey(self.path, x, y, z),
key=lambda self, x, y, z: hashkey(self.path, x, y, z, self.mosaicid),
)
def get_assets(self, x: int, y: int, z: int) -> List[str]:
"""Find assets."""
Expand Down