diff --git a/docs/parameters.rst b/docs/parameters.rst
index 15fc2c03..b2d7fa52 100644
--- a/docs/parameters.rst
+++ b/docs/parameters.rst
@@ -27,13 +27,15 @@ Here is the full list of configuration parameters you can specify in a ``config.
**imagery**: string
Label Maker expects to receive imagery tiles that are 256 x 256 pixels. You can specific the source of the imagery with one of:
+
+ - A template string for a tiled imagery service. Note that you will generally need an API key to obtain images and there may be associated costs. The above example requires a `Mapbox access token `_. Also see `OpenAerialMap `_ for open imagery. The access token for TMS image formats can be read from an environment variable ``'https://api.mapbox.com/v4/mapbox.satellite/{z}/{x}/{y}.jpg?access_token={ACCESS_TOKEN}'`` or added directly the imagery string.
+ - A GeoTIFF file location. Works with local files: ``'http://oin-hotosm.s3.amazonaws.com/593ede5ee407d70011386139/0/3041615b-2bdb-40c5-b834-36f580baca29.tif'``
+ - Remote files like a `WMS endpoint `_ ``GetMap`` request. Fill out all necessary parameters except ``bbox`` which should be set as ``{bbox}``. Ex:
+
+ .. code-block:: none
- A template string for a tiled imagery service. Note that you will generally need an API key to obtain images and there may be associated costs. The above example requires a `Mapbox access token `_. Also see `OpenAerialMap `_ for open imagery.
- The access token for TMS image formats can be read from an environment variable https://api.mapbox.com/v4/mapbox.satellite/{z}/{x}/{y}.jpg?access_token={ACCESS_TOKEN}" or added directly the imagery string.
+ 'https://basemap.nationalmap.gov/arcgis/services/USGSImageryOnly/MapServer/WMSServer?SERVICE=WMS&REQUEST=GetMap&VERSION=1.1.1&LAYERS=0&STYLES=&FORMAT=image%2Fjpeg&TRANSPARENT=false&HEIGHT=256&WIDTH=256&SRS=EPSG%3A3857&BBOX={bbox}'
- A GeoTIFF file location. Works with local files: ``'http://oin-hotosm.s3.amazonaws.com/593ede5ee407d70011386139/0/3041615b-2bdb-40c5-b834-36f580baca29.tif'``
-
- Remote files like a `WMS endpoint `_ ``GetMap`` request. Fill out all necessary parameters except ``bbox`` which should be set as ``{bbox}``. Ex: ``'https://basemap.nationalmap.gov/arcgis/services/USGSImageryOnly/MapServer/WMSServer?SERVICE=WMS&REQUEST=GetMap&VERSION=1.1.1&LAYERS=0&STYLES=&FORMAT=image%2Fjpeg&TRANSPARENT=false&HEIGHT=256&WIDTH=256&SRS=EPSG%3A3857&BBOX={bbox}'``
**http_auth**: list
Optional parameter to specify a username and password for restricted WMS services. For example, ``['my_username', 'my_password']``.
@@ -45,7 +47,7 @@ Here is the full list of configuration parameters you can specify in a ``config.
One of ``'classification'``, ``'object-detection'``, or ``'segmentation'``. This defines the output format for the final label numpy arrays (``y_train`` and ``y_test``).
``'classification'``
- Output is an array of ``len(classes) + 1``. Each array value will be either `1` or `0` based on whether it matches the class at the same index. The additional array element belongs to the background class, which will always be the first element.
+ Output is an array of ``len(classes) + 1``. Each array value will be either 1 or 0 based on whether it matches the class at the same index. The additional array element belongs to the background class, which will always be the first element.
``'object-detection'``
Output is an array of bounding boxes of the form ``[xmin, ymin, width, height, class_index]``. In this case, the values are pixel values measured from the upper left-hand corner (not latitude and longitude values). Each feature is tested against each class, so if a feature matches two or more classes, it will have the corresponding number of bounding boxes created.
@@ -57,18 +59,25 @@ Here is the full list of configuration parameters you can specify in a ``config.
Random generator seed. Optional, use to make results reproducible.
**split_vals**: list
- Default: `[0.8, 0.2]`
- Percentage of data to put in each category listed in split_names. Must be a list of floats that sum to one and match the length of `split-names`. For train, validate, and test data, a list like `[0.7, 0.2, 0.1]` is suggested.
+ Default: ``[0.8, 0.2]``
+
+ Percentage of data to put in each category listed in split_names. Must be a list of floats that sum to one and match the length of ``split-names``. For train, validate, and test data, a list like ``[0.7, 0.2, 0.1]`` is suggested.
**split_names**: list
- Default: `['train', 'test']`
- List of names for each subset of the data. Length of list must match length of `split_vals`.
+ Default: ``['train', 'test']``
+
+ List of names for each subset of the data. Length of list must match length of ``split_vals``.
**imagery_offset**: list of ints
An optional list of integers representing the number of pixels to offset imagery. For example ``[15, -5]`` will move the images 15 pixels right and 5 pixels up relative to the requested tile bounds.
**tms_image_format**: string
- An option string that has the downloaded imagery's format such as `.jpg` or `.png` when it isn't provided by the endpoint
+ An option string that has the downloaded imagery's format such as ``.jpg`` or ``.png`` when it isn't provided by the endpoint
**over_zoom**: int
- An integer greater than 0. If set for XYZ tiles, it will fetch tiles from `zoom` + `over_zoom`, to create higher resolution tiles which fill out the bounds of the original zoom level.
+ An integer greater than 0. If set for XYZ tiles, it will fetch tiles from ``zoom`` + ``over_zoom``, to create higher resolution tiles which fill out the bounds of the original zoom level.
+
+**band_indices**: list
+ Default: ``[1, 2, 3]``
+
+ A list of band indices to pull from a TIF. Using the `SpaceNet Roads Challenge Data `_ as an example, you can use ``[5, 3, 2, 7]`` to extract the Red, Green, Blue, and NIR bands respectively.
diff --git a/label_maker/package.py b/label_maker/package.py
index a3e7961f..cfd1c7a3 100644
--- a/label_maker/package.py
+++ b/label_maker/package.py
@@ -4,6 +4,7 @@
from os import path as op
from urllib.parse import urlparse
import numpy as np
+import rasterio
from PIL import Image
from label_maker.utils import is_tif, get_image_format
@@ -73,15 +74,17 @@ def package_directory(dest_folder, classes, imagery, ml_type, seed=False,
# open the images and load those plus the labels into the final arrays
if is_tif(imagery): # if a TIF is provided, use jpg as tile format
- image_format = '.jpg'
+ img_dtype = rasterio.open(imagery).profile['dtype']
+ image_format = '.tif'
else:
+ img_dtype = np.uint8
image_format = get_image_format(imagery, kwargs)
for tile in tiles:
image_file = op.join(dest_folder, 'tiles', '{}{}'.format(tile, image_format))
try:
- img = Image.open(image_file)
+ img = rasterio.open(image_file)
except FileNotFoundError:
# we often don't download images for each label (e.g. background tiles)
continue
@@ -89,9 +92,11 @@ def package_directory(dest_folder, classes, imagery, ml_type, seed=False,
print('Couldn\'t open {}, skipping'.format(image_file))
continue
- np_image = np.array(img)
+ i = np.array(img.read())
+ np_image = np.moveaxis(i, 0, 2)
img.close()
+
x_vals.append(np_image)
if ml_type == 'classification':
y_vals.append(labels[tile])
@@ -103,7 +108,9 @@ def package_directory(dest_folder, classes, imagery, ml_type, seed=False,
y_vals.append(labels[tile][..., np.newaxis]) # Add grayscale channel
# Convert lists to numpy arrays
- x_vals = np.array(x_vals, dtype=np.uint8)
+
+ #TO-DO flexible x_val dtype
+ x_vals = np.array(x_vals, dtype=img_dtype)
y_vals = np.array(y_vals, dtype=np.uint8)
# Get number of data samples per split from the float proportions
@@ -128,3 +135,4 @@ def package_directory(dest_folder, classes, imagery, ml_type, seed=False,
np.savez(op.join(dest_folder, 'data.npz'), **save_dict)
print('Saving packaged file to {}'.format(op.join(dest_folder, 'data.npz')))
+ print('Image dtype written in npz matches input image dtype: {}'.format(img_dtype))
diff --git a/label_maker/utils.py b/label_maker/utils.py
index c4ab809b..9a6e5925 100644
--- a/label_maker/utils.py
+++ b/label_maker/utils.py
@@ -106,6 +106,7 @@ def get_tile_tif(tile, imagery, folder, kwargs):
bound = bounds(*[int(t) for t in tile.split('-')])
imagery_offset = kwargs.get('imagery_offset') or [0, 0]
with rasterio.open(imagery) as src:
+ profile = src.profile
x_res, y_res = src.transform[0], src.transform[4]
# offset our imagery in the "destination pixel" space
@@ -137,15 +138,19 @@ def get_tile_tif(tile, imagery, folder, kwargs):
window = ((top, bottom), (left, right))
# read the first three bands (assumed RGB) of the TIF into an array
- data = np.empty(shape=(3, 256, 256)).astype(src.profile['dtype'])
- for k in (1, 2, 3):
- src.read(k, window=window, out=data[k - 1], boundless=True)
+ band_indices = kwargs.get('band_indices', (1, 2, 3))
+ band_count = len(band_indices)
- # save
- tile_img = op.join(folder, '{}{}'.format(tile, '.jpg'))
- img = Image.fromarray(np.moveaxis(data, 0, -1), mode='RGB')
- img.save(tile_img)
+ arr_shape = (band_count, 256, 256)
+ data = np.empty(shape=(arr_shape)).astype(profile['dtype'])
+ for i, k in enumerate(band_indices):
+ src.read(k, window=window, out=data[i], boundless=True)
+ # save
+ tile_img = op.join(folder, '{}{}'.format(tile, '.tif'))
+ with rasterio.open(tile_img, 'w', driver='GTiff', height=256,
+ width=256, count=band_count, dtype=profile['dtype']) as w:
+ w.write(data)
return tile_img
def get_tile_wms(tile, imagery, folder, kwargs):
diff --git a/label_maker/validate.py b/label_maker/validate.py
index fe086ed0..b78aa62b 100644
--- a/label_maker/validate.py
+++ b/label_maker/validate.py
@@ -35,5 +35,6 @@
'split_vals': {'type': 'list', 'schema': {'type': 'float'}},
'split_names': {'type': 'list', 'schema': {'type': 'string'}},
'tms_image_format': {'type': 'string'},
- 'over_zoom': {'type': 'integer', 'min': 1}
+ 'over_zoom': {'type': 'integer', 'min': 1},
+ 'band_indices': {'type': 'list'}
}
diff --git a/test/fixtures/1087767-1046604-21.jpg b/test/fixtures/1087767-1046604-21.jpg
deleted file mode 100644
index 6a6d3a11..00000000
Binary files a/test/fixtures/1087767-1046604-21.jpg and /dev/null differ
diff --git a/test/fixtures/1087767-1046604-21.tif b/test/fixtures/1087767-1046604-21.tif
new file mode 100644
index 00000000..13f58dd9
Binary files /dev/null and b/test/fixtures/1087767-1046604-21.tif differ
diff --git a/test/fixtures/1087767-1046604-21_offset.jpg b/test/fixtures/1087767-1046604-21_offset.jpg
deleted file mode 100644
index 71be8e22..00000000
Binary files a/test/fixtures/1087767-1046604-21_offset.jpg and /dev/null differ
diff --git a/test/fixtures/1087767-1046604-21_offset.tif b/test/fixtures/1087767-1046604-21_offset.tif
new file mode 100644
index 00000000..18e90e28
Binary files /dev/null and b/test/fixtures/1087767-1046604-21_offset.tif differ
diff --git a/test/fixtures/integration/config.integration.bands.json b/test/fixtures/integration/config.integration.bands.json
new file mode 100644
index 00000000..88d08535
--- /dev/null
+++ b/test/fixtures/integration/config.integration.bands.json
@@ -0,0 +1,12 @@
+{
+ "zoom": 18,
+ "classes": [
+ {"name": "paved", "filter": ["in", "paved", "1"]},
+ {"name": "unpaved", "filter": ["in", "paved", "2"]}
+ ],
+ "geojson": "integration-ms/ms-roads.geojson",
+ "imagery": "integration-ms/ms-img.tif",
+ "background_ratio": 0,
+ "ml_type": "classification",
+ "band_indices": [5, 7]
+ }
\ No newline at end of file
diff --git a/test/fixtures/integration/labels-ms.npz b/test/fixtures/integration/labels-ms.npz
new file mode 100644
index 00000000..7bb85aff
Binary files /dev/null and b/test/fixtures/integration/labels-ms.npz differ
diff --git a/test/fixtures/integration/ms_img.tif b/test/fixtures/integration/ms_img.tif
new file mode 100644
index 00000000..b117f744
Binary files /dev/null and b/test/fixtures/integration/ms_img.tif differ
diff --git a/test/fixtures/integration/roads_ms.geojson b/test/fixtures/integration/roads_ms.geojson
new file mode 100644
index 00000000..f4274ba9
--- /dev/null
+++ b/test/fixtures/integration/roads_ms.geojson
@@ -0,0 +1,44 @@
+{
+"type": "FeatureCollection",
+"crs": { "type": "name", "properties": { "name": "urn:ogc:def:crs:OGC:1.3:CRS84" } },
+"features": [
+{ "type": "Feature", "properties": { "bridge_type": "2", "heading": "0", "lane_numbe": "2", "lane_number": "2", "one_way_type": "2", "paved": "1", "road_id": 6624, "road_type": "3", "origarea": 0, "origlen": 0.00049762270337687863, "partialDec": 1, "truncated": 0 }, "geometry": { "type": "LineString", "coordinates": [ [ 32.501765305, 15.531368736 ], [ 32.501764601907517, 15.531361199939429 ] ] } },
+{ "type": "Feature", "properties": { "bridge_type": "2", "heading": "0", "lane_numbe": "2", "lane_number": "2", "one_way_type": "2", "paved": "2", "road_id": 2237, "road_type": "5", "origarea": 0, "origlen": 0.001120296618893227, "partialDec": 1, "truncated": 0 }, "geometry": { "type": "LineString", "coordinates": [ [ 32.5017159449771, 15.531361199939429 ], [ 32.501765305, 15.531368736 ] ] } },
+{ "type": "Feature", "properties": { "bridge_type": "2", "heading": "0", "lane_numbe": "1", "lane_number": "1", "one_way_type": "2", "paved": "2", "road_id": 12898, "road_type": "5", "origarea": 0, "origlen": 0.0004918126116161773, "partialDec": 1, "truncated": 0 }, "geometry": { "type": "LineString", "coordinates": [ [ 32.500938033, 15.531588986 ], [ 32.500782356976792, 15.531361199939429 ] ] } },
+{ "type": "Feature", "properties": { "bridge_type": "2", "heading": "0", "lane_numbe": "2", "lane_number": "2", "one_way_type": "2", "paved": "2", "road_id": 4294, "road_type": "5", "origarea": 0, "origlen": 0.001699185124345286, "partialDec": 1, "truncated": 0 }, "geometry": { "type": "LineString", "coordinates": [ [ 32.501765305, 15.531368736 ], [ 32.503388400014977, 15.531616522839315 ] ] } },
+{ "type": "Feature", "properties": { "bridge_type": "2", "heading": "0", "lane_numbe": "2", "lane_number": "2", "one_way_type": "2", "paved": "1", "road_id": 6936, "road_type": "3", "origarea": 0, "origlen": 0.001434105069154448, "partialDec": 1, "truncated": 0 }, "geometry": { "type": "LineString", "coordinates": [ [ 32.499878400014353, 15.531448954617677 ], [ 32.500156553, 15.531456647 ], [ 32.50037927, 15.531462358 ], [ 32.500542026, 15.531473779 ], [ 32.500711919, 15.531498049 ], [ 32.500856114, 15.531543736 ], [ 32.500938033, 15.531588986 ] ] } },
+{ "type": "Feature", "properties": { "bridge_type": "2", "heading": "0", "lane_numbe": "2", "lane_number": "2", "one_way_type": "2", "paved": "1", "road_id": 8674, "road_type": "3", "origarea": 0, "origlen": 0.0005144443811971296, "partialDec": 1, "truncated": 0 }, "geometry": { "type": "LineString", "coordinates": [ [ 32.501813093, 15.531880956 ], [ 32.501765305, 15.531368736 ] ] } },
+{ "type": "Feature", "properties": { "bridge_type": "2", "heading": "0", "lane_numbe": "2", "lane_number": "2", "one_way_type": "2", "paved": "2", "road_id": 949, "road_type": "5", "origarea": 0, "origlen": 0.0010891510053242974, "partialDec": 1, "truncated": 0 }, "geometry": { "type": "LineString", "coordinates": [ [ 32.501813093, 15.531880956 ], [ 32.502894542, 15.532010254 ] ] } },
+{ "type": "Feature", "properties": { "bridge_type": "2", "heading": "0", "lane_numbe": "2", "lane_number": "2", "one_way_type": "2", "paved": "2", "road_id": 6702, "road_type": "5", "origarea": 0, "origlen": 0.00045917190321641114, "partialDec": 1, "truncated": 0 }, "geometry": { "type": "LineString", "coordinates": [ [ 32.502894542, 15.532010254 ], [ 32.503350467, 15.532064763 ] ] } },
+{ "type": "Feature", "properties": { "bridge_type": "2", "heading": "0", "lane_numbe": "2", "lane_number": "2", "one_way_type": "2", "paved": "1", "road_id": 10730, "road_type": "3", "origarea": 0, "origlen": 0.00047040664432221313, "partialDec": 1, "truncated": 0 }, "geometry": { "type": "LineString", "coordinates": [ [ 32.501861713, 15.532348813 ], [ 32.501859326, 15.532329862 ], [ 32.501836155, 15.532128135 ], [ 32.501813093, 15.531880956 ] ] } },
+{ "type": "Feature", "properties": { "bridge_type": "2", "heading": "0", "lane_numbe": "2", "lane_number": "2", "one_way_type": "2", "paved": "1", "road_id": 11492, "road_type": "3", "origarea": 0, "origlen": 0.0014646567233461897, "partialDec": 1, "truncated": 0 }, "geometry": { "type": "LineString", "coordinates": [ [ 32.500938033, 15.531588986 ], [ 32.501190191, 15.53175646 ], [ 32.501374361, 15.531912076 ], [ 32.501564393, 15.532135358 ], [ 32.501668382, 15.532282081 ], [ 32.501807982, 15.532475812 ], [ 32.501929543, 15.532645838 ] ] } },
+{ "type": "Feature", "properties": { "bridge_type": "2", "heading": "0", "lane_numbe": "2", "lane_number": "2", "one_way_type": "2", "paved": "2", "road_id": 10807, "road_type": "5", "origarea": 0, "origlen": 0.00047089489476918357, "partialDec": 1, "truncated": 0 }, "geometry": { "type": "LineString", "coordinates": [ [ 32.502894542, 15.532010254 ], [ 32.502870009, 15.532232956 ], [ 32.502796246, 15.532468523 ] ] } },
+{ "type": "Feature", "properties": { "bridge_type": "2", "heading": "0", "lane_numbe": "2", "lane_number": "2", "one_way_type": "2", "paved": "2", "road_id": 124, "road_type": "5", "origarea": 0, "origlen": 0.00094216899344013577, "partialDec": 1, "truncated": 0 }, "geometry": { "type": "LineString", "coordinates": [ [ 32.502796246, 15.532468523 ], [ 32.501861713, 15.532348813 ] ] } },
+{ "type": "Feature", "properties": { "bridge_type": "2", "heading": "0", "lane_numbe": "2", "lane_number": "2", "one_way_type": "2", "paved": "1", "road_id": 3473, "road_type": "3", "origarea": 0, "origlen": 0.00030633265431616698, "partialDec": 1, "truncated": 0 }, "geometry": { "type": "LineString", "coordinates": [ [ 32.501929543, 15.532645838 ], [ 32.501881795, 15.532508213 ], [ 32.501861713, 15.532348813 ] ] } },
+{ "type": "Feature", "properties": { "bridge_type": "2", "heading": "0", "lane_numbe": "2", "lane_number": "2", "one_way_type": "2", "paved": "2", "road_id": 7621, "road_type": "5", "origarea": 0, "origlen": 0.00094516947840408083, "partialDec": 1, "truncated": 0 }, "geometry": { "type": "LineString", "coordinates": [ [ 32.503350467, 15.532064763 ], [ 32.503201943, 15.53299819 ] ] } },
+{ "type": "Feature", "properties": { "bridge_type": "2", "heading": "0", "lane_numbe": "2", "lane_number": "2", "one_way_type": "2", "paved": "2", "road_id": 1495, "road_type": "5", "origarea": 0, "origlen": 0.00046832820865854883, "partialDec": 1, "truncated": 0 }, "geometry": { "type": "LineString", "coordinates": [ [ 32.502796246, 15.532468523 ], [ 32.502725671, 15.532931503 ] ] } },
+{ "type": "Feature", "properties": { "bridge_type": "2", "heading": "0", "lane_numbe": "2", "lane_number": "2", "one_way_type": "2", "paved": "1", "road_id": 5530, "road_type": "3", "origarea": 0, "origlen": 0.00020808195160695143, "partialDec": 1, "truncated": 0 }, "geometry": { "type": "LineString", "coordinates": [ [ 32.50201942, 15.532832614 ], [ 32.501992737, 15.532794697 ], [ 32.501929543, 15.532645838 ] ] } },
+{ "type": "Feature", "properties": { "bridge_type": "2", "heading": "0", "lane_numbe": "2", "lane_number": "2", "one_way_type": "2", "paved": "2", "road_id": 2182, "road_type": "5", "origarea": 0, "origlen": 0.00071314059575769385, "partialDec": 1, "truncated": 0 }, "geometry": { "type": "LineString", "coordinates": [ [ 32.502725671, 15.532931503 ], [ 32.50201942, 15.532832614 ] ] } },
+{ "type": "Feature", "properties": { "bridge_type": "2", "heading": "0", "lane_numbe": "2", "lane_number": "2", "one_way_type": "2", "paved": "1", "road_id": 7589, "road_type": "3", "origarea": 0, "origlen": 0.00022146817931185573, "partialDec": 1, "truncated": 0 }, "geometry": { "type": "LineString", "coordinates": [ [ 32.5021012, 15.533037042 ], [ 32.502092445, 15.532988496 ], [ 32.50201942, 15.532832614 ] ] } },
+{ "type": "Feature", "properties": { "bridge_type": "2", "heading": "0", "lane_numbe": "2", "lane_number": "2", "one_way_type": "2", "paved": "2", "road_id": 12865, "road_type": "5", "origarea": 0, "origlen": 0.00048091805326194297, "partialDec": 1, "truncated": 0 }, "geometry": { "type": "LineString", "coordinates": [ [ 32.503201943, 15.53299819 ], [ 32.502725671, 15.532931503 ] ] } },
+{ "type": "Feature", "properties": { "bridge_type": "2", "heading": "0", "lane_numbe": "2", "lane_number": "2", "one_way_type": "2", "paved": "2", "road_id": 9675, "road_type": "5", "origarea": 0, "origlen": 0.00024399157913533898, "partialDec": 1, "truncated": 0 }, "geometry": { "type": "LineString", "coordinates": [ [ 32.503201943, 15.53299819 ], [ 32.503165835, 15.533239495 ] ] } },
+{ "type": "Feature", "properties": { "bridge_type": "2", "heading": "0", "lane_numbe": "2", "lane_number": "2", "one_way_type": "2", "paved": "1", "road_id": 10642, "road_type": "3", "origarea": 0, "origlen": 0.00084143103905213752, "partialDec": 1, "truncated": 0 }, "geometry": { "type": "LineString", "coordinates": [ [ 32.502721812, 15.533531367 ], [ 32.502657691, 15.533525304 ], [ 32.50256641, 15.533511962 ], [ 32.502475128, 15.533481067 ], [ 32.502395782, 15.533434022 ], [ 32.502328374, 15.533369422 ], [ 32.502260264, 15.533278843 ], [ 32.50219426, 15.533182646 ], [ 32.5021012, 15.533037042 ] ] } },
+{ "type": "Feature", "properties": { "bridge_type": "2", "heading": "0", "lane_numbe": "2", "lane_number": "2", "one_way_type": "2", "paved": "1", "road_id": 9643, "road_type": "3", "origarea": 0, "origlen": 0.00063268552763806661, "partialDec": 1, "truncated": 0 }, "geometry": { "type": "LineString", "coordinates": [ [ 32.502104382, 15.533666174 ], [ 32.502120855, 15.53350992 ], [ 32.502133171, 15.53329745 ], [ 32.502123341, 15.533159825 ], [ 32.5021012, 15.533037042 ] ] } },
+{ "type": "Feature", "properties": { "bridge_type": "2", "heading": "0", "lane_numbe": "2", "lane_number": "2", "one_way_type": "2", "paved": "2", "road_id": 11727, "road_type": "5", "origarea": 0, "origlen": 0.00029791694621247633, "partialDec": 1, "truncated": 0 }, "geometry": { "type": "LineString", "coordinates": [ [ 32.503165835, 15.533239495 ], [ 32.503156266, 15.533303444 ], [ 32.503192126, 15.533533927 ] ] } },
+{ "type": "Feature", "properties": { "bridge_type": "2", "heading": "0", "lane_numbe": "2", "lane_number": "2", "one_way_type": "2", "paved": "2", "road_id": 5562, "road_type": "5", "origarea": 0, "origlen": 0.00039697444257192802, "partialDec": 1, "truncated": 0 }, "geometry": { "type": "LineString", "coordinates": [ [ 32.503165835, 15.533239495 ], [ 32.50293262, 15.533518543 ], [ 32.502909791, 15.53354279 ] ] } },
+{ "type": "Feature", "properties": { "bridge_type": "2", "heading": "0", "lane_numbe": "2", "lane_number": "2", "one_way_type": "2", "paved": "1", "road_id": 1329, "road_type": "3", "origarea": 0, "origlen": 0.00011086048560384061, "partialDec": 1, "truncated": 0 }, "geometry": { "type": "LineString", "coordinates": [ [ 32.50283218, 15.533541805 ], [ 32.502721812, 15.533531367 ] ] } },
+{ "type": "Feature", "properties": { "bridge_type": "2", "heading": "0", "lane_numbe": "2", "lane_number": "2", "one_way_type": "2", "paved": "1", "road_id": 5446, "road_type": "3", "origarea": 0, "origlen": 0.00028266480525534593, "partialDec": 1, "truncated": 0 }, "geometry": { "type": "LineString", "coordinates": [ [ 32.503192126, 15.533533927 ], [ 32.503025628, 15.533544262 ], [ 32.502909791, 15.53354279 ] ] } },
+{ "type": "Feature", "properties": { "bridge_type": "2", "heading": "0", "lane_numbe": "2", "lane_number": "2", "one_way_type": "2", "paved": "1", "road_id": 3389, "road_type": "3", "origarea": 0, "origlen": 7.7617250313834397e-05, "partialDec": 1, "truncated": 0 }, "geometry": { "type": "LineString", "coordinates": [ [ 32.502909791, 15.53354279 ], [ 32.50283218, 15.533541805 ] ] } },
+{ "type": "Feature", "properties": { "bridge_type": "2", "heading": "0", "lane_numbe": "1", "lane_number": "1", "one_way_type": "2", "paved": "1", "road_id": 4474, "road_type": "3", "origarea": 0, "origlen": 0.00073877728421493406, "partialDec": 1, "truncated": 0 }, "geometry": { "type": "LineString", "coordinates": [ [ 32.502097192, 15.533790869 ], [ 32.502119831, 15.533738762 ], [ 32.502145109, 15.533684696 ], [ 32.502188643, 15.533631331 ], [ 32.502241306, 15.533597627 ], [ 32.502305202, 15.533567433 ], [ 32.502374717, 15.533547773 ], [ 32.502458977, 15.533535836 ], [ 32.502540429, 15.533534432 ], [ 32.502721812, 15.533531367 ] ] } },
+{ "type": "Feature", "properties": { "bridge_type": "2", "heading": "0", "lane_numbe": "2", "lane_number": "2", "one_way_type": "2", "paved": "2", "road_id": 2651, "road_type": "5", "origarea": 0, "origlen": 0.00036850600013569587, "partialDec": 1, "truncated": 0 }, "geometry": { "type": "LineString", "coordinates": [ [ 32.502876674, 15.533907615 ], [ 32.50283218, 15.533541805 ] ] } },
+{ "type": "Feature", "properties": { "bridge_type": "2", "heading": "0", "lane_numbe": "2", "lane_number": "2", "one_way_type": "2", "paved": "2", "road_id": 1531, "road_type": "5", "origarea": 0, "origlen": 0.00038732576773723554, "partialDec": 1, "truncated": 0 }, "geometry": { "type": "LineString", "coordinates": [ [ 32.502923441, 15.534292107 ], [ 32.502876674, 15.533907615 ] ] } },
+{ "type": "Feature", "properties": { "bridge_type": "2", "heading": "0", "lane_numbe": "2", "lane_number": "2", "one_way_type": "2", "paved": "2", "road_id": 592, "road_type": "5", "origarea": 0, "origlen": 0.00046976789311546256, "partialDec": 1, "truncated": 0 }, "geometry": { "type": "LineString", "coordinates": [ [ 32.502980162, 15.534758438 ], [ 32.502923441, 15.534292107 ] ] } },
+{ "type": "Feature", "properties": { "bridge_type": "2", "heading": "0", "lane_numbe": "2", "lane_number": "2", "one_way_type": "2", "paved": "2", "road_id": 1714, "road_type": "5", "origarea": 0, "origlen": 0.00050804781491254974, "partialDec": 1, "truncated": 0 }, "geometry": { "type": "LineString", "coordinates": [ [ 32.503388400014977, 15.534713779856281 ], [ 32.502980162, 15.534758438 ] ] } },
+{ "type": "Feature", "properties": { "bridge_type": "2", "heading": "0", "lane_numbe": "2", "lane_number": "2", "one_way_type": "2", "paved": "2", "road_id": 3590, "road_type": "5", "origarea": 0, "origlen": 0.0009252945373814397, "partialDec": 1, "truncated": 0 }, "geometry": { "type": "LineString", "coordinates": [ [ 32.502993877456262, 15.534871199940053 ], [ 32.502980162, 15.534758438 ] ] } },
+{ "type": "Feature", "properties": { "bridge_type": "2", "heading": "0", "lane_numbe": "2", "lane_number": "2", "one_way_type": "2", "paved": "1", "road_id": 12699, "road_type": "3", "origarea": 0, "origlen": 0.0004761020853063759, "partialDec": 1, "truncated": 0 }, "geometry": { "type": "LineString", "coordinates": [ [ 32.503388400014977, 15.533515414115682 ], [ 32.503229256, 15.533531623 ], [ 32.503192126, 15.533533927 ] ] } },
+{ "type": "Feature", "properties": { "bridge_type": "2", "heading": "0", "lane_numbe": "2", "lane_number": "2", "one_way_type": "2", "paved": "2", "road_id": 510, "road_type": "5", "origarea": 0, "origlen": 0.0012043419080481038, "partialDec": 1, "truncated": 0 }, "geometry": { "type": "LineString", "coordinates": [ [ 32.503388400014977, 15.533846905580408 ], [ 32.502876674, 15.533907615 ] ] } },
+{ "type": "Feature", "properties": { "bridge_type": "2", "heading": "0", "lane_numbe": "2", "lane_number": "2", "one_way_type": "2", "paved": "2", "road_id": 3539, "road_type": "5", "origarea": 0, "origlen": 0.00091995845404107168, "partialDec": 1, "truncated": 0 }, "geometry": { "type": "LineString", "coordinates": [ [ 32.502923441, 15.534292107 ], [ 32.503388400014977, 15.534224489468206 ] ] } },
+{ "type": "Feature", "properties": { "bridge_type": "2", "heading": "0", "lane_numbe": "2", "lane_number": "2", "one_way_type": "2", "paved": "2", "road_id": 10996, "road_type": "5", "origarea": 0, "origlen": 0.0004492455753427264, "partialDec": 1, "truncated": 0 }, "geometry": { "type": "LineString", "coordinates": [ [ 32.503350467, 15.532064763 ], [ 32.503388400014977, 15.53188853941435 ] ] } },
+{ "type": "Feature", "properties": { "bridge_type": "2", "heading": "0", "lane_numbe": "2", "lane_number": "2", "one_way_type": "2", "paved": "2", "road_id": 8753, "road_type": "5", "origarea": 0, "origlen": 0.00027350721239564728, "partialDec": 1, "truncated": 0 }, "geometry": { "type": "LineString", "coordinates": [ [ 32.503350467, 15.532064763 ], [ 32.503388400014977, 15.532069298373532 ] ] } }
+]
+}
diff --git a/test/integration/test_ms_bands_package.py b/test/integration/test_ms_bands_package.py
new file mode 100644
index 00000000..80f1e7ab
--- /dev/null
+++ b/test/integration/test_ms_bands_package.py
@@ -0,0 +1,46 @@
+"""Test that the following CLI command returns the expected outputs
+label-maker package -d integration-tif/sao_tome -c test/fixtures/integration/config.intergration.geotiff_package.json"""
+import unittest
+from os import makedirs
+from shutil import copyfile, copytree, rmtree
+import subprocess
+import os
+
+import numpy as np
+
+class TestClassificationPackage(unittest.TestCase):
+ """Tests for local GeoTIFF package creation"""
+ @classmethod
+ def setUpClass(cls):
+ makedirs('integration-ms')
+ copyfile('test/fixtures/integration/ms_img.tif', 'integration-ms/ms-img.tif')
+ copyfile('test/fixtures/integration/roads_ms.geojson', 'integration-ms/ms-roads.geojson')
+ copyfile('test/fixtures/integration/labels-ms.npz', 'integration-ms/labels.npz')
+
+ @classmethod
+ def tearDownClass(cls):
+ rmtree('integration-ms')
+
+ def test_cli(self):
+ """Verify data.npz produced by CLI"""
+ cmd = 'label-maker images --dest integration-ms --config test/fixtures/integration/config.integration.bands.json'
+ cmd = cmd.split(' ')
+ subprocess.run(cmd, universal_newlines=True)
+
+ cmd = 'label-maker package --dest integration-ms --config test/fixtures/integration/config.integration.bands.json'
+ cmd = cmd.split(' ')
+ subprocess.run(cmd, universal_newlines=True)
+
+ print(os.listdir('integration-ms/'))
+ data = np.load('integration-ms/data.npz')
+
+ # validate our label data with exact matches in shape
+ self.assertEqual(data['x_train'].shape, (8, 256, 256, 2))
+ self.assertEqual(data['x_test'].shape, (3, 256, 256, 2))
+
+ # validate our label data with exact matches in shape
+ self.assertEqual(data['y_train'].shape, (8, 3))
+ self.assertEqual(data['y_test'].shape, (3, 3))
+
+ #validate img dtype
+ self.assertEqual(np.uint16, data['x_train'].dtype)
\ No newline at end of file
diff --git a/test/tiles/1087767-1046604-21.jpg b/test/tiles/1087767-1046604-21.jpg
deleted file mode 100644
index 6a6d3a11..00000000
Binary files a/test/tiles/1087767-1046604-21.jpg and /dev/null differ
diff --git a/test/tiles/146-195-9.jpeg b/test/tiles/146-195-9.jpeg
deleted file mode 100644
index 3130abb1..00000000
Binary files a/test/tiles/146-195-9.jpeg and /dev/null differ
diff --git a/test/unit/test_utils.py b/test/unit/test_utils.py
index 0eee186c..4fd17f2a 100644
--- a/test/unit/test_utils.py
+++ b/test/unit/test_utils.py
@@ -82,8 +82,8 @@ def test_get_tile_tif(self):
makedirs(tiles_dir)
get_tile_tif(tile, 'test/fixtures/drone.tif', tiles_dir, {})
- test_tile = Image.open('test/tiles/{}.jpg'.format(tile))
- fixture_tile = Image.open('test/fixtures/{}.jpg'.format(tile))
+ test_tile = Image.open('test/tiles/{}.tif'.format(tile))
+ fixture_tile = Image.open('test/fixtures/{}.tif'.format(tile))
self.assertEqual(test_tile, fixture_tile)
def test_get_tile_tif_offset(self):
@@ -92,13 +92,14 @@ def test_get_tile_tif_offset(self):
# create tiles directory
dest_folder = 'test'
tiles_dir = op.join(dest_folder, 'tiles')
+ print(tiles_dir)
if not op.isdir(tiles_dir):
makedirs(tiles_dir)
get_tile_tif(tile, 'test/fixtures/drone.tif', tiles_dir,
{'imagery_offset': [128, 64]})
- test_tile = Image.open('test/tiles/{}.jpg'.format(tile))
- fixture_tile = Image.open('test/fixtures/{}_offset.jpg'.format(tile))
+ test_tile = Image.open('test/tiles/{}.tif'.format(tile))
+ fixture_tile = Image.open('test/fixtures/{}_offset.tif'.format(tile))
self.assertEqual(test_tile, fixture_tile)
def test_get_tile_vrt(self):
@@ -111,8 +112,8 @@ def test_get_tile_vrt(self):
makedirs(tiles_dir)
get_tile_tif(tile, 'test/fixtures/drone.vrt', tiles_dir, {})
- test_tile = Image.open('test/tiles/{}.jpg'.format(tile))
- fixture_tile = Image.open('test/fixtures/{}.jpg'.format(tile))
+ test_tile = Image.open('test/tiles/{}.tif'.format(tile))
+ fixture_tile = Image.open('test/fixtures/{}.tif'.format(tile))
self.assertEqual(test_tile, fixture_tile)
def test_get_tile_wms(self):