Skip to content

Commit

Permalink
CSW-T workflow (#197)
Browse files Browse the repository at this point in the history
* Changes for CNP CSW-T support

Adjusted indentation

Added elasticsearch check to Paw

[WIP] do not create services with supported extensions

More robust creation of layers from arcgis folders and services

Do not create layers for created services

Provisional documentation for cswt using paw

Verify that variable exists before accessing it, and give default value in case it doesn't

Move to cnp pycsw

Update migrations after uuid change

Fix getattr for REGISTRY_HARVEST_SERVICES

Updated cswt_insert.xml

Added uuid field

Only index Services if is_monitored is True

Allow Service update. Use uuid in pycsw mappings

Clear old pycsw

Replace id_string with uuid

Update tests

flake8

Updated sample cswt_insert with uuid

* flake8
  • Loading branch information
ingenieroariel committed Sep 6, 2016
1 parent 13c7652 commit 0e1608e
Show file tree
Hide file tree
Showing 16 changed files with 262 additions and 86 deletions.
6 changes: 3 additions & 3 deletions Dockerfile
Expand Up @@ -39,13 +39,13 @@ RUN pip install --no-cache-dir -r requirements.txt
ONBUILD COPY requirements.txt /usr/src/app/
ONBUILD RUN pip install --no-cache-dir -r requirements.txt

# Force a pycsw rebuild. Remove when pycsw 2.0.1 is released.
RUN pip uninstall -y pycsw
RUN pip install --upgrade git+git://github.com/geopython/pycsw.git@master#egg=pycsw
RUN rm -rf /usr/local/lib/python2.7/site-packages/pycsw/
RUN pip install --upgrade git+git://github.com/terranodo/pycsw.git@2aa428835ba27e8189ca9a7b8c2db940a687f135#egg=pycsw

ONBUILD COPY . /usr/src/app/
ONBUILD RUN pip install --no-deps --no-cache-dir -e /usr/src/app/

EXPOSE 8000
CMD ["python", "manage.py", "runserver", "0.0.0.0:8000"]


11 changes: 11 additions & 0 deletions _docs/cswt.md
@@ -0,0 +1,11 @@
## Layers creation applying CSW-T

HHypermap Registry comes with the ability to create layers giving a CSW transactions. Provisionally we will use the software Paw https://paw.cloud/.

### Using paw in development.

We will use the example files found in the data directory. First, we need to open the project **cswt_paw.paw**. Also, we will use the example catalog *cnp*. Make sure to create the catalog in the database before executing the csw transaction.

http://recordit.co/wIr24LD2EU

**Note.** The first time paw asks for credentials. It is possible to solve it by reset and clear of passphrase.
78 changes: 43 additions & 35 deletions data/cswt_insert.xml
@@ -1,18 +1,18 @@
<?xml version="1.0" encoding="UTF-8"?>
<csw:Transaction xmlns:csw="http://www.opengis.net/cat/csw/2.0.2" xmlns:ows="http://www.opengis.net/ows" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.opengis.net/cat/csw/2.0.2 http://schemas.opengis.net/csw/2.0.2/CSW-publication.xsd" service="CSW" version="2.0.2" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:dct="http://purl.org/dc/terms/" xmlns:registry="http://gis.harvard.edu/HHypermap/registry/0.1" >
<csw:Insert>

<csw:Record>
<dc:identifier>12</dc:identifier>
<csw:Insert>
<csw:Record>
<dc:identifier>23a09875-1ac8-4768-b799-512abc5ec0b2</dc:identifier>
<dc:title>Airports (OSM)</dc:title>
<dct:alternative>14</dct:alternative>
<dct:modified>2016-09-04T22:37:46Z</dct:modified>
<dct:modified>2016-09-06T00:54:14Z</dct:modified>
<dct:abstract>Airports - major airports only.</dct:abstract>
<dc:type>dataset</dc:type>
<dc:format>OGC:WMS</dc:format>
<dc:source>http://ngamaps.geointapps.org/arcgis/services/RIO/Rio_Foundation_Transportation/MapServer/WMSServer</dc:source>
<dc:relation>1</dc:relation>
<dc:relation>825b53bf-1066-408b-b38d-52fb80d2a35b</dc:relation>
<dct:references scheme="OGC:WMS">http://ngamaps.geointapps.org/arcgis/services/RIO/Rio_Foundation_Transportation/MapServer/WMSServer</dct:references>
<dct:references scheme="WWW:LINK">http://django:8000/registry/hypermap/layer/12/</dct:references>
<ows:BoundingBox crs="http://www.opengis.net/def/crs/EPSG/0/4326" dimensions="2">
<ows:LowerCorner>-23.63 -60.05</ows:LowerCorner>
<ows:UpperCorner>-3.03 -38.33</ows:UpperCorner>
Expand Down Expand Up @@ -41,16 +41,17 @@
<registry:property name="license/copyright" value="Private. For testing purposes."/>
</csw:Record>
<csw:Record>
<dc:identifier>11</dc:identifier>
<dc:identifier>888406e1-a813-44cc-aef1-99cbcd51580f</dc:identifier>
<dc:title>Stations (OSM)</dc:title>
<dct:alternative>13</dct:alternative>
<dct:modified>2016-09-04T22:37:46Z</dct:modified>
<dct:modified>2016-09-06T00:54:14Z</dct:modified>
<dct:abstract>Public transit stations (BRT and Metro)</dct:abstract>
<dc:type>dataset</dc:type>
<dc:format>OGC:WMS</dc:format>
<dc:source>http://ngamaps.geointapps.org/arcgis/services/RIO/Rio_Foundation_Transportation/MapServer/WMSServer</dc:source>
<dc:relation>1</dc:relation>
<dc:relation>825b53bf-1066-408b-b38d-52fb80d2a35b</dc:relation>
<dct:references scheme="OGC:WMS">http://ngamaps.geointapps.org/arcgis/services/RIO/Rio_Foundation_Transportation/MapServer/WMSServer</dct:references>
<dct:references scheme="WWW:LINK">http://django:8000/registry/hypermap/layer/11/</dct:references>
<ows:BoundingBox crs="http://www.opengis.net/def/crs/EPSG/0/4326" dimensions="2">
<ows:LowerCorner>-23.01 -43.44</ows:LowerCorner>
<ows:UpperCorner>-22.81 -43.18</ows:UpperCorner>
Expand Down Expand Up @@ -79,16 +80,17 @@
<registry:property name="license/copyright" value="Private. For testing purposes."/>
</csw:Record>
<csw:Record>
<dc:identifier>10</dc:identifier>
<dc:identifier>bb4a5d7e-4e45-465b-8376-e98abd659c90</dc:identifier>
<dc:title>Metro Lines (OSM)</dc:title>
<dct:alternative>12</dct:alternative>
<dct:modified>2016-09-04T22:37:46Z</dct:modified>
<dct:modified>2016-09-06T00:54:14Z</dct:modified>
<dct:abstract>Metro lines</dct:abstract>
<dc:type>dataset</dc:type>
<dc:format>OGC:WMS</dc:format>
<dc:source>http://ngamaps.geointapps.org/arcgis/services/RIO/Rio_Foundation_Transportation/MapServer/WMSServer</dc:source>
<dc:relation>1</dc:relation>
<dc:relation>825b53bf-1066-408b-b38d-52fb80d2a35b</dc:relation>
<dct:references scheme="OGC:WMS">http://ngamaps.geointapps.org/arcgis/services/RIO/Rio_Foundation_Transportation/MapServer/WMSServer</dct:references>
<dct:references scheme="WWW:LINK">http://django:8000/registry/hypermap/layer/10/</dct:references>
<ows:BoundingBox crs="http://www.opengis.net/def/crs/EPSG/0/4326" dimensions="2">
<ows:LowerCorner>-23.01 -43.37</ows:LowerCorner>
<ows:UpperCorner>-22.81 -43.17</ows:UpperCorner>
Expand Down Expand Up @@ -117,16 +119,17 @@
<registry:property name="license/copyright" value="Private. For testing purposes."/>
</csw:Record>
<csw:Record>
<dc:identifier>9</dc:identifier>
<dc:identifier>5bfdca4e-c150-4a60-8b58-ab28d579bce5</dc:identifier>
<dc:title>BRT Lines (OSM)</dc:title>
<dct:alternative>11</dct:alternative>
<dct:modified>2016-09-04T22:37:46Z</dct:modified>
<dct:modified>2016-09-06T00:54:14Z</dct:modified>
<dct:abstract>BRT Lines</dct:abstract>
<dc:type>dataset</dc:type>
<dc:format>OGC:WMS</dc:format>
<dc:source>http://ngamaps.geointapps.org/arcgis/services/RIO/Rio_Foundation_Transportation/MapServer/WMSServer</dc:source>
<dc:relation>1</dc:relation>
<dc:relation>825b53bf-1066-408b-b38d-52fb80d2a35b</dc:relation>
<dct:references scheme="OGC:WMS">http://ngamaps.geointapps.org/arcgis/services/RIO/Rio_Foundation_Transportation/MapServer/WMSServer</dct:references>
<dct:references scheme="WWW:LINK">http://django:8000/registry/hypermap/layer/9/</dct:references>
<ows:BoundingBox crs="http://www.opengis.net/def/crs/EPSG/0/4326" dimensions="2">
<ows:LowerCorner>-23.02 -43.68</ows:LowerCorner>
<ows:UpperCorner>-22.81 -43.23</ows:UpperCorner>
Expand Down Expand Up @@ -155,16 +158,17 @@
<registry:property name="license/copyright" value="Private. For testing purposes."/>
</csw:Record>
<csw:Record>
<dc:identifier>8</dc:identifier>
<dc:identifier>8bec4ef5-6ac4-4c03-aea4-387a2b2cbb80</dc:identifier>
<dc:title>Road Tunnels (OSM)</dc:title>
<dct:alternative>10</dct:alternative>
<dct:modified>2016-09-04T22:37:46Z</dct:modified>
<dct:modified>2016-09-06T00:54:14Z</dct:modified>
<dct:abstract>Road Tunnels</dct:abstract>
<dc:type>dataset</dc:type>
<dc:format>OGC:WMS</dc:format>
<dc:source>http://ngamaps.geointapps.org/arcgis/services/RIO/Rio_Foundation_Transportation/MapServer/WMSServer</dc:source>
<dc:relation>1</dc:relation>
<dc:relation>825b53bf-1066-408b-b38d-52fb80d2a35b</dc:relation>
<dct:references scheme="OGC:WMS">http://ngamaps.geointapps.org/arcgis/services/RIO/Rio_Foundation_Transportation/MapServer/WMSServer</dct:references>
<dct:references scheme="WWW:LINK">http://django:8000/registry/hypermap/layer/8/</dct:references>
<ows:BoundingBox crs="http://www.opengis.net/def/crs/EPSG/0/4326" dimensions="2">
<ows:LowerCorner>-23.02 -43.56</ows:LowerCorner>
<ows:UpperCorner>-22.67 -43.0</ows:UpperCorner>
Expand Down Expand Up @@ -193,16 +197,17 @@
<registry:property name="license/copyright" value="Private. For testing purposes."/>
</csw:Record>
<csw:Record>
<dc:identifier>7</dc:identifier>
<dc:identifier>d7e41e11-7b30-4508-9adc-7c76e99a7c0e</dc:identifier>
<dc:title>Bridges Tunnels (OSM)</dc:title>
<dct:alternative>9</dct:alternative>
<dct:modified>2016-09-04T22:37:46Z</dct:modified>
<dct:modified>2016-09-06T00:54:14Z</dct:modified>
<dct:abstract>bridges and tunnels identified from OpenStreetMap</dct:abstract>
<dc:type>dataset</dc:type>
<dc:format>OGC:WMS</dc:format>
<dc:source>http://ngamaps.geointapps.org/arcgis/services/RIO/Rio_Foundation_Transportation/MapServer/WMSServer</dc:source>
<dc:relation>1</dc:relation>
<dc:relation>825b53bf-1066-408b-b38d-52fb80d2a35b</dc:relation>
<dct:references scheme="OGC:WMS">http://ngamaps.geointapps.org/arcgis/services/RIO/Rio_Foundation_Transportation/MapServer/WMSServer</dct:references>
<dct:references scheme="WWW:LINK">http://django:8000/registry/hypermap/layer/7/</dct:references>
<ows:BoundingBox crs="http://www.opengis.net/def/crs/EPSG/0/4326" dimensions="2">
<ows:LowerCorner>-23.06 -43.64</ows:LowerCorner>
<ows:UpperCorner>-22.66 -42.85</ows:UpperCorner>
Expand Down Expand Up @@ -231,16 +236,17 @@
<registry:property name="license/copyright" value="Private. For testing purposes."/>
</csw:Record>
<csw:Record>
<dc:identifier>6</dc:identifier>
<dc:identifier>6d8e0da1-23d8-496c-979a-f43624f47a51</dc:identifier>
<dc:title>Rio Roads (OSM June 2015)</dc:title>
<dct:alternative>7</dct:alternative>
<dct:modified>2016-09-04T22:37:46Z</dct:modified>
<dct:modified>2016-09-06T00:54:14Z</dct:modified>
<dct:abstract>Rio roads</dct:abstract>
<dc:type>dataset</dc:type>
<dc:format>OGC:WMS</dc:format>
<dc:source>http://ngamaps.geointapps.org/arcgis/services/RIO/Rio_Foundation_Transportation/MapServer/WMSServer</dc:source>
<dc:relation>1</dc:relation>
<dc:relation>825b53bf-1066-408b-b38d-52fb80d2a35b</dc:relation>
<dct:references scheme="OGC:WMS">http://ngamaps.geointapps.org/arcgis/services/RIO/Rio_Foundation_Transportation/MapServer/WMSServer</dct:references>
<dct:references scheme="WWW:LINK">http://django:8000/registry/hypermap/layer/6/</dct:references>
<ows:BoundingBox crs="http://www.opengis.net/def/crs/EPSG/0/4326" dimensions="2">
<ows:LowerCorner>-23.07 -43.89</ows:LowerCorner>
<ows:UpperCorner>-22.52 -42.73</ows:UpperCorner>
Expand Down Expand Up @@ -269,16 +275,17 @@
<registry:property name="license/copyright" value="Private. For testing purposes."/>
</csw:Record>
<csw:Record>
<dc:identifier>5</dc:identifier>
<dc:identifier>fe7f1422-b23c-48e4-a442-9bfceff173c9</dc:identifier>
<dc:title>Belo Horizonte Roads (OSM May 2016)</dc:title>
<dct:alternative>6</dct:alternative>
<dct:modified>2016-09-04T22:37:46Z</dct:modified>
<dct:modified>2016-09-06T00:54:14Z</dct:modified>
<dct:abstract>Belo Horizonte Roads (OSM May 2016)</dct:abstract>
<dc:type>dataset</dc:type>
<dc:format>OGC:WMS</dc:format>
<dc:source>http://ngamaps.geointapps.org/arcgis/services/RIO/Rio_Foundation_Transportation/MapServer/WMSServer</dc:source>
<dc:relation>1</dc:relation>
<dc:relation>825b53bf-1066-408b-b38d-52fb80d2a35b</dc:relation>
<dct:references scheme="OGC:WMS">http://ngamaps.geointapps.org/arcgis/services/RIO/Rio_Foundation_Transportation/MapServer/WMSServer</dct:references>
<dct:references scheme="WWW:LINK">http://django:8000/registry/hypermap/layer/5/</dct:references>
<ows:BoundingBox crs="http://www.opengis.net/def/crs/EPSG/0/4326" dimensions="2">
<ows:LowerCorner>-20.1 -44.2</ows:LowerCorner>
<ows:UpperCorner>-19.55 -43.76</ows:UpperCorner>
Expand Down Expand Up @@ -307,16 +314,17 @@
<registry:property name="license/copyright" value="Private. For testing purposes."/>
</csw:Record>
<csw:Record>
<dc:identifier>4</dc:identifier>
<dc:identifier>ed55e6df-2048-4a94-8a12-bfcd60f65719</dc:identifier>
<dc:title>Brasilia Roads (OSM May 2016)</dc:title>
<dct:alternative>5</dct:alternative>
<dct:modified>2016-09-04T22:37:46Z</dct:modified>
<dct:modified>2016-09-06T00:54:14Z</dct:modified>
<dct:abstract>Brasilia Roads (OSM May 2016)</dct:abstract>
<dc:type>dataset</dc:type>
<dc:format>OGC:WMS</dc:format>
<dc:source>http://ngamaps.geointapps.org/arcgis/services/RIO/Rio_Foundation_Transportation/MapServer/WMSServer</dc:source>
<dc:relation>1</dc:relation>
<dc:relation>825b53bf-1066-408b-b38d-52fb80d2a35b</dc:relation>
<dct:references scheme="OGC:WMS">http://ngamaps.geointapps.org/arcgis/services/RIO/Rio_Foundation_Transportation/MapServer/WMSServer</dct:references>
<dct:references scheme="WWW:LINK">http://django:8000/registry/hypermap/layer/4/</dct:references>
<ows:BoundingBox crs="http://www.opengis.net/def/crs/EPSG/0/4326" dimensions="2">
<ows:LowerCorner>-16.05 -48.27</ows:LowerCorner>
<ows:UpperCorner>-15.49 -47.39</ows:UpperCorner>
Expand Down Expand Up @@ -345,16 +353,17 @@
<registry:property name="license/copyright" value="Private. For testing purposes."/>
</csw:Record>
<csw:Record>
<dc:identifier>3</dc:identifier>
<dc:identifier>2e853182-0f00-4ae5-b403-ec8759e576ac</dc:identifier>
<dc:title>Manaus Roads (OSM May 2016)</dc:title>
<dct:alternative>4</dct:alternative>
<dct:modified>2016-09-04T22:37:46Z</dct:modified>
<dct:modified>2016-09-06T00:54:14Z</dct:modified>
<dct:abstract>Manaus Roads (OSM May 2016)</dct:abstract>
<dc:type>dataset</dc:type>
<dc:format>OGC:WMS</dc:format>
<dc:source>http://ngamaps.geointapps.org/arcgis/services/RIO/Rio_Foundation_Transportation/MapServer/WMSServer</dc:source>
<dc:relation>1</dc:relation>
<dc:relation>825b53bf-1066-408b-b38d-52fb80d2a35b</dc:relation>
<dct:references scheme="OGC:WMS">http://ngamaps.geointapps.org/arcgis/services/RIO/Rio_Foundation_Transportation/MapServer/WMSServer</dct:references>
<dct:references scheme="WWW:LINK">http://django:8000/registry/hypermap/layer/3/</dct:references>
<ows:BoundingBox crs="http://www.opengis.net/def/crs/EPSG/0/4326" dimensions="2">
<ows:LowerCorner>-3.45 -60.47</ows:LowerCorner>
<ows:UpperCorner>-2.25 -59.7</ows:UpperCorner>
Expand Down Expand Up @@ -382,6 +391,5 @@
<registry:property name="license/restrictions" value="none"/>
<registry:property name="license/copyright" value="Private. For testing purposes."/>
</csw:Record>

</csw:Insert>
</csw:Insert>
</csw:Transaction>
Binary file modified data/cswt_paw.paw
Binary file not shown.
1 change: 1 addition & 0 deletions django.env
Expand Up @@ -9,4 +9,5 @@ REGISTRY_CHECK_PERIOD=120
REGISTRY_SKIP_CELERY=False
REGISTRY_LIMIT_LAYERS=0
REGISTRY_INDEX_CACHED_LAYERS_PERIOD=1
REGISTRY_HARVEST_SERVICES=True
C_FORCE_ROOT=1
2 changes: 2 additions & 0 deletions hypermap/aggregator/elasticsearch_client.py
Expand Up @@ -166,6 +166,8 @@ def layer_to_es(layer, with_bulk=False):
es_record['layer_date'] = es_date
es_record['layer_datetype'] = type

es_record['registry'] = layer.registry_tags()

if layer.get_tile_url():
es_record['tile_url'] = layer.get_tile_url()

Expand Down
35 changes: 35 additions & 0 deletions hypermap/aggregator/migrations/0003_uuid.py
@@ -0,0 +1,35 @@
# -*- coding: utf-8 -*-
from __future__ import unicode_literals

from django.db import migrations, models
import uuid


class Migration(migrations.Migration):

dependencies = [
('aggregator', '0002_multicatalog'),
]

operations = [
migrations.AddField(
model_name='layer',
name='uuid',
field=models.UUIDField(default=uuid.uuid4, editable=False),
),
migrations.AddField(
model_name='service',
name='uuid',
field=models.UUIDField(default=uuid.uuid4, editable=False),
),
migrations.AlterField(
model_name='layer',
name='csw_last_updated',
field=models.CharField(default=b'2016-09-05T22:37:56Z', max_length=32, null=True, blank=True),
),
migrations.AlterField(
model_name='service',
name='csw_last_updated',
field=models.CharField(default=b'2016-09-05T22:37:56Z', max_length=32, null=True, blank=True),
),
]
29 changes: 29 additions & 0 deletions hypermap/aggregator/migrations/0004_is_monitored.py
@@ -0,0 +1,29 @@
# -*- coding: utf-8 -*-
from __future__ import unicode_literals

from django.db import migrations, models


class Migration(migrations.Migration):

dependencies = [
('aggregator', '0003_uuid'),
]

operations = [
migrations.AddField(
model_name='service',
name='is_monitored',
field=models.BooleanField(default=True),
),
migrations.AlterField(
model_name='layer',
name='csw_last_updated',
field=models.CharField(default=b'2016-09-05T23:11:51Z', max_length=32, null=True, blank=True),
),
migrations.AlterField(
model_name='service',
name='csw_last_updated',
field=models.CharField(default=b'2016-09-05T23:11:51Z', max_length=32, null=True, blank=True),
),
]

0 comments on commit 0e1608e

Please sign in to comment.