Feature refactor #110

AaronV77 · 2018-09-12T19:31:56Z

No description provided.

# Conflicts: # test/data.py

sdc50

I've reviewed up until the provider plugin files.

sdc50 · 2018-09-12T19:36:50Z

examples/notebooks/Create GSSHA Model with Watershed Delineation - Cary.ipynb

@@ -150,9 +150,9 @@
    "    Returns quest uid for merged raster\n",
    "    \"\"\"\n",
    "    \n",
-    "    features = quest.api.get_features(service_uri, filters={'bbox': bbox})\n",
+    "    features = quest.api.search_catalog(service_uri, filters={'bbox': bbox})\n",


Let's change the variable name here from features to entries or something like that.

This is in the notebooks and will assume that you took care of this one. You mentioned in our reviewal process that you would. Let me know if you haven't. Thanks

sdc50 · 2018-09-12T19:48:56Z

quest/api/catalog.py

+
+    uris = []
+    for _, data in entries.iterrows():
+        uri = new_dataset(collection=collection, catalog_entry=data.to_frame().transpose(), source=DatasetSource.WEB_SERVICE)


I think we need to check to see if the cataloge_entries are from quest and if so make the source derived.

sdc50 · 2018-09-12T19:50:22Z

quest/api/catalog.py

+                        i.e. point/line/polygon
+                    parameter (string, optional): filter catalog_entries by parameter
+                    bbox (string, optional): filter catalog_entries by bounding box
+


We need to update this list of filters in the docstring to include all of the filters that are handled in the code.

sdc50 · 2018-09-12T19:59:58Z

quest/api/datasets.py

@@ -32,25 +31,25 @@ def download(feature, file_path, dataset=None, **kwargs):
            details of downloaded data

    """
-    service_uri = feature
+    service_uri = catalog_entry
    if not service_uri.startswith('svc://'):


Delete this section. We can now assume that service_uri will always start with 'svc://'.

sdc50 · 2018-09-12T20:00:57Z

quest/api/datasets.py

        df = df['service'] + '/' + df['service_id']
        service_uri = df.tolist()[0]

    if file_path is None:
        pass

-    provider, service, feature = parse_service_uri(service_uri)
+    provider, service, catalog_entry = parse_service_uri(service_uri)


change catalog_entry to catalog_id

sdc50 · 2018-09-12T21:16:53Z

quest/plugins/user_provider.py

-            fmt = self.features_format
-            paths = self._get_paths(self.features_file)
+        def search_catalog(self, **kwargs):
+            fmt = self.catalog_entries_format


self.catalog_file_format

sdc50 · 2018-09-12T21:17:05Z

quest/plugins/user_provider.py

-            paths = self._get_paths(self.features_file)
+        def search_catalog(self, **kwargs):
+            fmt = self.catalog_entries_format
+            paths = self._get_paths(self.catalog_entries_file)


self.catalog_file

sdc50 · 2018-09-12T21:19:09Z

quest/plugins/user_provider.py

-                            polys.append(Feature(geometry=util.bbox2poly(x1, y1, x2, y2, as_geojson=True), properties=properties, id=feature_id))
-                        features = FeatureCollection(polys)
-                        features = util.to_geodataframe(features)
+                            catalog_entry_id, x1, y1, x2, y2 = line.split()


change catalog_entry_id to catalog_id

sdc50 · 2018-09-12T21:21:23Z

quest/util/misc.py

    """Converts a list of uris into a pandas dataframe.

    Notes:
        Classified by resource type.

    Args:
        uris (list or string): List of Quest uris to classify into the following types: 'collections', 'services',
-        'publishers', 'datasets', or 'features'.
+        'publishers', 'datasets', or 'catalog_ids*'.


remove catalog_ids*

sdc50 · 2018-09-12T21:29:08Z

quest/util/misc.py

@@ -371,19 +373,16 @@ def uuid(resource_type):

    Notes:
        First character of uuid is replaced with 'f' or 'd' respectively


First character of uuid is replaced with 'd' for resource_type dataset.

stickler-ci · 2018-09-19T16:08:40Z

quest/api/__init__.py

-from .features import (
-    add_features,
-    get_features,
+from .catalog import (


E402 module level import not at top of file

stickler-ci · 2018-09-19T16:08:41Z

quest/api/catalog.py

+
+@add_async
+def search_catalog(uris=None, expand=False, as_dataframe=False, as_geojson=False,
+                 update_cache=False, filters=None, queries=None):


E128 continuation line under-indented for visual indent

stickler-ci · 2018-09-19T16:08:41Z

quest/api/catalog.py

+         Note: nested dicts are parsed out as a multi-index tag where keys for nested dicts are joined with ':'.
+    """
+    # group uris by type
+    grouped_uris = util.classify_uris(service_uris, as_dataframe=False, exclude=['collections', 'catalog_entries', 'datasets'])


E501 line too long (127 > 120 characters)

stickler-ci · 2018-09-19T16:08:41Z

quest/api/catalog.py

+
+    Args:
+        geometry (string or Shapely.geometry.shape, optional, Default=None):
+            well-known-text or Shapely shape representing the geometry of the catalog_entry. Alternatively `geom_type` and `geom_coords` can be passed.


E501 line too long (151 > 120 characters)

stickler-ci · 2018-09-19T16:08:41Z

quest/api/parameters.py

-
-    if isinstance(parameters,pd.DataFrame) and feature:
-        idx = parameters['service_id'] == feature
+    if isinstance(parameters,pd.DataFrame) and catalog_id:


E231 missing whitespace after ','

stickler-ci · 2018-09-19T16:08:44Z

quest/api/catalog.py

+            'service_id': catalog_id,
+            'geometry': geometry,
+            'metadata': metadata,
+            }


E123 closing bracket does not match indentation of opening bracket's line

stickler-ci · 2018-09-19T16:08:44Z

quest/api/manage.py

-from .datasets import get_datasets
-from quest.database.database import get_db, db_session
+from quest.database.database import get_db, db_session, select_datasets
+from ..static import DatasetStatus


F401 '..static.DatasetStatus' imported but unused

stickler-ci · 2018-09-19T16:08:44Z

test/test_catalog.py

+import pytest
+
+from data import SERVICES_FEATURE_COUNT, CACHED_SERVICES
+from quest.static import DatasetStatus


F401 'quest.static.DatasetStatus' imported but unused

stickler-ci · 2018-09-19T16:08:44Z

test/test_catalog.py

+pytestmark = pytest.mark.usefixtures('reset_projects_dir', 'set_active_project')
+
+SERVICE_URIS = [
+                'svc://usgs-ned:19-arc-second/581d2561e4b08da350d5a3b2',


E126 continuation line over-indented for hanging indent

stickler-ci · 2018-09-19T16:08:44Z

test/test_catalog.py

+                'svc://usgs-ned:19-arc-second/581d2561e4b08da350d5a3b2',
+                'svc://noaa-ncdc:gsod/028140-99999',
+                'svc://usgs-nwis:iv/01529950',
+                ]


E123 closing bracket does not match indentation of opening bracket's line

coveralls · 2018-09-19T16:48:11Z

Pull Request Test Coverage Report for Build 415

0 of 0 changed or added relevant lines in 0 files are covered.
210 unchanged lines in 11 files lost coverage.
Overall coverage increased (+0.2%) to 69.986%

Files with Coverage Reduction	New Missed Lines	%
/home/travis/miniconda/envs/test-environment/lib/python3.5/site-packages/quest/database/database.py	1	97.78%
/home/travis/miniconda/envs/test-environment/lib/python3.5/site-packages/quest/api/tools.py	2	20.41%
/home/travis/miniconda/envs/test-environment/lib/python3.5/site-packages/quest/util/units/init.py	4	50.0%
/home/travis/miniconda/envs/test-environment/lib/python3.5/site-packages/quest/api/parameters.py	10	28.21%
/home/travis/miniconda/envs/test-environment/lib/python3.5/site-packages/quest/api/manage.py	13	87.27%
/home/travis/miniconda/envs/test-environment/lib/python3.5/site-packages/quest/util/misc.py	15	81.21%
/home/travis/miniconda/envs/test-environment/lib/python3.5/site-packages/quest/plugins/base/service_base.py	16	56.96%
/home/travis/miniconda/envs/test-environment/lib/python3.5/site-packages/quest/api/metadata.py	23	68.54%
/home/travis/miniconda/envs/test-environment/lib/python3.5/site-packages/quest/plugins/base/provider_base.py	29	60.69%
/home/travis/miniconda/envs/test-environment/lib/python3.5/site-packages/quest/plugins/user_provider.py	37	45.51%

Totals
Change from base Build 387:	0.2%
Covered Lines:	1448
Relevant Lines:	2069

💛 - Coveralls

sdc50

There are some stickler issues still, but I'll fix those with my PR once this is merged in.

AaronV77 added 7 commits September 4, 2018 09:44

Finished filter renaming to tool throughout the Quest source code.

dc766b9

First initial commit for this PR.

44169ff

Merge branch 'filter_renaming' into feature_refactor

ce95c1d

# Conflicts: # test/data.py

Almost all the 'test-quick' are passing for this 3.0 refactor.

8e230d1

The tests are finally passing besides the 'new_catalog_entry'.

412fc81

Revert deleting the plugins.

1e73f86

Wrapping up all the refactor work.

2421d1a

sdc50 reviewed Sep 12, 2018

View reviewed changes

sdc50 mentioned this pull request Sep 19, 2018

Workflows #111

Merged

sdc50 and others added 2 commits September 19, 2018 09:16

rename feature to catalog tweaks

eb42da1

Finished refactor changes.

7d0662d

AaronV77 mentioned this pull request Sep 19, 2018

Clarify the use of "services" in the codebase #59

Closed

Test fix.

d8d5bc8

stickler-ci reviewed Sep 19, 2018

View reviewed changes

Test fix.

7d7e45b

sdc50 approved these changes Sep 19, 2018

View reviewed changes

sdc50 merged commit 4c66353 into erdc:master Sep 19, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature refactor #110

Feature refactor #110

AaronV77 commented Sep 12, 2018

sdc50 left a comment

sdc50 Sep 12, 2018

AaronV77 Sep 19, 2018

sdc50 Sep 12, 2018

sdc50 Sep 12, 2018

sdc50 Sep 12, 2018

sdc50 Sep 12, 2018

sdc50 Sep 12, 2018

sdc50 Sep 12, 2018

sdc50 Sep 12, 2018

sdc50 Sep 12, 2018

sdc50 Sep 12, 2018

stickler-ci Sep 19, 2018

stickler-ci Sep 19, 2018

stickler-ci Sep 19, 2018

stickler-ci Sep 19, 2018

stickler-ci Sep 19, 2018

stickler-ci Sep 19, 2018

stickler-ci Sep 19, 2018

stickler-ci Sep 19, 2018

stickler-ci Sep 19, 2018

stickler-ci Sep 19, 2018

coveralls commented Sep 19, 2018

sdc50 left a comment

		@@ -371,19 +373,16 @@ def uuid(resource_type):

		Notes:
		First character of uuid is replaced with 'f' or 'd' respectively

Feature refactor #110

Feature refactor #110

Conversation

AaronV77 commented Sep 12, 2018

sdc50 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coveralls commented Sep 19, 2018

Pull Request Test Coverage Report for Build 415

💛 - Coveralls

sdc50 left a comment

Choose a reason for hiding this comment