Add option to select different datasets and fid fields as input to OQT #37

matthiasschaub · 2021-05-26T09:53:16Z

Description

Add option to select different datasets and fid fields as input to OQT.

Available datasets and possible fid fields are defined in definitions.py.

Each dataset defaults to a primary fid field. The usage of other fields as identifier is possible.
The attribute/parameter/argument fid_field/fidField is added to the report class, oqt functions and API/CLI.

Rewrite save and load indicator from database logic to use one result table for all indicator results.

Corresponding issue

Closes #4

Checklist

I have updated my branch to main (e.g. through git rebase main)
My code follows the style guide and was checked with pre-commit before committing
I have commented my code
I have added sufficient unit and integration tests
I have updated the CHANGELOG.md
Check changes to VCR cassetts
Add sensible commits (squash, rebase, …)

CHANGELOG.md

joker234 · 2021-06-04T07:47:17Z

workers/ohsome_quality_analyst/base/report.py

@@ -40,9 +40,11 @@ def __init__(
        bpolys: FeatureCollection = None,
        dataset: str = None,
        feature_id: int = None,
+        fid_field: str = None,


Where did you document what fid_field is supposed to be? Do we have documentation for these report parameters in general?

Not really. The only documentation is provided through the CLI (cli_opts.py)

fid_field_opt = [ click.option( "--fid-field", type=str, help=( "Provide the feature id field of the dataset. " + "Use command list-fid-fields to view available " + "fid fields for each dataset" ), default=None, ) ]

workers/ohsome_quality_analyst/cli.py

workers/ohsome_quality_analyst/cli_opts.py

workers/ohsome_quality_analyst/geodatabase/create_results_table.sql

workers/ohsome_quality_analyst/geodatabase/client.py

workers/ohsome_quality_analyst/oqt.py

workers/tests/integrationtests/test_api_indicator.py

matthiasschaub · 2021-06-07T09:45:40Z

workers/ohsome_quality_analyst/api.py

-async def get_bpolys_from_db(dataset: str, featureId: int):
-    bpolys = await db_client.get_bpolys_from_db(dataset=dataset, feature_id=featureId)
-    return bpolys
+async def get_bpolys_from_db(


Tests are not sufficient for this function. But I will remove the function in a separate PR since it is not in use.

joker234

I think you should consider redoing the VCRs with a fresh cassette again. I got some changed/additional requests when running pytest with a fresh database.

workers/ohsome_quality_analyst/geodatabase/client.py

workers/ohsome_quality_analyst/geodatabase/create_results_table.sql

Rename get_fids to get_ids. Make get_ids dataset agnostic.

This is work in progress.

Update docstring Co-authored-by: Johannes Visintini <johannes.visintini@heigit.org> Update docstring Co-authored-by: Johannes Visintini <johannes.visintini@heigit.org>

Recreate VCR cassetts

Co-authored-by: Johannes Visintini <johannes.visintini@heigit.org> Make if stetement more precise Co-authored-by: Johannes Visintini <johannes.visintini@heigit.org>

matthiasschaub changed the title ~~WIP~~ Enable different datasets as input to OQT #4 May 26, 2021

matthiasschaub added the enhancement New feature or request label May 26, 2021

matthiasschaub self-assigned this May 26, 2021

matthiasschaub force-pushed the enable-different-datasets-as-input branch 4 times, most recently from 52ff5a0 to 575a0ef Compare June 1, 2021 15:26

matthiasschaub changed the title ~~Enable different datasets as input to OQT #4~~ Add option to select different datasets and fid fields as input to OQT #4 Jun 1, 2021

matthiasschaub changed the title ~~Add option to select different datasets and fid fields as input to OQT #4~~ Add option to select different datasets and fid fields as input to OQT Jun 1, 2021

matthiasschaub marked this pull request as ready for review June 1, 2021 15:40

matthiasschaub force-pushed the enable-different-datasets-as-input branch 3 times, most recently from 3ab57ed to 50c9fdd Compare June 3, 2021 13:56

matthiasschaub requested review from Hagellach37 and joker234 June 3, 2021 14:43

joker234 reviewed Jun 4, 2021

View reviewed changes

matthiasschaub commented Jun 7, 2021

View reviewed changes

matthiasschaub force-pushed the enable-different-datasets-as-input branch from 7b5bd3f to b5f0de8 Compare June 7, 2021 11:37

joker234 reviewed Jun 8, 2021

View reviewed changes

matthiasschaub and others added 10 commits June 9, 2021 10:32

Make geodatabase client agnostic to input dataset

18de088

Rename bpoly to bpolys for get_area_of_bpolys()

cf88af8

Create all indicators always runs with OQT regions

98769f4

Rename get_fids to get_ids. Make get_ids dataset agnostic.

Rewrite save and load indicator results functions

699ab88

Improve code readability

017aea7

Add test for get_connection of geodatase module

50243fd

Rename id_ to fid and id_field to fid_field

c1af64d

Rewrite of oqt.py to be compatible with db client

a757d15

Different dataset input through CLI

68b9e1b

This is work in progress.

Remove duplicated attrribute of JSON from DB

c5f62cf

matthiasschaub and others added 6 commits June 9, 2021 10:32

Fix not sufficiant conditional for fid fields

e5ecac7

Add and improve tests for oqt.py and cli.py

d3caa44

Change API parameter type of fid to int and str

3534864

Add and improve tests for the API

9a81213

Keep changelog order as before. Link to Issue.

607d868

Fix spelling

e6c6042

matthiasschaub force-pushed the enable-different-datasets-as-input branch from fb30766 to ebe34ad Compare June 9, 2021 08:36

matthiasschaub and others added 13 commits June 9, 2021 10:39

Improve docstring

de33296

Update docstring Co-authored-by: Johannes Visintini <johannes.visintini@heigit.org> Update docstring Co-authored-by: Johannes Visintini <johannes.visintini@heigit.org>

Add test for custom CLI parameter type

19af023

Init fid_field for tests in SetUp function

d34cd01

Revert fid changes of tests

cfb54ef

Add function to check sanity of user input

164596e

Removed try except statement due to empty except

7a44cb2

Revert variable name change

ccc89c1

Update VCR cassettes

532be3b

Recreate VCR cassetts

Update CHANGELOG

d96cb12

Rename attributes to avoid usage of SQL keywords

dbbbdd3

Make if stetement more precise

30babd8

Co-authored-by: Johannes Visintini <johannes.visintini@heigit.org> Make if stetement more precise Co-authored-by: Johannes Visintini <johannes.visintini@heigit.org>

Recreate relevant VCR cassettes

8ceb515

Fix wrong test region for tests of oqt.py

08288ce

matthiasschaub force-pushed the enable-different-datasets-as-input branch from ebe34ad to 08288ce Compare June 9, 2021 08:39

matthiasschaub and others added 3 commits June 9, 2021 10:44

Run black on geodatabase/client.py

13c5077

Fix order in Changelog

3bd6662

fix changelog order

e81340f

joker234 approved these changes Jun 9, 2021

View reviewed changes

joker234 added this to the Release 0.4.0 milestone Jun 9, 2021

matthiasschaub merged commit bdc414a into main Jun 9, 2021

matthiasschaub deleted the enable-different-datasets-as-input branch June 9, 2021 09:37

matthiasschaub mentioned this pull request Jun 9, 2021

Meaning and precision of variable names #50

Closed

matthiasschaub mentioned this pull request Jul 6, 2021

Bpoly feature #69

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add option to select different datasets and fid fields as input to OQT #37

Add option to select different datasets and fid fields as input to OQT #37

matthiasschaub commented May 26, 2021 •

edited

joker234 Jun 4, 2021

matthiasschaub Jun 4, 2021 •

edited

matthiasschaub Jun 7, 2021

joker234 left a comment

Add option to select different datasets and fid fields as input to OQT #37

Add option to select different datasets and fid fields as input to OQT #37

Conversation

matthiasschaub commented May 26, 2021 • edited

Description

Corresponding issue

Checklist

joker234 Jun 4, 2021

Choose a reason for hiding this comment

matthiasschaub Jun 4, 2021 • edited

Choose a reason for hiding this comment

matthiasschaub Jun 7, 2021

Choose a reason for hiding this comment

joker234 left a comment

Choose a reason for hiding this comment

matthiasschaub commented May 26, 2021 •

edited

matthiasschaub Jun 4, 2021 •

edited