Submit pydov for review

[stijnvanhoey]

Submitting Author: @Roel, @stijnvanhoey, @pjhaest
Package Name: pydov
One-Line Description of Package: Python package to retrieve data from Databank Ondergrond Vlaanderen (DOV)
Repository Link: https://github.com/DOV-Vlaanderen/pydov
Version submitted: 1.0.0
Editor: @lwasser
Reviewer 1: @NickleDave
Reviewer 2: @xmnlab
Archive: TBD
Version accepted: TBD

---

Description

pydov is a Python package to query and download data from Databank Ondergrond Vlaanderen (DOV). DOV aggregates data about soil, subsoil and groundwater of Flanders (Belgium) and makes them publicly available. Interactive and human-readable extraction and querying of the data is provided by a web application, whereas the focus of this package is to support machine-based extraction and conversion of the data.

Scope

• Please indicate which category or categories this package falls under:
  • Data retrieval
  • Data extraction
  • Data munging
  • Data deposition
  • Reproducibility
  • Geospatial
  • Education
  • Data visualization*

* Please fill out a pre-submission inquiry before submitting a data visualization package. For more info, see this section of our guidebook.

• Explain how the and why the package falls under these categories (briefly, 1-2 sentences):

pydov enables the access to the data services (both WFS services as well as the data itself) provided by DOV. It supports users in searching and downloading data on soil, subsoil and groundwater in Flanders in a reproducible way.

• Who is the target audience and what are scientific applications of this package?

The machine-based availability of the data can potentially serve a diverse community of researchers and students. Applications are in the field of geology, hydrogeology and geotechnics.

• Are there other Python packages that accomplish the same thing? If so, how does yours differ?

We have no knowledge about other packages that provide access to DOV data.

• If you made a pre-submission enquiry, please paste the link to the corresponding issue, forum post, or other discussion, or @tag the editor you contacted:

Technical checks

For details about the pyOpenSci packaging requirements, see our packaging guide. Confirm each of the following by checking the box. This package:

• does not violate the Terms of Service of any service it interacts with.
• has an OSI approved license
• contains a README with instructions for installing the development version.
• includes documentation with examples for all functions.
• contains a vignette with examples of its essential functions and uses. -> set of tutorials
• has a test suite.
• has continuous integration, such as Travis CI, AppVeyor, CircleCI, and/or others.

Publication options

• Do you wish to automatically submit to the Journal of Open Source Software? If so:

JOSS Checks

• The package has an obvious research application according to JOSS's definition in their submission requirements. Be aware that completing the pyOpenSci review process does not guarantee acceptance to JOSS. Be sure to read their submission requirements (linked above) if you are interested in submitting to JOSS.
• The package is not a "minor utility" as defined by JOSS's submission requirements: "Minor ‘utility’ packages, including ‘thin’ API clients, are not acceptable." pyOpenSci welcomes these packages under "Data Retrieval", but JOSS has slightly different criteria.
• The package contains a paper.md matching JOSS's requirements with a high-level description in the package root or in inst/.
• The package is deposited in a long-term repository with the DOI:

Note: Do not submit your package separately to JOSS

Are you OK with Reviewers Submitting Issues and/or pull requests to your Repo Directly?

This option will allow reviewers to open smaller issues that can then be linked to PR's rather than submitting a more dense text based review. It will also allow you to demonstrate addressing the issue via PR links.

• Yes I am OK with reviewers submitting requested changes as issues to my repo. Reviewers will then link to the issues in their submitted review.

Code of conduct

• I agree to abide by pyOpenSci's Code of Conduct during the review process and in maintaining my package should it be accepted.

P.S. Have feedback/comments about our review process? Leave a comment here

Editor and Review Templates

Editor and review templates can be found here

[lwasser]

Hi @stijnvanhoey !!! 👋 thank you for this submission!! i will add this to our list of packages to discuss in our next meeting which is next week - feb 6!!

[lwasser]

@NickleDave will be a reviewer for this submission!! @lwasser will try to find a second reviewer and editor (or she will be editor)

[xmnlab]

I would be happy to help here as a second reviewer :)

[lwasser]

hey @NickleDave and @xmnlab i know there is a lot going on. i just wanted to check in on this review to see how you guys were doing. I am very behind and know that ropensci is stopping new submissions for atleast a month. are you guys still able to review this package or should we take a step back?

Many thanks.

[xmnlab]

I can work on that. I will start that this week.

[NickleDave]

Same here, I think I can finish by next weekend at the latest.

[lwasser]

Thank you so much @xmnlab and @NickleDave !! much appreciated.

[lwasser]

ok let's shoot for both reviews in by Wed April 15th just in case you guys need a bit more time!! and reach out please if you need anything.

[NickleDave]

Hi @Roel @stijnvanhoey @pjhaest thank you for your patience. Please find below my review:

Package Review

Please check off boxes as applicable, and elaborate in comments below. Your review is not limited to these topics, as described in the reviewer guide

• As the reviewer I confirm that there are no conflicts of interest for me to review this work (If you are unsure whether you are in conflict, please speak to your editor before starting your review).

Documentation

The package includes all the following forms of documentation:

• A statement of need clearly stating problems the software is designed to solve and its target audience in README
• Installation instructions: for the development version of package and any non-standard dependencies in README
• Vignette(s) demonstrating major functionality that runs successfully locally
• Function Documentation: for all user-facing functions
• Examples for all user-facing functions
• Community guidelines including contribution guidelines in the README or CONTRIBUTING.
• Metadata including author(s), author e-mail(s), a url, and any other relevant metadata e.g., in a setup.py file or elsewhere.

Readme requirements
The package meets the readme requirements below:

• Package has a README.md file in the root directory.

The README should include, from top to bottom:

• The package name
• Badges for continuous integration and test coverage, the badge for pyOpenSci peer-review once it has started (see below), a repostatus.org badge, and any other badges. If the README has many more badges, you might want to consider using a table for badges, see this example, that one and that one. Such a table should be more wide than high.
• Short description of goals of package, with descriptive links to all vignettes (rendered, i.e. readable, cf the documentation website section) unless the package is small and there’s only one vignette repeating the README.
• Installation instructions
• Any additional setup required (authentication tokens, etc)
• Brief demonstration usage
• Direction to more detailed documentation (e.g. your documentation files or website).
• If applicable, how the package compares to other similar packages and/or how it relates to other packages
• Citation information

Functionality

• Installation: Installation succeeds as documented.
• Functionality: Any functional claims of the software been confirmed.
• Performance: Any performance claims of the software been confirmed.
• Automated tests: Tests cover essential functions of the package and a reasonable range of inputs and conditions. All tests pass on the local machine.
• Continuous Integration: Has continuous integration, such as Travis CI, AppVeyor, CircleCI, and/or others.
• Packaging guidelines: The package conforms to the pyOpenSci packaging guidelines.

For packages co-submitting to JOSS

• The package has an obvious research application according to JOSS's definition in their submission requirements.

Note: Be sure to check this carefully, as JOSS's submission requirements and scope differ from pyOpenSci's in terms of what types of packages are accepted.

The package contains a paper.md matching JOSS's requirements with:

• A short summary describing the high-level functionality of the software
• Authors: A list of authors with their affiliations
• A statement of need clearly stating problems the software is designed to solve and its target audience.
• References: with DOIs for all those that have one (e.g. papers, datasets, software).

Final approval (post-review)

• The author has responded to my review and made changes to my satisfaction. I recommend approving this package.

Estimated hours spent reviewing: 3

---

Review Comments

At a high-level pydov is a fully-developed packages that meets the PyOpenSci review criteria. It provides impressive functionality, yet is tightly scoped to querying and downloading data from Databank Ondergrond Vlaanderen (DOV).

I have one request for the documentation, and one question about the code, although I don't think the question about the code needs to be addressed before approving the package.

The documentation is very thorough, but I think one thing that would help new users is an "intro to working with pydov" tutorial. This tutorial would explains things at a level between the quickstart and the very detailed tutorials for each individual search object. What this tutorial would provide, that the current tutorials do not have, is an introduction to the core abstractions provided by the package. I notice that each DOV search tutorial follows the same structure, but there is not a lot of language about why a potential user might want to use some of the functionality, e.g. why get all the groundwater screens / boring holes within a bounding box.

The intro tutorial would ideally be a specific example of an analysis--it could even be one of the existing search tutorials--that introduces the concepts that repeat throughout the individual tutorials for each search object. This intro should be targeted at a "naive" researcher who is interested in having access to the data, but might not be familiar with some of the existing data structures and libraries that pydov leverages, e.g. the WFS protocol. I admit to a total lack of domain expertise in this area, so the devs' idea of what a "naive" researcher would know compared to mine might vary, but I think that adding an intro with more language about about what and why the package does what it does, in addition to how to do those things, would make it more welcoming to new users.

My question about the code relates to the search sub-package. I understand the need for the types sub-package, because each class in that sub-package encapsulates the metadata such as wfs and xml fields specific to each dataset type. However it's not quite clear to me why different classes are needed for the search objects that correspond to each type in the search package. Am I right that all the search classes have the exact same attributes and methods? My read of the code is that the only difference between the search classes is the default value for objecttype. E.g., a BoringSearch class has its objecttype default =Boring , which dictates what fields it has, but aside from that, it has the exact same methods as all the other search types. If I'm misunderstanding something about the code, please let me know and I'll just remove this question. But it seems like it might be possible to reduce code repetition by having another class that inherits from AbstractSearch with all the code that is common to the specific Search classes, and then just subclass this additional class for specific types.
Something like:

class MetaSearch(AbstractSearch):
    def __init__(self, layer, objecttype):
        super(MetaSearch, self).__init__('dov-pub:Boringen', objecttype)

    def __common_search_methods_here__():
    ...

import Boring

BoringSearch(MetaSearch)
    def __init__(self):
        super(BoringSearch, self).__init__(layer='dov-pub:Boringen', objecttype=Boring)

Thank you again for your patience during this crazy time and thank you for submitting the package, it was a pleasure to review. Looking forward to your response

[xmnlab]

hey everyone, I couldn't finish my review today. I will finish that tomorrow. sorry for the delay.