Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TypeError in get_onc_ferry worker #174

Closed
douglatornell opened this issue Apr 4, 2023 · 1 comment
Closed

TypeError in get_onc_ferry worker #174

douglatornell opened this issue Apr 4, 2023 · 1 comment
Assignees
@douglatornell
Labels
bug Something isn't working Workers
Milestone
v24.1

Comments

@douglatornell
Copy link
Member

Appearing in GHA pytest-with-coverage workflow.
Traceback:

    tests/workers/test_get_onc_ferry.py:156: in <module>
        class TestResampleNavCoord:
    tests/workers/test_get_onc_ferry.py:171: in TestResampleNavCoord
        resampled_coord = get_onc_ferry._resample_nav_coord(
    nowcast/workers/get_onc_ferry.py:204: in _resample_nav_coord
        resampled_coord = (nav_data[coord].resample(sampleTime="1Min").mean()).rename(
    /usr/share/miniconda3/envs/salishsea-nowcast-test/lib/python3.10/site-packages/xarray/core/dataarray.py:6654: in resample
        return self._resample(
    /usr/share/miniconda3/envs/salishsea-nowcast-test/lib/python3.10/site-packages/xarray/core/common.py:975: in _resample
        grouper = pd.Grouper(
    /usr/share/miniconda3/envs/salishsea-nowcast-test/lib/python3.10/site-packages/pandas/core/resample.py:1663: in __init__
        super().__init__(freq=freq, axis=axis, **kwargs)
    E   TypeError: Grouper.__init__() got an unexpected keyword argument 'base'

Seems to have started when GHA test env started using pandas=2.0.0.
@douglatornell douglatornell added bug Something isn't working Workers labels Apr 4, 2023
@douglatornell douglatornell added this to the v23.1 milestone Apr 4, 2023
douglatornell added a commit that referenced this issue Apr 5, 2023
@douglatornell
Mark as skip test that fails w/ pandas=2.0.0
be389e7 
See issue #174
douglatornell added a commit that referenced this issue Apr 7, 2023
@douglatornell
Mark as skip test that fails w/ pandas=2.0.0
509cb2f 
See issue #174
douglatornell added a commit that referenced this issue Apr 13, 2023
@douglatornell
Mark as skip test that fails w/ pandas=2.0.0
afa8cb6 
See issue #174
@douglatornell douglatornell modified the milestones: v23.1, v23.2 Sep 28, 2023
@douglatornell douglatornell modified the milestones: v23.2, v24.1 Jan 8, 2024
douglatornell added a commit that referenced this issue Feb 8, 2024
@douglatornell
Remove test skipping due to resolved issue #174
39a130f 
The pytest skip marker for "_resample_nav_coord()" function in
"tests/workers/test_get_onc_ferry.py" file has been removed. This has happened
following the successful resolution of issue number #174, making it unnecessary
to skip these specific unit tests anymore.

Fixes issue #174
douglatornell added a commit that referenced this issue Feb 29, 2024
@douglatornell
Update get_onc_ctd and get_onc_ferry workers to work with ONC API…
2749570 
… v3 (#234)

* Modernize test_get_onc_ctd

Replace unittest.mock.patch decorator with pytest.fixture for mock worker.

Add unit tests for production YAML config file elements related to worker;
re: issue #117.

Replace unittest.mock.patch decorator with pytest caplog fixture for tests of
logging; re: issue #82.

* Add unit test for YAML config file elements

Added unit test for production YAML configuration file ctd data observations
elements used in after_* () functions.

* Improve get_onc_ctd.main() function

Update docstring re: issue #121.

Return worker so that modernized unit tests for main() work; re issue #81.

* Update get_onc_ctd worker to use ONC API v3

Modified the get_onc_ctd worker to use ONC API v3 'getByLocation' method. The
new method uses a location code instead of a station name. Also, these changes
include adding a 'dateTo' parameter to limit the data to a specific time range
for the day. Additionally, small changes to variable and parameter names were
made to match the new method requirements.

* Remove test skipping due to resolved issue #174

The pytest skip marker for "_resample_nav_coord()" function in
"tests/workers/test_get_onc_ferry.py" file has been removed. This has happened
following the successful resolution of issue number #174, making it unnecessary
to skip these specific unit tests anymore.

Fixes issue #174

* Add unit tests for YAML config file elements

Add unit tests for production YAML config file elements related to worker;
re: issue #117.

* Update YAML config with Tsawwassen - Duke Point ferry route

The 'nowcast.yaml' configuration file has been updated to include the
Tsawwassen - Duke Point ferry route details including devices, sensors,
and other relevant data.

* Update dataset ID for TWDP-ferry

The dataset ID for TWDP-ferry was updated in the 'nowcast.yaml' configuration
file and in respective test. The change is reflected accurately to match the
new details for the Tsawwassen - Duke Point ferry route.

* Update get_onc_ferry worker to use ONC API v3

Modified the get_onc_ferry worker to use ONC API v3 'getByLocation' method. The
new method uses a location code instead of a station name. Also, these changes
include adding a 'dateTo' parameter to limit the data to a specific time range
for the day. Server-side averaging into 1-second bins is used to ensure that an
entire day's observations from each sensor can be obtained by a single API
request; there is a limit of 100,000 "rows" per sensor. Additionally, small
changes to variable and parameter names were made to match the new method
requirements.

* Update ONC_data_product_url attr in ferry datasets

Updated the ONC_data_product_url attribute in the datasets produced by the
get_onc_ferry worker to reflect changes in the data source's domain name and API
query parameter names. Code changes include updating the ONC data API domain
name, adding the 'locationCode' query parameter, and changing the
'deviceCategory' query parameter name to 'deviceCategoryCode'. All of those
changes are for compatibility with the ONC data API v3.

* Correct representation of relative humidity attribute

Changed the naming representation of the relative humidity attribute in the
'get_onc_ferry' worker. The attribute was initially named "REL_HUMIDITY" and it
was renamed to "rel_humidity", matching the common low-caps format of the other
attributes for consistency of naming convention.

* Update QA/QC filter criteria in get_onc_ferry worker

Adjust the QA/QC filter settings in the 'get_onc_ferry' worker to now include
data where the 'qaqcFlag' attribute is less than or equal to 1 or greater than
or equal to 7. This update is necessary because of the change to server-side
averaging in the request.

* Improve empty data array handling

ONC API v3 has more/different ways of returning empty responses to data requests
that we have to handle. It's not as clean as I would like, but it is good
enough for the purpose.
@douglatornell
Copy link
Member Author

Fixed as part of PR #234

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@douglatornell douglatornell

Labels
bug Something isn't working Workers
Projects
None yet
Milestone
v24.1
Development

No branches or pull requests
1 participant
@douglatornell