Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix birds not creating their wps output under each bird name #203

Merged
merged 7 commits into from
Feb 17, 2023
Merged

Fix birds not creating their wps output under each bird name #203

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
2b7d8d9
finch, raven, flyingpigeon: output under wps_output under their respe…
tlvu Oct 28, 2021
073ba02
finch, raven, flyingpigeon: create output dir under wps_output
tlvu Oct 28, 2021
88cc8ee
finch, raven, flyingpigeon: match outputurl with outputpath
tlvu Oct 28, 2021
dad44c7
Merge branch 'master' into fix-birds-not-creating-their-wps_output-un…
Zeitsperre Mar 16, 2022
a026423
Merge remote-tracking branch 'origin/master' into fix-birds-not-creat…
tlvu Feb 15, 2023
dfc1d0d
CHANGES: Fix birds not creating their wps output under each bird name
tlvu Feb 16, 2023
dcbe4d8
Bump version: 1.23.1 → 1.23.2
tlvu Feb 17, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
6 changes: 3 additions & 3 deletions .bumpversion.cfg
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
[bumpversion]
current_version = 1.23.1
current_version = 1.23.2
commit = True
tag = False
tag_name = {new_version}
Expand Down Expand Up @@ -30,11 +30,11 @@ search = {current_version}
replace = {new_version}

[bumpversion:file:RELEASE.txt]
search = {current_version} 2023-02-13T18:31:02Z
search = {current_version} 2023-02-17T02:54:37Z
replace = {new_version} {utcnow:%Y-%m-%dT%H:%M:%SZ}

[bumpversion:part:releaseTime]
values = 2023-02-13T18:31:02Z
values = 2023-02-17T02:54:37Z

[bumpversion:file(version):birdhouse/config/canarie-api/docker_configuration.py.template]
search = 'version': '{current_version}'
Expand Down
24 changes: 24 additions & 0 deletions CHANGES.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,30 @@

[//]: # (list changes here, using '-' for each new entry, remove this when items are added)

[1.23.2](https://github.com/bird-house/birdhouse-deploy/tree/1.23.2) (2023-02-17)
------------------------------------------------------------------------------------------------------------------

## Fixes
- Fix birds not creating their wps output under each bird name

Before this fix, finch, raven, flyingpigeon were dumping their output directly
under `https://PAVICS_HOST/wpsoutputs/`.

With this fix, it will be under each bird name, ex:
`https://PAVICS_HOST/wpsoutputs/finch/` which is cleaner and follows what
malleefowl and hummingbird already does.

Fixes https://github.com/bird-house/birdhouse-deploy/issues/11.
Fixes https://crim-ca.atlassian.net/browse/DAC-398

Requires PR https://github.com/Ouranosinc/pavics-sdi/pull/280,
https://github.com/bird-house/finch/pull/273 and
https://github.com/Ouranosinc/raven/pull/459.

If `optional-components/secure-data-proxy` is enabled, might need some
additional permissions for each bird in
https://github.com/bird-house/birdhouse-deploy/blob/master/birdhouse/optional-components/secure-data-proxy/config/magpie/config.yml.template.

[1.23.1](https://github.com/bird-house/birdhouse-deploy/tree/1.23.1) (2023-02-13)
------------------------------------------------------------------------------------------------------------------

Expand Down
2 changes: 1 addition & 1 deletion Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Generic variables
override SHELL := bash
override APP_NAME := birdhouse-deploy
override APP_VERSION := 1.23.1
override APP_VERSION := 1.23.2

# utility to remove comments after value of an option variable
override clean_opt = $(shell echo "$(1)" | $(_SED) -r -e "s/[ '$'\t'']+$$//g")
Expand Down
8 changes: 4 additions & 4 deletions README.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,13 +14,13 @@ for a full-fledged production platform.
* - releases
- | |latest-version| |commits-since|

.. |commits-since| image:: https://img.shields.io/github/commits-since/bird-house/birdhouse-deploy/1.23.1.svg
.. |commits-since| image:: https://img.shields.io/github/commits-since/bird-house/birdhouse-deploy/1.23.2.svg
:alt: Commits since latest release
:target: https://github.com/bird-house/birdhouse-deploy/compare/1.23.1...master
:target: https://github.com/bird-house/birdhouse-deploy/compare/1.23.2...master

.. |latest-version| image:: https://img.shields.io/badge/tag-1.23.1-blue.svg?style=flat
.. |latest-version| image:: https://img.shields.io/badge/tag-1.23.2-blue.svg?style=flat
:alt: Latest Tag
:target: https://github.com/bird-house/birdhouse-deploy/tree/1.23.1
:target: https://github.com/bird-house/birdhouse-deploy/tree/1.23.2

.. |readthedocs| image:: https://readthedocs.org/projects/birdhouse-deploy/badge/?version=latest
:alt: ReadTheDocs Build Status (latest version)
Expand Down
2 changes: 1 addition & 1 deletion RELEASE.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
1.23.1 2023-02-13T18:31:02Z
1.23.2 2023-02-17T02:54:37Z
8 changes: 4 additions & 4 deletions birdhouse/config/canarie-api/docker_configuration.py.template
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,8 @@ SERVICES = {
'info': {
'name': 'Node',
'synopsis': 'Nodes are data, compute and index endpoints accessed through the PAVICS platform or external clients. The Node service is the backend that allows: data storage, harvesting, indexation and discovery of local and federated data; authentication and authorization; server registration and management. Node service is therefore composed of several other services.',
'version': '1.23.1',
'releaseTime': '2023-02-13T18:31:02Z',
'version': '1.23.2',
'releaseTime': '2023-02-17T02:54:37Z',
'institution': 'Ouranos',
'researchSubject': 'Climatology',
'supportEmail': '${SUPPORT_EMAIL}',
Expand Down Expand Up @@ -242,8 +242,8 @@ PLATFORMS = {
'info': {
'name': 'PAVICS',
'synopsis': 'The PAVICS (Power Analytics for Visualization of Climate Science) platform is a collection of climate analysis services served through Open Geospatial Consortium (OGC) protocols. These services include data access, processing and visualization. Both data and algorithms can be accessed either programmatically, through OGC-compliant clients such as QGIS or ArcGIS, or a custom web interface.',
'version': '1.23.1',
'releaseTime': '2023-02-13T18:31:02Z',
'version': '1.23.2',
'releaseTime': '2023-02-17T02:54:37Z',
'institution': 'Ouranos',
'researchSubject': 'Climatology',
'supportEmail': '${SUPPORT_EMAIL}',
Expand Down
4 changes: 2 additions & 2 deletions birdhouse/config/finch/wps.cfg.template
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[server]
outputurl = https://${PAVICS_FQDN_PUBLIC}/wpsoutputs
outputpath = /data/wpsoutputs
outputurl = https://${PAVICS_FQDN_PUBLIC}/wpsoutputs/finch
outputpath = /data/wpsoutputs/finch

# default 3mb, fix "Broken pipe" between the proxy and the wps service
maxrequestsize = 400mb
Expand Down
4 changes: 2 additions & 2 deletions birdhouse/config/flyingpigeon/wps.cfg.template
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[server]
outputurl = https://${PAVICS_FQDN_PUBLIC}/wpsoutputs
outputpath = /data/wpsoutputs
outputurl = https://${PAVICS_FQDN_PUBLIC}/wpsoutputs/flyingpigeon
outputpath = /data/wpsoutputs/flyingpigeon
maxsingleinputsize = 2097152000.0

[logging]
Expand Down
4 changes: 2 additions & 2 deletions birdhouse/config/raven/wps.cfg.template
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[server]
outputurl = https://${PAVICS_FQDN_PUBLIC}/wpsoutputs
outputpath = /data/wpsoutputs
outputurl = https://${PAVICS_FQDN_PUBLIC}/wpsoutputs/raven
outputpath = /data/wpsoutputs/raven

# default 3mb, fix "Broken pipe" between the proxy and the wps service
maxrequestsize = 100mb
Expand Down
5 changes: 5 additions & 0 deletions birdhouse/pavics-compose.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -222,6 +222,11 @@ do
docker exec ${postgres_id} /postgres-setup.sh
fi

# because server.outputpath in wps.cfg do not create the dir
for bird in finch flyingpigeon raven; do
docker exec $bird mkdir -p /data/wpsoutputs/$bird
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not so proud of this hack. Question for someone with more WPS config knowledge, is there an option in the config file to tell the server to create the outputpath dir if it does not exist?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if we were to add this to the makefile for all birds instead?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm surprised because the FileStorage code uses os.makedirs, which as far as I understand should create the missing path. https://github.com/geopython/pywps/blob/711219792be8b3d6a175a81152282dc5046d412b/pywps/inout/storage/file.py#L97

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you know where PyWPS fails if you don't create this directory ?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh indeed ! Maybe the dir will be created on first use, not on bird startup so it's fine that's it it not there.

I saw this warning during bird startup server->outputpath configuration value /data/wpsoutputs/finch is not directory so I assumed too fast.

Will retest.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On my end, I'm ok to wait for a PyWPS fix if that avoids throw-away code.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we could then clean pavics-compose.sh further by moving the custom script section for geoserver and postgres to their own config folder. In the end the main compose script would be cleaner and each component would have the ability to run custom command in the context of pavics.

@dbyrns Unfortunately geoserver and postgres are not in a "component" layout. We have been adding new components using the new pluggable design but we have not migrate all the existing pieces to this pluggable design :(

On my end, I'm ok to wait for a PyWPS fix if that avoids throw-away code.

@huard no big deal with throw-away code, it's just this section of mkdir that is throw-away. Waiting for PyWPS means not only waiting for a release of PyWPS but also the integration into our birds then a new build of all our birds.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dbyrns Unfortunately geoserver and postgres are not in a "component" layout. We have been adding new components using the new pluggable design but we have not migrate all the existing pieces to this pluggable design :(

I know, I'm not suggesting to make them proper components, but only to extend the pre/post compose loop to include the "built-in" components in the config directory. This way any existing services could include custom scripts.

This time I agree that we can wait for PyWPS, but we should keep that in mind if component related stuff are to be included in the pavics-compose.sh

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

extend the pre/post compose loop to include the "built-in" components in the config directory. This way any existing services could include custom scripts.

Yeah that's a quick win. But I'd say also a potential throwaway code if we migrate all existing components to the new pluggable design. Potential usecase for a complete pluggable design is not all organisations deploying PAVICS will want all the current components activated. Maybe they just want Thredds and their birds, not our birds.

Anyways, back to this PR, it's a cheap throwaway and a quick win. I'd rather take it now than wait. I can make the pre/post for configs/ as well since we are into cheap throwaway for quick win. But I am fine if we prefer to wait.

Note the other issue bird-house/finch#160 probably also need this same mkdir hack or the matching proper fix on PyWPS side. So either we solve both issues now or we wait for both issues.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@huard Are you okay to merge this quick hack for this fix? Not sure how much time I'll need to debug and perform the proper fix in PyWPS. All the old birds (hummingbird, ...) using the old "Buildout way" are basically doing this same hack, ie mkdir themselve outside of PyWPS.

done

for adir in ${EXTRA_CONF_DIRS}; do
COMPONENT_POST_COMPOSE_UP="$adir/post-docker-compose-up"
if [ -x "$COMPONENT_POST_COMPOSE_UP" ]; then
Expand Down
4 changes: 2 additions & 2 deletions docs/source/conf.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,9 +69,9 @@
# built documents.
#
# The short X.Y version.
version = '1.23.1'
version = '1.23.2'
# The full version, including alpha/beta/rc tags.
release = '1.23.1'
release = '1.23.2'

# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
Expand Down