Esgf cwt process

[davidcaron]

• Executing the aggregate process using boreas data
• Executing the subset process using boreas data
• Add a unit test for subset (aggregate wouldn't be necessary, as subset requires more parameters)

(https://aims2.llnl.gov/wps/ is down)

• Executing the aggregate process using esgf data
• Executing the subset process using esgf data

The unchecked tasks will be in a different PR.

[davidcaron]

@fmigneault This branch would be ready to merge. I'll wait for your OK, because there are some things you may want to take a look at that should have been in another PR:

• in weaver/processes/wps1_process.py I changed some logic when converting from cwl inputs to pywps
• in weaver/wps_restapi/processes/processes.py I added a reset_connection argument, to be called from the forked celery process. I was getting a connection error from pymongo when I was accessing the connection for the first time in the celery worker.

I think the rest should not break this branch, although it's possible I missed some things when merging.

[davidcaron]

@fmigneault I would be ready to merge this, let me know if you have any questions.

[fmigneault]

@davidcaron @dbyrns
don't know what is causing it, but many tests fail.
they are all passing on master

[fmigneault]

this shouldn't be required
will cause problems with nginx that is currently doing it

[davidcaron]

Wouldn' t nginx override this? The way I see it it doesn't cause any harm because nginx catches the route before weaver does. And adding this makes it possible to run a local weaver without the need to have nginx running.

[dbyrns]

Good point David.

[davidcaron]

Also, isn't this the idea for #5?

[fmigneault]

If it works in both cases, great.
Actually, we could remove it altogether from nginx, because weaver.wps_output_dir and weaver.wps_output_path are hardcode in it.
This variant is better if workflows are functional with it.

[davidcaron]

I didn't think about large files... wouldn't nginx be more appropriate for large files?

Also, wouldn't setting weaver.wps_output_path = "/my-new-path" break the correct saving of outputs anyway, even in the current setup?

I think we can have both... keep the static route in weaver, remove nginx in the config/buildout of weaver, but when we deploy, configure another instance of nginx to serve static content on a specific route.

[fmigneault]

The idea with weaver.wps_output_path is to specify which weaver url to call to obtain the process results. I couldn't find a way to cleanly match nginx's, pywps's and weaver's configs together under this same setting. So currently, it just "works" because the values are all the same...
I managed to somehow patch the pywps/weaver configs with this setting, but not nginx. I was counting on #5 to just drop it.

[dbyrns]

There's not only large file, there is simultaneously access... Nginx will serve multiples files at the same time without issues while we will rapidly run out of python workers all strugling to serve the files.
For the config issue why not using an environment variable both used in the nginx and weaver config files with the template setup we have?

[fmigneault]

An external nginx in the compose to serve the files is an interesting idea.
We would need to make sure that add_static_view works standalone so that anyone can still run weaver by itself (although slower file transfer).

[dbyrns]

The external nginx is already serving the files using the volume wps_outputs.
so I think we got a consensus that #5 is the way to go and adding the static serve is helpful in debugging and not harmful in production. Config wise using the compose volume wps_outputs and mounting it in both docker image at a proper place should do it.

[davidcaron]

aims2 is up this morning. I will test the workflows requiring aims2.

[davidcaron]

aims2 is still down, I think it's been close to 2 weeks now...

[matprov]

This pull request will not be merge.
See esgf-cwt-process-rebase, for an updated version.
Closing this old PR.