Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Argo data tool #81

Merged
merged 22 commits into from Dec 21, 2023
Merged

Argo data tool #81

merged 22 commits into from Dec 21, 2023

Conversation

Marie59
Copy link
Collaborator

@Marie59 Marie59 commented Nov 17, 2023

1st batch tool for the Earth-System subdomain ! This tool allows user to retrieve argo data in netcdf file. It will be part of the ocean part of the Earth System. I think it should go in the section Get Data of the tool pannel.
I am not sure if I filed the .shed.yml correctly.
What do you think of it ?

@@ -0,0 +1,158 @@
# author: Marie Jossé
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you use the argparse library for that script?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes I think I can just need to check how it works ;)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it good now ?

</command>
<inputs>
<param name="user" type="select" label="Which kind of user are you ?">
<option value="standard">🏊 standard mode simplifies the dataset, remove most of its jargon and return a priori good data</option>
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do those emojis work? :)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes they do I really like the style ;)

@Marie59
Copy link
Collaborator Author

Marie59 commented Nov 24, 2023

I see that I need to add a category Earth-system somewhere with this warning message .. WARNING: Categories [['Earth-system']] unknown.
But I search in the repo github here and I don't find where I am supposed to do that. Could you give me a hand @yvanlebras or @bgruening ?

@Marie59
Copy link
Collaborator Author

Marie59 commented Nov 24, 2023

Weird my planemo t is working locally but it's not here...

@bgruening
Copy link
Collaborator

The warning about the missing category is because this category does not exist in the ToolShed. Use a different one from the TS please. https://toolshed.g2.bx.psu.edu/

@bgruening
Copy link
Collaborator

Are you running the tests locall with planemo test --biocontainers?

@yvanlebras
Copy link
Contributor

Yes, categoris are toolshed ones. You can use the ecology one

@Marie59
Copy link
Collaborator Author

Marie59 commented Nov 27, 2023

Are you running the tests locall with planemo test --biocontainers?

I was not using the option --biocontainers so I tried with it and I do have the same error. I search and tried out some stuffs and I think my tool is searching for a "galaxy user" to run and is not able to find it (because it does not exist I think). I don't really know how to fix that. I tried to add a user in the Cheetah but I don't know how to do it and it may not be the right thing to do.
Does one have you has an idea @bgruening and @yvanlebras ?

@Marie59
Copy link
Collaborator Author

Marie59 commented Nov 27, 2023

The warning about the missing category is because this category does not exist in the ToolShed. Use a different one from the TS please. https://toolshed.g2.bx.psu.edu/

Am I gonna be able to have this tool to one of the sections I have implemented in the earth system subdomain ? Because I don't want to be forced to add the entire set of ecology tools in the earth system...
Thinking about it maybe I should just add it in the 'Source Data' category ? and it will be in the 'Get Data' tool section ?

@bgruening
Copy link
Collaborator

ToolShed sections are independent of Galaxy tool panel sections. So don't worry about it here.

@bgruening
Copy link
Collaborator

Job in error state.. tool_id: argo_getdata, exit_code: 1, stderr: Traceback (most recent call last):
  File "/home/runner/work/tools-ecology/tools-ecology/tools/ocean/argo_getdata.py", line 187, in <module>
    argo_data = argo_data.load().data
                ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/argopy/fetchers.py", line 643, in load
    self._data = self.to_xarray(**kwargs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/argopy/fetchers.py", line 490, in to_xarray
    xds = self.fetcher.to_xarray(**kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/argopy/data_fetchers/erddap_data.py", line 716, in to_xarray
    results = self.post_process(results, add_dm=add_dm, URI=URI)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/argopy/data_fetchers/erddap_data.py", line 681, in post_process
    this_ds.attrs["Fetched_by"] = getpass.getuser()
                                  ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/getpass.py", line 169, in getuser
    return pwd.getpwuid(os.getuid())[0]
           ^^^^^^^^^^^^^^^^^^^^^^^^^
KeyError: 'getpwuid(): uid not found: 1001'

This looks really like an upstream problem of erddap_data.py

@bgruening
Copy link
Collaborator

I assume it is this line: https://github.com/euroargodev/argopy/blob/master/argopy/data_fetchers/erddap_data.py#L686

The function wants to get Return the password database entry for the given numeric user ID. not sure why this is needed. But in container settings and on HPC this is not a valid assumption.

@Marie59
Copy link
Collaborator Author

Marie59 commented Nov 27, 2023

So, I need to contact the authors of erddap to ask if they can modify that ?

@bgruening
Copy link
Collaborator

I think so. Otherwise we need to hack around this problem on our side :(

@Marie59
Copy link
Collaborator Author

Marie59 commented Dec 20, 2023

OOOH can I start my happy dance @bgruening ?

tools/ocean/macro.xml Outdated Show resolved Hide resolved
tools/ocean/macro.xml Outdated Show resolved Hide resolved
tools/ocean/macro.xml Outdated Show resolved Hide resolved
tools/ocean/argo_getdata.xml Outdated Show resolved Hide resolved
tools/ocean/argo_getdata.xml Outdated Show resolved Hide resolved
tools/ocean/argo_getdata.xml Outdated Show resolved Hide resolved
tools/ocean/argo_getdata.xml Outdated Show resolved Hide resolved
tools/ocean/argo_getdata.xml Outdated Show resolved Hide resolved
tools/ocean/argo_getdata.xml Outdated Show resolved Hide resolved
tools/ocean/argo_getdata.xml Outdated Show resolved Hide resolved
<param name="profile" type="text" multiple="true" value="0" label="Input profile cycle number" help="if you want to enter multiple ones just separate them with a comma (example : 3,12)"/>
</when>
</conditional>
<!--<conditional name="sources">
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should this be still expcluded?

tools/ocean/argo_getdata.xml Outdated Show resolved Hide resolved
tools/ocean/argo_getdata.xml Outdated Show resolved Hide resolved
tools/ocean/argo_getdata.xml Show resolved Hide resolved
tools/ocean/macro.xml Outdated Show resolved Hide resolved
@Marie59
Copy link
Collaborator Author

Marie59 commented Dec 21, 2023

I think I took all your @bgruening suggestions into account. Is there anything else that needs improvement ?

@bgruening
Copy link
Collaborator

Looks great!

@bgruening bgruening merged commit d3171d2 into galaxyecology:master Dec 21, 2023
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants