Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Beam NameError in weather-dl: save_main_session not applied #123

Closed
alxmrs opened this issue Mar 25, 2022 · 1 comment
Closed

Beam NameError in weather-dl: save_main_session not applied #123

alxmrs opened this issue Mar 25, 2022 · 1 comment

Comments

@alxmrs
Copy link
Collaborator

alxmrs commented Mar 25, 2022

When running the weather downloader as usual on the main branch:

weather-dl ./configs/era5_example_config_local_run.cfg --local-run

The following error occurs:

...
File "/Users/alxrsngrtn/homebrew/Caskroom/miniconda/base/lib/python3.8/site-packages/future/utils/__init__.py", line 446, in raise_with_traceback
    raise exc.with_traceback(traceback)
  File "apache_beam/runners/common.py", line 1213, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 570, in apache_beam.runners.common.SimpleInvoker.invoke_process
  File "/Users/alxrsngrtn/homebrew/Caskroom/miniconda/base/lib/python3.8/site-packages/apache_beam/transforms/core.py", line 1544, in <lambda>
    wrapper = lambda x: [fn(x)]
  File "/Users/alxrsngrtn/homebrew/Caskroom/miniconda/base/lib/python3.8/site-packages/apache_beam/transforms/core.py", line 2385, in <lambda>
    return pcoll | Map(lambda x: (self._key_func()(x), x)) | GroupByKey()
  File "/Users/alxrsngrtn/Github/weather-tools/weather_dl/download_pipeline/pipeline.py", line 86, in subsection_and_request
    subsection = t.cast(str, it.get('parameters', {}).get('__subsection__', 'default'))
NameError: name 't' is not defined [while running 'GroupBy request limits/Map(<lambda at core.py:2385>)'

Troubleshooting this bug is documented here: https://cloud.google.com/dataflow/docs/resources/faq#how_do_i_handle_nameerrors

PR #120 attempted to fix this; however, the error seems to have persisted.

pbattaglia added a commit to pbattaglia/weather-tools that referenced this issue Mar 26, 2022
Also fixed related bug where save_main_session arg for run() wasn't doing anything.
alxmrs pushed a commit that referenced this issue Mar 28, 2022
Also fixed related bug where save_main_session arg for run() wasn't doing anything.
@alxmrs
Copy link
Collaborator Author

alxmrs commented Mar 28, 2022

I've tested the fix in #124 both on a local and remote run. It seems to have fixed the issue.

@alxmrs alxmrs closed this as completed Mar 28, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant