Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

BUG: OOM in ./join-rows #374

Closed
corneliusroemer opened this issue Dec 13, 2022 · 1 comment
Closed

BUG: OOM in ./join-rows #374

corneliusroemer opened this issue Dec 13, 2022 · 1 comment
Labels
bug Something isn't working

Comments

@corneliusroemer
Copy link
Member

Current Behavior

    459 [batch] [2022-12-12T20:50:46+01:00] Traceback (most recent call last):
    460 [batch] [2022-12-12T20:50:46+01:00]   File "/nextstrain/build/./bin/join-rows", line 34, in <module>
    461 [batch] [2022-12-12T20:50:46+01:00]     main()
    462 [batch] [2022-12-12T20:50:46+01:00]   File "/nextstrain/build/./bin/join-rows", line 21, in main
    463 [batch] [2022-12-12T20:50:46+01:00]     left = pd.read_csv(args.first_file, sep='\t', low_memory=False, na_filter = False)
    464 [batch] [2022-12-12T20:50:46+01:00]   File "/usr/local/lib/python3.10/site-packages/pandas/util/_decorators.py", line 211, in wrapper
    465 [batch] [2022-12-12T20:50:46+01:00]     return func(*args, **kwargs)
    466 [batch] [2022-12-12T20:50:46+01:00]   File "/usr/local/lib/python3.10/site-packages/pandas/util/_decorators.py", line 331, in wrapper
    467 [batch] [2022-12-12T20:50:46+01:00]     return func(*args, **kwargs)
    468 [batch] [2022-12-12T20:50:46+01:00]   File "/usr/local/lib/python3.10/site-packages/pandas/io/parsers/readers.py", line 950, in read>
    469 [batch] [2022-12-12T20:50:46+01:00]     return _read(filepath_or_buffer, kwds)
    470 [batch] [2022-12-12T20:50:46+01:00]   File "/usr/local/lib/python3.10/site-packages/pandas/io/parsers/readers.py", line 611, in _read
    471 [batch] [2022-12-12T20:50:46+01:00]     return parser.read(nrows)
    472 [batch] [2022-12-12T20:50:46+01:00]   File "/usr/local/lib/python3.10/site-packages/pandas/io/parsers/readers.py", line 1778, in read
    473 [batch] [2022-12-12T20:50:46+01:00]     ) = self._engine.read(  # type: ignore[attr-defined]
    474 [batch] [2022-12-12T20:50:46+01:00]   File "/usr/local/lib/python3.10/site-packages/pandas/io/parsers/c_parser_wrapper.py", line 235>
    475 [batch] [2022-12-12T20:50:46+01:00]     data = self._reader.read(nrows)
    476 [batch] [2022-12-12T20:50:46+01:00]   File "pandas/_libs/parsers.pyx", line 790, in pandas._libs.parsers.TextReader.read
    477 [batch] [2022-12-12T20:50:46+01:00]   File "pandas/_libs/parsers.pyx", line 883, in pandas._libs.parsers.TextReader._read_rows
    478 [batch] [2022-12-12T20:50:46+01:00]   File "pandas/_libs/parsers.pyx", line 1973, in pandas._libs.parsers.raise_parser_error
    479 [batch] [2022-12-12T20:50:46+01:00] pandas.errors.ParserError: Error tokenizing data. C error: out of memory
@corneliusroemer corneliusroemer added the bug Something isn't working label Dec 13, 2022
corneliusroemer added a commit that referenced this issue Dec 13, 2022
@corneliusroemer
Use tsv-uniq instead of nextclade ./join-rows
08162a2 
This addresses an OOM error caused by
pandas loading multiple datasets into memory
in multiple rules in parallel
Helps with #374
Testing: ran locally successfully
This was referenced Dec 14, 2022
Merged
Open
@joverlee521
Copy link
Contributor

Resolved by #377

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@corneliusroemer @joverlee521