Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

read_parquet failed to read the dataset created by to_parquet with index = False. #6936

Closed
anmyachev opened this issue Feb 16, 2024 · 0 comments · Fixed by #6937
Closed

read_parquet failed to read the dataset created by to_parquet with index = False. #6936

anmyachev opened this issue Feb 16, 2024 · 0 comments · Fixed by #6937
Assignees
@anmyachev
Labels
bug 🦗 Something isn't working P1 Important tasks that we should complete soon pandas.io

Comments

@anmyachev
Copy link
Collaborator

From asv benchmark: io.parquet.TimeReadParquet.time_read_parquet. Modin 0.27.0

Traceback:

2024-02-16T12:19:04.9732956Z                For parameters: [10000, 10]
2024-02-16T12:19:04.9733967Z                2024-02-16 12:19:01,119	INFO worker.py:1724 -- Started a local Ray instance.
2024-02-16T12:19:04.9735230Z                UserWarning: Distributing <class 'list'> object. This may take some time.
2024-02-16T12:19:04.9737142Z                FutureWarning: Passing 'use_legacy_dataset' is deprecated as of pyarrow 15.0.0 and will be removed in a future version.
2024-02-16T12:19:04.9738395Z                Traceback (most recent call last):
2024-02-16T12:19:04.9739696Z                  File "/usr/share/miniconda3/lib/python3.10/site-packages/asv/benchmark.py", line 1435, in <module>
2024-02-16T12:19:04.9740752Z                    main()
2024-02-16T12:19:04.9741807Z                  File "/usr/share/miniconda3/lib/python3.10/site-packages/asv/benchmark.py", line 1428, in main
2024-02-16T12:19:04.9742850Z                    commands[mode](args)
2024-02-16T12:19:04.9744075Z                  File "/usr/share/miniconda3/lib/python3.10/site-packages/asv/benchmark.py", line 1167, in main_run
2024-02-16T12:19:04.9745169Z                    result = benchmark.do_run()
2024-02-16T12:19:04.9746396Z                  File "/usr/share/miniconda3/lib/python3.10/site-packages/asv/benchmark.py", line 573, in do_run
2024-02-16T12:19:04.9747507Z                    return self.run(*self._current_params)
2024-02-16T12:19:04.9748977Z                  File "/usr/share/miniconda3/lib/python3.10/site-packages/asv/benchmark.py", line 669, in run
2024-02-16T12:19:04.9750250Z                    samples, number = self.benchmark_timing(timer, min_repeat, max_repeat,
2024-02-16T12:19:04.9751832Z                  File "/usr/share/miniconda3/lib/python3.10/site-packages/asv/benchmark.py", line 736, in benchmark_timing
2024-02-16T12:19:04.9753020Z                    timing = timer.timeit(number)
2024-02-16T12:19:04.9754431Z                  File "/home/runner/work/modin/modin/asv_bench/.asv/env/1c925bb96e8284d2b96a9c0ca8d4675a/lib/python3.9/timeit.py", line 177, in timeit
2024-02-16T12:19:04.9755870Z                    timing = self.inner(it, self.timer)
2024-02-16T12:19:04.9756636Z                  File "<timeit-src>", line 6, in inner
2024-02-16T12:19:04.9757905Z                  File "/usr/share/miniconda3/lib/python3.10/site-packages/asv/benchmark.py", line 627, in <lambda>
2024-02-16T12:19:04.9759001Z                    func = lambda: self.func(*param)
2024-02-16T12:19:04.9760171Z                  File "/home/runner/work/modin/modin/asv_bench/benchmarks/io/parquet.py", line 48, in time_read_parquet
2024-02-16T12:19:04.9761287Z                    IMPL.read_parquet(
2024-02-16T12:19:04.9763061Z                  File "/home/runner/work/modin/modin/asv_bench/.asv/env/1c925bb96e8284d2b96a9c0ca8d4675a/lib/python3.9/site-packages/modin/utils.py", line 484, in wrapped
2024-02-16T12:19:04.9764929Z                    return func(*params.args, **params.kwargs)
2024-02-16T12:19:04.9767027Z                  File "/home/runner/work/modin/modin/asv_bench/.asv/env/1c925bb96e8284d2b96a9c0ca8d4675a/lib/python3.9/site-packages/modin/logging/logger_decorator.py", line 129, in run_and_log
2024-02-16T12:19:04.9768829Z                    return obj(*args, **kwargs)
2024-02-16T12:19:04.9770732Z                  File "/home/runner/work/modin/modin/asv_bench/.asv/env/1c925bb96e8284d2b96a9c0ca8d4675a/lib/python3.9/site-packages/modin/pandas/io.py", line 325, in read_parquet
2024-02-16T12:19:04.9772491Z                    query_compiler=FactoryDispatcher.read_parquet(
2024-02-16T12:19:04.9774920Z                  File "/home/runner/work/modin/modin/asv_bench/.asv/env/1c925bb96e8284d2b96a9c0ca8d4675a/lib/python3.9/site-packages/modin/core/execution/dispatching/factories/dispatcher.py", line 187, in read_parquet
2024-02-16T12:19:04.9777046Z                    return cls.get_factory()._read_parquet(**kwargs)
2024-02-16T12:19:04.9779468Z                  File "/home/runner/work/modin/modin/asv_bench/.asv/env/1c925bb96e8284d2b96a9c0ca8d4675a/lib/python3.9/site-packages/modin/core/execution/dispatching/factories/factories.py", line 212, in _read_parquet
2024-02-16T12:19:04.9781565Z                    return cls.io_cls.read_parquet(**kwargs)
2024-02-16T12:19:04.9783909Z                  File "/home/runner/work/modin/modin/asv_bench/.asv/env/1c925bb96e8284d2b96a9c0ca8d4675a/lib/python3.9/site-packages/modin/logging/logger_decorator.py", line 129, in run_and_log
2024-02-16T12:19:04.9785723Z                    return obj(*args, **kwargs)
2024-02-16T12:19:04.9787611Z                  File "/home/runner/work/modin/modin/asv_bench/.asv/env/1c925bb96e8284d2b96a9c0ca8d4675a/lib/python3.9/site-packages/modin/core/io/file_dispatcher.py", line 158, in read
2024-02-16T12:19:04.9789258Z                    query_compiler = cls._read(*args, **kwargs)
2024-02-16T12:19:04.9791203Z                  File "/home/runner/work/modin/modin/asv_bench/.asv/env/1c925bb96e8284d2b96a9c0ca8d4675a/lib/python3.9/site-packages/modin/logging/logger_decorator.py", line 129, in run_and_log
2024-02-16T12:19:04.9792924Z                    return obj(*args, **kwargs)
2024-02-16T12:19:04.9794991Z                  File "/home/runner/work/modin/modin/asv_bench/.asv/env/1c925bb96e8284d2b96a9c0ca8d4675a/lib/python3.9/site-packages/modin/core/io/column_stores/parquet_dispatcher.py", line 813, in _read
2024-02-16T12:19:04.9796788Z                    return cls.build_query_compiler(
2024-02-16T12:19:04.9798992Z                  File "/home/runner/work/modin/modin/asv_bench/.asv/env/1c925bb96e8284d2b96a9c0ca8d4675a/lib/python3.9/site-packages/modin/logging/logger_decorator.py", line 129, in run_and_log
2024-02-16T12:19:04.9800685Z                    return obj(*args, **kwargs)
2024-02-16T12:19:04.9802798Z                  File "/home/runner/work/modin/modin/asv_bench/.asv/env/1c925bb96e8284d2b96a9c0ca8d4675a/lib/python3.9/site-packages/modin/core/io/column_stores/parquet_dispatcher.py", line 693, in build_query_compiler
2024-02-16T12:19:04.9805125Z                    and dataset.pandas_metadata["column_indexes"][0]["numpy_type"] == "int64"
2024-02-16T12:19:04.9805991Z                IndexError: list index out of range
2024-02-16T12:19:04.9806631Z                asv: benchmark failed (exit status 1)
@anmyachev anmyachev added bug 🦗 Something isn't working pandas.io P1 Important tasks that we should complete soon labels Feb 16, 2024
anmyachev added a commit to anmyachev/modin that referenced this issue Feb 16, 2024
@anmyachev
FIX-modin-project#6936: Fix 'read_parquet' when dataset is created wi…
4a74380 
…th 'to_parquet' and 'index=False'

Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>
@anmyachev anmyachev self-assigned this Feb 16, 2024
anmyachev added a commit to anmyachev/modin that referenced this issue Feb 16, 2024
@anmyachev
FIX-modin-project#6936: Fix 'read_parquet' when dataset is created wi…
676514d 
…th 'to_parquet' and 'index=False'

Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>
This was referenced Feb 16, 2024
Merged
Merged
YarShev pushed a commit that referenced this issue Feb 16, 2024
@anmyachev
FIX-#6936: Fix 'read_parquet' when dataset is created with 'to_parque…
3f9a733 
…t' and 'index=False' (#6937)

Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@anmyachev anmyachev

Labels
bug 🦗 Something isn't working P1 Important tasks that we should complete soon pandas.io
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

FIX-#6936: Fix read_parquet when dataset is created with to_parquet and index=False anmyachev/modin
1 participant
@anmyachev