pyarrow.lib.ArrowInvalid: Not an Arrow file

[psrimanreddy]

While running the filetuning command for vqav2 using following command:

python run.py with data_root=/data2/dsets/dataset num_gpus=8 num_nodes=1 task_finetune_vqa_randaug per_gpu_batchsize=64 load_path="weights/vilt_200k_mlm_itm.ckpt"

I'm encountering the following error:

WARNING - root - Changed type of config entry "max_steps" from int to NoneType
WARNING - ViLT - No observers have been added to this run
INFO - ViLT - Running command 'main'
INFO - ViLT - Started
Global seed set to 0
INFO - lightning - Global seed set to 0
GPU available: True, used: True
INFO - lightning - GPU available: True, used: True
TPU available: None, using: 0 TPU cores
INFO - lightning - TPU available: None, using: 0 TPU cores
Using environment variable NODE_RANK for node rank (0).
INFO - lightning - Using environment variable NODE_RANK for node rank (0).
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
INFO - lightning - LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
Using native 16bit precision.
INFO - lightning - Using native 16bit precision.
Missing logger folder: result/finetune_vqa_randaug_seed0_from_vilt_200k_mlm_itm
WARNING - lightning - Missing logger folder: result/finetune_vqa_randaug_seed0_from_vilt_200k_mlm_itm
Global seed set to 0
INFO - lightning - Global seed set to 0
initializing ddp: GLOBAL_RANK: 0, MEMBER: 1/1
INFO - lightning - initializing ddp: GLOBAL_RANK: 0, MEMBER: 1/1
INFO - torch.distributed.distributed_c10d - Added key: store_based_barrier_key:1 to store for rank: 0
INFO - torch.distributed.distributed_c10d - Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
ERROR - ViLT - Failed after 0:00:06!
Traceback (most recent call last):
File "/home/imt2018525/.local/lib/python3.8/site-packages/sacred/experiment.py", line 312, in run_commandline
return self.run(
File "/home/imt2018525/.local/lib/python3.8/site-packages/sacred/experiment.py", line 276, in run
run()
File "/home/imt2018525/.local/lib/python3.8/site-packages/sacred/run.py", line 238, in call
self.result = self.main_function(*args)
File "/home/imt2018525/.local/lib/python3.8/site-packages/sacred/config/captured_function.py", line 42, in captured_function
result = wrapped(*args, **kwargs)
File "run.py", line 71, in main
trainer.fit(model, datamodule=dm)
File "/home/imt2018525/.local/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 473, in fit
results = self.accelerator_backend.train()
File "/home/imt2018525/.local/lib/python3.8/site-packages/pytorch_lightning/accelerators/ddp_accelerator.py", line 152, in train
results = self.ddp_train(process_idx=self.task_idx, model=model)
File "/home/imt2018525/.local/lib/python3.8/site-packages/pytorch_lightning/accelerators/ddp_accelerator.py", line 268, in ddp_train
self.trainer.call_setup_hook(model)
File "/home/imt2018525/.local/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 859, in call_setup_hook
self.datamodule.setup(stage_name)
File "/home/imt2018525/.local/lib/python3.8/site-packages/pytorch_lightning/core/datamodule.py", line 92, in wrapped_fn
return fn(*args, **kwargs)
File "/home/imt2018525/ViLT/vilt/datamodules/multitask_datamodule.py", line 34, in setup
dm.setup(stage)
File "/home/imt2018525/.local/lib/python3.8/site-packages/pytorch_lightning/core/datamodule.py", line 92, in wrapped_fn
return fn(*args, **kwargs)
File "/home/imt2018525/ViLT/vilt/datamodules/vqav2_datamodule.py", line 19, in setup
super().setup(stage)
File "/home/imt2018525/ViLT/vilt/datamodules/datamodule_base.py", line 138, in setup
self.set_val_dataset()
File "/home/imt2018525/ViLT/vilt/datamodules/datamodule_base.py", line 88, in set_val_dataset
self.val_dataset = self.dataset_cls(
File "/home/imt2018525/ViLT/vilt/datasets/vqav2_dataset.py", line 16, in init
super().init(
File "/home/imt2018525/ViLT/vilt/datasets/base_dataset.py", line 43, in init
tables = [
File "/home/imt2018525/ViLT/vilt/datasets/base_dataset.py", line 44, in
pa.ipc.RecordBatchFileReader(
File "/home/imt2018525/.local/lib/python3.8/site-packages/pyarrow/ipc.py", line 94, in init
self._open(source, footer_offset=footer_offset)
File "pyarrow/ipc.pxi", line 624, in pyarrow.lib._RecordBatchFileReader._open
File "pyarrow/error.pxi", line 122, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 84, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Not an Arrow file

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "run.py", line 11, in
def main(_config):
File "/home/imt2018525/.local/lib/python3.8/site-packages/sacred/experiment.py", line 190, in automain
self.run_commandline()
File "/home/imt2018525/.local/lib/python3.8/site-packages/sacred/experiment.py", line 347, in run_commandline
print_filtered_stacktrace()
File "/home/imt2018525/.local/lib/python3.8/site-packages/sacred/utils.py", line 493, in print_filtered_stacktrace
print(format_filtered_stacktrace(filter_traceback), file=sys.stderr)
File "/home/imt2018525/.local/lib/python3.8/site-packages/sacred/utils.py", line 528, in format_filtered_stacktrace
return "".join(filtered_traceback_format(tb_exception))
File "/home/imt2018525/.local/lib/python3.8/site-packages/sacred/utils.py", line 568, in filtered_traceback_format
current_tb = tb_exception.exc_traceback
AttributeError: 'TracebackException' object has no attribute 'exc_traceback'

I am not sure it's the issue with the pyarrow version. Can someone help me resolve this error? Thanks in advance.

[ThompsonISAT]

Hi I have encounter the same problem. I was wondering how we could fix it?
Thank you so much!

[psrimanreddy]

My issue got resolved when I generated the arrow files again. Probably there was some issue with the earlier arrow files. Try re-generating them using make_arrow function mentioned. Regards Sriman Reddy

…

________________________________ From: ThompsonISAT ***@***.***> Sent: Friday, January 13, 2023 9:36:46 AM To: dandelin/ViLT ***@***.***> Cc: IMT2018525 Sriman reddy Pingili ***@***.***>; Author ***@***.***> Subject: Re: [dandelin/ViLT] pyarrow.lib.ArrowInvalid: Not an Arrow file (Issue #78) Hi I have encounter the same problem. I was wondering how we could fix it? Thank you so much! — Reply to this email directly, view it on GitHub<#78 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AYEEAHTIFENLKFHH7KF3EHTWSDIFNANCNFSM6AAAAAATI244KM>. You are receiving this because you authored the thread.Message ID: ***@***.***>