Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

error in dataloding #47

Open
hlyyyyy opened this issue Jan 18, 2024 · 1 comment
Open

error in dataloding #47

hlyyyyy opened this issue Jan 18, 2024 · 1 comment

Comments

@hlyyyyy
Copy link

hlyyyyy commented Jan 18, 2024

My code:

import numpy as np
import mediapy
from tqdm import tqdm
import dataclasses

from waymax import config as _config
from waymax import dataloader
from waymax import datatypes
from waymax import visualization

config = _config.DatasetConfig(path ='/ssd2/waymo_open_dataset_motion_v_1_2_0/uncompressed/tf_example/testing/testing_tfexample.tfrecord@150',
    max_num_rg_points=30000,
    data_format=_config.DataFormat.TFRECORD,
    max_num_objects=32)

data_iter = dataloader.simulator_state_generator(config=config)
scenario = next(data_iter)

Error:

CUDA backend failed to initialize: Unable to load CUDA. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)
2024-01-18 13:29:21.716915: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.717835: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.718194: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.718219: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.718505: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.719481: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.720531: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.720556: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.720603: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.720632: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.720665: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.720695: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.720736: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.720842: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.721255: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.721299: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
Traceback (most recent call last):
  File "/ssd1/LightZero/zoo/waymax/envs/test.py", line 17, in <module>
    scenario = next(data_iter)
  File "/usr/local/lib/python3.10/dist-packages/waymo_waymax-0.1.0-py3.10.egg/waymax/dataloader/dataloader_utils.py", line 223, in get_data_generator
    for data in dataset:
  File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/data/ops/iterator_ops.py", line 809, in __next__
    return self._next_internal()
  File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/data/ops/iterator_ops.py", line 772, in _next_internal
    ret = gen_dataset_ops.iterator_get_next(
  File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/ops/gen_dataset_ops.py", line 3028, in iterator_get_next
    _ops.raise_from_not_ok_status(e, name)
  File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/framework/ops.py", line 5888, in raise_from_not_ok_status
    raise core._status_to_exception(e) from None  # pylint: disable=protected-access
tensorflow.python.framework.errors_impl.InvalidArgumentError: {{function_node __wrapped__IteratorGetNext_output_types_31_device_/job:localhost/replica:0/task:0/device:CPU:0}} Key: state/type.  Can't parse serialized Example.
         [[{{node ParseExample/ParseExampleV2}}]] [Op:IteratorGetNext] name: 
2024-01-18 13:29:21.722554: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.724826: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.724857: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.731598: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.734235: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.735144: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.735471: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.735531: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.736403: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.736785: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.739144: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.739335: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.739355: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.740241: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.740664: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.740718: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.740735: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.740828: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.740851: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.740896: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.740914: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.740931: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.740948: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.741291: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.741480: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.741523: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.741566: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.741588: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.741742: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.741769: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.741804: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.742204: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.742333: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.742428: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.742498: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.742512: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.742679: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.743450: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.743486: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.743510: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.743614: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.743883: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.744034: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.744415: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.744489: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.744557: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.744632: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.746093: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.748888: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.751985: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.755314: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.759302: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.763016: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.763520: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.769903: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.770057: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.770379: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.771022: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.774376: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.774463: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.776842: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.786968: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.

tensorflow version: 2.14.0-cpu
when I change the local file path to google cloud path "gs://xxx", the same error occurs, so it's not the problem of the downloaded file.

Do you have any suggestion about this err?
Thanks a lot.
@ryan-utopia
Copy link

maybe your downloaded TFrecord didn't align with the config you use
waymo_open_dataset_motion_v_1_2_0,
I just met this problem yesterday.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@hlyyyyy @ryan-utopia