Preprocessing issue #8

karanveersingh5623 · 2021-12-10T02:13:25Z

I tried running with all requirements , it keeps on reading the records but when it tries to write down .....here it fails , please help me to proceed further

root@6f4483fc53c1:/app/waymo-adas-main/waymo-motion-prediction-2021# python3 prerender.py --data /app/waymo-adas-main/waymo-dataset/original/validation/ --out /app/waymo-adas-main/data/train1
False
Namespace(data='/app/waymo-adas-main/waymo-dataset/original/validation/', each=0, n_jobs=20, n_shards=8, no_valid=False, out='/app/waymo-adas-main/data/train1', use_vectorize=False)
1215it [00:28, 42.50it/s]
  0%|                                                                                                                                                                                   | 0/1215 [00:00<?, ?it/s]
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "prerender.py", line 731, in merge
    parsed = tf.io.parse_single_example(data, features_description)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/execute.py", line 58, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position 40: invalid start byte
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "prerender.py", line 836, in <module>
    main()
  File "prerender.py", line 832, in main
    r.get()
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 771, in get
    raise self._value
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position 40: invalid start byte

The text was updated successfully, but these errors were encountered:

kbrodt · 2021-12-10T02:28:04Z

Try to remove multiprocessing and debug

karanveersingh5623 · 2021-12-10T06:22:04Z

Changed the code , got the same error

#    p = multiprocessing.Pool(args.n_jobs)
    proc_id = 0
#    res = []
    for data in tqdm(dataset.as_numpy_iterator()):
#        print(data)
        proc_id += 1
        kwds=dict(data=data, proc_id=proc_id, validate=not args.no_valid, out_dir=args.out, use_vectorize=args.use_vectorize)
#        print(kwds)
        merge(**kwds)

#    for r in tqdm(res):
#        r.get()


if __name__ == "__main__":
    main()

OUTPUT

root@6f4483fc53c1:/app/waymo-adas-main/waymo-motion-prediction-2021# python3 prerender1.py --data /app/waymo-adas-main/waymo-dataset/original/train/ --out /app/waymo-adas-main/data/train1
False
Namespace(data='/app/waymo-adas-main/waymo-dataset/original/train/', each=0, n_jobs=1, n_shards=8, no_valid=False, out='/app/waymo-adas-main/data/train1', use_vectorize=False)
<ShardDataset shapes: (), types: tf.string>
0it [00:00, ?it/s]
Traceback (most recent call last):
  File "prerender1.py", line 829, in <module>
    main()
  File "prerender1.py", line 822, in main
    merge(**kwds)
  File "prerender1.py", line 731, in merge
    parsed = tf.io.parse_single_example(data, features_description)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/execute.py", line 58, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position 40: invalid start byte

kbrodt · 2021-12-10T17:42:59Z

It's a problem of tensorflow. Are data and features_description in parsed = tf.io.parse_single_example(data, features_description) correct?

karanveersingh5623 · 2021-12-11T07:30:10Z

Thanks for reply
I just used one tf record with above code , it also failed . So data looks fine
Coming to features description , its defined in the script , so I couldn't figure out .
What else I can try ?

features_description = {}
features_description.update(roadgraph_features)
features_description.update(state_features)
features_description.update(traffic_light_features)

kbrodt · 2021-12-12T04:50:36Z

I've tested on my machine and there were no errors.

The tensorflow version:

tensorflow==2.5.0
tensorflow-addons==0.10.0
tensorflow-estimator==1.13.0
tensorflow-gpu==2.0.0

karanveersingh5623 · 2021-12-13T05:03:26Z

Same Error , below are the versions information . I am using Tensorflow 2.5.0 GPU docker image .

root@d9fa28761b2f:/app/waymo-adas-main/waymo-motion-prediction-2021# python3 prerender1.py --data /app/waymo-adas-main/waymo-dataset/original/train/ --out /app/waymo-adas-main/data/train1
Namespace(data='/app/waymo-adas-main/waymo-dataset/original/train/', each=0, n_jobs=1, n_shards=8, no_valid=False, out='/app/waymo-adas-main/data/train1', use_vectorize=False)
2021-12-13 04:53:46.102202: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2021-12-13 04:53:46.102231: E tensorflow/stream_executor/cuda/cuda_driver.cc:318] failed call to cuInit: UNKNOWN ERROR (34)
2021-12-13 04:53:46.102255: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:163] no NVIDIA GPU device is present: /dev/nvidia0 does not exist
2021-12-13 04:53:46.102490: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 AVX512F FMA
2021-12-13 04:53:46.141731: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2100000000 Hz
2021-12-13 04:53:46.157862: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x476d4d0 executing computations on platform Host. Devices:
2021-12-13 04:53:46.157922: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Host, Default Version
<ShardDataset shapes: (), types: tf.string>
0it [00:00, ?it/s]2021-12-13 04:53:46.248422: W tensorflow/core/framework/op_kernel.cc:1622] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Could not parse example input, value: '

&17642771458376274038_2080_000_2100_009@9@E@`l-;@s?ֿfP?EH9#-3
0it [00:00, ?it/s]
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gen_parsing_ops.py", line 1155, in parse_single_example
    "dense_shapes", dense_shapes)
tensorflow.python.eager.core._FallbackException: Expecting None or sequence value for element0 of attr dense_shapes, got TensorShapeProto

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "prerender1.py", line 830, in <module>
    main()
  File "prerender1.py", line 823, in main
    merge(**kwds)
  File "prerender1.py", line 731, in merge
    parsed = tf.io.parse_single_example(data, features_description)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/parsing_ops.py", line 1063, in parse_single_example_v2_unoptimized
    return parse_single_example_v2(serialized, features, name)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/parsing_ops.py", line 2093, in parse_single_example_v2
    dense_defaults, dense_shapes, name)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/parsing_ops.py", line 2210, in _parse_single_example_v2_raw
    name=name)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gen_parsing_ops.py", line 1164, in parse_single_example
    ctx=_ctx)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gen_parsing_ops.py", line 1260, in parse_single_example_eager_fallback
    attrs=_attrs, ctx=_ctx, name=name)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/execute.py", line 61, in quick_execute
    num_outputs)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position 40: invalid start byte

Packages Versions

root@d9fa28761b2f:/app/waymo-adas-main/waymo-motion-prediction-2021# python3 -m pip show tensorflow
Name: tensorflow
Version: 2.5.0
Summary: TensorFlow is an open source machine learning framework for everyone.
Home-page: https://www.tensorflow.org/
Author: Google Inc.
Author-email: packages@tensorflow.org
License: Apache 2.0
Location: /usr/local/lib/python3.6/dist-packages
Requires: absl-py, astunparse, flatbuffers, gast, google-pasta, grpcio, h5py, keras-nightly, keras-preprocessing, numpy, opt-einsum, protobuf, six, tensorboard, tensorflow-estimator, termcolor, typing-extensio
ns, wheel, wrapt
Required-by:
root@d9fa28761b2f:/app/waymo-adas-main/waymo-motion-prediction-2021#
root@d9fa28761b2f:/app/waymo-adas-main/waymo-motion-prediction-2021#
root@d9fa28761b2f:/app/waymo-adas-main/waymo-motion-prediction-2021# python3 -m pip show tensorflow-addons
Name: tensorflow-addons
Version: 0.10.0
Summary: TensorFlow Addons.
Home-page: UNKNOWN
Author: Google Inc.
Author-email: opensource@google.com
License: Apache 2.0
Location: /usr/local/lib/python3.6/dist-packages
Requires: typeguard
Required-by:
root@d9fa28761b2f:/app/waymo-adas-main/waymo-motion-prediction-2021# python3 -m pip show tensorflow-gpu
Name: tensorflow-gpu
Version: 2.0.0
Summary: TensorFlow is an open source machine learning framework for everyone.
Home-page: https://www.tensorflow.org/
Author: Google Inc.
Author-email: packages@tensorflow.org
License: Apache 2.0
Location: /usr/local/lib/python3.6/dist-packages
Requires: absl-py, astor, gast, google-pasta, grpcio, keras-applications, keras-preprocessing, numpy, opt-einsum, protobuf, six, tensorboard, tensorflow-estimator, termcolor, wheel, wrapt
Required-by:
root@d9fa28761b2f:/app/waymo-adas-main/waymo-motion-prediction-2021# python3 -m pip show tensorflow-estimator
Name: tensorflow-estimator
Version: 2.0.1
Summary: TensorFlow Estimator.
Home-page: https://www.tensorflow.org/
Author: Google Inc.
Author-email: UNKNOWN
License: Apache 2.0
Location: /usr/local/lib/python3.6/dist-packages
Requires:
Required-by: tensorflow, tensorflow-gpu

karanveersingh5623 · 2021-12-13T05:20:32Z

if you have any docker image , can you share ?

kbrodt · 2021-12-13T05:37:25Z

Did you try with another samples or redownload new uncompressed/tf_example/{training,validation,testing}?

No docker.

karanveersingh5623 · 2021-12-13T12:13:46Z

yea , I thought the same ….. so downloaded just 2 TF records again and tried …..same error .
Reading TF records runs fine but when the time comes to write the records , there it fails...…..
Don't know how can I proceed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Preprocessing issue #8

Preprocessing issue #8

karanveersingh5623 commented Dec 10, 2021 •

edited

kbrodt commented Dec 10, 2021

karanveersingh5623 commented Dec 10, 2021 •

edited

kbrodt commented Dec 10, 2021

karanveersingh5623 commented Dec 11, 2021

kbrodt commented Dec 12, 2021

karanveersingh5623 commented Dec 13, 2021

karanveersingh5623 commented Dec 13, 2021

kbrodt commented Dec 13, 2021

karanveersingh5623 commented Dec 13, 2021

Preprocessing issue #8

Preprocessing issue #8

Comments

karanveersingh5623 commented Dec 10, 2021 • edited

kbrodt commented Dec 10, 2021

karanveersingh5623 commented Dec 10, 2021 • edited

kbrodt commented Dec 10, 2021

karanveersingh5623 commented Dec 11, 2021

kbrodt commented Dec 12, 2021

karanveersingh5623 commented Dec 13, 2021

karanveersingh5623 commented Dec 13, 2021

kbrodt commented Dec 13, 2021

karanveersingh5623 commented Dec 13, 2021

karanveersingh5623 commented Dec 10, 2021 •

edited

karanveersingh5623 commented Dec 10, 2021 •

edited