Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(WIP) Tfdsv4 carla mot #1907

Draft
wants to merge 6 commits into
base: tfdsv4
Choose a base branch
from

Conversation

jprokos26
Copy link
Contributor

No description provided.

@@ -31,6 +31,7 @@
from armory.art_experimental.attacks import patch
from armory.art_experimental.attacks.sweep import SweepAttack
from armory.datasets.generator import ArmoryDataGenerator
from armory.data.datasets import EvalGenerator # TODO: Remove before PR merge
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

EvalGenerator is seemingly never reached as armory runs fine without this import, but flake8 complains about this line without this import:

return EvalGenerator(dataset, num_eval_batches=1)

EvalGenerator should be completely removed from this file as suggested by #1836 (comment)

Comment on lines +171 to +222
def mot_array_to_coco(batch):
"""
Map from 3D array (batch_size x detections x 9) to extended coco format
of dimension (batch_size x frames x detections_per_frame)
NOTE: 'image_id' is given as the frame of a video, so is not unique
"""
if len(batch.shape) == 2:
not_batch = True
batch = tf.expand_dims(batch, axis=0)
elif len(batch.shape) == 3:
not_batch = False
else:
raise ValueError(f"batch.ndim {len(batch.shape)} is not in (2, 3)")

# output = tf.TensorArray(dtype=tf.float32, size=batch.shape[0], dynamic_size=False)
output = []
for i in range(batch.shape[0]):
array = batch[i]
# if not tf.math.greater(tf.shape(array)[0], 0):
if array.shape[0] == 0:
# no object detections
# output = output.write(i, [])
output.append([])
continue

# frames = tf.TensorArray(dtype=tf.float32, size=tf.shape(array)[0], dynamic_size=False)
frames = []
for detection in array:
frame = tf.lookup.StaticHashTable(
{
# TODO: should image_id include video number as well?
"image_id": tf.cast(tf.math.round(detection[0]), tf.int32),
"category_id": tf.cast(tf.math.round(detection[7]), tf.int32),
"bbox": tf.cast(detection[2:6], float),
"score": tf.cast(detection[6], float),
# The following are extended fields
"object_id": tf.cast(
tf.math.round(detection[1]), tf.int32
), # for a specific object across frames
"visibility": tf.cast(detection[8], float),
}
)
frames.append(frame)
# frames = frames.write(frames.size(), frame)
# output = output.write(i, frames)
output.append(frames)

if not_batch:
output = output[0]

raise NotImplementedError("This does not work yet")
return output
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function is translated from the linked function below, however tensorflow does not support storing dictionaries as elements of a tensor (whereas previously numpy had no problem with this). As such the return of this preprocessing function cannot be a tensor containing lists of dictionaries (which contain tensors as values) since this is not supported even with RaggedTensors. From this I only see two viable options:

  1. Keep the logic of this function as a preprocessing step and encode the key/value pairs of the dictionary using some encoding schema to be decoded during the next call
  2. Move this function to be applied in CarlaMOT in the next call
  3. Some tensorflow magical operations to get this to work without touching scenario code

Thoughts?

def mot_array_to_coco(batch):
"""
Map from 3D array (batch_size x detections x 9) to extended coco format
of dimension (batch_size x frames x detections_per_frame)
NOTE: 'image_id' is given as the frame of a video, so is not unique
"""
if batch.ndim == 2:
not_batch = True
batch = [batch]
elif batch.ndim == 3:
not_batch = False
else:
raise ValueError(f"batch.ndim {batch.ndim} is not in (2, 3)")
output = np.empty(len(batch), dtype=object)
for i, array in enumerate(batch):
if not len(array):
# no object detections
output.append([])
continue
frames = []
for detection in array:
frames.append(
{
# TODO: should image_id include video number as well?
"image_id": int(np.round(detection[0])),
"category_id": int(np.round(detection[7])),
"bbox": [float(x) for x in detection[2:6]],
"score": float(detection[6]),
# The following are extended fields
"object_id": int(
np.round(detection[1])
), # for a specific object across frames
"visibility": float(detection[8]),
}
)
output[i] = frames
if not_batch:
output = output[0]
return output

Copy link
Contributor Author

@jprokos26 jprokos26 Mar 30, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that the rest of the function works up until the NotImplementedError; after that point it either throws an error when trying to tf.convert_to_tensor(output) or when the function is mapped to the dataloader it complains that the output is not a valid return type (since it is a list of dictionaries).
Also the static hash table makes no difference compared to using a generic dictionary.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally (3) is best but I think (2) may be preferable to (1)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed; assuming you don't have a recommendation of how to proceed with (3) I will implement (2).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not, we've run into a similar issue with other datasets that has required modifying the scenario code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants