Adds video support to color based augmentations #3580

JanuszL · 2021-12-17T14:07:45Z

extends brightness_contrast operator family to support sequences
extends color_twist operator family to support sequences
removes old_color_twist operator
makes Yiq2Rgb a runtime computed inversion of the Rgb2Yiq matrix in the color twist operator

Signed-off-by: Janusz Lisiecki jlisiecki@nvidia.com

Description

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Refactoring (Redesign of existing code that doesn't affect functionality)
Other (e.g. Documentation, Tests, Configuration)

What happened in this PR

Additional information

Affected modules and functionalities:
- brightness_contrast
- color_twist
- old_color_twist

Key points relevant for the review:
- NA

Checklist

Tests

Documentation

DALI team only

Requirements

Implements new requirements
Affects existing requirements
N/A

REQ IDs: CTSTOPS.04, BRICON.05

JIRA TASK: DALI-2509

mzient · 2021-12-17T14:10:02Z

dali/kernels/imgproc/pointwise/multiply_add.h

+    int num_channels = in.shape[2];
+    int image_width = in.shape[1];
+    int image_hight = in.shape[0];


Suggested change

int num_channels = in.shape[2];

int image_width = in.shape[1];

int image_hight = in.shape[0];

int num_channels = in.shape[ndims-1];

int image_width = in.shape[ndims-2];

int image_height = in.shape[ndims-3];

mzient · 2021-12-17T14:10:33Z

dali/kernels/imgproc/pointwise/multiply_add.h

+      num_channels = in.shape[3];
+      image_width = in.shape[2];
+      image_hight = in.shape[1];


Suggested change

num_channels = in.shape[3];

image_width = in.shape[2];

image_hight = in.shape[1];

mzient · 2021-12-17T14:13:21Z

dali/kernels/imgproc/pointwise/multiply_add.h

+      num_channels = in.shape[3];
+      image_width = in.shape[2];
+      image_hight = in.shape[1];
+      // wer cannot use .z as it is field, [] is resolved in the runtime


In principle, unless we include this file in a .cu unit, we could use .z.

mzient · 2021-12-17T14:16:23Z

dali/kernels/imgproc/pointwise/multiply_add.h

+    auto *row = in.data + adjusted_roi.lo.y * row_stride + img_start * row_stride * image_hight;
+    for (int z = img_start; z < img_end; z++) {
+      for (int y = adjusted_roi.lo.y; y < adjusted_roi.hi.y; y++) {
+        for (int xc = adjusted_roi.lo.x * num_channels; xc < adjusted_roi.hi.x * num_channels; xc++)
+          *ptr++ = ConvertSat<OutputType>(row[xc] * multiplier + addend);
+        row += row_stride;
+      }
+      row += img_stride;


Suggested change

auto *row = in.data + adjusted_roi.lo.y * row_stride + img_start * row_stride * image_hight;

for (int z = img_start; z < img_end; z++) {

for (int y = adjusted_roi.lo.y; y < adjusted_roi.hi.y; y++) {

for (int xc = adjusted_roi.lo.x * num_channels; xc < adjusted_roi.hi.x * num_channels; xc++)

*ptr++ = ConvertSat<OutputType>(row[xc] * multiplier + addend);

row += row_stride;

}

row += img_stride;

auto *img = in.data + adjusted_roi.lo.y * row_stride + img_start * row_stride * image_hight;

for (int z = img_start; z < img_end; z++) {

auto *row = img;

for (int y = adjusted_roi.lo.y; y < adjusted_roi.hi.y; y++) {

for (int xc = adjusted_roi.lo.x * num_channels; xc < adjusted_roi.hi.x * num_channels; xc++)

*ptr++ = ConvertSat<OutputType>(row[xc] * multiplier + addend);

row += row_stride;

}

img += img_stride;

mzient · 2021-12-17T14:19:25Z

dali/test/python/test_operator_brightness_contrast.py

  batch_size=32
  n_iters = 8
-  ri1 = RandomDataIterator(batch_size, shape=(128, 32, 3), dtype=dali_type_to_np(inp_dtype))
-  ri2 = RandomDataIterator(batch_size, shape=(128, 32, 3), dtype=dali_type_to_np(inp_dtype))
+  shape = (128, 32, 3) if not is_video else (random.randint(2-5), 128, 32, 3)


Suggested change

shape = (128, 32, 3) if not is_video else (random.randint(2-5), 128, 32, 3)

shape = (128, 32, 3) if not is_video else (random.randint(2, 5), 128, 32, 3)

mzient · 2021-12-17T16:06:28Z

dali/operators/image/color/brightness_contrast.cc

+                using Kernel = TheKernel<OutputType, InputType>;
+                TensorListShape<> sh = input.shape();
+                auto num_dims = sh.sample_dim();
+                int frame_num = num_dims == 3 ? 1 : sh[0][0];


Suggested change

int frame_num = num_dims == 3 ? 1 : sh[0][0];

int num_frames = num_dims == 3 ? 1 : sh[0][0];

frame_num reads too much as "frame number" which is ambiguous

mzient · 2021-12-17T16:10:19Z

dali/operators/image/color/brightness_contrast.cc

+                                                                      brightness_[sample_id],
+                                                                      brightness_shift_[sample_id],
+                                                                      contrast_[sample_id]);
+                            kernel_manager_.Run<Kernel>(thread_id, sample_id, ctx, tvout, tvin,


Technically, this is wrong. Now we've multipled the number of samples, so it's sum(num_frames[i] for i in range(num_samples). Since the kernel is stateless, we can stop pretending that there are multiple samples and just pass 0 as sample_id all the time. It will be less confusing.

JanuszL · 2021-12-18T03:04:54Z

!build

dali-automaton · 2021-12-18T03:12:31Z

CI MESSAGE: [3617555]: BUILD STARTED

dali-automaton · 2021-12-18T04:03:17Z

CI MESSAGE: [3617555]: BUILD FAILED

JanuszL · 2021-12-18T04:09:31Z

!build

dali-automaton · 2021-12-18T04:11:58Z

CI MESSAGE: [3617736]: BUILD STARTED

dali-automaton · 2021-12-18T05:23:33Z

CI MESSAGE: [3617736]: BUILD FAILED

- extends brightness_contrast operator family - extends hue operator Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com>

Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com>

Signed-off-by: Joaquin Anton <janton@nvidia.com>

jantonguirao · 2022-01-04T14:16:52Z

!build

dali-automaton · 2022-01-04T14:22:00Z

CI MESSAGE: [3694441]: BUILD STARTED

dali-automaton · 2022-01-04T14:44:57Z

CI MESSAGE: [3694441]: BUILD FAILED

Signed-off-by: Joaquin Anton <janton@nvidia.com>

jantonguirao · 2022-01-04T15:15:18Z

!build

dali-automaton · 2022-01-04T15:21:10Z

CI MESSAGE: [3694673]: BUILD STARTED

dali-automaton · 2022-01-04T15:44:20Z

CI MESSAGE: [3694673]: BUILD FAILED

Signed-off-by: Joaquin Anton <janton@nvidia.com>

jantonguirao · 2022-01-04T15:53:19Z

!build

dali-automaton · 2022-01-04T15:56:45Z

CI MESSAGE: [3694778]: BUILD STARTED

dali-automaton · 2022-01-04T17:09:00Z

CI MESSAGE: [3694778]: BUILD PASSED

Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com> Co-authored-by: Joaquin Anton <janton@nvidia.com>

JanuszL marked this pull request as draft December 17, 2021 14:07

mzient reviewed Dec 17, 2021

View reviewed changes

JanuszL force-pushed the video_contrast branch from 3d69f2c to c3f1371 Compare December 17, 2021 14:29

mzient reviewed Dec 17, 2021

View reviewed changes

JanuszL force-pushed the video_contrast branch 3 times, most recently from 25db40d to 0e0d2a7 Compare December 18, 2021 02:59

JanuszL force-pushed the video_contrast branch from f7a1fe0 to dd72aa9 Compare December 18, 2021 04:09

JanuszL force-pushed the video_contrast branch from 7afdfe9 to 2adeaf8 Compare December 18, 2021 05:09

JanuszL force-pushed the video_contrast branch 2 times, most recently from 07da4fd to b963aae Compare December 20, 2021 18:21

JanuszL added 7 commits December 28, 2021 15:06

Adds video support to color based augmentations

99acbd0

- extends brightness_contrast operator family - extends hue operator Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com>

Ver 2

ad8d664

Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com>

Improved

b0f8134

Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com>

Color

0f8997a

Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com>

Impr

66add0e

Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com>

Formating

1f448b5

Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com>

Tests

b17c733

Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com>

JanuszL force-pushed the video_contrast branch from b963aae to d7194d5 Compare December 28, 2021 14:07

Simplify code and address code review issues

5cd4cce

Signed-off-by: Joaquin Anton <janton@nvidia.com>

fix clang build

24aa18b

Signed-off-by: Joaquin Anton <janton@nvidia.com>

Remove unnecessary variables

3f616d9

Signed-off-by: Joaquin Anton <janton@nvidia.com>

jantonguirao approved these changes Jan 4, 2022

View reviewed changes

jantonguirao merged commit c4c2e32 into NVIDIA:main Jan 4, 2022

cyyever pushed a commit to cyyever/DALI that referenced this pull request Jan 23, 2022

Adds video support to color based augmentations (NVIDIA#3580)

b6c92d3

Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com> Co-authored-by: Joaquin Anton <janton@nvidia.com>

cyyever pushed a commit to cyyever/DALI that referenced this pull request Jan 23, 2022

Adds video support to color based augmentations (NVIDIA#3580)

8648797

Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com> Co-authored-by: Joaquin Anton <janton@nvidia.com>

cyyever pushed a commit to cyyever/DALI that referenced this pull request Jan 23, 2022

Adds video support to color based augmentations (NVIDIA#3580)

ed1f37c

Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com> Co-authored-by: Joaquin Anton <janton@nvidia.com>

cyyever pushed a commit to cyyever/DALI that referenced this pull request Jan 23, 2022

Adds video support to color based augmentations (NVIDIA#3580)

c0f06c5

Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com> Co-authored-by: Joaquin Anton <janton@nvidia.com>

cyyever pushed a commit to cyyever/DALI that referenced this pull request Jan 23, 2022

Adds video support to color based augmentations (NVIDIA#3580)

18973f4

Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com> Co-authored-by: Joaquin Anton <janton@nvidia.com>

cyyever pushed a commit to cyyever/DALI that referenced this pull request Jan 23, 2022

Adds video support to color based augmentations (NVIDIA#3580)

02e932b

Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com> Co-authored-by: Joaquin Anton <janton@nvidia.com>

cyyever pushed a commit to cyyever/DALI that referenced this pull request Jan 23, 2022

Adds video support to color based augmentations (NVIDIA#3580)

8215ba9

Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com> Co-authored-by: Joaquin Anton <janton@nvidia.com>

cyyever pushed a commit to cyyever/DALI that referenced this pull request Jan 23, 2022

Adds video support to color based augmentations (NVIDIA#3580)

2b20c2f

Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com> Co-authored-by: Joaquin Anton <janton@nvidia.com>

cyyever pushed a commit to cyyever/DALI that referenced this pull request Jan 23, 2022

Adds video support to color based augmentations (NVIDIA#3580)

5f0b6f9

Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com> Co-authored-by: Joaquin Anton <janton@nvidia.com>

cyyever pushed a commit to cyyever/DALI that referenced this pull request Jan 23, 2022

Adds video support to color based augmentations (NVIDIA#3580)

eca030c

Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com> Co-authored-by: Joaquin Anton <janton@nvidia.com>

cyyever pushed a commit to cyyever/DALI that referenced this pull request Jan 23, 2022

Adds video support to color based augmentations (NVIDIA#3580)

b79b5a3

Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com> Co-authored-by: Joaquin Anton <janton@nvidia.com>

cyyever pushed a commit to cyyever/DALI that referenced this pull request Jan 23, 2022

Adds video support to color based augmentations (NVIDIA#3580)

689cf65

Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com> Co-authored-by: Joaquin Anton <janton@nvidia.com>

cyyever pushed a commit to cyyever/DALI that referenced this pull request Jan 23, 2022

Adds video support to color based augmentations (NVIDIA#3580)

de3f4e0

Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com> Co-authored-by: Joaquin Anton <janton@nvidia.com>

cyyever pushed a commit to cyyever/DALI that referenced this pull request Feb 21, 2022

Adds video support to color based augmentations (NVIDIA#3580)

75e9143

Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com> Co-authored-by: Joaquin Anton <janton@nvidia.com>

cyyever pushed a commit to cyyever/DALI that referenced this pull request May 13, 2022

Adds video support to color based augmentations (NVIDIA#3580)

9d45a13

Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com> Co-authored-by: Joaquin Anton <janton@nvidia.com>

cyyever pushed a commit to cyyever/DALI that referenced this pull request Jun 7, 2022

Adds video support to color based augmentations (NVIDIA#3580)

0372400

Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com> Co-authored-by: Joaquin Anton <janton@nvidia.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adds video support to color based augmentations #3580

Adds video support to color based augmentations #3580

JanuszL commented Dec 17, 2021 •

edited by jantonguirao

Loading

mzient Dec 17, 2021 •

edited

Loading

mzient Dec 17, 2021

mzient Dec 17, 2021

mzient Dec 17, 2021 •

edited

Loading

mzient Dec 17, 2021

mzient Dec 17, 2021

JanuszL Dec 18, 2021

mzient Dec 17, 2021

JanuszL Dec 18, 2021

JanuszL commented Dec 18, 2021

dali-automaton commented Dec 18, 2021

dali-automaton commented Dec 18, 2021

JanuszL commented Dec 18, 2021

dali-automaton commented Dec 18, 2021

dali-automaton commented Dec 18, 2021

jantonguirao commented Jan 4, 2022

dali-automaton commented Jan 4, 2022

dali-automaton commented Jan 4, 2022

jantonguirao commented Jan 4, 2022

dali-automaton commented Jan 4, 2022

dali-automaton commented Jan 4, 2022

jantonguirao commented Jan 4, 2022

dali-automaton commented Jan 4, 2022

dali-automaton commented Jan 4, 2022

	num_channels = in.shape[3];
	image_width = in.shape[2];
	image_hight = in.shape[1];

	shape = (128, 32, 3) if not is_video else (random.randint(2-5), 128, 32, 3)
	shape = (128, 32, 3) if not is_video else (random.randint(2, 5), 128, 32, 3)

	int frame_num = num_dims == 3 ? 1 : sh[0][0];
	int num_frames = num_dims == 3 ? 1 : sh[0][0];

Adds video support to color based augmentations #3580

Adds video support to color based augmentations #3580

Conversation

JanuszL commented Dec 17, 2021 • edited by jantonguirao Loading

Description

What happened in this PR

Additional information

Checklist

Tests

Documentation

DALI team only

Requirements

mzient Dec 17, 2021 • edited Loading

Choose a reason for hiding this comment

mzient Dec 17, 2021

Choose a reason for hiding this comment

mzient Dec 17, 2021

Choose a reason for hiding this comment

mzient Dec 17, 2021 • edited Loading

Choose a reason for hiding this comment

mzient Dec 17, 2021

Choose a reason for hiding this comment

mzient Dec 17, 2021

Choose a reason for hiding this comment

JanuszL Dec 18, 2021

Choose a reason for hiding this comment

mzient Dec 17, 2021

Choose a reason for hiding this comment

JanuszL Dec 18, 2021

Choose a reason for hiding this comment

JanuszL commented Dec 18, 2021

dali-automaton commented Dec 18, 2021

dali-automaton commented Dec 18, 2021

JanuszL commented Dec 18, 2021

dali-automaton commented Dec 18, 2021

dali-automaton commented Dec 18, 2021

jantonguirao commented Jan 4, 2022

dali-automaton commented Jan 4, 2022

dali-automaton commented Jan 4, 2022

jantonguirao commented Jan 4, 2022

dali-automaton commented Jan 4, 2022

dali-automaton commented Jan 4, 2022

jantonguirao commented Jan 4, 2022

dali-automaton commented Jan 4, 2022

dali-automaton commented Jan 4, 2022

JanuszL commented Dec 17, 2021 •

edited by jantonguirao

Loading

mzient Dec 17, 2021 •

edited

Loading

mzient Dec 17, 2021 •

edited

Loading