Fix layout broadcasting arithm ops #4951

stiepan · 2023-07-20T13:25:06Z

Category:

Bug fix (non-breaking change which fixes an issue)

Description:

Without this change, it is possible to produce tensors whose layout does not match their dimensionality.

The arithm ops layout propagation works by picking just any non-empty layout and making sure that any other operands have either empty or equal layout. This leads to two issues:

layout of wrong length: if more ndim-ed tensor has no layout and the less ndim-ed tensor does, the output ends up with layout of incorrect lengt. Take a of shape (1, 2, 3, 4) and empty layout, and b of shape (4,) and layout "C". The output ends up to have 4dims and layout "C".
errors when the layout match but are not equal, for example FHWC and C.

The fix is to 1. check if the picked layout is not too short and if so, discard it, 2. allow for different layouts as long as one if suffix of the other.

Additional information:

It would be nice to follow-up the layout propagation with more involved mechanism that would:

not discard partial information, i.e. when bigger dim-ed tensor does not have layout, but the samller dim-ed has, we should retain the information about layout suffix. Possibly with "*" wildcards in the layout (that would have consitently special treatment in equality).
generalize the matching from common suffix to one of the layouts being subsequence of the other.

Affected modules and functionalities:

Key points relevant for the review:

Tests:

Checklist

Documentation

DALI team only

Requirements

Implements new requirements
Affects existing requirements
N/A

REQ IDs: N/A

JIRA TASK: DALI-3556

Signed-off-by: Kamil Tokarski <ktokarski@nvidia.com>

dali/test/python/operator_1/test_arithmetic_ops.py

stiepan · 2023-07-20T14:03:52Z

!build

dali-automaton · 2023-07-20T14:05:09Z

CI MESSAGE: [9043057]: BUILD STARTED

dali-automaton · 2023-07-20T16:40:09Z

CI MESSAGE: [9043057]: BUILD FAILED

Signed-off-by: Kamil Tokarski <ktokarski@nvidia.com>

stiepan · 2023-07-20T19:51:25Z

!build

dali-automaton · 2023-07-20T19:55:13Z

CI MESSAGE: [9048568]: BUILD STARTED

dali-automaton · 2023-07-20T22:13:41Z

CI MESSAGE: [9048568]: BUILD PASSED

Signed-off-by: Kamil Tokarski <ktokarski@nvidia.com>

mzient · 2023-07-21T10:29:24Z

dali/operators/math/expressions/arithmetic.h

+ * @brief Checks if the child is a (possibly improper) suffix of the parent.
+ * The child must not be longer than the parent.
+ */
+inline bool HasSuffix(const TensorLayout &parent, const TensorLayout &child) {


More common name: EndsWith

Suggested change

inline bool HasSuffix(const TensorLayout &parent, const TensorLayout &child) {

inline bool EndsWith(const TensorLayout &parent, const TensorLayout &child) {

done, nice name.

mzient · 2023-07-21T10:31:04Z

dali/operators/math/expressions/arithmetic.h

+ * The child must not be longer than the parent.
+ */
+inline bool HasSuffix(const TensorLayout &parent, const TensorLayout &child) {
+  assert(parent.size() >= child.size());


Unnecessary and actually harmful. Obviously a longer string is not a suffix of a shorter one.

Suggested change

assert(parent.size() >= child.size());

if (parent.size() < child.size())

return false;

Signed-off-by: Kamil Tokarski <ktokarski@nvidia.com>

stiepan · 2023-07-21T11:18:51Z

!build

dali-automaton · 2023-07-21T11:20:22Z

CI MESSAGE: [9058724]: BUILD STARTED

mzient · 2023-07-21T12:08:09Z

dali/operators/math/expressions/arithmetic.h

    if (!types_layout_inferred_) {
      result_type_id_ = PropagateTypes<Backend>(*expr_, ws);
      result_layout_ = GetCommonLayout<Backend>(*expr_, ws);
+      // sample_ndim is assumed to be constant between iterations.
+      if (result_layout_.size() != result_shape_.sample_dim()) {
+        // If the largest dimentional tensors do not have layout set but some


Suggested change

// If the largest dimentional tensors do not have layout set but some

// If the tensors with the most dimensions do not have a layout set but some

mzient · 2023-07-21T12:10:12Z

dali/operators/math/expressions/arithmetic_test.cc

@@ -134,7 +134,7 @@ TEST(ArithmeticOpsTest, TreePropagationLayoutError) {
    ptr->Resize({{1}, {2}}, DALI_INT32);
  }
  in[0]->SetLayout(TensorLayout());
-  in[1]->SetLayout(TensorLayout("HW"));
+  in[1]->SetLayout(TensorLayout("DH"));


Actually, we should test both cases - the old one should not throw, but instead extend the layout to "DHW".

We have tests for that in python though.

I added the case.

dali/test/python/operator_1/test_arithmetic_ops.py

mzient · 2023-07-21T12:22:27Z

dali/operators/math/expressions/arithmetic.h

+/**
+ * @brief Checks if the child is a (possibly improper) suffix of the parent.
+ */
+inline bool EndsWith(const TensorLayout &parent, const TensorLayout &child) {


Nitpick:
parent and child don't capture the relation of the arguments well.

Suggested change

inline bool EndsWith(const TensorLayout &parent, const TensorLayout &child) {

inline bool EndsWith(const TensorLayout &what, const TensorLayout &with) {

or

Suggested change

inline bool EndsWith(const TensorLayout &parent, const TensorLayout &child) {

inline bool EndsWith(const TensorLayout &whole, const TensorLayout &suffix) {

stiepan · 2023-07-21T12:45:23Z

!build

dali-automaton · 2023-07-21T12:50:08Z

CI MESSAGE: [9059365]: BUILD STARTED

dali-automaton · 2023-07-21T14:39:47Z

CI MESSAGE: [9059365]: BUILD PASSED

Fix layout broadcasting in arithm ops * Allow layouts of different lenghts as long as they share common suffix * Use empty layout if the inferred one is too short --------- Signed-off-by: Kamil Tokarski <ktokarski@nvidia.com>

stiepan added 3 commits July 20, 2023 13:09

Fix layout broadcasting in arithm ops

95a0bb0

Signed-off-by: Kamil Tokarski <ktokarski@nvidia.com>

Test failures

f4c85ef

Signed-off-by: Kamil Tokarski <ktokarski@nvidia.com>

Fix unused variable in the tests

797191f

Signed-off-by: Kamil Tokarski <ktokarski@nvidia.com>

github-advanced-security bot found potential problems Jul 20, 2023

View reviewed changes

dali/test/python/operator_1/test_arithmetic_ops.py Fixed Show resolved Hide resolved

Fix arithm op cpp test

b0d0879

Signed-off-by: Kamil Tokarski <ktokarski@nvidia.com>

Review remarks

1fe7b3c

Signed-off-by: Kamil Tokarski <ktokarski@nvidia.com>

jantonguirao assigned szalpal and mzient Jul 21, 2023

mzient reviewed Jul 21, 2023

View reviewed changes

stiepan added 2 commits July 21, 2023 12:56

More review remarks

3696790

Signed-off-by: Kamil Tokarski <ktokarski@nvidia.com>

Clarify the cooment

abed91f

Signed-off-by: Kamil Tokarski <ktokarski@nvidia.com>

mzient reviewed Jul 21, 2023

View reviewed changes

dali/test/python/operator_1/test_arithmetic_ops.py Show resolved Hide resolved

mzient reviewed Jul 21, 2023

View reviewed changes

dali/test/python/operator_1/test_arithmetic_ops.py Show resolved Hide resolved

mzient reviewed Jul 21, 2023

View reviewed changes

mzient approved these changes Jul 24, 2023

View reviewed changes

szalpal approved these changes Jul 25, 2023

View reviewed changes

stiepan merged commit 7b954bf into NVIDIA:main Jul 25, 2023
5 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix layout broadcasting arithm ops #4951

Fix layout broadcasting arithm ops #4951

stiepan commented Jul 20, 2023 •

edited

Loading

stiepan commented Jul 20, 2023

dali-automaton commented Jul 20, 2023

dali-automaton commented Jul 20, 2023

stiepan commented Jul 20, 2023

dali-automaton commented Jul 20, 2023

dali-automaton commented Jul 20, 2023

mzient Jul 21, 2023

stiepan Jul 21, 2023 •

edited

Loading

mzient Jul 21, 2023

stiepan Jul 21, 2023

stiepan commented Jul 21, 2023

dali-automaton commented Jul 21, 2023

mzient Jul 21, 2023

stiepan Jul 21, 2023

mzient Jul 21, 2023

stiepan Jul 21, 2023

stiepan Jul 21, 2023

mzient Jul 21, 2023 •

edited

Loading

stiepan Jul 21, 2023

stiepan commented Jul 21, 2023

dali-automaton commented Jul 21, 2023

dali-automaton commented Jul 21, 2023

	inline bool HasSuffix(const TensorLayout &parent, const TensorLayout &child) {
	inline bool EndsWith(const TensorLayout &parent, const TensorLayout &child) {

	assert(parent.size() >= child.size());
	if (parent.size() < child.size())
	return false;

	// If the largest dimentional tensors do not have layout set but some
	// If the tensors with the most dimensions do not have a layout set but some

Fix layout broadcasting arithm ops #4951

Fix layout broadcasting arithm ops #4951

Conversation

stiepan commented Jul 20, 2023 • edited Loading

Category:

Description:

Additional information:

Affected modules and functionalities:

Key points relevant for the review:

Tests:

Checklist

Documentation

DALI team only

Requirements

stiepan commented Jul 20, 2023

dali-automaton commented Jul 20, 2023

dali-automaton commented Jul 20, 2023

stiepan commented Jul 20, 2023

dali-automaton commented Jul 20, 2023

dali-automaton commented Jul 20, 2023

Choose a reason for hiding this comment

stiepan Jul 21, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stiepan commented Jul 21, 2023

dali-automaton commented Jul 21, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mzient Jul 21, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stiepan commented Jul 21, 2023

dali-automaton commented Jul 21, 2023

dali-automaton commented Jul 21, 2023

stiepan commented Jul 20, 2023 •

edited

Loading

stiepan Jul 21, 2023 •

edited

Loading

mzient Jul 21, 2023 •

edited

Loading