Features/510 split #677

lenablind · 2020-09-25T11:41:07Z

Description

Implementation of functions split(), vsplit(), hsplit(), dsplit() based on np analogues.
For process-local operations, I use torch.split

Docs numpy:

Docs pytorch: https://pytorch.org/docs/stable/generated/torch.split.html

differences numpy and torch

This affects the input parameter indices_or_sections (np) / split_size_or_sections (torch).
As the variations in the given names might already reveal, the parameters contain different types of information depending on the used library.

Numpy:

integer:
In this case, indices_or_sections indicates in how many parts the input ary shall be split.
Therefore, indices_or_sections = 2 will result into a list containing 2 DNDarrays.
array_like
On the other hand, indices_or_sections can be handled as a list containing the boundaries of a series of slices
For instance, indices_or_sections = [1, 3, 4] will indicate the following data chunks:
- [: 1]
- [1: 3]
- [3: 4]
- [4: ]

Torch:
In contrary to numpy, the torch analogue always contains the size(s) of the resulting tensor(s).
If integer, all (function-)split tensors will have the same size, otherwise the possibly varying split sizes might be defined in an array_like.

According to these differences, the parameter semantics of numpy are mapped to those of torch as explained in the following.

Strategy

As vsplit, dsplit, hsplit are no more than calls of split with a specific axis, I'll reduce the explanation to the algorithm of the latter.

Depending on whether ary is distributed in the same dimension as it is split within the function (therefore ary.split == axis), the strategy varies, as the data chunks on each node have to be reorganized correctly in the resulting list of (function-) split DNDarrays. (Challenge: Data of (function-) split DNDarrays is split via MPI)

axis != ary.split

Mapping np -> torch:
If indices_or_sections is integer, the resulting size for all DNDarrays will be calculated using the size of the affected axis.
If indices_or_sections is array_like, the resulting split sizes are calculated usinght.diff (1st discrete difference).

axis == ary.split

indices_or_sections is integer:
- CASE 1 : number of processes == indices_or_selections
  In this case, the split matches the distribution and the data on the node can be used directly and inserted into the resulting list.
  The remaining DNDarrays are filled with empty DNDarrays of the needed shape.
- CASE 2: number of processes != indices_or_selections
  The goal is, to use torch.split with the process-adapted/correctly calculated indices_or_sections parameter (new_indices)
  1. Calculate the index of the first element in the resulting list which needs data of the current process
  2. Calculate the amount of data which is needed to fill this chunk up
  3. Use the remaining data of the process for the following chunks
indices_or_sections is array_like / DNDarray after sanitation
The goal is, to use torch.split with the process-adapted/correctly calculated indices_or_sections parameter.
Therefore, reduce the input information of indices_or_sections to the process relevant/ the given indices (of the global DNDarray) which correspond to the (local) data on the node. The needed information is provided via the slice out of ht.comm.chunk
Afterwards, map the numpy to the torch semantics, as described above in axis != ary.split

In all cases, every DNDarray within the list is balanced before being returned.

Overview split functions and related axis

Function	Axis
vsplit	0
hsplit	1
dsplit	2

Hint: In contrary to the corresponding general split function call with axis 1, the input within hsplit is allowed to be 1-dimensional.
This results into some dimension changing operations in the HeAT version, more specifically a reshape adding one dimension and a flattening afterwards, to meet full consistency with numpy.

Issue/s resolved: #510

Changes proposed:

Implemented new functions: manipulations.split(), manipulations.hsplit(), manipulations.vsplit(), manipulations.dsplit()
Implemented corresponding tests

Type of change

New feature (non-breaking change which adds functionality)

Due Diligence

All split configurations tested
Multiple dtypes tested in relevant functions
Documentation updated (if needed)
Updated changelog.md under the title "Pending Additions"

Does this change modify the behaviour of other functions? If so, which?

no

…d case

mtar · 2020-09-25T11:41:10Z

GPU cluster tests are currently disabled on this Pull Request.

codecov · 2020-09-25T11:53:59Z

Codecov Report

Merging #677 (454cdee) into master (170592a) will increase coverage by 0.04%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master     #677      +/-   ##
==========================================
+ Coverage   97.54%   97.59%   +0.04%     
==========================================
  Files          87       87              
  Lines       17698    18030     +332     
==========================================
+ Hits        17264    17596     +332     
  Misses        434      434

Impacted Files	Coverage Δ
heat/core/manipulations.py	`99.22% <100.00%> (+0.06%)`	⬆️
heat/core/tests/test_manipulations.py	`99.94% <100.00%> (+<0.01%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 170592a...454cdee. Read the comment docs.

mtar · 2020-09-29T08:04:24Z

run tests

coquelin77

this all looks good to me. However, there is a larger comment to be discussed.

Do we want to have this return actual views of the DNDarray? If so, it would mean that we cannot balance the DNDarrays, and it might change some of the logic involved in the function itself. @ClaudiaComito @Markus-Goetz , thoughts?

heat/core/manipulations.py

coquelin77 · 2020-10-26T09:49:52Z

heat/core/manipulations.py

+    if len(ary.lshape) < 2:
+        ary = reshape(ary, (1, ary.lshape[0]))
+        result = split(ary, indices_or_sections, 1)
+        result = [flatten(sub_array) for sub_array in result]
+    else:
+        result = split(ary, indices_or_sections, 1)


is this needed in dsplit as well?

@coquelin77 No, it isn't. hsplit is some kind of a special function among the three split variations. In contrary to the corresponding general split function call with axis 1, the ary is allowed to be 1-dimensional (in numpy) even if (functionally) split among the second axis. As it isn't in torch, ary has to be reshaped (adding one dimension). The subsequent flattening is needed to meet full consistency with numpy.
As this exception doesn't occur within the other split functions, no shape manipulating is needed there.

heat/core/manipulations.py

coquelin77 · 2020-10-26T10:02:32Z

heat/core/manipulations.py

+    for sub_DNDarray in sub_arrays_ht:
+        sub_DNDarray.balance_()


i think that this should be communicated in the docs. Since all of the sub-arrays are balanced, they are not linked to the original data points...this spawns a larger question.

@coquelin77 I agree, but I guess this strongly depends on the general discussion above.

coquelin77 · 2020-10-26T14:46:10Z

rerun tests

ClaudiaComito

Hi Lena,

before I look into the details, something important jumps to my eye: np.split & co. return lists of subarrays as views into the original array. The current heat implementation also says so, but really it returns copies. To exemplify this:

>>> import numpy as np
>>> import heat as ht
>>> a = np.arange(16).reshape(4,4)
>>> a
array([[ 0,  1,  2,  3],
       [ 4,  5,  6,  7],
       [ 8,  9, 10, 11],
       [12, 13, 14, 15]])
>>> b, c = np.split(a, 2)
>>> b[0,0] = 10
>>> b
array([[10,  1,  2,  3],
       [ 4,  5,  6,  7]])
>>> a
array([[10,  1,  2,  3],
       [ 4,  5,  6,  7],
       [ 8,  9, 10, 11],
       [12, 13, 14, 15]])

>>> ht_a = ht.arange(16).reshape((4,4))
>>> ht_a
DNDarray([[ 0,  1,  2,  3],
          [ 4,  5,  6,  7],
          [ 8,  9, 10, 11],
          [12, 13, 14, 15]], dtype=ht.int32, device=cpu:0, split=None)
>>> ht_b, ht_c = ht.split(ht_a, 2)
>>> ht_b[0,0] = 10
>>> ht_b
DNDarray([[10,  1,  2,  3],
          [ 4,  5,  6,  7]], dtype=ht.int32, device=cpu:0, split=None)
>>> ht_a
DNDarray([[ 0,  1,  2,  3],
          [ 4,  5,  6,  7],
          [ 8,  9, 10, 11],
          [12, 13, 14, 15]], dtype=ht.int32, device=cpu:0, split=None)

ClaudiaComito · 2020-10-27T08:31:24Z

heat/core/manipulations.py

+            DNDarray([])
+        ]
+
+    """


here we should raise an exception if ary.ndim < 3

If that is the case, an exception is raised within split, so I think raising an additional one in dsplit would be superfluous

heat/core/manipulations.py

ClaudiaComito · 2020-10-27T09:46:09Z

this all looks good to me. However, there is a larger comment to be discussed.

Do we want to have this return actual views of the DNDarray? If so, it would mean that we cannot balance the DNDarrays, and it might change some of the logic involved in the function itself. @ClaudiaComito @Markus-Goetz , thoughts?

I'm with you @coquelin77, no balancing. We need ht.view

coquelin77 · 2020-10-27T09:53:08Z

this all looks good to me. However, there is a larger comment to be discussed.
Do we want to have this return actual views of the DNDarray? If so, it would mean that we cannot balance the DNDarrays, and it might change some of the logic involved in the function itself. @ClaudiaComito @Markus-Goetz , thoughts?

I'm with you @coquelin77, no balancing. We need ht.view

this isnt a major change from the code, it could be done with relative ease if the balance is dropped. however, if the hard splits result in a partial view on two processes, this would be hard to communicate to users

Markus-Goetz · 2020-10-27T10:00:30Z

@coquelin77 @ClaudiaComito For now it should return a copy, generally however I would obviously like to have something like a ht.view of the data, especially silent return of it. I am not 100% sure whether we can actually achieve this easily, due to the additional properties that we carry around and would have to watch out for. Yet, if somebody would successfully implement this, that would be a solid success for squeezing out further performance

coquelin77

looks like the comment about views vs copies didnt make it into the docs of dsplit, hsplit. or vsplit

coquelin77 · 2020-11-17T08:26:10Z

heat/core/manipulations.py

+    Returns
+    -------
+    sub_arrays : list of DNDarrays
+        A list of sub-DNDarrays as views into ary.


copies not views, also ary an be a code block (ary)

also can you add the copy statement to the description of the function as well. i think that people will often only read that part

Oh dear, sorry I missed that. I'll fix it and wrap ary into a code block. Are all markdown features available in the dosctrings/correctly rendered in sphinx? And you're right, the general description is probably the most relevant part to most, I'll add the hint

lenablind · 2020-11-17T12:41:57Z

rerun tests

coquelin77 · 2020-11-17T14:23:33Z

rerun tests

coquelin77

looks like the comment about views vs copies didnt make it into the docs of dsplit, hsplit. or vsplit

coquelin77 · 2020-11-17T14:42:11Z

looks like the comment about views vs copies didnt make it into the docs of dsplit, hsplit. or vsplit

sorry this was a remnant of before, ignore this

lenablind added 24 commits September 21, 2020 11:06

Sanitizing parameters

ca8a106

First approach to function

be6ab99

indices_or_sections = int, undistributed case

7dc205a

Undistributed indices_or_sections (mapping np -> torch), undistribute…

d273872

…d case

Additional tests

16f465e

indices_or sections distributed, undistributed case

ed75e8b

Merge branch 'master' into features/510-split

8173eb1

Distributed case, axis != a.split, int axis == a.split

f7adbc0

Implementation for all cases

2cbad85

ary.split == axis, ary.comm.size == indices

7093e3b

Merge branch 'master' into features/510-split

62c6e5b

ary.split == axis, ary.comm.size >= indices

b949e26

ary.split == axis, indices = int

a5f49f8

United cases 1 & 2, replaced for loop

5a363d9

ary.split == axis, indices array_like

7307340

Expanded Docstrings

fcd4834

Changed algorithm using where expression

ffbbc19

Additional test case, first draft hsplit, vsplit, dsplit

1c65be1

dsplit, hsplit, vsplit implemented + docstrings

6286635

Tests for hsplit, vsplit, dsplit + flatten in hsplit

a98c329

Correction of docstring example

bd7af0b

Added warning to docstring

057d430

Clarifying modifications

439540a

Merge branch 'master' into features/510-split

c138ef0

lenablind requested review from mtar, coquelin77 and ClaudiaComito September 25, 2020 11:41

Added PR to changelog

dd22292

Merge branch 'master' into features/510-split

1c30da6

coquelin77 reviewed Oct 26, 2020

View reviewed changes

lenablind added 4 commits October 26, 2020 16:29

Added reference to split in docs

5806e69

Restructured documentation (Notes section)

ddb9bb7

Moved PR to section

1fcd1e1

Merge branch 'master' into features/510-split

54a0159

ClaudiaComito requested changes Oct 27, 2020

View reviewed changes

ClaudiaComito reviewed Oct 27, 2020

View reviewed changes

heat/core/manipulations.py Outdated Show resolved Hide resolved

ClaudiaComito mentioned this pull request Oct 27, 2020

Implement DNDarray.view() #689

Closed

lenablind and others added 2 commits October 27, 2020 14:26

Adapted requested changes & changed description of 'views' to 'copies'

fd58cc8

Merge branch 'master' into features/510-split

054aabd

coquelin77 requested changes Nov 17, 2020

View reviewed changes

lenablind added 2 commits November 17, 2020 13:03

Merge branch 'master' into features/510-split

5fba248

Added docstrings & updated to current master

454cdee

coquelin77 approved these changes Nov 17, 2020

View reviewed changes

coquelin77 merged commit debeaf7 into master Nov 17, 2020

coquelin77 deleted the features/510-split branch November 17, 2020 14:43

ClaudiaComito mentioned this pull request Nov 27, 2020

Features/689 view #701

Closed

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Features/510 split #677

Features/510 split #677

lenablind commented Sep 25, 2020 •

edited

mtar commented Sep 25, 2020

codecov bot commented Sep 25, 2020 •

edited

mtar commented Sep 29, 2020

coquelin77 left a comment

coquelin77 Oct 26, 2020

lenablind Oct 27, 2020

coquelin77 Oct 26, 2020

lenablind Oct 27, 2020

coquelin77 commented Oct 26, 2020

ClaudiaComito left a comment •

edited

ClaudiaComito Oct 27, 2020

lenablind Oct 27, 2020

ClaudiaComito commented Oct 27, 2020

coquelin77 commented Oct 27, 2020

Markus-Goetz commented Oct 27, 2020

coquelin77 left a comment

coquelin77 Nov 17, 2020

lenablind Nov 17, 2020

lenablind commented Nov 17, 2020

coquelin77 commented Nov 17, 2020

coquelin77 left a comment

coquelin77 commented Nov 17, 2020

Features/510 split #677

Features/510 split #677

Conversation

lenablind commented Sep 25, 2020 • edited

Description

differences numpy and torch

Strategy

axis != ary.split

axis == ary.split

Overview split functions and related axis

Changes proposed:

Type of change

Due Diligence

Does this change modify the behaviour of other functions? If so, which?

mtar commented Sep 25, 2020

codecov bot commented Sep 25, 2020 • edited

Codecov Report

mtar commented Sep 29, 2020

coquelin77 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coquelin77 commented Oct 26, 2020

ClaudiaComito left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ClaudiaComito commented Oct 27, 2020

coquelin77 commented Oct 27, 2020

Markus-Goetz commented Oct 27, 2020

coquelin77 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lenablind commented Nov 17, 2020

coquelin77 commented Nov 17, 2020

coquelin77 left a comment

Choose a reason for hiding this comment

coquelin77 commented Nov 17, 2020

lenablind commented Sep 25, 2020 •

edited

codecov bot commented Sep 25, 2020 •

edited

ClaudiaComito left a comment •

edited