Realm not completing gather copy on the GPU

[piyueh]

Problem

Advanced indexing of a relatively huge (e.g., length 10K) 1D array returns UnboundLocalError: local variable 'shardfn' referenced before assignment, rather than NotImplementedError.

I understand that advanced indexing is mostly not yet implemented. Most related routines raise NotImplementedError to let users know about this situation. However, this particular use case raises this different error, which seems to be a bug to me.

To reproduce

1. step 1: prepare test.py:

   from legate import numpy
   a = numpy.arange(10000)
   print(a[(1, 2, 3), ]) 

2. step 2: run with, for example

   $ legate --cpus 1 test.py

Output

Traceback (most recent call last):
  File "<blahblah>/lib/python3.8/site-packages/legion_top.py", line 394, in legion_python_main
    run_path(args[start], run_name='__main__')
  File "<blahblah>/lib/python3.8/site-packages/legion_top.py", line 193, in run_path
    exec(code, module.__dict__, module.__dict__)
  File "./test.py", line 3, in <module>
    print(a[(1, 2, 3), ])
  File "<blahblah>/lib/python3.8/site-packages/legate/numpy/array.py", line 381, in __getitem__
    shape=None, thunk=self._thunk.get_item(key, stacklevel=2)
  File "<blahblah>/lib/python3.8/site-packages/legate/numpy/deferred.py", line 414, in get_item
    copy = Copy(mapper=self.runtime.mapper_id, tag=shardfn)
UnboundLocalError: local variable 'shardfn' referenced before assignment

Expected results

Either [1, 2, 3] or NotImplementedError.

Notes

• Interestingly, smaller arrays do not have this issue. For example, if a = numpy.arange(100), the code works fine.
• Another way to make it works is to use GPUs instead of CPUs. For example, legate --gpus 1 test.py works fine. This is interesting, as the GPU implementation seems to be more stable than CPU implementation?

[piyueh]

I added notes regarding using GPUs.

[lightsighter]

Fixed with f227ed7431. Was just a typo. You would need a bigger array to see the same effect on GPUs. Legate won't offload to accelerators unless it thinks that it will be a performance win.

[piyueh]

Hi, I was still not able to get the expected result when using flag -lg:numpy:test with GPU. Also, with CPU, I got the correct output but also got a warning from Legion. Here's my test code:

from legate import numpy
a = numpy.arange(50000)
idx = numpy.random.randint(0, 10000, 5000, int)
print(a[idx, ])
print(a[[1, 2, 3], ])

1. Using CPU (legate --cpus 1 ./test.py -lg:numpy:test), I got warnings and also the correct answer:

   [0 - 7f980aa7aa80]    0.790074 {4}{runtime}: [warning 1071] LEGION WARNING: Region requirement 1 of
   operation Copy (UID 7) in parent task legion_python_main (UID 1) is using uninitialized data for field(s)
   1048578 of logical region (2,1,2) (from file <prefix>/legate.core/legion/runtime/legion/legion_ops.cc:1192)
   For more information see:
   http://legion.stanford.edu/messages/warning_code.html#warning_code_1071
   
   [4250 3719 2489 ... 5808 6931 9231]
   [0 - 7f980aa7aa80]    0.821821 {4}{runtime}: [warning 1071] LEGION WARNING: Region requirement 1 of
   operation Copy (UID 12) in parent task legion_python_main (UID 1) is using uninitialized data for field(s)
   1048578 of logical region (3,1,3) (from file <prefix>/legate.core/legion/runtime/legion/legion_ops.cc:1192)
   For more information see:
   http://legion.stanford.edu/messages/warning_code.html#warning_code_1071
   
   [1 2 3]
   

2. Using GPU (legate --gpus 1 ./test.py -lg:numpy:test), I first got this warning:

   [0 - 7fa3536cfa80]    1.620641 {4}{runtime}: [warning 1071] LEGION WARNING: Region requirement 1 of
   operation Copy (UID 7) in parent task legion_python_main (UID 1) is using uninitialized data for field(s)
   1048578 of logical region (2,1,2) (from file <prefix>/legate.core/legion/runtime/legion/legion_ops.cc:1192)
   For more information see:
   http://legion.stanford.edu/messages/warning_code.html#warning_code_1071
   

   And then the program just hung. No more outputs, and it was not terminated either.

[lightsighter]

@piyueh You can safely ignore the warnings. We'll try to make them go away in the future by telling Legion the right privileges, but in this case they are not causing any issues.

@streichler The single GPU case is hanging because Realm is not completing a gather copy on the GPU. Please see the output below:

../LegateCore/install/bin/legate --gpus 1 ./test1.py -lg:numpy:test --fbmem 2048 -lg:inorder -level dma=1,copy=1

[0 - 7f3c3024f000]    1.646389 {2}{dma}: dma request 0x7f3a74004f10 created - plan=0x7f3a74004960 before=0 after=8000000000400011
[0 - 7f3c3024f000]    1.646398 {2}{dma}: dma request 0x7f3a74004f10 ready - plan=0x7f3a74004960 before=0 after=8000000000400011
[0 - 7f3c3024f000]    1.646522 {2}{dma}: dma request 0x7f3a74004f10 started - plan=0x7f3a74004960 before=0 after=8000000000400011

The copy was issued to Realm and didn't have any event precondition. Realm clearly started the DMA request, but it never finished.

[streichler]

@lightsighter @piyueh can I ask one of you to run with -level dma=2,xplan=1 and paste the log messages? This may be an instance of StanfordLegion/realm#87.

[piyueh]

@streichler

$ legate --gpus 1 ./test.py -lg:numpy:test --fbmem 2048 -lg:inorder -level dma=2,xplan=1
[0 - 7f21f5629700]    2.307125 {4}{runtime}: [warning 1071] LEGION WARNING: Region requirement 1 of operation Copy (UID 7) in parent task legion_python_main (UID 1) is using uninitialized data for field(s) 1048578 of logical region (2,1,2) (from file /home/u00u92m5kmdwylez30357/Downloads/legate/legate.core/legion/runtime/legion/legion_ops.cc:1200)
For more information see:
http://legion.stanford.edu/messages/warning_code.html#warning_code_1071

[0 - 7f21f561b700]    2.307267 {2}{xplan}: created: plan=0x7f2024004da0 domain=IS:<0>..<4999>,dense srcs=1 dsts=1
[0 - 7f21f561b700]    2.307284 {1}{xplan}: created: plan=0x7f2024004da0 srcs[0]=field(1048577, ind=0, size=8)
[0 - 7f21f561b700]    2.307289 {1}{xplan}: created: plan=0x7f2024004da0 dsts[0]=field(1048578, inst=4000000000400003, size=8)
[0 - 7f21f561b700]    2.307300 {1}{xplan}: created: plan=0x7f2024004da0 indirects[0]=4000000000400002[1048577+0] -> IS:<0>..<49999>,dense:4000000000400001
[0 - 7f21f561b700]    2.307337 {1}{xplan}: analysis: plan=0x7f2024004da0 dim_order=[0] xds=3 ibs=3
[0 - 7f21f561b700]    2.307345 {1}{xplan}: analysis: plan=0x7f2024004da0 xds[0]: target=0 inputs=[inst(4000000000400002,1+1)] outputs=[edge(0)]
[0 - 7f21f561b700]    2.307351 {1}{xplan}: analysis: plan=0x7f2024004da0 xds[1]: target=0 inputs=[edge(0)] outputs=[edge(1), edge(2)]
[0 - 7f21f561b700]    2.307356 {1}{xplan}: analysis: plan=0x7f2024004da0 xds[2]: target=0 inputs=[ind(0,2,4000000000400001,0+1), edge(2), edge(1)] outputs=[inst(4000000000400003,0+1)]
[0 - 7f21f561b700]    2.307360 {1}{xplan}: analysis: plan=0x7f2024004da0 ibs[0]: memory=1a00000000000000 size=65536
[0 - 7f21f561b700]    2.307364 {1}{xplan}: analysis: plan=0x7f2024004da0 ibs[1]: memory=1a00000000000000 size=65536
[0 - 7f21f561b700]    2.307368 {1}{xplan}: analysis: plan=0x7f2024004da0 ibs[2]: memory=1a00000000000000 size=65536
[0 - 7f21f561b700]    2.307374 {2}{dma}: dma request 0x7f2024005580 created - plan=0x7f2024004da0 before=0 after=8000000000f0000e
[0 - 7f21f561b700]    2.307379 {2}{dma}: dma request 0x7f2024005580 ready - plan=0x7f2024004da0 before=0 after=8000000000f0000e
[0 - 7f21f561b700]    2.307391 {2}{dma}: dma request 0x7f2024005580 started - plan=0x7f2024004da0 before=0 after=8000000000f0000e

[streichler]

Ok, not the same thing as StanfordLegion/realm#87. I'll see if I can reproduce this hang on my system.