New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The problem of ABI compatibility in MXTVMBridge #14030

Closed
wkcn opened this Issue Jan 30, 2019 · 7 comments

Comments

Projects
None yet
6 participants
@wkcn
Copy link
Member

wkcn commented Jan 30, 2019

Description

Hi, there.

I would like to use MXTVMBridge to accelerate my project, but there is a flaky problem.
The example works when MXNet is built from source by myself, but it doesn't work when MXNet is installed by PIP.

Environment info (Required)

----------Python Info----------                                                                                                            
Version      : 3.7.2                                                                                                                       
Compiler     : GCC 8.2.1 20181127                                                                                                          
Build        : ('default', 'Jan 10 2019 23:51:51')
Arch         : ('64bit', '')
------------Pip Info-----------
Version      : 18.1                                                                                                                        
Directory    : /usr/lib/python3.7/site-packages/pip
----------MXNet Info-----------
Version      : 1.5.0
Directory    : /home/wkcn/proj/incubator-mxnet/python/mxnet
Hashtag not found. Not installed from pre-built package.
----------System Info----------
Platform     : Linux-4.20.5-arch1-1-ARCH-x86_64-with-arch
system       : Linux
node         : MiraiT
release      : 4.20.5-arch1-1-ARCH
version      : #1 SMP PREEMPT Sat Jan 26 12:59:18 UTC 2019
----------Hardware Info----------
machine      : x86_64
processor    :
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
Address sizes:       39 bits physical, 48 bits virtual
CPU(s):              4
On-line CPU(s) list: 0-3
Thread(s) per core:  2
Core(s) per socket:  2
Socket(s):           1
NUMA node(s):        1
Vendor ID:           GenuineIntel
CPU family:          6
Model:               142
Model name:          Intel(R) Core(TM) i7-7500U CPU @ 2.70GHz
Stepping:            9
CPU MHz:             3500.069
CPU max MHz:         3500.0000
CPU min MHz:         400.0000
BogoMIPS:            5810.00
Virtualization:      VT-x
L1d cache:           32K
L1i cache:           32K
L2 cache:            256K
L3 cache:            4096K
NUMA node0 CPU(s):   0-3
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe
 syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni
 pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsa
ve avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_ad
just bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp
_notify hwp_act_window hwp_epp

Package used (Python/R/Scala/Julia):
Python

Build info (Required if built from source)

Compiler (gcc/clang/mingw/visual studio): gcc

MXNet commit hash:
e37ff53

Build config:
make -j 5 USE_OPENCV=0 USE_BLAS=openblas

Minimum reproducible example

https://github.com/wkcn/test_tvm_bridge

In this example, tvm_packed_func.h is simplyfied from TVM.

Steps to reproduce

  1. clone the example
  2. change the path of libmxnet.so in the line 55 of the code.
  3. make
  4. ./test

What have you tried to solve it?

In the function SetMXTVMBridge of the reproducible example,
args.num_args is wrong and args.values[0].v_str is sometimes wrong when MXNet is installed by pip, namely pip install mxnet --pre.
However, the result is correct when MXNet is built by myself, make -j 5 USE_OPENCV=1 USE_BLAS=openblas in the latest MXNet source.

In the PR #9880, merrymercy met a similar problem. #9880 (comment)

It seems the temporary variable TVMArgs {"WrapAsyncCall", PackedFunc(mxnet::WrapAsyncCall)} is released before calling fregister in the function MXTVMBridge because of optimization of compiler.

Thanks!

@mxnet-label-bot

This comment has been minimized.

Copy link
Contributor

mxnet-label-bot commented Jan 30, 2019

Hey, this is the MXNet Label Bot.
Thank you for submitting the issue! I will try and suggest some labels so that the appropriate MXNet community members can help resolve it.
Here are my recommended labels: Bug

@szha

This comment has been minimized.

Copy link
Member

szha commented Jan 31, 2019

In pip we use symbol whitelist https://github.com/apache/incubator-mxnet/blob/master/make/config/libmxnet.ver. Is any required function missing from this list? @tqchen

@tqchen

This comment has been minimized.

Copy link
Member

tqchen commented Jan 31, 2019

Because the bridge uses c++ std::function, it requires a common ABI between the two in order to make things work, so build from source is indeed the best way

@vdantu

This comment has been minimized.

Copy link
Contributor

vdantu commented Jan 31, 2019

@mxnet-label-bot add [question]

@marcoabreu marcoabreu added the Question label Jan 31, 2019

@wkcn

This comment has been minimized.

Copy link
Member Author

wkcn commented Jan 31, 2019

@szha @tqchen @vdantu
Thank you!
I know that the reason is about ABI compatibility now.
It will be better if there is a solution without rebuilding the sources : )

@wkcn wkcn closed this Jan 31, 2019

@wkcn

This comment has been minimized.

Copy link
Member Author

wkcn commented Jan 31, 2019

@tqchen
Is it possible to provide an API like this?
It may be available to avoid the problem of ABI compatibility.

class PackedFunc {
 public:
  using FType = std::function<void(TVMArgs args, TVMRetValue* rv)>;
  PackedFunc() {
    SetRemoteCallPacked();
  };
  explicit PackedFunc(FType body) : body_(body) {
    SetRemoteCallPacked();
  }
  void CallPacked(TVMArgs args, TVMRetValue* rv) const {
    _RemoteCallPacked(this, args, rv);
  }
  template <typename... Args>
  inline TVMRetValue operator()(Args&&... args) const {
    const int kNumArgs = sizeof...(Args);
    const int kArraySize = kNumArgs > 0 ? kNumArgs : 1;
    TVMValue values[kArraySize];
    int type_codes[kArraySize];
    detail::for_each(TVMArgsSetter(values, type_codes),
                     std::forward<Args>(args)...);
    TVMRetValue rv;
    _RemoteCallPacked(this, TVMArgs(values, type_codes, kNumArgs), &rv);
    return rv;
  }
 private:
  void SetRemoteCallPacked() {
    _RemoteCallPacked = [](const PackedFunc* func, TVMArgs args, TVMRetValue* rv) {
      func->body_(args, rv);
    };
  }
  void (*_RemoteCallPacked)(const PackedFunc* func, TVMArgs args, TVMRetValue* rv);
 private:
  FType body_;
...
};

@wkcn wkcn reopened this Feb 1, 2019

@wkcn wkcn changed the title The bug in MXTVMBridge The problem of ABI compatibility in MXTVMBridge Feb 1, 2019

@wkcn

This comment has been minimized.

Copy link
Member Author

wkcn commented Feb 1, 2019

The problem is related to TVM, so I open an issue in TVM project.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment