New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RELEASE] Announcing v0.9 Release Candidate 1 #3509

Closed
piiswrong opened this Issue Oct 12, 2016 · 11 comments

Comments

Projects
None yet
8 participants
@piiswrong
Contributor

piiswrong commented Oct 12, 2016

We are announcing v0.9 rc1 https://github.com/dmlc/mxnet/tree/v0.9rc1. This includes backend refactor to use NNVM for operator registration and graph optimization.

Schedule

  • Due to the big scope of changes, we first release a release candidate to collect feedbacks. Initial feedback period is two weeks.
  • If no issue is raised during this period we will merge v0.9 to master on Oct 28th. If you need more time, please respond to this issue (preferably with specific concerns) to ask for an extension.
  • Immediately before v0.9 is merged in to master, we will tag v0.8, which is the last release before NNVM refactor.
  • We will continue to develop v0.9 during this period. If you are interested in cutting edge development, please use https://github.com/dmlc/mxnet/tree/nnvm.

List of Changes in v0.9 (excluding those in v0.8)


  • Refactor Symbolic (GraphExecutor, StaticGraph), and NDArray Imperative (MXFuncInvoke) modules to use NNVM for registration and optimization.
    • mxnet/nnvm is added, GraphExecutor is rewritten. StaticGraph is removed.
    • Simple op (registered with MXNET_REGISTER_SIMPLE_OP) is deprecated (but still supported through legacy support) and all simple ops in master are factored to register with NNVM. They are also moved to src/operator/tensor.
    • OperatorProperty, which is the old way of registering layers, is also deprecated (but still supported through legacy support) and we will gradually move them to NNVM
    • Guide for registering operators with NNVM is available here #3248.
    • MXFuncInvoke is replaced with MXImperativeInvoke. Now ALL operators (including symbols like Convolution) are available under both mx.nd.* and mx.sym.* for ndarray operation and symbolic operation.
    • Cython interface for mx.ndarray and mx.symbol are added for reduced python side overhead. Enable with make cython, make cython2, and make cython3 are added.
  • cc optimizers are deprecated and removed. ccsgd is now an alias of sgd. SGD and Adam are now implemented as one step ndarray operation to match the speed of ccsgd.
  • mx.image and mx.image.ImageIter are added for high performance image IO and augmentation. This is meant to replace cv2. This should make writing high performance image data iterators much easier. Try mx.image when you find mx.io.ImageRecordIter too restrictive.
  • MXProfiler is added (example/profiler) for profiling performance of layers.

List of Pending Changes

These are not available in v0.9 rc1 and may or may not be available in final v0.9 release.

  • Vastly (10x - 100x) improved CPU speed with MKLDNN optimizations from Intel.
  • Improved RNN support. More details later.
  • Layout support for nn layers (NCHW, NHWC, HCDHW)

List of Changes in v0.8

@starimpact

This comment has been minimized.

Show comment
Hide comment
@starimpact

starimpact Oct 12, 2016

Contributor

wonderful!!!

Contributor

starimpact commented Oct 12, 2016

wonderful!!!

@tornadomeet

This comment has been minimized.

Show comment
Hide comment
@tornadomeet

tornadomeet Oct 12, 2016

Contributor

@piiswrong Does all examples in mxnet/example will run ok without any change in branch v0.9 ?

Contributor

tornadomeet commented Oct 12, 2016

@piiswrong Does all examples in mxnet/example will run ok without any change in branch v0.9 ?

@piiswrong

This comment has been minimized.

Show comment
Hide comment
@piiswrong

piiswrong Oct 12, 2016

Contributor

@tornadomeet They should. If not we should fix them

Contributor

piiswrong commented Oct 12, 2016

@tornadomeet They should. If not we should fix them

@starimpact

This comment has been minimized.

Show comment
Hide comment
@starimpact

starimpact Oct 12, 2016

Contributor

@tqchen @piiswrong @tornadomeet
i just pull the newest mxnet, and compiled it on tx1 with cuda 8.0.
i'm afraid it still has a lot of warnings when compiling on tx1 with cuda 8.0.
And, it just happens on TX1. it is ok on pc. Why ?
Oh,,,,,It is ok on TK1 also.
maybe the TX1's compiler has problem.

the newest TX1's compiler is g++-5.0. mxnet has a lot of problems on it.

Using g++-4.8 is ok for me.

/home/ubuntu/mingzhang/mxnet/dmlc-core/include/dmlc/./logging.h:62:9: note: in definition of macro ‘CHECK’
   if (!(x))                                                \
         ^
In file included from src/operator/./matrix_op-inl.h:9:0,
                 from src/operator/matrix_op.cc:7:
include/mxnet/operator_util.h:481:3: warning: ‘mxnet::op::__make_SimpleOpRegEntry_transpose__XPU__’ defined but not used [-Wunused-variable]
   __make_ ## SimpleOpRegEntry ## _ ## Name ## __ ## DEV ##__ =          \
   ^
src/operator/./matrix_op-inl.h:640:1: note: in expansion of macro ‘MXNET_REGISTER_SIMPLE_OP’
 MXNET_REGISTER_SIMPLE_OP(transpose, XPU)
 ^
include/mxnet/operator_util.h:481:3: warning: ‘mxnet::op::__make_SimpleOpRegEntry_expand_dims__XPU__’ defined but not used [-Wunused-variable]
   __make_ ## SimpleOpRegEntry ## _ ## Name ## __ ## DEV ##__ =          \
   ^
src/operator/./matrix_op-inl.h:649:1: note: in expansion of macro ‘MXNET_REGISTER_SIMPLE_OP’
 MXNET_REGISTER_SIMPLE_OP(expand_dims, XPU)
 ^
include/mxnet/operator_util.h:481:3: warning: ‘mxnet::op::__make_SimpleOpRegEntry_crop__XPU__’ defined but not used [-Wunused-variable]
   __make_ ## SimpleOpRegEntry ## _ ## Name ## __ ## DEV ##__ =          \
   ^
src/operator/./matrix_op-inl.h:658:1: note: in expansion of macro ‘MXNET_REGISTER_SIMPLE_OP’
 MXNET_REGISTER_SIMPLE_OP(crop, XPU)
 ^
include/mxnet/operator_util.h:481:3: warning: ‘mxnet::op::__make_SimpleOpRegEntry_slice_axis__XPU__’ defined but not used [-Wunused-variable]
   __make_ ## SimpleOpRegEntry ## _ ## Name ## __ ## DEV ##__ =          \
   ^
src/operator/./matrix_op-inl.h:666:1: note: in expansion of macro ‘MXNET_REGISTER_SIMPLE_OP’
 MXNET_REGISTER_SIMPLE_OP(slice_axis, XPU)
 ^
include/mxnet/operator_util.h:481:3: warning: ‘mxnet::op::__make_SimpleOpRegEntry_flip__XPU__’ defined but not used [-Wunused-variable]
   __make_ ## SimpleOpRegEntry ## _ ## Name ## __ ## DEV ##__ =          \
   ^
src/operator/./matrix_op-inl.h:676:1: note: in expansion of macro ‘MXNET_REGISTER_SIMPLE_OP’
 MXNET_REGISTER_SIMPLE_OP(flip, XPU)
 ^
include/mxnet/operator_util.h:481:3: warning: ‘mxnet::op::__make_SimpleOpRegEntry_dot__XPU__’ defined but not used [-Wunused-variable]
   __make_ ## SimpleOpRegEntry ## _ ## Name ## __ ## DEV ##__ =          \
   ^
src/operator/./matrix_op-inl.h:684:1: note: in expansion of macro ‘MXNET_REGISTER_SIMPLE_OP’
 MXNET_REGISTER_SIMPLE_OP(dot, XPU)
 ^
include/mxnet/operator_util.h:481:3: warning: ‘mxnet::op::__make_SimpleOpRegEntry_batch_dot__XPU__’ defined but not used [-Wunused-variable]
   __make_ ## SimpleOpRegEntry ## _ ## Name ## __ ## DEV ##__ =          \
   ^
src/operator/./matrix_op-inl.h:691:1: note: in expansion of macro ‘MXNET_REGISTER_SIMPLE_OP’
 MXNET_REGISTER_SIMPLE_OP(batch_dot, XPU)

Contributor

starimpact commented Oct 12, 2016

@tqchen @piiswrong @tornadomeet
i just pull the newest mxnet, and compiled it on tx1 with cuda 8.0.
i'm afraid it still has a lot of warnings when compiling on tx1 with cuda 8.0.
And, it just happens on TX1. it is ok on pc. Why ?
Oh,,,,,It is ok on TK1 also.
maybe the TX1's compiler has problem.

the newest TX1's compiler is g++-5.0. mxnet has a lot of problems on it.

Using g++-4.8 is ok for me.

/home/ubuntu/mingzhang/mxnet/dmlc-core/include/dmlc/./logging.h:62:9: note: in definition of macro ‘CHECK’
   if (!(x))                                                \
         ^
In file included from src/operator/./matrix_op-inl.h:9:0,
                 from src/operator/matrix_op.cc:7:
include/mxnet/operator_util.h:481:3: warning: ‘mxnet::op::__make_SimpleOpRegEntry_transpose__XPU__’ defined but not used [-Wunused-variable]
   __make_ ## SimpleOpRegEntry ## _ ## Name ## __ ## DEV ##__ =          \
   ^
src/operator/./matrix_op-inl.h:640:1: note: in expansion of macro ‘MXNET_REGISTER_SIMPLE_OP’
 MXNET_REGISTER_SIMPLE_OP(transpose, XPU)
 ^
include/mxnet/operator_util.h:481:3: warning: ‘mxnet::op::__make_SimpleOpRegEntry_expand_dims__XPU__’ defined but not used [-Wunused-variable]
   __make_ ## SimpleOpRegEntry ## _ ## Name ## __ ## DEV ##__ =          \
   ^
src/operator/./matrix_op-inl.h:649:1: note: in expansion of macro ‘MXNET_REGISTER_SIMPLE_OP’
 MXNET_REGISTER_SIMPLE_OP(expand_dims, XPU)
 ^
include/mxnet/operator_util.h:481:3: warning: ‘mxnet::op::__make_SimpleOpRegEntry_crop__XPU__’ defined but not used [-Wunused-variable]
   __make_ ## SimpleOpRegEntry ## _ ## Name ## __ ## DEV ##__ =          \
   ^
src/operator/./matrix_op-inl.h:658:1: note: in expansion of macro ‘MXNET_REGISTER_SIMPLE_OP’
 MXNET_REGISTER_SIMPLE_OP(crop, XPU)
 ^
include/mxnet/operator_util.h:481:3: warning: ‘mxnet::op::__make_SimpleOpRegEntry_slice_axis__XPU__’ defined but not used [-Wunused-variable]
   __make_ ## SimpleOpRegEntry ## _ ## Name ## __ ## DEV ##__ =          \
   ^
src/operator/./matrix_op-inl.h:666:1: note: in expansion of macro ‘MXNET_REGISTER_SIMPLE_OP’
 MXNET_REGISTER_SIMPLE_OP(slice_axis, XPU)
 ^
include/mxnet/operator_util.h:481:3: warning: ‘mxnet::op::__make_SimpleOpRegEntry_flip__XPU__’ defined but not used [-Wunused-variable]
   __make_ ## SimpleOpRegEntry ## _ ## Name ## __ ## DEV ##__ =          \
   ^
src/operator/./matrix_op-inl.h:676:1: note: in expansion of macro ‘MXNET_REGISTER_SIMPLE_OP’
 MXNET_REGISTER_SIMPLE_OP(flip, XPU)
 ^
include/mxnet/operator_util.h:481:3: warning: ‘mxnet::op::__make_SimpleOpRegEntry_dot__XPU__’ defined but not used [-Wunused-variable]
   __make_ ## SimpleOpRegEntry ## _ ## Name ## __ ## DEV ##__ =          \
   ^
src/operator/./matrix_op-inl.h:684:1: note: in expansion of macro ‘MXNET_REGISTER_SIMPLE_OP’
 MXNET_REGISTER_SIMPLE_OP(dot, XPU)
 ^
include/mxnet/operator_util.h:481:3: warning: ‘mxnet::op::__make_SimpleOpRegEntry_batch_dot__XPU__’ defined but not used [-Wunused-variable]
   __make_ ## SimpleOpRegEntry ## _ ## Name ## __ ## DEV ##__ =          \
   ^
src/operator/./matrix_op-inl.h:691:1: note: in expansion of macro ‘MXNET_REGISTER_SIMPLE_OP’
 MXNET_REGISTER_SIMPLE_OP(batch_dot, XPU)

@starimpact

This comment has been minimized.

Show comment
Hide comment
@starimpact

starimpact Oct 12, 2016

Contributor

v0.9.0rc1 compiling warnings[CPU mode]:

src/operator/tensor/elemwise_sum.cc: In function ‘std::vector<nnvm::NodeEntry> mxnet::op::ElementWiseSumGrad(const NodePtr&, const std::vector<nnvm::NodeEntry>&)’:
src/operator/tensor/elemwise_sum.cc:30:14: warning: unused variable ‘h’ [-Wunused-variable]
   for (auto& h : n->inputs) {
              ^
In file included from /home/mingzhang/work/dmlc/mxnet/dmlc-core/include/dmlc/registry.h:13:0,
                 from include/mxnet/operator_util.h:18,
                 from src/operator/tensor/./broadcast_reduce_op.h:9,
                 from src/operator/tensor/broadcast_reduce_op.cc:6:
src/operator/tensor/./broadcast_reduce_op.h: In function ‘bool mxnet::op::ReduceAxisShape(const nnvm::NodeAttrs&, std::vector<nnvm::TShape>*, std::vector<nnvm::TShape>*)’:
/home/mingzhang/work/dmlc/mxnet/dmlc-core/include/dmlc/./logging.h:65:38: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
 #define CHECK_LT(x, y) CHECK((x) < (y))
                                      ^
/home/mingzhang/work/dmlc/mxnet/dmlc-core/include/dmlc/./logging.h:62:9: note: in definition of macro ‘CHECK’
   if (!(x))                                                \
         ^
src/operator/tensor/./broadcast_reduce_op.h:82:5: note: in expansion of macro ‘CHECK_LT’
     CHECK_LT(param.axis, ishape.ndim())
     ^
In file included from src/operator/tensor/broadcast_reduce_op.cc:6:0:
src/operator/tensor/./broadcast_reduce_op.h:92:50: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
       for (int i = param.axis+1; i < ishape.ndim(); ++i) oshape[i-1] = ishape[i];
Contributor

starimpact commented Oct 12, 2016

v0.9.0rc1 compiling warnings[CPU mode]:

src/operator/tensor/elemwise_sum.cc: In function ‘std::vector<nnvm::NodeEntry> mxnet::op::ElementWiseSumGrad(const NodePtr&, const std::vector<nnvm::NodeEntry>&)’:
src/operator/tensor/elemwise_sum.cc:30:14: warning: unused variable ‘h’ [-Wunused-variable]
   for (auto& h : n->inputs) {
              ^
In file included from /home/mingzhang/work/dmlc/mxnet/dmlc-core/include/dmlc/registry.h:13:0,
                 from include/mxnet/operator_util.h:18,
                 from src/operator/tensor/./broadcast_reduce_op.h:9,
                 from src/operator/tensor/broadcast_reduce_op.cc:6:
src/operator/tensor/./broadcast_reduce_op.h: In function ‘bool mxnet::op::ReduceAxisShape(const nnvm::NodeAttrs&, std::vector<nnvm::TShape>*, std::vector<nnvm::TShape>*)’:
/home/mingzhang/work/dmlc/mxnet/dmlc-core/include/dmlc/./logging.h:65:38: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
 #define CHECK_LT(x, y) CHECK((x) < (y))
                                      ^
/home/mingzhang/work/dmlc/mxnet/dmlc-core/include/dmlc/./logging.h:62:9: note: in definition of macro ‘CHECK’
   if (!(x))                                                \
         ^
src/operator/tensor/./broadcast_reduce_op.h:82:5: note: in expansion of macro ‘CHECK_LT’
     CHECK_LT(param.axis, ishape.ndim())
     ^
In file included from src/operator/tensor/broadcast_reduce_op.cc:6:0:
src/operator/tensor/./broadcast_reduce_op.h:92:50: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
       for (int i = param.axis+1; i < ishape.ndim(); ++i) oshape[i-1] = ishape[i];
@blackball

This comment has been minimized.

Show comment
Hide comment
@blackball

blackball Oct 17, 2016

Contributor

Maybe this is off the topic.

From above, I see:

"Vastly (10x - 100x) improved CPU speed with MKLDNN optimizations from Intel."

Where does "10x-100x" come from ?

I roughly read the source codes of MKL-DNN before. MKL-DNN trys to employ AVX2 and OpenMP to make things faster, and it will try to generate the operators' codes by using JIT at run-time. By using JIT, it could take some advantages by optimally hand coding some of the operators. It will be faster, but it shouldn't be more than 10x faster.

If it's really that fast on single machine, I am really wondering what is the cause. Or I missed something important while I was reading the MKL-DNN or mxnet source codes.

Thanks!

Contributor

blackball commented Oct 17, 2016

Maybe this is off the topic.

From above, I see:

"Vastly (10x - 100x) improved CPU speed with MKLDNN optimizations from Intel."

Where does "10x-100x" come from ?

I roughly read the source codes of MKL-DNN before. MKL-DNN trys to employ AVX2 and OpenMP to make things faster, and it will try to generate the operators' codes by using JIT at run-time. By using JIT, it could take some advantages by optimally hand coding some of the operators. It will be faster, but it shouldn't be more than 10x faster.

If it's really that fast on single machine, I am really wondering what is the cause. Or I missed something important while I was reading the MKL-DNN or mxnet source codes.

Thanks!

@gengyifeng

This comment has been minimized.

Show comment
Hide comment
@gengyifeng

gengyifeng Oct 20, 2016

Contributor

Failed to run "python train_cifar10_resnet.py --gpus 0"

Traceback (most recent call last):
  File "train_cifar10_resnet.py", line 391, in <module>
    fit(args, net, get_iterator)
  File "train_cifar10_resnet.py", line 387, in fit
    epoch_end_callback=checkpoint
  File "/home/yifeng/merge_test/mxnet_0.9/example/image-classification/../../python/mxnet/model.py", line 744, in fit
    self._init_params(dict(data.provide_data+data.provide_label))
  File "/home/yifeng/merge_test/mxnet_0.9/example/image-classification/../../python/mxnet/model.py", line 500, in _init_params
    self.initializer(k, v)
  File "/home/yifeng/merge_test/mxnet_0.9/example/image-classification/../../python/mxnet/initializer.py", line 156, in __call__
    init(name, arr)
  File "train_cifar10_resnet.py", line 163, in __call__
    self._init_weight(name, arr)
  File "/home/yifeng/merge_test/mxnet_0.9/example/image-classification/../../python/mxnet/initializer.py", line 268, in _init_weight
    random.normal(0, scale, out=arr)
  File "/home/yifeng/merge_test/mxnet_0.9/example/image-classification/../../python/mxnet/_ctypes/ndarray.py", line 127, in generic_ndarray_function
    c_array(ctypes.c_char_p, [c_str(str(i)) for i in kwargs.values()])))
  File "/home/yifeng/merge_test/mxnet_0.9/example/image-classification/../../python/mxnet/base.py", line 75, in check_call
    raise MXNetError(py_str(_LIB.MXGetLastError()))
mxnet.base.MXNetError: Required parameter shape of Shape(tuple) is not presented
Contributor

gengyifeng commented Oct 20, 2016

Failed to run "python train_cifar10_resnet.py --gpus 0"

Traceback (most recent call last):
  File "train_cifar10_resnet.py", line 391, in <module>
    fit(args, net, get_iterator)
  File "train_cifar10_resnet.py", line 387, in fit
    epoch_end_callback=checkpoint
  File "/home/yifeng/merge_test/mxnet_0.9/example/image-classification/../../python/mxnet/model.py", line 744, in fit
    self._init_params(dict(data.provide_data+data.provide_label))
  File "/home/yifeng/merge_test/mxnet_0.9/example/image-classification/../../python/mxnet/model.py", line 500, in _init_params
    self.initializer(k, v)
  File "/home/yifeng/merge_test/mxnet_0.9/example/image-classification/../../python/mxnet/initializer.py", line 156, in __call__
    init(name, arr)
  File "train_cifar10_resnet.py", line 163, in __call__
    self._init_weight(name, arr)
  File "/home/yifeng/merge_test/mxnet_0.9/example/image-classification/../../python/mxnet/initializer.py", line 268, in _init_weight
    random.normal(0, scale, out=arr)
  File "/home/yifeng/merge_test/mxnet_0.9/example/image-classification/../../python/mxnet/_ctypes/ndarray.py", line 127, in generic_ndarray_function
    c_array(ctypes.c_char_p, [c_str(str(i)) for i in kwargs.values()])))
  File "/home/yifeng/merge_test/mxnet_0.9/example/image-classification/../../python/mxnet/base.py", line 75, in check_call
    raise MXNetError(py_str(_LIB.MXGetLastError()))
mxnet.base.MXNetError: Required parameter shape of Shape(tuple) is not presented
@sbodenstein

This comment has been minimized.

Show comment
Hide comment
@sbodenstein

sbodenstein Oct 28, 2016

Contributor

@piiswrong: Is this still being merged today?

Contributor

sbodenstein commented Oct 28, 2016

@piiswrong: Is this still being merged today?

@piiswrong

This comment has been minimized.

Show comment
Hide comment
@piiswrong

piiswrong Oct 28, 2016

Contributor

We are going to delay it. Need more time to verify that there are no bugs

Contributor

piiswrong commented Oct 28, 2016

We are going to delay it. Need more time to verify that there are no bugs

@taliesinb

This comment has been minimized.

Show comment
Hide comment
@taliesinb

taliesinb Oct 31, 2016

Contributor

@piiswrong: Any information about a new estimated timeline?

Contributor

taliesinb commented Oct 31, 2016

@piiswrong: Any information about a new estimated timeline?

@tqchen

This comment has been minimized.

Show comment
Hide comment
@tqchen

tqchen Oct 31, 2016

Member

@gengyifeng Your error was due to the front-end python package not getting updated.

Member

tqchen commented Oct 31, 2016

@gengyifeng Your error was due to the front-end python package not getting updated.

@tqchen tqchen modified the milestones: v0.9, v1.0 Oct 31, 2016

@tqchen tqchen closed this Oct 19, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment