mxnet_coreml_converter unsupported type error

[matthewberryman]

Description

Traceback (most recent call last):
  File "/usr/local/bin/mxnet_coreml_converter.py", line 113, in <module>
    coreml_model = convert(model=mod, mode=mode, class_labels=class_labels, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/converter/_mxnet_converter.py", line 212, in convert
    converter_func = _get_layer_converter_fn(op)
  File "/usr/local/lib/python2.7/site-packages/converter/_mxnet_converter.py", line 98, in _get_layer_converter_fn
    raise TypeError("MXNet layer of type %s is not supported." % layer)
TypeError: MXNet layer of type _copy is not supported.

Environment info (Required)

➜ python diagnose.py                                                                                                                <aws:default>
----------Python Info----------
('Version      :', '2.7.15')
('Compiler     :', 'GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.11.45.5)')
('Build        :', ('default', 'Jan 12 2019 21:07:57'))
('Arch         :', ('64bit', ''))
------------Pip Info-----------
('Version      :', '18.1')
('Directory    :', '/usr/local/lib/python2.7/site-packages/pip')
----------MXNet Info-----------
('Version      :', '1.5.0')
('Directory    :', '/usr/local/lib/python2.7/site-packages/mxnet')
('Commit Hash   :', 'bff0fdce13057ecd6d8b79c0c3d4dbefd5b85bd8')
----------System Info----------
('Platform     :', 'Darwin-18.2.0-x86_64-i386-64bit')
('system       :', 'Darwin')
('node         :', 'vlan-2715-10-17-218-5.guest.wireless.sydney.edu.au')
('release      :', '18.2.0')
('version      :', 'Darwin Kernel Version 18.2.0: Thu Dec 20 20:46:53 PST 2018; root:xnu-4903.241.1~1/RELEASE_X86_64')
----------Hardware Info----------
('machine      :', 'x86_64')
('processor    :', 'i386')
machdep.cpu.brand_string: Intel(R) Core(TM) i7-7567U CPU @ 3.50GHz
machdep.cpu.features: FPU VME DE PSE TSC MSR PAE MCE CX8 APIC SEP MTRR PGE MCA CMOV PAT PSE36 CLFSH DS ACPI MMX FXSR SSE SSE2 SS HTT TM PBE SSE3 PCLMULQDQ DTES64 MON DSCPL VMX EST TM2 SSSE3 FMA CX16 TPR PDCM SSE4.1 SSE4.2 x2APIC MOVBE POPCNT AES PCID XSAVE OSXSAVE SEGLIM64 TSCTMR AVX1.0 RDRAND F16C
machdep.cpu.leaf7_features: SMEP ERMS RDWRFSGS TSC_THREAD_OFFSET BMI1 AVX2 BMI2 INVPCID SMAP RDSEED ADX IPT SGX FPU_CSDS MPX CLFSOPT
machdep.cpu.extfeatures: SYSCALL XD 1GBPAGE EM64T LAHF LZCNT PREFETCHW RDTSCP TSCI
----------Network Test----------
Setting timeout: 10
Timing for MXNet: https://github.com/apache/incubator-mxnet, DNS: 0.0153 sec, LOAD: 1.2856 sec.
Timing for PYPI: https://pypi.python.org/pypi/pip, DNS: 0.0215 sec, LOAD: 0.9972 sec.
Timing for FashionMNIST: https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/dataset/fashion-mnist/train-labels-idx1-ubyte.gz, DNS: 0.0794 sec, LOAD: 0.9364 sec.
Timing for Conda: https://repo.continuum.io/pkgs/free/, DNS: 0.0142 sec, LOAD: 1.0024 sec.
Timing for Gluon Tutorial(en): http://gluon.mxnet.io, DNS: 0.3045 sec, LOAD: 0.4526 sec.
Timing for Gluon Tutorial(cn): https://zh.gluon.ai, DNS: 0.0696 sec, LOAD: 0.8024 sec.

Package used (Python/R/Scala/Julia):
Python.

Build info (Required if built from source)

Compiler (gcc/clang/mingw/visual studio):
clang

MXNet commit hash:
Have installed via pip install mxnet-to-coreml which returns the following version numbers:

Requirement already satisfied: mxnet-to-coreml in /usr/local/lib/python2.7/site-packages (0.1.3)
Requirement already satisfied: coremltools in /usr/local/lib/python2.7/site-packages (from mxnet-to-coreml) (2.0)
Requirement already satisfied: mxnet in /usr/local/lib/python2.7/site-packages (from mxnet-to-coreml) (1.5.0b20190101)
Requirement already satisfied: pyyaml in /usr/local/lib/python2.7/site-packages (from mxnet-to-coreml) (3.13)
Requirement already satisfied: protobuf>=3.1.0 in /usr/local/lib/python2.7/site-packages (from coremltools->mxnet-to-coreml) (3.6.1)
Requirement already satisfied: six==1.10.0 in /usr/local/lib/python2.7/site-packages (from coremltools->mxnet-to-coreml) (1.10.0)
Requirement already satisfied: numpy>=1.10.0 in /usr/local/lib/python2.7/site-packages (from coremltools->mxnet-to-coreml) (1.16.0)
Requirement already satisfied: requests>=2.20.0 in /usr/local/lib/python2.7/site-packages (from mxnet->mxnet-to-coreml) (2.21.0)
Requirement already satisfied: graphviz<0.9.0,>=0.8.1 in /usr/local/lib/python2.7/site-packages (from mxnet->mxnet-to-coreml) (0.8.4)
Requirement already satisfied: setuptools in /usr/local/lib/python2.7/site-packages (from protobuf>=3.1.0->coremltools->mxnet-to-coreml) (40.6.3)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python2.7/site-packages (from requests>=2.20.0->mxnet->mxnet-to-coreml) (2018.11.29)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python2.7/site-packages (from requests>=2.20.0->mxnet->mxnet-to-coreml) (3.0.4)
Requirement already satisfied: idna<2.9,>=2.5 in /usr/local/lib/python2.7/site-packages (from requests>=2.20.0->mxnet->mxnet-to-coreml) (2.8)
Requirement already satisfied: urllib3<1.25,>=1.21.1 in /usr/local/lib/python2.7/site-packages (from requests>=2.20.0->mxnet->mxnet-to-coreml) (1.24.1)

Error Message:

Per description

Minimum reproducible example & steps to reproduce:

command

mxnet_coreml_converter.py --model-prefix=image-classification --epoch=1 --input-shape='{"data":"3,224,224"}' --mode=classifier --pre-processing-arguments='{"image_input_names":"data"}' --class-labels synset.txt --label-names="softmax_label" --output-file="nn.mlmodel"

I can't share all of the model file but the problem appears to be right at the start of image-classification-symbol.json

{
  "nodes": [
    {
      "op": "null",
      "name": "data",
      "inputs": []
    },
    {
      "op": "_copy",
      "name": "id",
      "inputs": [[0, 0, 0]]
    },
    {
      "op": "null",
      "name": "bn_data_gamma",
      "attrs": {
        "eps": "2e-05",
        "fix_gamma": "True",
        "momentum": "0.9"
      },
      "inputs": []
    },

[frankfliu]

@mxnet-label-bot add [Python]

[ChaiBapchya]

@matthewberryman

Short answer - op variable needs to contain layers that are in MXNET_LAYER_REGISTRY

Why?
Long answer -
Upon digging in the file https://github.com/apache/incubator-mxnet/blob/master/tools/coreml/converter/_mxnet_converter.py

I found that : Most likely reason why it's giving unsupported type error is because in your image-classification-symbol.json

"op" : "_copy"

is mentioned

Your error is thrown because of this line
https://github.com/apache/incubator-mxnet/blob/5dc138d95f6ac302e3f0e1c9dc9dcb774d83f69e/tools/coreml/converter/_mxnet_converter.py#L98

which is triggered when the layer variable isn't part of the MXNet layer registry (i.e. permissible layers)
https://github.com/apache/incubator-mxnet/blob/5dc138d95f6ac302e3f0e1c9dc9dcb774d83f69e/tools/coreml/converter/_mxnet_converter.py#L95

Permissible layers are
https://github.com/apache/incubator-mxnet/blob/5dc138d95f6ac302e3f0e1c9dc9dcb774d83f69e/tools/coreml/converter/_mxnet_converter.py#L28

    'FullyConnected' : _layers.convert_dense,
    'Activation'     : _layers.convert_activation,
    'SoftmaxOutput'  : _layers.convert_softmax,
    'Convolution'    : _layers.convert_convolution,
    'Pooling'        : _layers.convert_pooling,
    'Flatten'        : _layers.convert_flatten,
    'transpose'      : _layers.convert_transpose,
    'Concat'         : _layers.convert_concat,
    'BatchNorm'      : _layers.convert_batchnorm,
    'elemwise_add'   : _layers.convert_elementwise_add,
    'Reshape'        : _layers.convert_reshape,
    'Deconvolution'  : _layers.convert_deconvolution,

Hope this helps.

Solution - Use only above permitted layers.

[matthewberryman]

_copy is an mxnet operator (see e.g. https://github.com/apache/incubator-mxnet/blob/25e915bd401f7ac4c639c935f775deccebec96d3/python/mxnet/contrib/onnx/mx2onnx/_op_translations.py#L720 ), and above json snippet in #13965 (comment) is what I am getting from mxnet.

Either there a way of telling mxnet to avoid using the _copy operator, in which case the solution needs to be to mention it in the docs as I couldn't find it there, https://mxnet.incubator.apache.org/versions/master/search.html?q=_copy+layer&check_keywords=yes&area=default# https://mxnet.incubator.apache.org/versions/master/search.html?q=_copy+operator# being the searches I tried, or if there's no way of telling mxnet not to use the _copy operator, then the solution would be add it to this conversion tool as a feature request (since convolution is supported by above list, perhaps first rewrite it as convolution with Kronecker delta function?).
Otherwise #13965 (comment), while informative, doesn't solve the problem.

[ChaiBapchya]

Yes, copy is an MXNet operator. My previous comment doesn't deny that.
But coreml convertor takes in layers as defined in

incubator-mxnet/tools/coreml/converter/_layers.py

Only 12 layers are currently mentioned here which can be converted from MXNet to CoreML, at the moment.

I have created an issue as a Feature Request to add "Copy" operator to MXNet to CoreML convertor. Thanks for bringing this issue up.

[matthewberryman]

Hi @ChaiBapchya - many thanks for opening the feature request. Closing this issue in favour of that one.