Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

A problem has occurred in Numba's internals #6098

Closed
munib94 opened this issue Aug 6, 2020 · 5 comments
Closed

A problem has occurred in Numba's internals #6098

munib94 opened this issue Aug 6, 2020 · 5 comments
Labels
no action required No action was needed to resolve.

Comments

@munib94
Copy link

munib94 commented Aug 6, 2020

Hello,

I am trying to train the following network from scratch on my system: https://github.com/laoreja/HPLFlowNet.

To install numba, I first did conda install --channel=numba llvmlite, followed by pip install numba==0.38.1. To install vtk I used python -m pip install --upgrade --user vtk according to https://stackoverflow.com/questions/48623136/installing-vtk-in-python

My system:
OS: Ubuntu 20.04 LTS
Python version: 3.5.4
CUDA/cuDNN version: 9.0/7.6.4
GPU: Nvidia Titan RTX
CUDA driver version: 10.2

When I execute the command python3 main.py configs/train_ours.yaml I get the following error:

/home/munib/git_repos/HPLFlowNet/cmd_args.py:61: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  args = yaml.load(fd)
Attention!!!, ckpt_dir already exists!                                        Whether to continue? [y/n] y
sys.argv:
main.py configs/train_ours.yaml
NUMBA NUM THREADS	16
DEVICE               cuda
allow_less_points    False
arch                 HPLFlowNet
aug_pc2              {'jitter_sigma': 0.01, 'shift_range': 0.3, 'degree_range': 0.0, 'jitter_clip': 0.0}
aug_together         {'jitter_sigma': 0.01, 'shift_range': 1.0, 'scale_low': 0.95, 'scale_high': 1.05, 'degree_range': 0.1745329252, 'jitter_clip': 0.0}
batch_size           1
bcn_use_bias         True
bcn_use_norm         True
ckpt_dir             checkpoints/train/ours
custom_lr            True
data_process         {'DEPTH_THRESHOLD': 35.0, 'NO_CORR': True}
data_root            /mnt/HDD_1/datasets/FlyingThings3D_subset_processed_35m
dataset              FlyingThings3DSubset
dim                  3
epochs               800
evaluate             False
gain                 1.0
init                 xavier
last_relu            False
lr                   0.0001
lr_switch_epochs     [700, 615, 420, 380, 340, 300, 260, 220, 110, 0]
lrs                  [4.9e-06, 7e-06, 1e-05, 1.17649e-05, 1.6807e-05, 2.401e-05, 3.43e-05, 4.9e-05, 7e-05, 0.0001]
num_points           8192
print_freq           50
resume               False
scales_filter_map    [[3.0, 1, -1, -1], [2.0, 1, -1, -1], [1.0, 1, 1, 1], [0.5, 1, 1, 1], [0.25, 1, 1, 1], [0.125, 1, 1, 1], [0.0625, 1, 1, 1]]
use_leaky            True
workers              16

Traceback (most recent call last):
  File "/home/munib/.pyenv/versions/miniconda3-latest/envs/hplflownet/lib/python3.5/site-packages/numba/errors.py", line 491, in new_error_context
    yield
  File "/home/munib/.pyenv/versions/miniconda3-latest/envs/hplflownet/lib/python3.5/site-packages/numba/lowering.py", line 216, in lower_block
    self.lower_inst(inst)
  File "/home/munib/.pyenv/versions/miniconda3-latest/envs/hplflownet/lib/python3.5/site-packages/numba/lowering.py", line 265, in lower_inst
    val = self.lower_assign(ty, inst)
  File "/home/munib/.pyenv/versions/miniconda3-latest/envs/hplflownet/lib/python3.5/site-packages/numba/lowering.py", line 411, in lower_assign
    return self.lower_expr(ty, value)
  File "/home/munib/.pyenv/versions/miniconda3-latest/envs/hplflownet/lib/python3.5/site-packages/numba/lowering.py", line 743, in lower_expr
    res = self.lower_call(resty, expr)
  File "/home/munib/.pyenv/versions/miniconda3-latest/envs/hplflownet/lib/python3.5/site-packages/numba/lowering.py", line 709, in lower_call
    res = impl(self.builder, argvals)
  File "/home/munib/.pyenv/versions/miniconda3-latest/envs/hplflownet/lib/python3.5/site-packages/numba/targets/base.py", line 1043, in __call__
    return self._imp(self._context, builder, self._sig, args)
  File "/home/munib/.pyenv/versions/miniconda3-latest/envs/hplflownet/lib/python3.5/site-packages/numba/targets/arrayobj.py", line 3688, in array_copy
    return _array_copy(context, builder, sig, args)
  File "/home/munib/.pyenv/versions/miniconda3-latest/envs/hplflownet/lib/python3.5/site-packages/numba/targets/arrayobj.py", line 3667, in _array_copy
    ary.itemsize, align=1)
  File "/home/munib/.pyenv/versions/miniconda3-latest/envs/hplflownet/lib/python3.5/site-packages/numba/cgutils.py", line 951, in raw_memcpy
    return _raw_memcpy(builder, 'llvm.memcpy', dst, src, count, itemsize, align)
  File "/home/munib/.pyenv/versions/miniconda3-latest/envs/hplflownet/lib/python3.5/site-packages/numba/cgutils.py", line 943, in _raw_memcpy
    is_volatile])
  File "/home/munib/.pyenv/versions/miniconda3-latest/envs/hplflownet/lib/python3.5/site-packages/llvmlite/ir/builder.py", line 841, in call
    cconv=cconv, tail=tail, fastmath=fastmath)
  File "/home/munib/.pyenv/versions/miniconda3-latest/envs/hplflownet/lib/python3.5/site-packages/llvmlite/ir/instructions.py", line 84, in __init__
    raise TypeError(msg)
TypeError: Type of #4 arg mismatch: i1 != i32

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "main.py", line 290, in <module>
    main()
  File "main.py", line 63, in main
    gen_func=transforms.GenerateDataUnsymmetric(args),
  File "/home/munib/git_repos/HPLFlowNet/transforms/transforms.py", line 297, in __init__
    traversal.go(center, hash_table)
  File "/home/munib/git_repos/HPLFlowNet/transforms/transforms.py", line 119, in go
    self.walk_cuboid(start_key, 0, False, walking_keys, hash_table_list)
  File "/home/munib/git_repos/HPLFlowNet/transforms/transforms.py", line 127, in walk_cuboid
    self.walk_cuboid(walking_keys[d], d + 1, has_zero or (i == 0), walking_keys, hash_table_list)
  File "/home/munib/git_repos/HPLFlowNet/transforms/transforms.py", line 127, in walk_cuboid
    self.walk_cuboid(walking_keys[d], d + 1, has_zero or (i == 0), walking_keys, hash_table_list)
  File "/home/munib/git_repos/HPLFlowNet/transforms/transforms.py", line 127, in walk_cuboid
    self.walk_cuboid(walking_keys[d], d + 1, has_zero or (i == 0), walking_keys, hash_table_list)
  File "/home/munib/git_repos/HPLFlowNet/transforms/transforms.py", line 128, in walk_cuboid
    walking_keys[d] = advance_in_dimension(self.d + 1, 1, d, walking_keys[d])
  File "/home/munib/.pyenv/versions/miniconda3-latest/envs/hplflownet/lib/python3.5/site-packages/numba/dispatcher.py", line 360, in _compile_for_args
    raise e
  File "/home/munib/.pyenv/versions/miniconda3-latest/envs/hplflownet/lib/python3.5/site-packages/numba/dispatcher.py", line 311, in _compile_for_args
    return self.compile(tuple(argtypes))
  File "/home/munib/.pyenv/versions/miniconda3-latest/envs/hplflownet/lib/python3.5/site-packages/numba/dispatcher.py", line 618, in compile
    cres = self._compiler.compile(args, return_type)
  File "/home/munib/.pyenv/versions/miniconda3-latest/envs/hplflownet/lib/python3.5/site-packages/numba/dispatcher.py", line 83, in compile
    pipeline_class=self.pipeline_class)
  File "/home/munib/.pyenv/versions/miniconda3-latest/envs/hplflownet/lib/python3.5/site-packages/numba/compiler.py", line 871, in compile_extra
    return pipeline.compile_extra(func)
  File "/home/munib/.pyenv/versions/miniconda3-latest/envs/hplflownet/lib/python3.5/site-packages/numba/compiler.py", line 365, in compile_extra
    return self._compile_bytecode()
  File "/home/munib/.pyenv/versions/miniconda3-latest/envs/hplflownet/lib/python3.5/site-packages/numba/compiler.py", line 802, in _compile_bytecode
    return self._compile_core()
  File "/home/munib/.pyenv/versions/miniconda3-latest/envs/hplflownet/lib/python3.5/site-packages/numba/compiler.py", line 789, in _compile_core
    res = pm.run(self.status)
  File "/home/munib/.pyenv/versions/miniconda3-latest/envs/hplflownet/lib/python3.5/site-packages/numba/compiler.py", line 251, in run
    raise patched_exception
  File "/home/munib/.pyenv/versions/miniconda3-latest/envs/hplflownet/lib/python3.5/site-packages/numba/compiler.py", line 243, in run
    stage()
  File "/home/munib/.pyenv/versions/miniconda3-latest/envs/hplflownet/lib/python3.5/site-packages/numba/compiler.py", line 676, in stage_nopython_backend
    self._backend(lowerfn, objectmode=False)
  File "/home/munib/.pyenv/versions/miniconda3-latest/envs/hplflownet/lib/python3.5/site-packages/numba/compiler.py", line 626, in _backend
    lowered = lowerfn()
  File "/home/munib/.pyenv/versions/miniconda3-latest/envs/hplflownet/lib/python3.5/site-packages/numba/compiler.py", line 613, in backend_nopython_mode
    self.flags)
  File "/home/munib/.pyenv/versions/miniconda3-latest/envs/hplflownet/lib/python3.5/site-packages/numba/compiler.py", line 990, in native_lowering_stage
    lower.lower()
  File "/home/munib/.pyenv/versions/miniconda3-latest/envs/hplflownet/lib/python3.5/site-packages/numba/lowering.py", line 135, in lower
    self.lower_normal_function(self.fndesc)
  File "/home/munib/.pyenv/versions/miniconda3-latest/envs/hplflownet/lib/python3.5/site-packages/numba/lowering.py", line 176, in lower_normal_function
    entry_block_tail = self.lower_function_body()
  File "/home/munib/.pyenv/versions/miniconda3-latest/envs/hplflownet/lib/python3.5/site-packages/numba/lowering.py", line 201, in lower_function_body
    self.lower_block(block)
  File "/home/munib/.pyenv/versions/miniconda3-latest/envs/hplflownet/lib/python3.5/site-packages/numba/lowering.py", line 216, in lower_block
    self.lower_inst(inst)
  File "/home/munib/.pyenv/versions/miniconda3-latest/envs/hplflownet/lib/python3.5/contextlib.py", line 77, in __exit__
    self.gen.throw(type, value, traceback)
  File "/home/munib/.pyenv/versions/miniconda3-latest/envs/hplflownet/lib/python3.5/site-packages/numba/errors.py", line 499, in new_error_context
    six.reraise(type(newerr), newerr, tb)
  File "/home/munib/.pyenv/versions/miniconda3-latest/envs/hplflownet/lib/python3.5/site-packages/numba/six.py", line 659, in reraise
    raise value
numba.errors.LoweringError: Failed at nopython (nopython mode backend)
Type of #4 arg mismatch: i1 != i32

File "transforms/transforms.py", line 105:
def advance_in_dimension(d1, increment, adv_dim, key):
    key_cp = key.copy()
    ^
[1] During: lowering "$0.3 = call $0.2(vararg=None, args=[], func=$0.2, kws=[])" at /home/munib/git_repos/HPLFlowNet/transforms/transforms.py (105)
-------------------------------------------------------------------------------
This should not have happened, a problem has occurred in Numba's internals.

Please report the error message and traceback, along with a minimal reproducer
at: https://github.com/numba/numba/issues/new

If you need help writing a minimal reproducer please see:
http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports

If more help is needed please feel free to speak to the Numba core developers
directly at: https://gitter.im/numba/numba

Thanks in advance for your help in improving Numba!

I am reporting this here due to the error message. Please help, thank you.
@stuartarchibald
Copy link
Contributor

Thanks for the report. The current version of Numba is 0.50, please could you try with that? Thanks.

@stuartarchibald
Copy link
Contributor

From memory of seeing Type of #4 arg mismatch: i1 != i32, I've a vague recollection of this being due to mixing incompatible llvmlite and numba. In the above llvmlite (latest) came from the numba channel and numba 0.38 from pip, these are not compatible.

@munib94
Copy link
Author

munib94 commented Aug 6, 2020 •
edited

I uninstalled the old Numba version and reinstalled using pip install numba. It installed version 0.47.0, but when I ran it, I did not get the same Numba error. I did get another error, but it may not be related to Numba. Thanks!

@stuartarchibald
Copy link
Contributor

Thanks for trying. Numba and llvmlite are generally released in lockstep, very specific combinations of Numba and llvmlite will work together. If you install Numba using conda you'll get a guaranteed working combination.

@stuartarchibald stuartarchibald added no action required No action was needed to resolve. and removed needtriage labels Aug 7, 2020
@munib94
Copy link
Author

munib94 commented Aug 9, 2020

Thanks for the tip! If you want, you can close this issue.

@esc esc closed this as completed Aug 10, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
no action required No action was needed to resolve.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@esc @stuartarchibald @munib94