train yolov3-tiny error

[ou525]

when i train yolov3-tiny, appear error matrix_wrap.cpp:800: error: (-215:Assertion failed) (flags & FIXED_TYPE) != 0 in function 'type'. opencv version 3.4.3. please how to solve it @AlexeyAB

[AlexeyAB]

• Can you show screenshot of this error?
• What OS do you use?
• What parameters do you use in the Makefile?

[ou525]

@AlexeyAB
OS:
Linux version 4.15.0-42-generic (buildd@lgw01-amd64-023) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu116.04.10)) #4516.04.1-Ubuntu SMP Mon Nov 19 13:02:27 UTC 2018

makefile:
GPU=1
CUDNN=1
CUDNN_HALF=0
OPENCV=1
AVX=0
OPENMP=0
LIBSO=1

set GPU=1 and CUDNN=1 to speedup on GPU

set CUDNN_HALF=1 to further speedup 3 x times (Mixed-precision using Tensor Cores) on GPU Tesla V100, Titan V, DGX-2

set AVX=1 and OPENMP=1 to speedup on CPU (if error occurs then set AVX=0)

DEBUG=0

ARCH= -gencode arch=compute_30,code=sm_30 \

-gencode arch=compute_35,code=sm_35 \

-gencode arch=compute_50,code=[sm_50,compute_50] \

-gencode arch=compute_52,code=[sm_52,compute_52] \

-gencode arch=compute_61,code=[sm_61,compute_61]

OS := $(shell uname)

Tesla V100

ARCH= -gencode arch=compute_61,code=[sm_61,compute_61]

GeForce RTX 2080 Ti, RTX 2080, RTX 2070 Quadro RTX 8000, Quadro RTX 6000, Quadro RTX 5000 Tesla T4

ARCH= -gencode arch=compute_75,code=[sm_75,compute_75]

Jetson XAVIER

ARCH= -gencode arch=compute_72,code=[sm_72,compute_72]

GTX 1080, GTX 1070, GTX 1060, GTX 1050, GTX 1030, Titan Xp, Tesla P40, Tesla P4

ARCH= -gencode arch=compute_61,code=sm_61 -gencode arch=compute_61,code=compute_61

GP100/Tesla P100 – DGX-1

ARCH= -gencode arch=compute_60,code=sm_60

For Jetson TX1, Tegra X1, DRIVE CX, DRIVE PX - uncomment:

ARCH= -gencode arch=compute_53,code=[sm_53,compute_53]

For Jetson Tx2 or Drive-PX2 uncomment:

ARCH= -gencode arch=compute_62,code=[sm_62,compute_62]

VPATH=./src/
EXEC=darknet
OBJDIR=./obj/

ifeq ($(LIBSO), 1)
LIBNAMESO=darknet.so
APPNAMESO=uselib
endif

CC=gcc
CPP=g++
NVCC=nvcc
OPTS=-Ofast
LDFLAGS= -lm -pthread
COMMON=
CFLAGS=-Wall -Wfatal-errors -Wno-unused-result -Wno-unknown-pragmas

ifeq ($(DEBUG), 1)
OPTS= -O0 -g
else
ifeq ($(AVX), 1)
CFLAGS+= -ffp-contract=fast -mavx -mavx2 -msse3 -msse4.1 -msse4.2 -msse4a
endif
endif

CFLAGS+=$(OPTS)

ifeq ($(OPENCV), 1)
COMMON+= -DOPENCV
CFLAGS+= -DOPENCV
LDFLAGS+= pkg-config --libs opencv
COMMON+= pkg-config --cflags opencv
endif

ifeq ($(OPENMP), 1)
CFLAGS+= -fopenmp
LDFLAGS+= -lgomp
endif

ifeq ($(GPU), 1)
COMMON+= -DGPU -I/usr/local/cuda/include/
CFLAGS+= -DGPU
ifeq ($(OS),Darwin) #MAC
LDFLAGS+= -L/usr/local/cuda/lib -lcuda -lcudart -lcublas -lcurand
else
LDFLAGS+= -L/usr/local/cuda/lib64 -lcuda -lcudart -lcublas -lcurand
endif
endif

ifeq ($(CUDNN), 1)
COMMON+= -DCUDNN
ifeq ($(OS),Darwin) #MAC
CFLAGS+= -DCUDNN -I/usr/local/cuda/include
LDFLAGS+= -L/usr/local/cuda/lib -lcudnn
else
CFLAGS+= -DCUDNN -I/usr/local/cudnn/include
LDFLAGS+= -L/usr/local/cudnn/lib64 -lcudnn
endif
endif

ifeq ($(CUDNN_HALF), 1)
COMMON+= -DCUDNN_HALF
CFLAGS+= -DCUDNN_HALF
ARCH+= -gencode arch=compute_70,code=[sm_70,compute_70]
endif

OBJ=http_stream.o gemm.o utils.o cuda.o convolutional_layer.o list.o image.o activations.o im2col.o col2im.o blas.o crop_layer.o dropout_layer.o maxpool_layer.o softmax_layer.o data.o matrix.o network.o connected_layer.o cost_layer.o parser.o option_list.o darknet.o detection_layer.o captcha.o route_layer.o writing.o box.o nightmare.o normalization_layer.o avgpool_layer.o coco.o dice.o yolo.o detector.o layer.o compare.o classifier.o local_layer.o swag.o shortcut_layer.o activation_layer.o rnn_layer.o gru_layer.o rnn.o rnn_vid.o crnn_layer.o demo.o tag.o cifar.o go.o batchnorm_layer.o art.o region_layer.o reorg_layer.o reorg_old_layer.o super.o voxel.o tree.o yolo_layer.o upsample_layer.o
ifeq ($(GPU), 1)
LDFLAGS+= -lstdc++
OBJ+=convolutional_kernels.o activation_kernels.o im2col_kernels.o col2im_kernels.o blas_kernels.o crop_layer_kernels.o dropout_layer_kernels.o maxpool_layer_kernels.o network_kernels.o avgpool_layer_kernels.o
endif

OBJS = $(addprefix $(OBJDIR), $(OBJ))
DEPS = $(wildcard src/*.h) Makefile

all: obj backup results $(EXEC) $(LIBNAMESO) $(APPNAMESO)

ifeq ($(LIBSO), 1)
CFLAGS+= -fPIC

$(LIBNAMESO): $(OBJS) src/yolo_v2_class.hpp src/yolo_v2_class.cpp
$(CPP) -shared -std=c++11 -fvisibility=hidden -DYOLODLL_EXPORTS $(COMMON) $(CFLAGS) $(OBJS) src/yolo_v2_class.cpp -o $@ $(LDFLAGS)

$(APPNAMESO): $(LIBNAMESO) src/yolo_v2_class.hpp src/yolo_console_dll.cpp
$(CPP) -std=c++11 $(COMMON) $(CFLAGS) -o $@ src/yolo_console_dll.cpp $(LDFLAGS) -L ./ $(LIBNAMESO)
endif

$(EXEC): $(OBJS)
$(CPP) -std=c++11 $(COMMON) $(CFLAGS) $^ -o $@ $(LDFLAGS)

$(OBJDIR)%.o: %.c $(DEPS)
$(CC) $(COMMON) $(CFLAGS) -c $< -o $@

$(OBJDIR)%.o: %.cpp $(DEPS)
$(CPP) $(COMMON) $(CFLAGS) -c $< -o $@

$(OBJDIR)%.o: %.cu $(DEPS)
$(NVCC) $(ARCH) $(COMMON) --compiler-options "$(CFLAGS)" -c $< -o $@

obj:
mkdir -p obj
backup:
mkdir -p backup
results:
mkdir -p results

.PHONY: clean

clean:
rm -rf $(OBJS) $(EXEC) $(LIBNAMESO) $(APPNAMESO)

error:
99: 661.598022, 531.605225 avg loss, 0.000000 rate, 2.199952 seconds, 6336 images
Loaded: 0.000027 seconds
Region 16 Avg IOU: 0.182536, Class: 0.534901, Obj: 0.720300, No Obj: 0.489493, .5R: 0.000000, .75R: 0.000000, count: 2
Region 23 Avg IOU: 0.160294, Class: 0.490894, Obj: 0.516987, No Obj: 0.497239, .5R: 0.058824, .75R: 0.000000, count: 34
Region 16 Avg IOU: 0.367009, Class: 0.651940, Obj: 0.554976, No Obj: 0.489369, .5R: 0.333333, .75R: 0.000000, count: 3
Region 23 Avg IOU: 0.260651, Class: 0.527383, Obj: 0.476481, No Obj: 0.497853, .5R: 0.111111, .75R: 0.055556, count: 18
Region 16 Avg IOU: 0.249406, Class: 0.633855, Obj: 0.546919, No Obj: 0.487636, .5R: 0.062500, .75R: 0.062500, count: 16
Region 23 Avg IOU: 0.190888, Class: 0.555747, Obj: 0.513262, No Obj: 0.497771, .5R: 0.000000, .75R: 0.000000, count: 36
Region 16 Avg IOU: 0.357677, Class: 0.603474, Obj: 0.372896, No Obj: 0.488863, .5R: 0.125000, .75R: 0.000000, count: 8
Region 23 Avg IOU: 0.231945, Class: 0.476930, Obj: 0.474184, No Obj: 0.497165, .5R: 0.157895, .75R: 0.000000, count: 19
Region 16 Avg IOU: 0.221145, Class: 0.651452, Obj: 0.607109, No Obj: 0.491284, .5R: 0.125000, .75R: 0.000000, count: 8
Region 23 Avg IOU: 0.160383, Class: 0.486856, Obj: 0.497579, No Obj: 0.497273, .5R: 0.047619, .75R: 0.000000, count: 21
Region 16 Avg IOU: 0.280254, Class: 0.633501, Obj: 0.536651, No Obj: 0.489481, .5R: 0.000000, .75R: 0.000000, count: 9
Region 23 Avg IOU: 0.220955, Class: 0.537242, Obj: 0.434310, No Obj: 0.497846, .5R: 0.000000, .75R: 0.000000, count: 12
Region 16 Avg IOU: 0.379991, Class: 0.668899, Obj: 0.518288, No Obj: 0.489482, .5R: 0.142857, .75R: 0.000000, count: 14
Region 23 Avg IOU: 0.239220, Class: 0.509071, Obj: 0.530566, No Obj: 0.497130, .5R: 0.065217, .75R: 0.000000, count: 46
Region 16 Avg IOU: 0.392891, Class: 0.684374, Obj: 0.424502, No Obj: 0.490795, .5R: 0.375000, .75R: 0.000000, count: 8
Region 23 Avg IOU: 0.243940, Class: 0.535487, Obj: 0.467535, No Obj: 0.497445, .5R: 0.000000, .75R: 0.000000, count: 21

100: 644.027222, 542.847412 avg loss, 0.000000 rate, 2.199051 seconds, 6400 images
terminate called after throwing an instance of 'cv::Exception'
what(): OpenCV(3.4.3) /home/ou/opencv/opencv-3.4.3/opencv-3.4.3/modules/core/src/matrix_wrap.cpp:800: error: (-215:Assertion failed) (flags & FIXED_TYPE) != 0 in function 'type'

Aborted (core dumped)

[ou525]

gdb result:
[Thread 0x7fff01bb8700 (LWP 6194) exited]
[Thread 0x7fff033bb700 (LWP 6191) exited]
terminate called after throwing an instance of 'cv::Exception'
what(): OpenCV(3.4.3) /home/ou/opencv/opencv-3.4.3/opencv-3.4.3/modules/core/src/matrix_wrap.cpp:800: error: (-215:Assertion failed) (flags & FIXED_TYPE) != 0 in function 'type'

Thread 1 "darknet" received signal SIGABRT, Aborted.
0x00007fffd9ab8428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
54 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) where
#0 0x00007fffd9ab8428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
#1 0x00007fffd9aba02a in __GI_abort () at abort.c:89
#2 0x00007fffda61884d in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#3 0x00007fffda6166b6 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#4 0x00007fffda616701 in std::terminate() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5 0x00007fffda616919 in __cxa_throw () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#6 0x00007ffff3da224a in cv::error(cv::Exception const&) () from /usr/local/lib/libopencv_core.so.3.4
#7 0x00007ffff3da2344 in cv::error(int, cv::String const&, char const*, char const*, int) () from /usr/local/lib/libopencv_core.so.3.4
#8 0x00007ffff3ce39b9 in cv::errorNoReturn () from /usr/local/lib/libopencv_core.so.3.4
#9 0x00007ffff3ce696d in cv::_InputArray::type(int) const () from /usr/local/lib/libopencv_core.so.3.4
#10 0x00007ffff3bad06c in cv::Mat::copyTo(cv::_OutputArray const&) const () from /usr/local/lib/libopencv_core.so.3.4
#11 0x00007ffff736ed8b in cvSaveImage () from /usr/local/lib/libopencv_imgcodecs.so.3.4
#12 0x000000000043ced4 in draw_train_loss ()
#13 0x000000000046e0ac in train_detector ()
#14 0x0000000000471ffd in run_detector ()
#15 0x00000000004059f4 in main ()

[ou525]

when i use -dont_show comand, its run ok