Index error when training Omnipose in colab

[XinChen1020]

I want to training a Omnipose based on my own dataset on colab but it keeps giving me index error

2021-11-21 17:14:46,035 [INFO] WRITING LOG OUTPUT TO /root/.cellpose/run.log
2021-11-21 17:14:48,953 [INFO] ** TORCH CUDA version installed and working. **
2021-11-21 17:14:48,953 [INFO] >>>> using GPU
2021-11-21 17:14:48,953 [INFO] >>>> Omnipose enabled. See https://raw.githubusercontent.com/MouseLand/cellpose/master/cellpose/omnipose/license.txt for licensing details.
2021-11-21 17:14:49,097 [INFO] >>>> training from scratch
2021-11-21 17:14:49,097 [INFO] >>>> during training rescaling images to fixed diameter of 30.0 pixels
2021-11-21 17:14:49,174 [INFO] Training with rescale = 1.00
2021-11-21 17:14:49,346 [INFO] train channels = 2
2021-11-21 17:14:49,363 [INFO] test channels = 2
2021-11-21 17:14:49,363 [INFO] flows precomputed
2021-11-21 17:14:49,404 [INFO] flows precomputed
2021-11-21 17:14:49,409 [INFO] >>> Using RAdam optimizer
2021-11-21 17:14:49,547 [INFO] >>>> median diameter set to = 30
2021-11-21 17:14:49,547 [INFO] >>>> training network with 2 channel input <<<<
2021-11-21 17:14:49,547 [INFO] >>>> saving every 100 epochs
2021-11-21 17:14:49,547 [INFO] >>>> LR: 0.00020, batch_size: 2, weight_decay: 0.00001
2021-11-21 17:14:49,547 [INFO] >>>> ntrain = 58
2021-11-21 17:14:49,547 [INFO] >>>> rescale is 1
2021-11-21 17:14:49,547 [INFO] >>>> ntest = 6
2021-11-21 17:14:49,547 [INFO] (2, 258, 252)
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/usr/local/lib/python3.7/dist-packages/cellpose/main.py", line 436, in
main()
File "/usr/local/lib/python3.7/dist-packages/cellpose/main.py", line 415, in main
batch_size=args.batch_size, omni=args.omni)
File "/usr/local/lib/python3.7/dist-packages/cellpose/models.py", line 850, in train
batch_size, rescale)
File "/usr/local/lib/python3.7/dist-packages/cellpose/core.py", line 930, in _train_net
rescale=rsc, scale_range=scale_range, unet=self.unet, inds=inds, omni=self.omni)
File "/usr/local/lib/python3.7/dist-packages/cellpose/transforms.py", line 659, in random_rotate_and_resize
xy=xy, do_flip=do_flip, rescale=rescale, inds=inds)
File "/usr/local/lib/python3.7/dist-packages/cellpose/omnipose/core.py", line 324, in random_rotate_and_resize
inds is None if inds is None else inds[n], dist_bg)
File "/usr/local/lib/python3.7/dist-packages/cellpose/omnipose/core.py", line 434, in random_crop_warp
lbl[k] = cv2.warpAffine(labels[k], M, (xy[1],xy[0]), borderMode=mode, flags=method)
IndexError: index 6 is out of bounds for axis 0 with size 6

[schienstockd]

I get the same error message when trying to train a model on our HPC system.
I have a few 100 2D images + masks; all are 200x200.

2021-12-07 11:01:18,543 [INFO] ** TORCH CUDA version installed and working. **
2021-12-07 11:01:18,544 [INFO] >>>> using GPU
2021-12-07 11:01:18,544 [INFO] >>>> Omnipose enabled. See https://raw.githubusercontent.com/MouseLand/cellpose/master/cellpose/omnipose/license.txt for licensing details.
2021-12-07 11:01:22,197 [INFO] >>>> training from scratch
2021-12-07 11:01:22,197 [INFO] >>>> during training rescaling images to fixed diameter of 30.0 pixels
2021-12-07 11:01:22,285 [INFO] Training with rescale = 1.00
2021-12-07 11:01:23,121 [INFO] train channels = 2
2021-12-07 11:01:23,121 [INFO] NOTE: test data not provided OR labels incorrect OR not same number of channels as train data
2021-12-07 11:01:23,121 [INFO] flows precomputed
2021-12-07 11:01:23,323 [INFO] >>> Using RAdam optimizer
2021-12-07 11:01:23,754 [INFO] >>>> median diameter set to = 30
2021-12-07 11:01:23,754 [INFO] >>>> training network with 2 channel input <<<<
2021-12-07 11:01:23,755 [INFO] >>>> saving every 100 epochs
2021-12-07 11:01:23,755 [INFO] >>>> LR: 0.20000, batch_size: 8, weight_decay: 0.00001
2021-12-07 11:01:23,755 [INFO] >>>> ntrain = 530
2021-12-07 11:01:23,755 [INFO] >>>> rescale is 1
2021-12-07 11:01:23,755 [INFO] (2, 200, 200)
Traceback (most recent call last):
  File "/usr/local/easybuild-2019/easybuild/software/compiler/gcccore/10.2.0/python/3.8.6/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/easybuild-2019/easybuild/software/compiler/gcccore/10.2.0/python/3.8.6/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/schienstockd/.local/lib/python3.8/site-packages/cellpose/__main__.py", line 436, in <module>
    main()
  File "/home/schienstockd/.local/lib/python3.8/site-packages/cellpose/__main__.py", line 409, in main
    cpmodel_path = model.train(images, labels, train_files=image_names,
  File "/home/schienstockd/.local/lib/python3.8/site-packages/cellpose/models.py", line 846, in train
    model_path = self._train_net(train_data, train_flows, 
  File "/home/schienstockd/.local/lib/python3.8/site-packages/cellpose/core.py", line 928, in _train_net
    imgi, lbl, scale = transforms.random_rotate_and_resize(
  File "/home/schienstockd/.local/lib/python3.8/site-packages/cellpose/transforms.py", line 658, in random_rotate_and_resize
    return omnipose.core.random_rotate_and_resize(X, Y=Y, scale_range=scale_range, gamma_range=gamma_range, 
  File "/home/schienstockd/.local/lib/python3.8/site-packages/cellpose/omnipose/core.py", line 321, in random_rotate_and_resize
    imgi[n], lbl[n], scale[n] = random_crop_warp(img, y, nt, xy, nchan, scale[n], 
  File "/home/schienstockd/.local/lib/python3.8/site-packages/cellpose/omnipose/core.py", line 431, in random_crop_warp
    return random_crop_warp(img, Y, nt, xy, nchan, scale, rescale, scale_range, gamma_range, do_flip, ind, dist_bg, depth=depth+1)
  File "/home/schienstockd/.local/lib/python3.8/site-packages/cellpose/omnipose/core.py", line 434, in random_crop_warp
    lbl[k] = cv2.warpAffine(labels[k], M, (xy[1],xy[0]), borderMode=mode, flags=method)
IndexError: index 6 is out of bounds for axis 0 with size 6

[authman]

Out of the curious, @schienstockd + @XinChen1020, are you guys training in grayscale?

[schienstockd]

Yes. Is that an issue? It seemed to work when using "normal" cellpose. We have GFP fluorescent cells and would like to use that for prediction.

[pascal-pfeiffer]

I think, i found the issue:

cellpose/cellpose/models.py

Lines 840 to 844 in 45f1a3c

	train_flows = dynamics.labels_to_flows(train_labels, files=train_files, use_gpu=self.gpu, device=self.device, omni=omni)
	if run_test:
	test_flows = dynamics.labels_to_flows(test_labels, files=test_files)
	else:
	test_flows = None

this line is missing the omni argument hand over
test_flows = dynamics.labels_to_flows(test_labels, files=test_files, omni=omni)

add it, delete test flows and try again

[schienstockd]

I think, i found the issue:

cellpose/cellpose/models.py

Lines 840 to 844 in 45f1a3c

	train_flows = dynamics.labels_to_flows(train_labels, files=train_files, use_gpu=self.gpu, device=self.device, omni=omni)
	if run_test:
	test_flows = dynamics.labels_to_flows(test_labels, files=test_files)
	else:
	test_flows = None

this line is missing the omni argument hand over test_flows = dynamics.labels_to_flows(test_labels, files=test_files, omni=omni)

add it, delete test flows and try again

Thanks - that worked

[authman]

Eh, not sure if this worked. I deleted the flows, and altered L842 such that it reads test_flows = dynamics.labels_to_flows(test_labels, files=test_files, omni=omni) but I get the following error:

python -m cellpose --train --train_size --use_gpu --dir ~/train/ --test_dir ~/valid/  --pretrained_model cyto2_omni --omni --chan 0 --chan2 0
2021-12-08 21:39:11,026 [INFO] ** TORCH CUDA version installed and working. **
2021-12-08 21:39:11,026 [INFO] >>>> using GPU
2021-12-08 21:39:11,026 [INFO] >>>> Omnipose enabled. See https://raw.githubusercontent.com/MouseLand/cellpose/master/cellpose/omnipose/license.txt for licensing details.
Not all flows are present. Run flow generation again.
2021-12-08 21:39:13,438 [INFO] >>>> pretrained model /home/user/.cellpose/models/cyto2_omnitorch_0 is being used
2021-12-08 21:39:13,438 [INFO] >>>> during training rescaling images to fixed diameter of 30.0 pixels
2021-12-08 21:39:13,555 [INFO] Training with rescale = 1.00
2021-12-08 21:39:16,802 [INFO] train channels = 2
2021-12-08 21:39:17,612 [INFO] test channels = 2
2021-12-08 21:39:17,612 [INFO] NOTE: computing flows for labels (could be done before to save time)
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 485/485 [00:10<00:00, 48.34it/s]
2021-12-08 21:39:30,151 [INFO] flows precomputed
2021-12-08 21:39:30,221 [INFO] >>> Using RAdam optimizer
2021-12-08 21:39:32,830 [INFO] >>>> median diameter set to = 30
2021-12-08 21:39:32,830 [INFO] >>>> training network with 2 channel input <<<<
2021-12-08 21:39:32,830 [INFO] >>>> saving every 20 epochs
2021-12-08 21:39:32,830 [INFO] >>>> LR: 0.20000, batch_size: 32, weight_decay: 0.00001
2021-12-08 21:39:32,830 [INFO] >>>> ntrain = 485
2021-12-08 21:39:32,830 [INFO] >>>> rescale is 1
2021-12-08 21:39:32,830 [INFO] >>>> ntest = 121
2021-12-08 21:39:32,830 [INFO] (2, 520, 704)
~/venv/lib/python3.8/site-packages/torch/nn/functional.py:718: UserWarning: Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (Triggered internally at  /pytorch/c10/core/TensorImpl.h:1156.)
  return torch.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode)
Traceback (most recent call last):
  File "/home/user/anaconda3/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/user/anaconda3/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "~/venv/lib/python3.8/site-packages/cellpose/__main__.py", line 436, in <module>
    main()
  File "~/venv/lib/python3.8/site-packages/cellpose/__main__.py", line 409, in main
    cpmodel_path = model.train(images, labels, train_files=image_names,
  File "~/venv/lib/python3.8/site-packages/cellpose/models.py", line 846, in train
    model_path = self._train_net(train_data, train_flows, 
  File "~/venv/lib/python3.8/site-packages/cellpose/core.py", line 947, in _train_net
    imgi, lbl, scale = transforms.random_rotate_and_resize(
  File "~/venv/lib/python3.8/site-packages/cellpose/transforms.py", line 658, in random_rotate_and_resize
    return omnipose.core.random_rotate_and_resize(X, Y=Y, scale_range=scale_range, gamma_range=gamma_range, 
  File "~/venv/lib/python3.8/site-packages/cellpose/omnipose/core.py", line 321, in random_rotate_and_resize
    imgi[n], lbl[n], scale[n] = random_crop_warp(img, y, nt, xy, nchan, scale[n], 
  File "~/venv/lib/python3.8/site-packages/cellpose/omnipose/core.py", line 468, in random_crop_warp
    lbl[7] = (gaussian(1-np.clip(bg_edt,0,cutoff)/cutoff, 1)+0.5)
IndexError: index 7 is out of bounds for axis 0 with size 7

UPDATE: Apologies, I had deleted the train flows but had forgotten to delete the validation flows as well. After their removal, everything worked fine.

[sky1ove]

It works!
I think after changing the code, you need to generate flows again then it can process successfully.