Protect LineFollower from None cam_img

[Ezward]

User anbello is getting an error trying to start the cv_control template.:

Traceback (most recent call last):
  File "/home/pi/projects/donkeycar/donkeycar/vehicle.py", line 151, in start
    self.update_parts()
  File "/home/pi/projects/donkeycar/donkeycar/vehicle.py", line 199, in update_parts
    outputs = p.run(*inputs)
  File "./manage.py", line 80, in run
    max_yellow, confidense, mask = self.get_i_color(cam_img)
  File "./manage.py", line 59, in get_i_color
    scan_line = cam_img[iSlice : iSlice + self.scan_height, :, :]
TypeError: 'NoneType' object is not subscriptable

My theory is that when the template first starts, the camera is not 'warmed' so it doesn't immediately return an image, which causes the LineFollower to fail. We should 1) warm the camera (wait until it starts providing frames) 2) protect LineFollower from a None image by just returning.

• Add a check in line 80 of

  donkeycar/donkeycar/templates/cv_control.py

  Line 80 in 3297771

  max_yellow, confidense, mask = self.get_i_color(cam_img)

  , if cam_img is not None: and put all the the code except for that last return statement inside the if statement.
• Add the camera warming code as in complete.py

Discovered bug: cv_control.py will not run on a headless OS install because it tries to open a window with debug info using opencv. The fix is to make this debug window optional.

Tasks

• Add camera warming code to PiCamera
• Add camera warming code to Webcam
• Add camera warming code to CISCamera
• Make cv_control.py debug window optional

Testing

• reproduce reported bug in cv_control.py template on Raspberry Pi with 'PICAM' configuration with dev branch (commit a26bb73)
• verify cv_control.py bug cannot be repeated using fix branch with camera warming fix for PiCamera, 'PICAM' configuration on Raspberry Pi

Note: these tests require installation of 'imgaug' on the donkeycar, which is an optional install pip install imgaug

• reproduce newly found bug in 'CROP' image transform configuration in complete.py dev branch (commit a26bb73) on Raspberry Pi with 'PICAM' camera

• reproduce 'CROP' transform bug in dev branch (commit a26bb73) on Jetons Nano 2GB with 'CSIC' camera configuration

• verify complete.py 'CROP' transform bug cannot be repeated using fix branch and PICAM camera on Raspbery Pi

• verify complete.py 'CROP' transform bug cannot be repeated using fix branch and CSIC camera on Jetson Nano

• verify complete.py 'CROP' transform bug cannot be repeated using fix branch and WEBCAM camera (not requires optional install of pygame; sudo apt-get install libsdl2-mixer-2.0-0 libsdl2-image-2.0-0 libsdl2-2.0-0, pip install pygame

• Verify that cv_control.py template can run without debug window on headless OS. Tested on Raspberry Pi with PICAM configuration and headless Raspberry Pi OS (tested via ssh without HDMI connected)

[Ezward]

It looks like the picam already handles it's own warming; perhaps it should wait until frames get returned.

donkeycar/donkeycar/parts/camera.py

Line 35 in 3297771

print('PiCamera loaded.. .warming camera')

[Ezward]

Calling the run() method on the camera part does NOT set self.frame. It should probably set self.frame and return that if it does not get a frame from the camera. That's not the issue here, but probably still a good idea. Then we can call the run method until a non-empty frame is returned and then we know positively that the camera is warmed OR if we do this for 2 seconds and do not get a frame then we know the camera is not going to work. Then when enter the vehicle loop the camera part will already have one frame ready.

[Ezward]

https://github.com/autorope/donkeycar/tree/961-protect-from-empty-camera-image

• this branch updates the PiCamera and CISCamera to warm until they get a frame
• this branch protects the LineFollower from None image

[Ezward]

I get a related bug using the complete.py template from dev branch and a model that uses 'CROP' transformation. In that case it appears that imgaug get's a None image and becomes unhappy

INFO:donkeycar.vehicle:Starting vehicle at 20 Hz
/usr/lib/python3/dist-packages/picamera/encoders.py:544: PiCameraResolutionRounded: frame size rounded up from 160x120 to 160x128
  width, height, fwidth, fheight)))
Traceback (most recent call last):
  File "/home/pi/projects/autorope/donkeycar/donkeycar/vehicle.py", line 154, in start
    self.update_parts()
  File "/home/pi/projects/autorope/donkeycar/donkeycar/vehicle.py", line 202, in update_parts
    outputs = p.run(*inputs)
  File "/home/pi/projects/autorope/donkeycar/donkeycar/pipeline/augmentations.py", line 120, in run
    aug_img_arr = self.augmentations.augment_image(img_arr)
  File "/home/pi/env/lib/python3.7/site-packages/imgaug/augmenters/meta.py", line 766, in augment_image
    type(image).__name__),)
TypeError: %d format: a number is required, not str
INFO:donkeycar.vehicle:Shutting down vehicle and its parts...

If I remove 'CROP' transformation then the model runs without the error (of course it does not perform well) So that shows it is the imgaug that is erroring.
If I run the fix branch https://github.com/autorope/donkeycar/tree/961-protect-from-empty-camera-image then the error goes away. That indicates the imgaug was crapping out on a None camera image. So fixing the camera parts to make sure they return non-none images fixes more than one kind of bug.

[Ezward]

I was able to repeat the cv_control bug using the current dev branch (commit a26bb73) . I then switch the donkeycar project folder to the fix branch, https://github.com/autorope/donkeycar/tree/961-protect-from-empty-camera-image and the error went array. So this confirms that the issue was with a None image and the fix branch fixes that. I need to test with CISC camera and Web cam as well.

[Ezward]

Got the imgaug error on a Jetson Nano with CSIC camera config trying to run a model that uses 'CROP' configuration using the complete.py template and the dev branch;
relevant configuration in myconfig.py:

CAMERA_TYPE = "CSIC"   # (PICAM|WEBCAM|CVCAM|CSIC|V4L|D435|MOCK|IMAGE_LIST)
IMAGE_W = 160
IMAGE_H = 120
IMAGE_DEPTH = 3         # default RGB=3, make 1 for mono

TRANSFORMATIONS = ['CROP']
ROI_CROP_TOP = 45               # the number of rows of pixels to ignore on the top of the image

python manage.py drive --type=tflite_linear --model=models/linear_crop_multiply.tflite results in

INFO:donkeycar.vehicle:Starting vehicle at 20 Hz
Traceback (most recent call last):
  File "/home/ed/projects/donkeycar/donkeycar/vehicle.py", line 154, in start
    self.update_parts()
  File "/home/ed/projects/donkeycar/donkeycar/vehicle.py", line 202, in update_parts
    outputs = p.run(*inputs)
  File "/home/ed/projects/donkeycar/donkeycar/pipeline/augmentations.py", line 120, in run
    aug_img_arr = self.augmentations.augment_image(img_arr)
  File "/home/ed/env/lib/python3.6/site-packages/imgaug/augmenters/meta.py", line 766, in augment_image
    type(image).__name__),)
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/home/ed/projects/donkeycar/donkeycar/parts/camera.py", line 174, in update
    self.init_camera()
  File "/home/ed/projects/donkeycar/donkeycar/parts/camera.py", line 169, in init_camera
    self.poll_camera()
  File "/home/ed/projects/donkeycar/donkeycar/parts/camera.py", line 181, in poll_camera
    self.frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
cv2.error: OpenCV(4.5.4) /tmp/pip-req-build-bbq4c33o/opencv/modules/imgproc/src/color.cpp:182: error: (-215:Assertion failed) !_src.empty() in function 'cvtColor'

TypeError: %d format: a number is required, not str

INFO:donkeycar.vehicle:Shutting down vehicle and its parts...
stopping CSICamera

[Ezward]

Pull Request 967 addresses this issue and a little more.

[Ezward]

We are unable to test 'CROP' with CSIC camera on the Jetson Nano because of a bug in imgaug on the Jetson Nano; see Issue 970. We can verify that the CSIC camera passes the warm up phase and returns frames. So the 'CROP' code would work if imgaug worked and given we have confirmed 'CROP' works on RaspberryPi with PICAM and we have not actually changed any 'CROP' code, then we can presume that it would work on the Nano if imgaug is fixed.

[Ezward]

Merged #967