Ethernet problems with Arduino H7 and Vision shield

[RobiTobi]

If I run the example image_transfer_jpg_as_the_remote_device_for_your_computer.py on the camera and rpc_image_tansfer_jpg_as_the_controller_device.py on PC, with ethernet connectivity, I get just one or none Image in the pygame window. One image if I run call back img = get_frame_buffer_call_back("sensor.GRAYSCALE", "sensor.QVGA", cutthrough=False, silent=True) then the pygame window freezes with no response. I get none picture (black image and 0 FPS) if I run call back img = get_frame_buffer_call_back("sensor.GRAYSCALE", "sensor.QVGA", cutthrough=True, silent=True)

Steps to reproduce the behavior:

1. Configure IPv4 to ethernet adapter (192.168.1.100 / 255.255.255.0)
2. Connect patch cable from ethernet adapter to the Vision Board
3. Connect camera via USB C Cable with the computer
4. Double press hardware reset on H7
5. Connect to camera via OpenMV IDE (in my case COM4)
6. Flash firmware
7. Wait a few secounds after success Messagebox
8. Paste camera script into OpenMV IDE and save the script
9. Upload the Camera script as main.py, without converting (MessageBox -> No)
10. Eject drive from PC
11. Disconnect and reconnect USB
12. Download master branch from git
13. Create new Py Charm project here D:\openmv-master\tools\rpc
14. Overwrite the code of rpc_image_tansfer_jpg_as_the_controller_device.py with the PC script and save
15. Run RPC script on PC

I expected to get image by image with every call, which is then shown in the pygame window.

After some testing, it turned out, that putting the interface init from top to the while loop, it gives better result. I get multiple images in low speed.
But just with this call back:
img = get_frame_buffer_call_back("sensor.GRAYSCALE", "sensor.QVGA", cutthrough=False, silent=True)

If I try cutthrough=True I get no image, as I mentioned before. After switching back to cutthrough=False there is the same behavior (no image). If you unplug and reconnect the H7 via USB cutthrough=False it is working again.

PC script:

# Image Transfer - As The Controller Device
#
# This script is meant to talk to the "image_transfer_jpg_as_the_remote_device_for_your_computer.py" on the OpenMV Cam.
#
# This script shows off how to transfer the frame buffer to your computer as a jpeg image.

import io, pygame, rpc, serial, serial.tools.list_ports, socket, struct, sys

# Fix Python 2.x.
try: input = raw_input
except NameError: pass

# The RPC library above is installed on your OpenMV Cam and provides mutliple classes for
# allowing your OpenMV Cam to control over USB or WIFI.

##############################################################
# Choose the interface you wish to control an OpenMV Cam over.
##############################################################

# Uncomment the below lines to setup your OpenMV Cam for controlling over a USB VCP.
#
# * port - Serial Port Name.
#
# print("\nAvailable Ports:\n")
# for port, desc, hwid in serial.tools.list_ports.comports():
#     print("{} : {} [{}]".format(port, desc, hwid))
# sys.stdout.write("\nPlease enter a port name: ")
# sys.stdout.flush()
# interface = rpc.rpc_usb_vcp_master(port=input())
# print("")
# sys.stdout.flush()

# Uncomment the below line to setup your OpenMV Cam for controlling over WiFi.
#
# * slave_ip - IP address to connect to.
# * my_ip - IP address to bind to ("" to bind to all interfaces...)
# * port - Port to route traffic to.
#
#interface = rpc.rpc_network_master(slave_ip="192.168.1.41", my_ip="192.168.1.100", port=0x1DBA)

##############################################################
# Call Back Handlers
##############################################################

def get_frame_buffer_call_back(pixformat_str, framesize_str, cutthrough, silent):
    if not silent: print("Getting Remote Frame...")

    result = interface.call("jpeg_image_snapshot", "%s,%s" % (pixformat_str, framesize_str))
    if result is not None:

        size = struct.unpack("<I", result)[0]
        img = bytearray(size)

        if cutthrough:
            # Fast cutthrough data transfer with no error checking.

            # Before starting the cut through data transfer we need to sync both the master and the
            # slave device. On return both devices are in sync.
            result = interface.call("jpeg_image_read")
            if result is not None:

                # GET BYTES NEEDS TO EXECUTE NEXT IMMEDIATELY WITH LITTLE DELAY NEXT.

                # Read all the image data in one very large transfer.
                interface.get_bytes(img, 5000) # timeout

        else:
            # Slower data transfer with error checking.

            # Transfer 32 KB chunks.
            chunk_size = (1 << 15)

            if not silent: print("Reading %d bytes..." % size)
            for i in range(0, size, chunk_size):
                ok = False
                for j in range(3): # Try up to 3 times.
                    result = interface.call("jpeg_image_read", struct.pack("<II", i, chunk_size))
                    if result is not None:
                        img[i:i+chunk_size] = result # Write the image data.
                        if not silent: print("%.2f%%" % ((i * 100) / size))
                        ok = True
                        break
                    if not silent: print("Retrying... %d/2" % (j + 1))
                if not ok:
                    if not silent: print("Error!")
                    return None

        return img

    else:
        if not silent: print("Failed to get Remote Frame!")

    return None

pygame.init()
screen_w = 640
screen_h = 480
try:
    screen = pygame.display.set_mode((screen_w, screen_h), flags=pygame.RESIZABLE)
except TypeError:
    screen = pygame.display.set_mode((screen_w, screen_h))
pygame.display.set_caption("Frame Buffer")
clock = pygame.time.Clock()

while(True):
    interface = rpc.rpc_network_master(slave_ip="192.168.1.41", my_ip="192.168.1.100", port=0x1DBA)
    sys.stdout.flush()

    # You may change the pixformat and the framesize of the image transfered from the remote device
    # by modifying the below arguments.
    #
    # When cutthrough is False the image will be transferred through the RPC library with CRC and
    # retry protection on all data moved. For faster data transfer set cutthrough to True so that
    # get_bytes() and put_bytes() are called after an RPC call completes to transfer data
    # more quicly from one image buffer to another. Note: This works because once an RPC call
    # completes successfully both the master and slave devices are synchronized completely.
    #
    img = get_frame_buffer_call_back("sensor.GRAYSCALE", "sensor.QVGA", cutthrough=False, silent=True)
    if img is not None:
        try:
            screen.blit(pygame.transform.scale(pygame.image.load(io.BytesIO(img), "jpg"), (screen_w, screen_h)), (0, 0))
            pygame.display.update()
            clock.tick()
        except pygame.error: pass

    print(clock.get_fps())

    for event in pygame.event.get():
        if event.type == pygame.QUIT:
            pygame.quit()
            quit()

Camera script:

# Image Transfer - As The Remote Device
#
# This script is meant to talk to the "image_transfer_jpg_as_the_controller_device.py" on your computer.
#
# This script shows off how to transfer the frame buffer to your computer as a jpeg image.

import image, network, omv, rpc, sensor, struct

sensor.reset()
sensor.set_pixformat(sensor.GRAYSCALE)
sensor.set_framesize(sensor.QVGA)
sensor.skip_frames(time = 2000)

# Turn off the frame buffer connection to the IDE from the OpenMV Cam side.
#
# This needs to be done when manually compressing jpeg images at higher quality
# so that the OpenMV Cam does not try to stream them to the IDE using a fall back
# mechanism if the JPEG image is too large to fit in the IDE JPEG frame buffer on the OpenMV Cam.

omv.disable_fb(True)

# The RPC library above is installed on your OpenMV Cam and provides mutliple classes for
# allowing your OpenMV Cam to be controlled over USB or LAN/WLAN.

################################################################
# Choose the interface you wish to control your OpenMV Cam over.
################################################################

# Uncomment the below line to setup your OpenMV Cam for control over a USB VCP.
#
#interface = rpc.rpc_usb_vcp_slave()

# Uncomment the below line to setup your OpenMV Cam for control over the lan.
#
network_if = network.LAN()
network_if.active(True)
network_if.ifconfig(('192.168.1.41', '255.255.255.0', '192.168.1.1', '8.8.8.8'))

interface = rpc.rpc_network_slave(network_if)

# Uncomment the below line to setup your OpenMV Cam for control over the wlan.
#
# network_if = network.WLAN(network.STA_IF)
# network_if.active(True)
# network_if.connect('your-ssid', 'your-password')
#
# interface = rpc.rpc_network_slave(network_if)

################################################################
# Call Backs
################################################################

# When called sets the pixformat and framesize, takes a snapshot
# and then returns the frame buffer jpg size to store the image in.
#
# data is a pixformat string and framesize string.
def jpeg_image_snapshot(data):
    pixformat, framesize = bytes(data).decode().split(",")
    sensor.set_pixformat(eval(pixformat))
    sensor.set_framesize(eval(framesize))
    img = sensor.snapshot().compress(quality=90)
    return struct.pack("<I", img.size())

def jpeg_image_read_cb():
    interface.put_bytes(sensor.get_fb().bytearray(), 5000) # timeout

# Read data from the frame buffer given a offset and size.
# If data is empty then a transfer is scheduled after the RPC call finishes.
#
# data is a 4 byte size and 4 byte offset.
def jpeg_image_read(data):
    if not len(data):
        interface.schedule_callback(jpeg_image_read_cb)
        return bytes()
    else:
        offset, size = struct.unpack("<II", data)
        return memoryview(sensor.get_fb().bytearray())[offset:offset+size]

# Register call backs.

interface.register_callback(jpeg_image_snapshot)
interface.register_callback(jpeg_image_read)

# Once all call backs have been registered we can start
# processing remote events. interface.loop() does not return.

interface.loop()

• IDE version: 2.9.2
• Firmware version: 4.3.1
• Board: Portenta H7 + Arduino Vision shield
• OS: Windows 10/11
• Python 3.10.5
• PyCharm 2022.1.2
• pyserial 3.5
• pygame 2.1.2

Wireshark with default example:
Filter: (ip.src==192.168.1.100 and ip.dst==192.168.1.41) or ( ip.src==192.168.1.41 and ip.dst==192.168.1.100)
cutthrough=True
cutthrough=False

Wireshark file with interface init in while loop:

Interface init in while loop

Initial posted here:
https://forums.openmv.io/t/portenta-h7-and-vision-shield-ethernet-doesnt-work/7212

[iabdalkader]

This issue has been mentioned on OpenMV Forums. There might be relevant details there:

https://forums.openmv.io/t/portenta-h7-and-vision-shield-ethernet-doesnt-work/7212/5

[RobiTobi]

I also figured out, that some TCP Ports are reused. If this happend, there is no transmission and no image in the pygame. This can happen several times in a row. It would be nice, if the port could be defined. Instead of random using ports from a list.

Wireshark interface init while loop port reused

[kwagyeman]

Hi, I have to update the RPC code since the way sockets work has changed. Doing the RTSP scripts now and getting that bug free and then we will update the RPC one.