Delay/lag with Amcrest IP4M-1051

[CZonin]

I recently replaced my cameras with an Amcrest AD410 and IP4M-1051. They are the only 2 cameras I have integrated into Home Assistant/Frigate running on a Pi 4 with a Coral USB Accelerator. The live stream from the AD410 seems to be working great, but the IP4M-1051 is constantly refreshing pretty slow, followed by fast forwarding to catch up. I loaded the rtsp stream in VLC to see if it has the same delay, but it's working perfectly. I've tried dropping the stream to 1080p @ 30fps but it didn't seem to help. Any idea why this is happening? Is my Pi 4 not able to handle 2 streams?

Figured this might be better as an issue: #2379