Adding support for audio/video

[sashahilton00]

Hi,

Opening an issue here after @Elektroarzt and I emailed for a bit. Relevant parts of the the email trail pasted below:

@sashahilton00 - Out of interest did you get anywhere with the audio/video component? From what I’ve read it’s some sort of composite video, I’m wondering if it would be possible to add a decoder chip and create an RTSP server on the device when the camera is activated, then send the server address over MQTT to enable a remote feed.

@Elektroarzt - Concerning the video option, we (Gurkenflieger and me) did not investigate yet but I’d love such a feature. This would be a Piggy-Back on the existing PCB or a complete new design with maybe 4 layers instead of 2, as the current design is at the lower end of JLCPCBs 2 layer design rules in terms of trace width, micro vias and so on.

@sashahilton00 - For the audio/video I’ve done a small amount of reading - it seems to be using some sort of analog audio and NTSC/PAL video on the bus from what I can tell. The latter would need a dedicated decoder chip that fed digital out into the ESP32 via i2s, need to investigate the audio. The comelit-simplebus1 project on GitHub may offer some insight, as I’d be surprised if they’d changed much between the two.
You may be right about the 4 layers being needed, though it would be good to keep it as 2 to keep costs down and allow people with a home setup to manufacture it. Also one would need to give some consideration to power draw as I’m sure video decoding will use a fair bit more than the current system.

@Elektroarzt - From what I read, the video feature would bring necessity for an external PSRAM attached to the ESP, as the 400kB or so would’t be sufficient to store a single frame in the onboard RAM.

@sashahilton00 - Both https://www.espboards.dev/blog/esp32-soc-options/ and https://gist.github.com/sekcompsci/2bf39e715d5fe47579fa184fa819f421 seem to suggest that the ESP32-S2 has both an internal ADC and DAC. My PCB skills are a little rusty, but looking at a similar project (https://github.com/vvigilante/comelit-simplebus1), it looks as though that’s what vvigilante has done - used the internal ADC/DACs for the audio with a potentiometer to allow for manual adjustment (could be neat if this were a digital potentiometer) - I don’t have an oscilloscope to measure, but vvigilante mentions that peak to peak amplitude is approximately 2.5v.
My thinking is that one could incorporate the analogue write/analogue write enable circuitry into your design, and at the same time switch from the Seeed Studio XIAO ESP32C3 to something like the Adafruit QT Py ESP32-S2 (https://shop.pimoroni.com/products/adafruit-qt-py-esp32-s2-wifi-dev-board-with-ufl-antenna-port-stemma-qt?variant=39658449764435), which keeps the xiao footprint, type c and external antenna, or equally one could move to having the esp32 soldered directly by JLC or similar (eg. ESP32-S2-MINI-2U-N4), and including the type C connector manually.
One thought I did have if one goes down the soldered board route, is to make the type C connector (and possibly the bus connector) vertical, eg. https://jlcpcb.com/partdetail/Xunpu-TYPEC_300ABCP16H175/C5454935, as when in the standard euro switch box the USB would then be accessible without needing to remove the device for firmware updates.
If one went down this route it looks as though esp-adf (https://github.com/espressif/esp-adf) would provide the necessary support to implement two way audio streaming (have a look at the following for an RTSP implementation: https://github.com/espressif/esp-adf/tree/48dea7711391e0abf00c65aca18557572fc5189a/examples/protocols/esp-rtsp), where a ring of the bell would set up the rtsp server and send the ringing MQTT message, whilst the client if a microphone is available can send an RTSP address in an MQTT message for the microphone stream. There’s also RTC/VoIP possibilities if that’s preferred.
If that sounds of interest and you’re happy to modify the PCB to use an S2 variant chip with the ADC/DAC wired up to the bus, I’d be happy to get JLC to make me up a board and see if I can write some software to get two way audio working.

@Elektroarzt - I had to read a while into your links. Here is my opinion:

• the Adafruit QT Py ESP32-S2 seems to be a good unit for audio, as it has ADC / DAC, but has only 2MB PSRAM, I don’t know how much is needed for capturing the video later on
• maybe the XIAO ESP32-S3 could be interesting, it comes with 8MB PSRAM, but without the DAC and is pin compatible. Maybe a PWM with RC-filtering will do? If so, it would be possible to start on the current hardware with a piggy-back, as there are 2 spare GPIO pins of the XIAO exposed to the expansion header J2 which maybe could be used for the audio part and on the other side the bus signal is wired to the debug header J3
• an external antenna is a must from my point of view, to keep stable operation in the wall box
• concerning the vertical connector, it would be necessary to go on a 4 layer design from what I see in this tight environment - the hardware design guide also looks not really compatible for a cheap JLCPCB job in terms of track width, spacings and drill diameter (design guide attached), so I like to keep a dev board as long as possible; I also think there is no necessity for contacting the USB port too often, as there is the OTA possibility
  All in all I’m not into software, so I can’t say anything to your rtsp server suggestions. Up to now gurkenflieger and me split the work. Gurkenflieger made the schematic and the software. I made the CAD work for PCB in KiCAD, 3d modeling in Fusion 360 and printing the housing. Would be good to lift this discussion on Git, to involve him. I’m sure gurkenflieger has also some ideas.

[sashahilton00]

@Gurkenflieger if you have any thoughts they would be most welcome.

Having done some more reading up today on the ESP32 and video capture, it looks as though it will be a stretch to manage it. If you have any insights as to the video format that Comelit is using that would be great, everything I read suggeests it's NTSC/PAL.

With that in mind, the ESP32 has support for the ESP32-CAM module, which seems to support an 8 or 16 bit DVP image sensor. There are also low power PAL/NTSC decoder chips available (eg. TVP5150AM1), which output the same 8-bit YCbCr 4:2:2 as the esp32 cam to my knowledge. The esp32 apparently has a 20MHz limit on the pixel clock from the sensor, whilst PAL/NTSC requires a 27Mhz clock, though after a bit of searching theres this rather neat workaround by the looks of it from ESP_igrr, which uses a frequency divider and two octal latching ICs to convert the 8-bit 27Mhz stream into a 16 bit 14.5Mhz stream, essentially halving the overhead.

However, this is where I'm not sure the video component is solvable - the raw video stream is too large to stream (as highlighted here), if you've got any thoughts as to whether this is doable without building a fully fledged streaming SBC, I'm all ears, though dealing with video decoding/encoding feels like it's getting a bit beyond the scope of a simple retrofit device.

[Elektroarzt]

@sashahilton00 I can say that the video option would be out of my comfort zone concerning the RF layout and debugging. If you like to contribute and go for a prototype with the audio part, you are welcome. I then could integrate in the PCB, if tests run ok.