Add custom encryption with aiortc python webRTC

[SourceCodeZone]

I am tying to use webrtc insertable stream. Right now my sender peer is aiortc https://github.com/aiortc/aiortc python webrtc lib and receiver is normal browser(latest chrome which support insertable stream).

Right now I am adding simple change on the trasnmistted encoded byte, that is subtract 1(at python-webrtc-aiortc). And at receiver side add 1 to reconstruct the frame.

Java script

     pc.ontrack = function (event) {
     var remoteVideo = document.getElementById("video_");
     remoteVideo.srcObject =event.streams[0];

	    let receiverTransform = new TransformStream({
	    start() {},
	    flush() {},
	    async transform(encodedFrame, controller) {
	      // Reconstruct the original frame.
	      let view = new DataView(encodedFrame.data);

	      let newData = new ArrayBuffer(encodedFrame.data.byteLength);
	      let newView = new DataView(newData);

	      
	      for (let i = 0; i < encodedFrame.data.byteLength ; i++){

                    var sub = 0;
		if(view.getUint8(i)>0&&view.getUint8(i)<255)
		sub = 1;
		newView.setUint8(i, view.getUint8(i)+sub);
                  }
                  encodedFrame.data = newData;
	      controller.enqueue(encodedFrame);
	      },
	    });


	  let receiverStreams = event.receiver.track.kind === 'video' ? event.receiver.createEncodedVideoStreams() : event.receiver.createEncodedAudioStreams();
	  receiverStreams.readableStream.pipeThrough(receiverTransform).pipeTo(receiverStreams.writableStream);

 }

Python code:

I am editing following file https://github.com/aiortc/aiortc/blob/c0504b6962484ac26ba8ad065191794ac6f607a4/src/aiortc/rtcrtpsender.py#L284 and corresponding code, where the decode frame get

                tmpdata = list(payload) //decode frame packet
                for x in range(len(payload)):
                   if(x>3) and (tmpdata[x]>0) and (tmpdata[x]<255):      # first four byte is some inf data, not received at the js section preprocess so avoid it.
                      tmpdata[x] = tmpdata[x] -1
                print(tmpdata)
                packet.ssrc = self._ssrc
                packet.payload = bytearray(tmpdata)

I am getting correct data at js side, but the video not playing.

Few doubts having about python side encryption.

1. Where should I apply the encryption, just after frame encoded on frame data, before adding header, sequence number and timestamp.

2. Why the video not playing while the frame send is received correctly when I apply the above changes.

Please give some advice, how I can add encryption at aiortc side decryption at browser side. I have implemented it like above and data received same as sent but video not playing. If I remove the subtract operation(simple-encryption) from both side then it works.

[alvestrand]

Sorry, this seems to be an aiortc problem.
Does it work when you don't apply any en/decryption?

[alvestrand]

you need to make sure you apply the encryption before the frame is split into RTP packets.

[SourceCodeZone]

Thanks for the feedback, it works fine when I apply no change on sender side and receiver side(no en/decryption).

But still some doubts.
Does the object encodedFrame in the function async transform(encodedFrame, controller) used somewhere else other than this function. Because if I modify the frame at sender and reverted that change inside transform function the video not play, showing progress indicator in player. I have verified it by printing the send and received data and observed both data are same.

I got some progress(that is received distorted video) when I discard first 10 byte from encryption/decryption at sender/receiver. Based on the webrtc example https://github.com/webrtc/samples/blob/gh-pages/src/content/peerconnection/endtoend-encryption/js/worker.js the variable cryptoOffset has maximum value 10 according to frame type so I chose 10. But still received distorted video, but at least playing it.

I will make sure whether packet is sending before or after RTP packet split.

[murillo128]

The encryption is done on full media frames, just after the encoder and before the rtp packetization.
So you have to do the encryption here:
https://github.com/aiortc/aiortc/blob/c0504b6962484ac26ba8ad065191794ac6f607a4/src/aiortc/codecs/vpx.py#L350
on the self.buffer

[SourceCodeZone]

I have done on the same place but receive distorted frame.

[Sean-Der]

Hey @SourceCodeZone I am also discussing this on #37

It sounds like this will not work unless you implement the GFD extension. The docs aren't public yet, but when I get access to them will share the PR I make to https://github.com/pion/webrtc

Right now even if I properly decrypt everything my video fails to play. I ran two streams and compare the buffers (one didn't decode before send, and the other did) and the buffers were exactly the same.

[SourceCodeZone]

Thanks for the information, for me the goal was implement custom encryption between two peers, so as a solution I have implemented without insertable stream using aiortc only. Like the sender peer(aiortc-peer) send the frame by encrypting. The receiver(aiortc-peer) peer received the data and decrypted, then decode and displayed in webserver using Flask as described here https://medium.com/datadriveninvestor/video-streaming-using-flask-and-opencv-c464bf8473d6

[Sean-Der]

@SourceCodeZone that is very cool! Nice work, I bet you learned a lot building that.

I will try my best to help with aiortc after I land my change in Pion's code base. I know enough Python to be dangerous :)

[SourceCodeZone]

Thanks man, and all the very best with your Pion's code base work.

[murillo128]

It sounds like this will not work unless you implement the GFD extension. The docs aren't public yet, but when I get access to them will share the PR I make to https://github.com/pion/webrtc

Right now even if I properly decrypt everything my video fails to play. I ran two streams and compare the buffers (one didn't decode before send, and the other did) and the buffers were exactly the same.

If you are skipping the first bytes of the payload, it should work and most likely you are not doing something correctly. Implementing GFD will most probably not going to solve the problem for you.

[alvestrand]

Closing issue as "not requiring any spec change".