Using FFmpeg without write/read file path?

[Mamena2020]

My primary objective is to record audio and alter its pitch in real-time. The steps I'm currently working on are as follows:

1. Recording:
   For recording, I'm using this package flutter sound and streaming audio data in pcm16 format (Uint8List).

2. Auto Pitch:
   To achieve pitch shifting, I'm using this ffmpeg kit. The process involves several steps:

   • Write the audio data to files.
   • execute FFmpeg arguments for applying the desired pitch.
   • Read the output audio data from file.
   • Play output audio player.feedFromStream(outputAudio) using FlutterSoundPlayer from flutter sound
     I have successfully accomplished this, but the real-time pitch shifting experience is not as smooth as I expected. This is likely too much steps used affecting real time performance.

i've try to use ffmpeg using the Dart Process API, but seems not luck.

    
   Future<Uint8List> testAudio({required Uint8List audioData}) async {
    
      final additionalArguments = [
        '-af',
        'volume=1'  // example argument
      ]; 

      final ffmpegProcess = await io.Process.start(
        'ffmpeg',
        [
          '-i', 'pipe:0', // input
          ...additionalArguments,
          '-f', 'wav', 'pipe:1', // output
        ],
        mode: io.ProcessStartMode.detachedWithStdio,
        runInShell: true,
      );
      ffmpegProcess.stdin.add(audioData);
      await ffmpegProcess.stdin.flush();
      await ffmpegProcess.stdin.close();
     
      final bytes = await ffmpegProcess.stdout.toBytes();
      ffmpegProcess.kill();
      return Uint8List.fromList(bytes);
  }

   extension StreamToBytes on Stream<List<int>> {
      Future<List<int>> toBytes() async {
        final chunks = await toList();
        final bytesBuilder = BytesBuilder();
        for (final chunk in chunks) {
          bytesBuilder.add(chunk);
        }
        return bytesBuilder.takeBytes();
      }
    }

How i can filter audio in real-time using ffmpeg and get result in Uint8list? I mean, if using ffmpeg with path to write/read, it too much step to get result in realtime. any solution for this?

[ersha80]

why close this issue? Did you find any solutions?

[Mamena2020]

@ersha80 Did I find a solution? Yes, if I use ffmpeg without writing/reading file paths. Instead, I use UDP/TCP running on the local machine. However, this does not resolve my problem because ffmpeg cannot update the arguments during execution. What I need is to change the pitch in real-time encoding.

Here is an example code for using TCP input/output. I am not using UDP because UDP sends data quickly but sometimes unordered. This can be handled by adding "atempo" to get ordered results when using UDP output. Therefore, the better option is TCP.

1. Create TCP receiver

      String tcpAddress = "127.0.0.1";
      io.ServerSocket? tcpReceiver;
      io.Socket? tcpSocketReceiver;
     
      Queue<Uint8List> queueData = Queue<Uint8List>();
      Uint8List receivedData = Uint8List(0); 
     
      tcpReceiver = await io.ServerSocket.bind(tcpAddress, tcpOutputPort);
      tcpReceiver!.listen((socket) {
        tcpSocketReceiver = socket;
        int expectedDataSize = 640; 
  
        tcpSocketReceiver!.listen((event) {
          // combine new data with old data
          receivedData = Uint8List.fromList(receivedData + event);
          // processing data -> output from ffmpeg fluctuates -> need to set audio streams to be the same size.
          while (receivedData.length >= expectedDataSize) {
            final chunk = receivedData.sublist(0, expectedDataSize);
            queueData.add(chunk);  // receiving data and add to queue.
            receivedData = receivedData.sublist(expectedDataSize);
          }
        });
      });

2. Create Stream data

      Stream<Queue<Uint8List>>? streamQueue;
      StreamSubscription? streamSub;
 
       streamQueue =
            Stream<Queue<Uint8List>>.periodic(const Duration(milliseconds: 5), (_) {
            return queueData;
       });
      streamSub = streamQueue!.listen((data) {
            if(data.isNotEmpty){
             // you can put audio player here to consume the data stream  
             // data.removeFirst()
            }
         }
      });

3. Execute Ffmpeg

      FFmpegSession? session; 
      int tcpOutputPort = 65000;// example output port
      int tcpInputPort = 65001;// example input port


      final arguments = [
      "-f",
      "s16le",
      "-i",
      "tcp://$tcpAddress:$tcpInputPort?listen",
      "-filter_complex",
      "aecho=1.00:0.72:300:0.5, vibrato=f=12:d=0.3, asetrate=44000*1.2,aresample=44000,atempo=1/1.2", // example audio effect
      "-f",
      "s16le",
      "tcp://$tcpAddress:$tcpOutputPort",
    ];
     // running ffmpeg session as async -> need to close after adding audio!
     session = await FFmpegKit.executeWithArgumentsAsync(arguments);

4. Add audio input

    io.Socket? tcpSocketSender;
    
    tcpSocketSender ??= await io.Socket.connect(tcpAddress, tcpInputPort);
    tcpSocketSender!.add(audioData); // You can add them all at once or use streaming audio data.
    tcpSocketSender!.flush();