SecretStream Push method

[cremfert]

I just have a question: from my understanding of the docs, the provided push method of the SecretStream API handles all the tag and header creation and ordering, right? So, instead of deciding for each chunk whether this is a final message or not (and hence set the respective tag), I only have to provide a stream of Uint8List, right? Or is there anything else I need to consider? I'm trying to implement this: https://libsodium.gitbook.io/doc/secret-key_cryptography/secretstream#example-stream-encryption

Thanks a lot!

EDIT:
I tried:

Future<Map<String, String>> encryptFile(File sourceFile, File targetFile) async {
    IOSink sink = targetFile.openWrite();
    SecureKey key = _sodium.crypto.secretStream.keygen();
    _sodium.crypto.secretStream.push(messageStream: sourceFile.openRead().map<Uint8List>((chunk) => Uint8List.fromList(chunk)), key: key).listen(
          (event) => sink.add(event),
        );

    debugPrint("DONE");
    return {"key": hex.encode(key.extractBytes())};
  }

  decryptFile(File sourceFile, File targetFile, SecureKey key) async {
    IOSink sink = targetFile.openWrite();
    _sodium.crypto.secretStream.pull(cipherStream: sourceFile.openRead().map<Uint8List>((chunk) => Uint8List.fromList(chunk)), key: key).listen(
          (event) => debugPrint("CHUNK"),
        );
  }

but I always receive: Unhandled Exception: cipher stream was closed before the final push message was received.

What do I wrong?

[Skycoder42]

Sorry for the late reply!

I think you might not be using the dart stream interface correctly. It should look something like this:

final source = sourceFile.openRead();
final sink = targetFile.openWrite();
final key = _sodium.crypto.secretStream.keygen();

await source
  .map<Uint8List>((chunk) => Uint8List.fromList(chunk))
  .transform(_sodium.crypto.secretStream.createPush(key))
  .pipe(sink);

This will use the stream transformation APIs to completely pipe all data through the stream encryption into the output file. Using listen directly will sometimes cause problems, as you need to manually take care of when to wait for and close certain streams. The transformation APIs handle all that logic for you.

Decryption works the same way, just use secretStream.createPull instead.

[cremfert]

Thanks again for your help! I'll try that and will report (and close the issue).

[cremfert]

Unfortunately, it doesn't work. The message is:

The argument type 'IOSink' can't be assigned to the parameter type 'StreamConsumer<Uint8List>'.

openWrite is of type IOSink, but pipe expects StreamConsumer. Do you have an idea how to proceed? Thanks a lot!

[cremfert]

Finally found it - the stream needs to be casted before calling pipe on it:
await source.map<Uint8List>((chunk) => Uint8List.fromList(chunk)).transform(_sodium.crypto.secretStream.createPush(key)).cast<List<int>>().pipe(sink);

Thanks a lot!

[cremfert]

Now decryption is not working :( - sorry:

IOSink sink = targetFile.openWrite();
    final Stream<List<int>> source = sourceFile.openRead();
    await source.map<Uint8List>((chunk) => Uint8List.fromList(chunk)).transform(_sodium.crypto.secretStream.createPull(key)).cast<List<int>>().pipe(sink);
    sink.close();
    debugPrint("DONE");

Error message is: InvalidHeaderException (Expected secretstream header with 24 bytes, but received 120 bytes)

I just changed from createPush to createPull - code is the same like for encryption. Is this wrong? 120 bytes is the full size of the encrypted file.

[Skycoder42]

Okay, so the casting is correct, yes. I forgot about that, sorry.

Regarding the decryption: I looked in the documentation again and it seems that the cipher expects predefined message "packages". In other words, if you encrypt 100 bytes at once and get, for example 128 encrypted bytes, you must pass these exact 128 bytes as complete "block" back to the decryption - this means streaming the file like that will not work.

You have 2 options now:

1. If the file is not that large, just encrypt it in one go with sodium.crypto.secretBox. While a little slower and more memory heavy, it's the easiest way. You can use readAsBytes and writeAsBytes to get the whole file at once.
2. If that is not an option due to file size, you can still use the stream api, but you have to read the file in chunks. The flow would be:
   1. Define a "blocksize", i.e. 1024 byte
   2. Read the file in blocks of that size
   3. encrypt each block and write it to the output file
   4. To decrypt, you would then read blocks with a size of blocksize + sodium.crypto.secretStream.aBytes
   5. decrypt each of these blocks and you have your original file back

To transform the stream into such blocks, you could use something like rxdarts buffer method (https://pub.dev/documentation/rxdart/latest/rx/BufferExtensions/bufferCount.html):

final source = sourceFile.openRead()
  .expand(bytes => bytes)
  .buffer(100);

[cremfert]

Ok, thanks a lot for pointing me to the right direction. I now use for encryption:

final Stream<List<int>> source = sourceFile.openRead().expand((bytes) => bytes).bufferCount(100);
await source.map<Uint8List>((chunk) => Uint8List.fromList(chunk)).transform(_sodium.crypto.secretStream.createPush(key)).cast<List<int>>().pipe(sink);

and for decryption:

final Stream<List<int>> source = sourceFile.openRead().expand((bytes) => bytes).bufferCount(100 + _sodium.crypto.secretStream.aBytes);
await source.map<Uint8List>((chunk) => Uint8List.fromList(chunk)).transform(_sodium.crypto.secretStream.createPull(key)).cast<List<int>>().pipe(sink);

but unfortunately, it still says: InvalidHeaderException (Expected secretstream header with 24 bytes, but received 117 bytes)

Any further ideas? Thanks a lot for your help!

[Skycoder42]

okay so I read a little further, and apparently the encryption stream always starts with a 24 byte header, followed by the encrypted data. so, what you have to do is something like that:

var counter = 0;
var headerSent = false;
final source = sourceFile.openRead()
  .expand((bytes) => bytes)
  .bufferTest((e) {
    counter++;
    if (!headerSent && counter == _sodium.crypto.secretStream.headerBytes) {
      counter = 0;
      headerSent = true;
      return true;
    } else if (headerSent && counter == 100 + _sodium.crypto.secretStream.aBytes) {
      counter = 0;
      return true;
    }
    return false;
  });

This would collect 24 bytes for the header once and the always a chunk of encrypted data.

Sorry for not beeing much of a help, but I only wrapped the APIs for dart, so I don't no all of their behaviours in every detail

[cremfert]

Really thanks for your help - your code correctly create a stream with a header and the cipher data. However, I now get Unhandled Exception: A low-level libsodium operation has failed - I looked through your code, I can't see any problems. Is it possible to get the detailed error message? Then I can try to fix it and make a PR. Thanks a lot!

[Skycoder42]

Hi. These errors happen if a sodium operation returns -1 - the underlying library does not provide any other error codes for security reasons, so no, there is no way to get any more details. However, if you can get the stacktrace of the unhandeled exception, at least I could figure out which operation fails.

[cremfert]

Hi, I got it working finally. Seems like I did something wrong with the streams. For everyone coming here having the same question, here is an example:

import 'dart:async';
import 'dart:typed_data';

import 'package:flutter/foundation.dart';
import 'package:sodium_libs/sodium_libs.dart';

class EncTest {
 late Sodium sodium;
 runTest() async {
   sodium = await SodiumInit.init();

   debugPrint("HEADER BYTES: " + sodium.crypto.secretStream.headerBytes.toString());
   debugPrint("APPENDING BYTES: " + sodium.crypto.secretStream.aBytes.toString());

   encrypt();
 }

 encrypt() {
   SecureKey key = sodium.crypto.secretStream.keygen();
   StreamController<Uint8List> controller = StreamController();

   List<int> encBytes = [];

   sodium.crypto.secretStream.push(messageStream: controller.stream, key: key).listen((encryptedBytes) {
     encBytes.addAll(encryptedBytes);
   }, onDone: (() {
     decrypt(Uint8List.fromList(encBytes), key);
   }));

   // Following lines are only an example; normally you would iterate over file blocks
   controller.add(Uint8List.fromList("1234".codeUnits));
   controller.add(Uint8List.fromList("test".codeUnits));
   controller.add(Uint8List.fromList("5678".codeUnits));

   controller.close();
 }

 decrypt(Uint8List encryptedData, SecureKey key) {
   StreamController<Uint8List> controller = StreamController();

   List<int> decBytes = [];

   sodium.crypto.secretStream.pull(cipherStream: controller.stream, key: key).listen((decryptedBytes) {
     decBytes.addAll(decryptedBytes);
   }, onDone: () {
     debugPrint(decBytes.toString());
   });

   controller.add(encryptedData.sublist(0, 24)); // Header
   // Each block has string length + 17 (aBytes.length) = 4 + 17 = 21
   controller.add(encryptedData.sublist(24, 45)); // String 1 = "1234"
   controller.add(encryptedData.sublist(45, 66)); // String 2 = "test"
   controller.add(encryptedData.sublist(66, 87)); // String 3 = "5678"
   controller.add(encryptedData.sublist(87)); // Final Tag

   controller.close();
 }
}

Thanks a lot for your help, @Skycoder42 !

[ramvdixit]

Hello,

I have modified the code from @cremfert to suit my need of chunking files to be able to cover different file sizes and the Encryption works perfectly. The Decryption also works and I am able to retrieve the original content of the file; however, there is a nagging error at the end of it (in spite of successful decryption). The error is get is: cipher stream was closed before the final push message was received on the onError() callback. Here is my code.

static Future<void> streamDecryptAsset2(String key) async {
    ensureSodiumInit();
    final useKey =
        SecureKey.fromList(sodium, Uint8List.fromList(hexToBytes(key)));
    final appDocsDirectory = await getApplicationDocumentsDirectory();
    String inputFilePath = "${appDocsDirectory.path}/txt/crude-oil-prices.csv";
    final outDirectory = Directory('${appDocsDirectory.path}/txt/decrypted');
    if (await outDirectory.exists()) {
    } else {
      await outDirectory.create(recursive: true);
    }

    final file = File(inputFilePath);
    final source = file.openRead();

    final outputFile = File("${outDirectory.path}/${path.basename(file.path)}");
    final sink = outputFile.openWrite();
    StreamController<Uint8List> controller = StreamController();
    List<int> decBytes = [];

    sodium.crypto.secretStream
        .pull(cipherStream: controller.stream, key: useKey)
        .listen((decryptedBytes) {
      decBytes.addAll(decryptedBytes);
    }, onDone: (() {
      debugPrint("DECRYPTION: onDone()");
    }), onError: (e) {
      debugPrint("DECRYPTION: $e");
    });
    final content = await source.first;
    final contentWithoutHeader = content.toList(growable: true).sublist(24);
    controller.add(Uint8List.fromList(content.sublist(0, 24)));
    for (int i = 4096; i < y.length; i += 4096) {
      final chunk = contentWithoutHeader.sublist(0, i + 17);
      controller.add(Uint8List.fromList(chunk));
    }
    sink.add(decBytes);
    await sink.flush();
    await sink.close();
    controller.close();
  }

If I do not include the onError() callback, the decryption still happens, I DO NOT get any runtime exceptions, which is scarier.

If I comment out the last line controller.close(), the error goes away, but I am not sure if that's the right thing to do and if it might end up with a memory / memory leak issue.

Does anyone has any pointer so I can look? I am not sure if I should just ignore the exception as it might lead to memory leaks if the stream is not properly closed. Any help would be awesome!

Thanks and cheers!
Ram