PGPainless produces broken messages for some signature / data format combinations.

[vanitasvitae]

A signature over data can either be of SignatureType.BINARY_DOCUMENT or SignatureType.CANONICAL_TEXT_DOCUMENT.
The body of a literal data packet can either be ´StreamEncoding.BINARY´, ´StreamEncoding.TEXT´ or ´StreamEncoding.UTF8´.

It turns out, that PGPainless only produces valid output for (BINARY,BINARY) combination.
Others result in errors, or lead to interoperability problems.

This is being investigated in https://github.com/pgpainless/pgpainless/tree/canonicalizedDatza

UPDATE

Some of the error cases have been fixed:

• Binary Data, Binary Signature
• Binary Data, Text Signature
• Text Data, Binary Signature
• Text Data, Text Signature
• UTF8 Data, Binary Signature
• UTF8 Data, Text Signature

[vanitasvitae]

For now users are advised to avoid using StreamEncoding.TEXT/StreamEncoding.UTF8 in combination with DocumentSignatureType.BINARY_DOCUMENT.

All other combinations should be fine now.

[vanitasvitae]

After checking back with some other OpenPGP developers it was decided that the best way forward is to deprecate ProducerOptions.setEncoding(StreamEncoding).

This would mean that all messages would use StreamEncoding.BINARY by default, which produces valid signatures for all combinations with DocumentSignatureTypes.

PGPainless release 1.2.X will remove ProducerOptions.setEncoding(StreamEncoding) completely.

[vanitasvitae]

@teythoon suggested that the Literal Data packets format option (StreamEncoding) is merely intended as a hint for the consumer.

As a consequence I decided to stop automatically CRLF-encoding messages based on the StreamEncoding.
That means that applications that rely on messages to be CRLF-encoded (e.g. to match gpg --textmode output) would need to do the CRLF-encoding themselves before feeding the output to PGPainless.

For that purpose I copied a class from Bouncy Castle that gets the job done.
An application that needs textmode output can apply the CRLFGeneratorStream as follows:

InputStream plaintext = ... // message input
StreamEncoding encoding = StreamEncoding.UTF8: // or StreamEncoding.TEXT. THIS IS NOW MERELY CONSIDERED METADATA!
ProducerOptions options = ...; // configure how the message shall be created
options.setEncoding(encoding); // set encoding metadata 

EncryptionStream signerOrEncryptor = PGPainless.encryptAndOrSign()
    .onOutputStream(...)
    .withOptions(options);

// Wrap encryptionStream in CRLF generator
CRLFGeneratorStream crlfOut = new CRLFGeneratorStream(signerOrEncryptor, encoding);

// Feed the message to the CRLF stream instead of the encryptor
Streams.pipeAll(plaintext, crlfOut);

crlfOut.close; // closes both crlf and encryption stream
// continue as usual
EncryptionResult result = signerOrEncryptor.getResult();

UPDATE: This is no longer necessary.

[vanitasvitae]

Update:
I added ProducerOptions.applyCRLFEncoding() which will handle CRLF encoding for the user.
So applying CRLF encoding is now a matter of calling said method.