`OpenSSL::BN` is used both as a signed and unsigned value, binary encoding should support both

[rickmark]

A high bit when binary encoding a BN can signal the number is negative. There should be a way when encoding / decoding OpenSSL::BN to indicate if the encoded value is signed or unsigned.

Parsing a unsigned : OpenSSL::BN.new('FF', 16, signed: false) -> .to_i => 255
Parsing a signed : OpenSSL::BN.new('FF', 16, signed: true) -> .negative? => true
Omitting the value for signed: maintains compatibility

Similar parameters may be needed for #to_s as if it is a unsigned with the most significant bit set occurring on the byte boundary it needs an additional "\x00" prefix to disambiguate its sign

[rhenium]

What is the use case?

For hexadecimal number, BN_hex2bn() takes a string prefixed by '-': OpenSSL::BN.new("-ff", 16) will be -255.

[rickmark]

Much of encoding done in blockchain tech is occurring with using power 2 not 16 serialization / deserialization. And for much of it you need to be assured you’re being parsed as a positive integer. This is that joyous problem of ASN1 encodings and seco256k1 signatures in Bitcoin being varying lengths depending on the MSBit. Get Outlook for iOS<https://aka.ms/o0ukef>

…

________________________________ From: Kazuki Yamaguchi ***@***.***> Sent: Tuesday, March 30, 2021 10:54:41 PM To: ruby/openssl ***@***.***> Cc: Rick Mark ***@***.***>; Author ***@***.***> Subject: Re: [ruby/openssl] `OpenSSL::BN` is used both as a signed and unsigned value, binary encoding should support both (#431) What is the use case? For hexadecimal number, BN_hex2bn() takes a string prefixed by '-': OpenSSL::BN.new("-ff", 16) will be -255. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fruby%2Fopenssl%2Fissues%2F431%23issuecomment-810787684&data=04%7C01%7C%7Ccacb35b80b934b14063808d8f4097a6b%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637527668826535498%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=lI0q%2BZQr4HgQFiyc2O1gqOuHG%2FthxND55zTBxTPngRc%3D&reserved=0>, or unsubscribe<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAAA6TW2GWQISNUHQ7WFEUDTTGK2KDANCNFSM42DE7L6A&data=04%7C01%7C%7Ccacb35b80b934b14063808d8f4097a6b%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637527668826545497%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=D0PSS94MenKE8%2FACCqD%2FiRKMoQgAXwFNyo3i5alR4y8%3D&reserved=0>.

[rhenium]

OpenSSL::BN.new(octet_string, 2) will always parse octet_string as a positive number encoded in big-endian. So, if the positive number is the only concern, it should be good already.

I don't see the format (two's complement, variable length) is widely used outside ASN.1 BER's context. For decoding such encoding I'd suggest prepending a proper header and using the ASN.1 routine:

bin = "\xff\x00"
der = [2, bin.bytesize, bin].pack("Cwa*")
OpenSSL::ASN1.decode(der).value #=> -256

[rickmark]

Yeah - the unfortunate bit is bad conformance has lead some implementations to place a 32 octet big int with a high most significant bit directly into the integer type - causing it to be mis-interpreted as a negative value. For ECDSA signatures you know that both values are in fact positive so being able to hint to decode them as unsigned would help....

I know its awful to build features around non-conformance though :-/