BigInt: Problems with the output in hexadecimal representation and negative numbers

[felix-mk]

Output of BigInts in hexadecimal representation

If you output BigInts with std::hex via operator<<(std::ostream&, const BigInt&) (in src/lib/math/bigint/big_io.cpp) the prefix 0x is automatically added:

std::cout << std::hex << Botan::BigInt{0x123} << '\n';
// Output: 0x123

However, this does not happen with the standard library:

std::cout << std::hex << 0x123 << '\n';
// Output: 123 and not 0x123

std::cout << "0x" << std::hex << 0x123 << '\n';
// Output: 0x123

This is probably the reason why it was not noticed that this code from the cli outputs the prefix twice.

// src/cli/math.cpp, line: 66
const Botan::BigInt p = Botan::random_prime(rng(), bits);

if(hex)
    output() << "0x" << std::hex << p << "\n";
    //          ^^^^^^^^^^^^^^^^^^^^^
else
    output() << p << "\n";

$ botan gen_prime --hex 2
# Output: 0x0x02 and not 0x02

Nevertheless, the current implementation of BigInt makes sense in that it allows a negative sign to be placed in front. (Which of course does not happen with normal integers in the two's complement). However, there is another problem.

Output of negative BigInts

When displaying negative BigInts, the sign is output twice:

std::cout << -Botan::BigInt{10} << '\n';
// Output: --10 and not -10

This happens because both operator<<(std::ostream&, const BigInt&) and the functions called from it to_dec_string/to_hex_string (in src/lib/math/bigint/big_code.cpp) prepend a negative sign.

This leads to the output being far from the expected/normal:

std::cout << "0x" << std::hex << -Botan::BigInt{10} << '\n';
// Output: 0x--0x0A and not 0x-0A or -0x0A

My ideas for the solution

1. No more prefix 0x in operator<<(std::ostream&, const BigInt&).
2. No more sign in operator<<(std::ostream&, const BigInt&), but still in to_dec_string/to_hex_string, as some functions call to_dec_string/to_hex_string directly.

I don't know if these changes are realistic, because I haven't found a way to find all the places in the code that depend on the overloaded operator <<.

---

This is my first issue on GitHub. I hope I haven't missed anything obvious.

[randombit]

Addressed now on master with #3121 thank you @felix-mk for the detailed report