Store proof info (keys, uptime, version) directly in LMDB

[jagerman]

Edit: see revamped version below.

The map was being initialized too early, from before recent state proof
data had been set, and so on startup quorumnet SN authentication was
failing to recognize incoming SN connections until a proof had been
received from the remote SN.

This also adds a shared mutex around the x25519 map access.

[Doy-lee]

It looks like there are several issues. First we were restoring the keys from state_archive which has old proof information (10k blocks old). But supposing we load the latest proof info, that information that get serializeds to DB is actually 12hrs old (since we re-derive everything from 12hrs back), if you shut down and restart your node.

It also seems like a commit got lost somewhere, I thought I remember you deleting the checkpointing part in this function. But right now we're still only storing quorums from the last immutable checkpoint -> chain height in short_term_state_cull_height

  static uint64_t short_term_state_cull_height(uint8_t hf_version, cryptonote::BlockchainDB const *db, uint64_t block_height)
  {
    size_t constexpr DEFAULT_SHORT_TERM_STATE_HISTORY = 6 * STATE_CHANGE_TX_LIFETIME_IN_BLOCKS;
    static_assert(DEFAULT_SHORT_TERM_STATE_HISTORY >= BLOCKS_EXPECTED_IN_HOURS(12), // Arbitrary, but raises a compilation failure if it gets shortened.
        "not enough short term state storage for blink quorum retrieval!");
    uint64_t result =
        (block_height < DEFAULT_SHORT_TERM_STATE_HISTORY) ? 0 : block_height - DEFAULT_SHORT_TERM_STATE_HISTORY;

    if (hf_version >= cryptonote::network_version_13_enforce_checkpoints)
    {
      uint64_t latest_height = db->height() - 1;
      cryptonote::checkpoint_t checkpoint;
      if (db->get_immutable_checkpoint(&checkpoint, latest_height))
        result = std::min(result, checkpoint.height - 1);
    }
    return result;
  }

So actually, the proof information being stored is probably only 12 blocks old.

The missing x25519 pubkeys might be from restoring from the archive and not the latest proof. I don't think the moving of x25519 map init to the end of the init function is the fix, rather that it was restoring from the wrong state and also this decouples the x25519 from the load step which doesn't feel right.

[jagerman]

I think we should just move to proof info to a completely separate database; it doesn't really have much to do with state and isn't something that we ever care about more than the latest value (also it isn't affected by rollbacks).

[jagerman]

Mostly rewrote this PR to remove proofs from stored states:

This extracts uptime proof data entirely from service node states,
instead storing (some) proof data as its own first-class object in the
code and backed by the database. We now persistently store:

• timestamp
• version
• ip & ports
• ed25519 pubkey

and update it every time a proof is received. Upon restart, we load all
the proofs from the database, which means we no longer lose last proof
received times on restart, and always have the most recently received ip
and port information (rather than only having whatever it was in the
most recently stored state, which could be ~12 blocks ago). It also
means we don't need to spam the network with proofs for up to half an
hour after a restart because we now know when we last sent (and
received) our own proof before restarting.

This separation required some restructuring: most notably changing
state_t to be constructed with a service_node_list pointer, which it
needs both directly and for BlockchainDB access. Related to this is
also eliminating the separate BlockchainDB pointer stored in
service_node_list in favour of just accessing it via m_blockchain (which
now has a new has_db() method to detect when it has been initialized).

[Doy-lee]

I'm just not too sure about restoring the uptime proofs. Reason being we didn't trust that the P2P layer is a fully connected graph, so some nodes might never see the uptime proof at all- so we made it relay aggressively in general.

I don't know. It's hard to say, the only way I'd really trust it is when we convert the system to use your zmq layer.

[Doy-lee]

Technically we don't even need old migration code and the various version, v3 is more than 2 hard forks old at this point.

[jagerman]

Since it's there already I'm kind of tempted to leave it in, mainly because it won't break the case where someone finds some ancient lmdb they forgot about and tries to update lokid. Not too concerned about that, but like I said, the support is already there.

[Doy-lee]

I've removed the blockchain dependency in the constructor in #956 I'd prefer we don't give state_t access to the service node list if we don't have to.

[jagerman]

It's not really going to be feasible with a much more significant rearchitecting: state_t is the place where we process state changes and registrations but it needs to be able to get at proof data to update it. Taking it in the constructor like this is slightly cleaner than the alternative of needing to take it as an argument in multiple methods.

[jagerman]

Ideally processing of state changes should be done in service_node_list, which then sets up the state_t as needed, but like I said, that's a bigger change than I wanted to introduce at this point.

[jagerman]

I'm just not too sure about restoring the uptime proofs. Reason being we didn't trust that the P2P layer is a fully connected graph, so some nodes might never see the uptime proof at all- so we made it relay aggressively in general.

I've added a commit to restore that aggressive sending by clearing our own timestamp when loading.

[jagerman]

Rebased and resolved conflicts (and squashed the last couple fixes into the previous commits)