feat: add /v2/micro-blocks/:hash endpoint

[sborrazas]

refs #888

[thepiwo]

again docs seem missing

[jyeshe]

No operational findings, just some enhancements.

[jyeshe]

Nice trick using the mb hash instead of next kb hash 👏 Depending on how much use the Explorer gonna make of this endpoint, maybe the AeMdw.Node.Db could have a similar micro block walker function that only uses :aec_blocks.prev_hash(block). It might be (or not) an early optimization but since it's a simple code I would go this way instead of profiling and monitoring the use.

[sborrazas]

Not sure I understand what you mean. :aec_blocks.prev_hash(block) returns a hash, so I can't do :aec_blocks.prev_hash(:aec_blocks.prev_hash(hash)). I need to use :aec_db.get_block(hash) or :aec_db.get_header(hash) too to get the prev_hash from it

[jyeshe]

It's because the current micro_block_walker defines the unfolding function expected by unfold(acc(), (acc() -> {element(), acc()} | nil)) in a way that the element accumulated is the block:

ae_mdw/lib/ae_mdw/node/db.ex

Line 245 in 0540074

{block, :aec_blocks.prev_hash(block)}

On the other hand with a simple recursion the accumulator could be the index counter.

[sborrazas]

The only difference between using a number accumulated and having this stream reduce blocks is that it will keep 1 block in-memory, while the other alternative will keep 1 number in-memory. TBH, I don't think it's worth a separate implementation since the block is quite tiny in size

[jyeshe]

What caught my attention on the first moment was Enum.count being used on the blocks. The unfold keeps all previous microblocks, including its transactions in memory accumulating them to create a list. So one related factor that popped up when I saw it was the comment on the amount of microblocks, that there might thousands in a single generation (decided by the leader limited only by the delay of 3 seconds). On top of this taking a more objective number, thinking only on a 100 TPS and supposing all transactions are spend_tx (a small kind of transaction with ~ 1KB) it would allocate and reallocate up to 3 * 60 * 100 * 1000 = 18MB per request. In order to handle 1000 simultaneous requests it would require 18GB only for this function call. As the goal of these new endpoints is to provide lighter ones, I think it's a good opportunity to save some resources.

[sborrazas]

This is not how streams work, streams are lazy, which is why there will be at most 1 micro block in memory at any given time. Please note that at no point the stream is turned into a list

stream
|> Enum.to_list()
|> Enum.count()

is very different from

stream
|> Enum.count()

The former will contain all the elements of the Enum in memory, while the latter will contain at most 1 element of the Enum at any given time

[jyeshe]

While Stream.map operates on each element Enum.map depends on the whole list. Almost sure that Enum.count would behave the same but I am checking it.

[jyeshe]

It's true when there is a Enum.reverse() like the micro_block_walker:

iex(aeternity@localhost)1> Task.async(fn -> Stream.unfold(1_000_000, fn 0 -> nil; n -> {n, n-1} end) |> Enum.reverse() |> Enum.count(); :erlang.process_info(self(), :memory) end) |> Task.await()
{:memory, 22445712}

[jyeshe]

Could we have also the "micro_block_index"?

[sborrazas]

Done