Feature/streaming report #389
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This function streams the wanted transactionIds from the db, then for each txId, the full transaction is fetch and returned to the user using backpressure. As a user, the rest endpoint response almost immediately with a 200 and then it start to download the transactions. This allows to easily download 1M tx without timeouts.
simerplaha
reviewed
Nov 22, 2022
tdroxler
changed the title
polarker
reviewed
Nov 22, 2022
app/src/main/scala/org/alephium/explorer/api/model/Transaction.scala
Outdated
Show resolved
Hide resolved
app/src/main/scala/org/alephium/explorer/persistence/queries/TransactionQueries.scala
Outdated
Show resolved
Hide resolved
app/src/main/scala/org/alephium/explorer/persistence/queries/TransactionQueries.scala
Outdated
Show resolved
Hide resolved
app/src/main/scala/org/alephium/explorer/service/TransactionService.scala
Outdated
Show resolved
Hide resolved
app/src/main/scala/org/alephium/explorer/service/TransactionService.scala
Show resolved
Hide resolved
app/src/test/scala/org/alephium/explorer/service/TransactionServiceSpec.scala
Outdated
Show resolved
Hide resolved
app/src/main/scala/org/alephium/explorer/api/model/Transaction.scala
Outdated
Show resolved
Hide resolved
Keeping the code behind it in services as it doesn't hurt.
When not providing a parent, we were always generated a genesis block, which isn't what we want for most of the tests.
Without batching we download between 500-900KB/s and with a batching at 100 it's around 1.5-2MB 🚀
This was referenced Nov 24, 2022
|
With the two latest commits, we are now refusing to export addresses with more than 10K txs (within the time range). Currently the biggest exportable address has 8749 txs, and it takes 4s to dl the 6M csv file. |
|
Fighting a flaky test and after that we should be good. |
polarker
reviewed
Dec 1, 2022
polarker
reviewed
Dec 1, 2022
app/src/test/scala/org/alephium/explorer/api/model/ApiModelSpec.scala
Outdated
Show resolved
Hide resolved
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Resolves: #374
We are still waiting
tapirto release a new version, but my PR got merged few days ago .I still miss the tests, but I think it can be good to start reviewing it, as it's a new concept.
With streaming, when you call the endpoint, you get a response directly:
from the backend side we first fetch the txIds as a stream and as soon psql have the first txIds ready, it starts to push them down the flow, for each txId we fetch the tx detail and it goes down the stream to the http response. The user is dowloading each tx, like any download, we don't wait to have ALL tx to return them.
As it's reactive streams, the all things is using backpressure.
Using streams allows to be able to download for example the 1M+ txs from our biggest miner without issue.
cc/ @mvaivre for the two new endpoints:
max time interval is 365 days