generically decode DEX swap transactions, including event log decoding #14
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
queries to the moonscan.io API started failing. Using Wireshark I tracked it down to use of the "requests" python package which doesn't support HTTPS2. It was submitting the moonscan.io GET request with "HTTP/1.1" and then moonscan.io responds with "HTTP/1.1 403 Forbidden" and the cloudflare error message. Whereas if I hand-crafted the same GET request as a URL string, it would be accepted and I received JSON back with the transaction history. Wireshark showed that these handcrafted URLs were being submitted as HTTP2. The solution was to migrate GET and POST HTTP messages to use the "httpx" package instead of "requests". (and with pip you must install the httpx optional support for http2) I went ahead and changed subscan.io calls to use httpx also, but with minimal testing.
* add rate limiting to our block explorer API calls, to make sure we always get a valid response from the APIs. * retrieve the ABIs for each contract interface, instead of archiving copies of each one. This allows us to support all APIs without needing to archive each one. * refactor moonbeam_scraper.py's definition of the params to be passed to Moonscan.io in an API call. The specific parameter names should be encapsulated in the wrapper methods. * detect and log when transaction input data isn't decoded properly (since we're trying to decode all transactions now).
* Get token name, symbol, and calculate real transaction values. But this required adding support for the Blockscout API. * adaptive rate limit for subscan.io depending on whether an API key is provided. * migrate URLs to be encapsulated within the block explorer wrappers. contributes to ChaosDAO-org#8 decoding DEX swap transactions
* Support most dex swap transactions except "swapETHForTokens" transactions because they don't specify input quantity. contributes to ChaosDAO-org#8 decoding DEX swap transactions
* decode errors for DPS contracts were due to the initial "convert_to_hex" functionality not being able to handle more complex data structures. When working down through nested structures, it missed buried byte arrays and didn't convert them to hex. "json.dumps(decoded_func_params)" was throwing an exception because the bytes weren't JSON serializable. The solution was to look for lists embedded inside a tuple/struct. * decode errors for 0xTaylor's puzzle contract were intentionally caused by poor utf8 string handling. Therefore still print out a warning the first time that contract is encountered and can't be decoded. Include full traceback and diagnostic messages instead of swallowing them. Now able to make the script more generic by removing references to specific problem contracts.
indicate contributors to each script file
Previously, we were able to decode the input data to a contract interaction. But in a DEX exchange, there could be slippage so the actual amount swapped might be slightly different. Therefore, we need to get the actual event logs to see the exact quantities exchanged. This commit uses @yifei_huang's script to decode a transaction log file. Some testing remains, to figure out to handle multiple input values to a swap. contributes to ChaosDAO-org#8
moonbeam_scraper.py is now able to retrieve transaction receipts for contract interactions so that it can decode the internal dex 'trace' transactions to determine the exact input and output quantities for swap transactions. Previously, we were using the internal 'Swap' transactions, but those were too difficult to match up for a multi-hop swap (i.e. ROME -> MOVR -> SOLAR). So instead the code now only sums up the Transfer input quantities from the source acct. This is done in a generic way without requiring classes for each DEX contract implementation which works as long as DEX contracts all follow general patterns and naming conventions. contributes to ChaosDAO-org#8
spazcoin
force-pushed
the
decodeLogs
branch
2 times, most recently
from
6666b1a
to
19304d0
Compare
* Include details about Moonscan.io and Blockscout.io API keys and how they align. * Include description of the current JSON output format and how to interpret it. * Include some common warning messages to see in the console output while running the tool, just to help users know whether they should be worried about the messages.
especially for Moonriver/Moonbeam `account_transactions` processing configuration
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description:
This PR uses @yifei_huang's scripts to decode contract input data as well interpret a transaction log file (to figure out the result of a contract interaction, not just the inputs).
See article explanation: https://towardsdatascience.com/decoding-ethereum-smart-contract-data-eed513a65f76
Swapinputs. So instead the code now only sums up the Transfer input quantities from the source acct. This is done in a generic way without requiring classes for each DEX contract implementation which works as long as DEX contracts all follow general patterns and naming conventions.convert_to_hexfunctionality not being able to handle more complex data structures. When working down through nested structures, it missed buried byte arrays and didn't convert them to hex.json.dumps(decoded_func_params)was throwing an exception because the bytes weren't JSON serializable. The solution was to look for lists embedded inside a tuple/struct.utf8string handling. Therefore still print out a warning the first time that contract is encountered and can't be decoded. Include full traceback and diagnostic messages instead of swallowing them.httpxfor GET and POST instead ofrequestslibrary. Queries to the moonscan.io API started failing. Using Wireshark I tracked it down to use of the "requests" python package which doesn't support HTTPS2. It was submitting the moonscan.io GET request withHTTP/1.1and then moonscan.io responds withHTTP/1.1 403 Forbiddenand the cloudflare error message. Whereas if I hand-crafted the same GET request as a URL string, it would be accepted and I received JSON back with the transaction history. Wireshark showed that these handcrafted URLs were being submitted as HTTP2. The solution was to migrate GET and POST HTTP messages to use thehttpxpackage instead ofrequests. (and with pip you must install the httpx optional support for http2) I went ahead and changed subscan.io calls to use httpx also, but with minimal testing.moonbeam_scraperworks now, how to interpret its output data, what typical warnings/errors are.