How to run/develop locally? #14

davezuch · 2022-07-10T19:46:09Z

davezuch
Jul 10, 2022

Apologies for any dumb questions, I don't normally write python and have no experience writing anything web3 related. I wanted to try running dfkreport locally so that I could play around with the code and make tweaks. There are a couple things I'm confused on though.

First, if I try to run main.py I get this error:

$ ./main.py 0x... 2021-01-01 2021-12-31
Traceback (most recent call last):
  File "/.../dfkreport/./main.py", line 110, in <module>
    main()
  File "/.../dfkreport/./main.py", line 41, in main
    with open('../transactions/{0}'.format(reportInfo[8]), 'rb') as file:
FileNotFoundError: [Errno 2] No such file or directory: '../transactions/74e0c7ead1384985a8a68a3c3ba767ec'

Why does the app expect a transactions directory one level up? Is it supposed to be run in some sort of VM or Docker container?

Second, how do I serve the UI? I saw the other discussion about running ../watch.py from the web directory, but that doesn't seem to have anything to do with running an HTTP server that will interpret python files when I navigate to e.g. localhost:8000/home.py.

Any help would be appreciated!

Answered by davezuch

Jul 10, 2022

Why does the app expect a transactions directory one level up? Is it supposed to be run in some sort of VM or Docker container?

Okay I figured main.py is supposed to be ran from web and got past the FileNotFoundError by adding a transactions directory to the root. Now I'm getting this error:

$ ../main.py 0x... 2021-01-01 2021-12-31

reportInfo: ('0x...', '2021-01-01', '2021-12-31', 1657494143, 890, 0, 890, 0, '', '', 1, 'fifo', 1, None)
Traceback (most recent call last):
  File "/.../dfkreport/.venv/lib/python3.9/site-packages/web3/contract.py", line 1513, in call_contract_function
    output_data = web3.codec.decode_abi(output_types, return_data)
  File "/.../dfkreport/.venv/lib/python…

View full answer

davezuch · 2022-07-10T23:11:05Z

davezuch
Jul 10, 2022
Author

Why does the app expect a transactions directory one level up? Is it supposed to be run in some sort of VM or Docker container?

Okay I figured main.py is supposed to be ran from web and got past the FileNotFoundError by adding a transactions directory to the root. Now I'm getting this error:

$ ../main.py 0x... 2021-01-01 2021-12-31

reportInfo: ('0x...', '2021-01-01', '2021-12-31', 1657494143, 890, 0, 890, 0, '', '', 1, 'fifo', 1, None)
Traceback (most recent call last):
  File "/.../dfkreport/.venv/lib/python3.9/site-packages/web3/contract.py", line 1513, in call_contract_function
    output_data = web3.codec.decode_abi(output_types, return_data)
  File "/.../dfkreport/.venv/lib/python3.9/site-packages/eth_abi/codec.py", line 181, in decode_abi
    return decoder(stream)
  File "/.../dfkreport/.venv/lib/python3.9/site-packages/eth_abi/decoding.py", line 127, in __call__
    return self.decode(stream)
  File "/.../dfkreport/.venv/lib/python3.9/site-packages/eth_utils/functional.py", line 45, in inner
    return callback(fn(*args, **kwargs))
  File "/.../dfkreport/.venv/lib/python3.9/site-packages/eth_abi/decoding.py", line 173, in decode
    yield decoder(stream)
  File "/.../dfkreport/.venv/lib/python3.9/site-packages/eth_abi/decoding.py", line 127, in __call__
    return self.decode(stream)
  File "/.../dfkreport/.venv/lib/python3.9/site-packages/eth_abi/decoding.py", line 198, in decode
    raw_data = self.read_data_from_stream(stream)
  File "/.../dfkreport/.venv/lib/python3.9/site-packages/eth_abi/decoding.py", line 305, in read_data_from_stream
    raise InsufficientDataBytes(
eth_abi.exceptions.InsufficientDataBytes: Tried to read 32 bytes.  Only got 0 bytes

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/.../dfkreport/web/../main.py", line 91, in main
    reportData = taxmap.buildTaxMap(txData, args.wallet, datetime.datetime.strptime(args.startDate, '%Y-%m-%d').date(
  File "/.../dfkreport/taxmap.py", line 61, in buildTaxMap
    eventMap = events.checkTransactions(txns[0], account, startDate, endDate, 'harmony')
  File "/.../dfkreport/events.py", line 166, in checkTransactions
    results = extractSwapResults(w3, tx, account, timestamp, receipt)
  File "/.../dfkreport/events.py", line 737, in extractSwapResults
    fiatValues = getSwapFiatValues(timestamp, sentToken[0], sentAmount[0], rcvdToken[0], rcvdAmount[0])
  File "/.../dfkreport/events.py", line 783, in getSwapFiatValues
    fiatReceiveValue = prices.priceLookup(timestamp, rcvdToken) * rcvdAmount
  File "/.../dfkreport/prices.py", line 64, in priceLookup
    return decimal.Decimal(getPrice(token, lookupDate, fiatType))
  File "/.../dfkreport/prices.py", line 126, in getPrice
    prices = fetchPriceData(token, date)
  File "/.../dfkreport/prices.py", line 92, in fetchPriceData
    result = fetchItemPrice(token, date)
  File "/.../dfkreport/prices.py", line 108, in fetchItemPrice
    price = getCurrentPrice(token, '0x72Cb10C6bfA5624dD07Ef608027E366bd690048F')
  File "/.../dfkreport/prices.py", line 149, in getCurrentPrice
    tokenDecimals = getTokenInfo(w3, token)[1]
  File "/.../dfkreport/prices.py", line 174, in getTokenInfo
    contract = w3.eth.contract(address=address, abi=ABI)
  File "/.../dfkreport/.venv/lib/python3.9/site-packages/web3/eth.py", line 720, in contract
    return ContractFactory(address)
  File "/.../dfkreport/.venv/lib/python3.9/site-packages/web3/contract.py", line 338, in __init__
    self.address = normalize_address(self.web3.ens, address)
  File "/.../dfkreport/.venv/lib/python3.9/site-packages/web3/_utils/normalizers.py", line 250, in normalize_address
    validate_name_has_address(ens, address)
  File "/.../dfkreport/.venv/lib/python3.9/site-packages/web3/_utils/ens.py", line 45, in validate_name_has_address
    addr = ens.address(name)
  File "/.../dfkreport/.venv/lib/python3.9/site-packages/ens/main.py", line 133, in address
    return cast(ChecksumAddress, self.resolve(name, 'addr'))
  File "/.../dfkreport/.venv/lib/python3.9/site-packages/ens/main.py", line 235, in resolve
    resolver = self.resolver(normal_name)
  File "/.../dfkreport/.venv/lib/python3.9/site-packages/ens/main.py", line 247, in resolver
    resolver_addr = self.ens.caller.resolver(normal_name_to_hash(normal_name))
  File "/.../dfkreport/.venv/lib/python3.9/site-packages/web3/contract.py", line 1453, in call_function
    return fn(*args, **kwargs).call(transaction, block_identifier)
  File "/.../dfkreport/.venv/lib/python3.9/site-packages/web3/contract.py", line 957, in call
    return call_contract_function(
  File "/.../dfkreport/.venv/lib/python3.9/site-packages/web3/contract.py", line 1530, in call_contract_function
    raise BadFunctionCallOutput(msg) from e
web3.exceptions.BadFunctionCallOutput: Could not transact with/call contract function, is contract deployed correctly and chain synced?

Any idea what the issue is?

6 replies

davezuch Jul 17, 2022
Author

I didn't make any changes to the source code, but I did make changes to the version of pyhmy installed by pip. I was getting an exception about pyhmy.bech32 not being found, and in fact the version of pyhmy installed didn't have that module, so I think I manually copied down what's on the library's GitHub.

My gut tells me the issue has to do with having the wrong versions of dependencies, but no versions are pinned in requirements.txt, and like I said, I don't normally write python, so wasn't sure what else to try there.

pwillworth Jul 18, 2022
Maintainer

Ah yes, I had to pull in updated code for pyhmy as well. I guess they are not pushing new package versions to pypi anymore or something. I think at this point it might be good to debug exactly what it's trying to look up at this point and see if it should be succeeding or not. It is possible for someone to have tokens they do swaps on for harmony that are not in the DFK dex, so that would return this kind of error. However, the default code is setup to just swallow and log those, so not sure why things are crashing here just because a price lookup failed.

davezuch Jul 19, 2022
Author

Well, it looks like the token it's trying to look up is JEWEL:

  File "/.../dfkreport/prices.py", line 108, in fetchItemPrice
    price = getCurrentPrice(token, '0x72Cb10C6bfA5624dD07Ef608027E366bd690048F')

https://explorer.harmony.one/address/0x72cb10c6bfa5624dd07ef608027e366bd690048f

So it should def be supported.

pwillworth Jul 23, 2022
Maintainer

Hmmm, that is odd that it is looking up Jewel price on the dex. It should be going to coingecko first, but you might be exhausting your allowed traffic rate to coingecko. I can see how there might be a problem if it's trying to lookup jewel on the dex because most of the code assumes that will not happen and it tries to use jewel as a route to calculate the price of whatever non-jewel thing is being looked up assuming there is a pair with it to jewel in the dex.

davezuch Aug 14, 2022
Author

Okay, I just had time to come back to this. This last response actually helped a lot.

The function fetchPriceData in prices.py has this conditional:

dfkreport/prices.py

Lines 70 to 71 in 989d455

    
           # Coin Gecko only has prices from October 20th 2021 forward for JEWEL 
        
           if (realDate > datetime.datetime.strptime('19-10-2021', '%d-%m-%Y') or token != 'defi-kingdoms') and token[0] != '0':

I'm trying to get data for the whole year of 2021, so there are transactions before October 20 2021 which it was failing on, since it has to look up those prices on the DEX.

I ended up fixing this by removing this block from priceLookup:

 def priceLookup(timestamp, token, fiatType='usd'):
     lookupDate = datetime.date.fromtimestamp(timestamp).strftime('%d-%m-%Y')
-    # if token is in map, switch to gecko token name instead
-    if token in token_map:
-        token = token_map[token]

And moving that logic into fetchPriceData:

 def fetchPriceData(token, date):
     realDate = datetime.datetime.strptime(date, '%d-%m-%Y')
     prices = None
+    gecko_token = None
+    # if token is in map, switch to gecko token name instead
+    if token in token_map:
+        gecko_token = token_map[token]
     # Coin Gecko only has prices from October 20th 2021 forward for JEWEL
-    if (realDate > datetime.datetime.strptime('19-10-2021', '%d-%m-%Y') or token != 'defi-kingdoms') and token[0] != '0':
-        gecko_uri = "https://api.coingecko.com/api/v3/coins/{0}/history?date={1}&localization=false".format(token, date)
+    if (realDate > datetime.datetime.strptime('19-10-2021', '%d-%m-%Y') or gecko_token != 'defi-kingdoms') and gecko_token != None:
+        gecko_uri = "https://api.coingecko.com/api/v3/coins/{0}/history?date={1}&localization=false".format(gecko_token, date)

With those changes, it seems like I'm getting working transactions/reports.

mrlowe · 2022-10-06T14:47:25Z

mrlowe
Oct 6, 2022

Additional info that might be useful: I've got a vagrant box running ubuntu focal (20) and finally have the app up. I did have to use Python 3.8 (because otherwise pip was getting unfortunate combinations of dependencies) ... the pyhmy issue seems to be due to their changing/resetting their versioning and pip getting the older (but yanked) versions of the library. I also had to explicitly install eth-hash<0.4.0,>=0.3.1 because there was a compatibility issue with a newer version, but aside from that it seems happy.

I haven't run anything through 2021 yet though, so I haven't run into the gecko issue.

0 replies

mrlowe · 2022-10-18T20:45:22Z

mrlowe
Oct 18, 2022

@pwillworth when I run an address with just DFK selected on the live site, I get the data I expect. But when I run locally, I only get a few records. (So, for example, I see 12 wallet transactions on production, but 0 locally.)

I've run blockCrawler, and Harmony looks fine, but DFKChain is acting differently. Anything obvious I might be missing? Do I need to use the covalent branch?

2 replies

pwillworth Oct 18, 2022
Maintainer

The avax subnet explorer endpoint ( https://subnet-explorer-api.avax.network/v1.0/53935 ) has kind of a restrictive limit on it I think. Depending on what activity you are doing locally, you might have hit the limit for yourself. You might check the main.log and see what's happening when it tries to fetch the transactions from there. It will still show an incomplete report even if it wasn't able to get the full tx list.

You can try the covalent branch if you want, it should work. You would have to get an api key from their site though. I'm working on moving to that instead of avax subnet explorer because they told me it is being deprecated and might go away when I asked them how to adjust the page size.

mrlowe Oct 19, 2022

Aha,

https://subnet-explorer-api.avax.network/v1.0/53935/transactions?address=0xWHATEVER&sort=asc

Is returning 404s. Updating to 1.1 looks like it will resolve the issue (as per https://subnet-explorer-api.avax.network/). Thank you! ❤️

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

How to run/develop locally? #14

{{title}}

Replies: 3 comments 8 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

How to run/develop locally? #14

davezuch Jul 10, 2022

Replies: 3 comments · 8 replies

davezuch Jul 10, 2022 Author

davezuch Jul 17, 2022 Author

pwillworth Jul 18, 2022 Maintainer

davezuch Jul 19, 2022 Author

pwillworth Jul 23, 2022 Maintainer

davezuch Aug 14, 2022 Author

mrlowe Oct 6, 2022

mrlowe Oct 18, 2022

pwillworth Oct 18, 2022 Maintainer

mrlowe Oct 19, 2022

davezuch
Jul 10, 2022

Replies: 3 comments 8 replies

davezuch
Jul 10, 2022
Author

davezuch Jul 17, 2022
Author

pwillworth Jul 18, 2022
Maintainer

davezuch Jul 19, 2022
Author

pwillworth Jul 23, 2022
Maintainer

davezuch Aug 14, 2022
Author

mrlowe
Oct 6, 2022

mrlowe
Oct 18, 2022

pwillworth Oct 18, 2022
Maintainer