feat: added ape cache plugin

[johnson2427]

What I did

Added CacheQueryProvider to ape cache and created initial database structure and methods to instantiate an sqlite database for caching provider data.

fixes: #632

How I did it

created cli and added init, purge and query functions

How to verify it

I used the infura plugin

ape plugins install infura

Then instantiate the caching database

ape cache init --network ethereum:mainnet:infura
ape console --network ethereum:mainnet:infura

Run this multiple times:

In [1]: chain.blocks.query("number", start_block=0, stop_block=100)

You should see that the query occurs quicker after the first call. This will take a few seconds to grab the data from the provider

Also run transaction calls multiple times:

In [2]: chain.blocks[-2].transactions

You should see that the query is quicker after the first call. That block number can change, so you may not see it after the first attempt. But you will have DynamicTransactionAPI datatypes in the return if it comes from the provider. You will just receive a dictionary if it comes from the database.

Then exit the IPython kernel
And run the below commands

ape cache query --network ethereum:mainnet:infura "SELECT *  FROM blocks"
ape cache query --network ethereum:mainnet:infura "SELECT * FROM transactions"

Both should return data from the database.

Also run pytests

Checklist

• All changes are completed
• New test cases have been added
• Documentation has been updated

[antazoey]

Coming along!

My main two comments:

• Avoid string interpolation in SQL - use query parameters instead... It is better practice and has more security checks
• What happened to DefaultQueryProvider? Are we required to have a DB now? This would be a pretty big breaking change, right?

[antazoey]

I think the docs in ape_cache can just be deleted! We don't really generate anything from those.

Address @fubuloubu 's thing and make the tests happy and it is an approval from me :)

Bonus if you use the logger though! It is better because I sometimes like to disable output using the --verbosity flag and it's more consistent.

Thank you :)

[johnson2427]

Should we force the query to pull all data from the provider if it doesn't exist? That way we could populate the cache db. If you wanted to only see the number column, then pandas will return that column at the end, yet the rest of that data will be stored. It prevents you from having to hit the provider multiple times for data that you are already there for.

[fubuloubu]

Should we force the query to pull all data from the provider if it doesn't exist? That way we could populate the cache db. If you wanted to only see the number column, then pandas will return that column at the end, yet the rest of that data will be stored. It prevents you from having to hit the provider multiple times for data that you are already there for.

By default, it already is pulling all the data, however it is then filtering out all the data that wasn't asked for. So pushing that data to the cache db prior to that filter being applied would be an acceptable way to proceed. Since it's an iterator, you would need itertools.tee to fork the iterator into two different iterators

[johnson2427]

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
ape-infura 0.2.0 requires eth-ape<0.3.0,>=0.2.1, but you have eth-ape 0.1.0b3.dev192+g9a790289.d20220502 which is incompatible.

Not sure how to handle this issue.

This this is a quick fix that I'm not seeing, let me know

[NotPeopling2day]

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.

ape-infura 0.2.0 requires eth-ape<0.3.0,>=0.2.1, but you have eth-ape 0.1.0b3.dev192+g9a790289.d20220502 which is incompatible.

Not sure how to handle this issue.

This this is a quick fix that I'm not seeing, let me know

Sounds like your environment is behind? Try fetching the latest version of ape core from upstream and see if that fixes it.

[johnson2427]

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.

ape-infura 0.2.0 requires eth-ape<0.3.0,>=0.2.1, but you have eth-ape 0.1.0b3.dev192+g9a790289.d20220502 which is incompatible.

Not sure how to handle this issue.
This this is a quick fix that I'm not seeing, let me know

Sounds like your environment is behind? Try fetching the latest version of ape core from upstream and see if that fixes it.

yeah, it is, I'm up to date with the main, rebased, yet when I install, it still installs 0.1... not sure why. I'll remake my venv

[antazoey]

Added some small code optimizations.

I have an idea for some tests, but it might not be easy:

1. Populate the .db files on your machine for various providers.
2. Add those files to test fixtures and use them in tests

And so that leads to me another idea: Allow connecting to HistoricalProvider which only requires the .db files and no active connection. This will help with testing against historical data.

Edit: the testing / historical-provider design can happen later, but tests would be nice somehow.

[johnson2427]

Only issue left is I cannot store the data as is. We get an error for unsupported types.

[antazoey]

Some of my other comments were resolved though I don't see the resolution (maybe not pushed yet?).

Looking fantastic though! I can't wait to use, especially for mainnet-fork.
I want to make a pre-populated mainnet-fork .db for testing.

[fubuloubu]

A little bit more refactoring and we can merge this!

Then move optimization work to another PR (partial row support, mgirations, etc.)

[antazoey]

🎉

hold up, seeing something with a logger?

[antazoey]

Minimally, go to tests/integration/cli/test_misc.py and add cache to that list. I am concerned that the plugin currently does not work yet it has passing tests.

[johnson2427]

@fubuloubu I respectfully argue against using decode_receipt on the caching side. There's really no need to do so at the moment. We just need the data to return a dataframe at the moment.

[fubuloubu]

Just one major refactor and a bunch of minor comments, and we are good

[fubuloubu]

Suggested change

	def perform_query(self, query: QueryType) -> Optional[Iterator]: # type: ignore
	def perform_query(self, query: QueryType) -> Iterator:

[johnson2427]

21465f7

Still need the type ignore. Without it I get:

src/ape_cache/query.py:314: error: Signature of "perform_query" incompatible with supertype "QueryAPI"

[fubuloubu]

hmm

[johnson2427]

@fubuloubu I'm not touching that Squash and Merge button! Too much pressure!

[fubuloubu]

Approved, with many issues coming out of this which @NotPeopling2day is tracking

[antazoey]