Option to specify location of '.cache' folder [APE-703]

[0rtis]

Overview

Add an option to specify th location of the .cache folder. At the moment, it is located at the root of the contracts folder. This can cause issue with other framework (ex: Hardhat will parse all contracts in the contracts folder, including .cache resulting in very long processing time)

Specification

Ideally, it would be an option on the ape config file like cache_folder: mycache (similar to contracts_folder).

[antazoey]

This may also affect compiler plugins like ape-solidity. We would need to make sure they work when the .cache folder is outside of the contracts/ folder may require providing multiple base-paths.

[mikeshultz]

So what all ends up in the .cache folder? Is it just ethpm/dependency stuff or is there other uses?

[antazoey]

So what all ends up in the .cache folder? Is it just ethpm/dependency stuff or is there other uses?

Primarily it gets used by ape-solidity. Your dependency source files get plopped there from the cache and then ape-soldity uses remappings to refer to the files there from your import statements. That is why it defaults to the contracts/ folder - it works best for importing

[mikeshultz]

So what all ends up in the .cache folder? Is it just ethpm/dependency stuff or is there other uses?

Primarily it gets used by ape-solidity. Your dependency source files get plopped there from the cache and then ape-soldity uses remappings to refer to the files there from your import statements. That is why it defaults to the contracts/ folder - it works best for importing

Wondering if solc going to have a problem with this. I vaguely recall some sticking points that might make this challenging. For instance, directory paths affecting some kind of source hashing (libraries maybe?) or contract validation (memory is foggy). Also, I didn't think solc would compile source files outside of the contracts path.

Maybe there's some workarounds. Just need to wire it up and test it, I guess.

[antazoey]

So what all ends up in the .cache folder? Is it just ethpm/dependency stuff or is there other uses?

Primarily it gets used by ape-solidity. Your dependency source files get plopped there from the cache and then ape-soldity uses remappings to refer to the files there from your import statements. That is why it defaults to the contracts/ folder - it works best for importing

Wondering if solc going to have a problem with this. I vaguely recall some sticking points that might make this challenging. For instance, directory paths affecting some kind of source hashing (libraries maybe?) or contract validation (memory is foggy). Also, I didn't think solc would compile source files outside of the contracts path.

Maybe there's some workarounds. Just need to wire it up and test it, I guess.

Alternative idea: is there a way to tell Hardhat to ignore these files via CLI or config file?

[mikeshultz]

Background on the paths restrictions. I haven't dug into ape-solidity's compiler code yet so maybe if it's building JSON bundles or something it doesn't matter. https://docs.soliditylang.org/en/latest/path-resolution.html#allowed-paths

The hashing thing I mentioned, I think it had to do with solc producing different bytecode if the directory structure of contract imports were changed. That memory is super old and may not be relevant anymore. I also can't seem to find any modern reference to the issue, so may also not be relevant.

I think we just need to try this out and see what happens.

[antazoey]

I think we just need to try this out and see what happens.

Ya, definitely need some research and answers to questions.
But it seems like the real issue is that Ape isn't playing nice with other frameworks?

Does HH or Foundry cache dependency-source files for importing somewhere similarly?

[mikeshultz]

HH stores dep contracts in the project root's node_modules/ dir.

Looks like foundry stores deps in a lib/ dir in the root, also separate from user sources. They also seem to require some remappings for OZ internal/clean imports.

[antazoey]

Solc has a --include-path CLI option. Perhaps we just need to make sure the new .cache directory is included there when compiling

[fubuloubu]

@Ninjagod1251 can share some of our preliminary findings with the both of you based on our conversations with the HH team

[mikeshultz]

Found .cache referenced in ape-solidity and ape-cairo. ape-vyper does not appear to leverage this. Any others I might be missing?

[antazoey]

Found .cache referenced in ape-solidity and ape-cairo. ape-vyper does not appear to leverage this. Any others I might be missing?

• Don't worry about ape-cairo, is no longer really works. You can think of it as theoretical though.
• ape-vyper does not use it - i think it is does differently, like linking the compiled JSONs directly

ape-solidity for sure does use it.

[fubuloubu]

ape-vyper does not appear to leverage this. Any others I might be missing?

It might in the future as vyper is adding module libraries

[mikeshultz]

Frustrating experimentation. The couple of potentially useful compiler options are only available for newer versions, and trying to conditionally support project-level features depending on compiler version probably isn't really going to work (e.g. --no-cbor-metadata and --import-paths).

The hashing thing I mentioned, I think it had to do with solc producing different bytecode if the directory structure of contract imports were changed. That memory is super old and may not be relevant anymore. I also can't seem to find any modern reference to the issue, so may also not be relevant.

Changing the paths of files, changes the metadata (and JSON inputs) which changes the "garbage" appended to the contracts' bytecode. More background on this: https://docs.soliditylang.org/en/latest/metadata.html#encoding-of-the-metadata-hash-in-the-bytecode

It's kind of a cool feature, but super frustrating and I'm not aware of anyone that actually leverages it (though seems like it could be a cool Ape feature).

So, moving the location of the .cache directory changes the bytecode output, which may be unexpected to users. Maybe this isn't a big deal, since it's a setting in project-level config, but it may be pretty confusing and is likely to break things like contract verification if it ever changes during the life of the project.

I think that's probably fine and we just document it and let the users adjust their project structure as they see fit. If they manually had deps in their source tree and moved them they'd see the same behavior.

---

Long term thinking, maybe we could maybe create a temporary package structure that we build from on the fly every time we compile. This lets us assemble dependencies in a relative path however we see fit. This would likely increases complexity, and have a performance impact but having reproducible build steps is an interesting idea.

[antazoey]

Long term thinking, maybe we could maybe create a temporary package structure that we build from on the fly every time we compile.

I think we kinda do this for dependencies already so it might not be a huge leap.

Regarding the flags that are available on some versions but not others: we do similar things already:
https://github.com/ApeWorX/ape-solidity/blob/main/ape_solidity/compiler.py#L461-L462

[mikeshultz]

Regarding the flags that are available on some versions but not others: we do similar things already: https://github.com/ApeWorX/ape-solidity/blob/main/ape_solidity/compiler.py#L461-L462

If we did go that route, it would be just "throw error if config option is used and compiler < version" which isn't ideal, and probably not necessary.

[antazoey]

We could add those configs separately to the ape-solidity config so the user could sometimes do this:

compile:
  cache_folder: my/custom/cache

solidity:
    import-paths:
      - my/custom/cache

[mikeshultz]

We could add those configs separately to the ape-solidity config so the user could sometimes do this:

I don't think this makes much sense (for our problem). This basically requires user configuration to support some internal behavior of ape-solidity rather than something the user wanted to do. And only if they happen to not have any contracts with an older pragma.

[mikeshultz]

Also, thinking about this and testing a bit more, we probably shouldn't allow absolute paths here (nor resolve anything to absolute paths) because they are likely to be different across machines. And different paths inputs would cause different metadata hashes altering the bytecode...

But, maybe we can use the project root as a base path for compilers instead of the contracts/ directory.

[mikeshultz]

• Core side PR: feat: adds compiler_cache_folder config var #1897
• ape-solidity PR: feat: support configured dependency cache directory ape-solidity#133

Need to get the docs correct with the latest revision but the code should be ready for review.

[mikeshultz]

Just adding some context I ran into for anyone that needs to reference in this issue.

Was just looking around and it looks like Vyper also had some CBOR metadata appended to the bytecode. Then removed it later. Doesn't look like it was influenced by input metadata at all, so settings changes or path changes shouldn't influence it at all like with Solidity.

[fubuloubu]

Doesn't look like it was influenced by input metadata at all, so settings changes or path changes shouldn't influence it at all like with Solidity.

Correct, Vyper's CBOR only contained the version of the compiler it was compiled with

[mikeshultz]

This feature will be availible in the next release of the ape-solidity plugin. For some information on how to use it, see the compiler documentation.