Using --deploy-fixture (hardhat-deploy option) results in empty gas report #86

rlogiacco · 2022-01-16T23:59:48Z

It looks like by adding a call to ethers my gas report goes from being populated to completely empty.

Here is the deploy script disabling the gas reporting: if I comment out the lines I've marked gar report comes back.

import { HardhatRuntimeEnvironment } from "hardhat/types";
import { DeployFunction } from "hardhat-deploy/types";

const func: DeployFunction = async function (hre: HardhatRuntimeEnvironment) {
  const { upgrades } = hre;

  const factory = await hre.ethers.getContractFactory("ReceiversHolder"); // this is the culprit
  const instance = await upgrades.deployProxy(factory, [[]]); // this must be comment to avoid compilation errors
  await instance.deployed(); // this must be comment to avoid compilation errors
};
export default func;

My dev deps are:

    "devDependencies": {
        "@nomiclabs/hardhat-ethers": "^2.0.4",
        "@nomiclabs/hardhat-etherscan": "^2.1.8",
        "@nomiclabs/hardhat-waffle": "^2.0.1",
        "@openzeppelin/contracts": "^4.4.0",
        "@openzeppelin/contracts-upgradeable": "^4.4.0",
        "@openzeppelin/hardhat-upgrades": "^1.13.0",
        "@typechain/ethers-v5": "^7.2.0",
        "@typechain/hardhat": "^2.3.1",
        "@types/chai": "^4.3.0",
        "@types/data-urls": "^2.0.1",
        "@types/mocha": "^9.0.0",
        "@types/node": "^16.4.0",
        "@typescript-eslint/eslint-plugin": "^4.33.0",
        "@typescript-eslint/parser": "^4.33.0",
        "base64-sol": "^1.1.0",
        "chai": "^4.3.4",
        "chai-as-promised": "^7.1.1",
        "data-urls": "^3.0.0",
        "dotenv": "^10.0.0",
        "eslint": "^7.32.0",
        "eslint-config-prettier": "^8.3.0",
        "eslint-config-standard": "^16.0.3",
        "eslint-plugin-import": "^2.25.3",
        "eslint-plugin-node": "^11.1.0",
        "eslint-plugin-prettier": "^3.4.1",
        "eslint-plugin-promise": "^5.2.0",
        "ethereum-waffle": "^3.4.0",
        "ethers": "^5.5.3",
        "hardhat": "^2.7.1",
        "hardhat-deploy": "^0.9.14",
        "hardhat-deploy-ethers": "^0.3.0-beta.11",
        "hardhat-gas-reporter": "^1.0.6",
        "hardhat-packager": "^1.2.1",
        "npm-prepare-dist": "^0.3.3",
        "prettier": "^2.5.1",
        "prettier-plugin-solidity": "^1.0.0-beta.13",
        "recursive-copy": "^2.0.13",
        "solhint": "^3.3.6",
        "solidity-coverage": "^0.7.17",
        "ts-node": "^10.1.0",
        "typescript": "^4.3.5"
      }

The text was updated successfully, but these errors were encountered:

cgewecke · 2022-01-19T23:47:57Z

@rlogiacco Do you have a repo or branch I could look at to reproduce this? I suspect the problem may be coming from upgradeability & proxying rather than ethers.

rlogiacco · 2022-01-23T11:29:35Z

Sure I do: https://github.com/agileware-org/nft-collection branch upgradeable.
I would agree with you if only some invocations were missing, like those going through a proxy, but they all disappear, including cost of contract deployment...

Let me know if you need any help in understanding the repo

cgewecke · 2022-01-23T17:32:18Z

@rlogiacco I wasn't able to reproduce this problem (on the hardhat network).

After installing your repo, I commented out rinkeby, mainnet and default network settings in hardhat.config.ts (because I wasn't using a private key and it throws an error when loading)

Then I ran npx hardhat test, setting the REPORT_GAS env variable to true (because your reporter config says gas reporting is only enabled when that variable is set).

I saw:

Is something missing from this report?

[EDIT: updated screenshot to show case when coinmarketcap env variable is set]

rlogiacco · 2022-01-23T22:39:31Z

have you switched to the upgradeable branch? I believe you are still on the main branch because I don't see the invocation of the proxy upgrade operation.

The entire command I'm using is REPORT_GAS=y npx hardhat test --network hardhat --deploy-fixture

cgewecke · 2022-01-23T23:00:56Z

@rlogiacco Sorry, you're right, I was on the wrong branch. On upgradeable without the --deploy-fixture flag I see a report which suggests the problem comes from the way you're using hardhat-deploy. Could you verify this is the case on your end?

The --deploy-fixture flag tells hardhat-deploy to execute the deployments before the tests launch and the mocha reporter has no record of those contracts. It begins tracking deployments and matching contract invocations to contract addresses when the tests start.

This should be fixable fwiw. There's special logic at eth-gas-reporter to handle a similar issue with Truffle migrations. We just need to inspect hardhat-deploy's artifacts and match contracts to their pre-deployed addresses.

In the interim, is it viable for you to run your tests without that flag enabled if you want a gas report? Both commands seem to work (although --deploy-fixture may speed things up a bit)

rlogiacco · 2022-01-24T19:24:13Z

Sure, but the report becomes empty only if I use the ethers library in any of those deployment fixtures, while it works perfectly if I don't use the ethers lib in there. To verify just open the file deploy/2.DroppableCollectionFactory.ts and comment line 11 (and anything related to that, obviously). Then the report will be there again even if you use the --deploy-fixture flag.

In other words, the use of the ethers library in any deploy fixture breaks the gas report. I'm about to create a new branch (I'll call it with-gas-report) with the same content but a few variations which do not require the use of the ethers library.

Thanks a million for taking the time to help me on this.

rlogiacco · 2022-01-24T19:33:29Z

Here you go, and at this PR you can see the difference agileware-org/nft-collection#1

rlogiacco · 2022-01-27T23:44:46Z

@cgewecke have you had any chance to review the link I provided? I believe it makes more clear why I'm considering this an issue with the ethers library...

cgewecke · 2022-01-28T00:38:36Z

Apologies @rlogiacco

I'm unsure what's happening in the new branch but perhaps the get method executes the deployments if they're missing during the test. That would result in them being detected by the reporter.

get is part of hardhat-deploy.

Does it work with ethers and without the --deploy-fixutre flag?

If so the differences you're seeing are very likely due to how hardhat-deploy behaves.

rlogiacco · 2022-01-28T22:18:10Z

Actually it does work and it reports gas correctly if I remove the --deploy-fixture flag!!!
That's pretty surprising!
So I guess the get method somewhat triggers something parallel to the what the --deploy-fiXture does!

Do you suggest I close this and open an issue on the hardhat-deploy project?

cgewecke · 2022-01-28T23:28:06Z

Do you suggest I close this and open an issue on the hardhat-deploy project?

@rlogiacco No, I think it's something we should fix here (e.g at eth-gas-reporter). Thanks for discovering it.

Am going to edit the title a bit to reflect the underlying issue.

cameel · 2022-02-04T22:44:27Z

Just to add more data points for testing - I have recently seen that in an NFT project called Bleeps. Here's how to reproduce it:

git clone https://github.com/wighawag/bleeps
cd bleeps/common-lib/
npm install
npm run build
cd ../contracts
sed -i 's|"bleeps-common": "workspace:\*",|"bleeps-common": "file:../common-lib/",|g' package.json
rm test/BleepsDAO.governor.test.ts # Quick'n'dirty workaround for https://github.com/wighawag/bleeps/issues/2
npm install npm-run-all
npm install
HARDHAT_DEPLOY_FIXTURE=true REPORT_GAS=true npx --no hardhat test

·--------------------------------------------------|---------------------------|----------------|-----------------------------·
|               Solc version: 0.8.9                ·  Optimizer enabled: true  ·  Runs: 999999  ·  Block limit: 50000000 gas  │
···················································|···························|················|······························
|  Methods                                                                                                                    │
·····························|·····················|·············|·············|················|·············|················
|  Contract                  ·  Method             ·  Min        ·  Max        ·  Avg           ·  # calls    ·  usd (avg)    │
·····························|·····················|·············|·············|················|·············|················
|  Bleeps                    ·  delegateBySig      ·      76512  ·     156596  ·        118510  ·          3  ·            -  │
·····························|·····················|·············|·············|················|·············|················
|  Bleeps                    ·  mint               ·      93503  ·     127468  ·        119195  ·        152  ·            -  │
·····························|·····················|·············|·············|················|·············|················
|  Bleeps                    ·  permit             ·      40878  ·      57966  ·         49422  ·          2  ·            -  │
·····························|·····················|·············|·············|················|·············|················
|  Bleeps                    ·  permitForAll       ·      58066  ·      75174  ·         66620  ·          2  ·            -  │
·····························|·····················|·············|·············|················|·············|················
|  Bleeps                    ·  safeTransferFrom   ·     141843  ·     147854  ·        144011  ·         16  ·            -  │
·····························|·····················|·············|·············|················|·············|················
|  Bleeps                    ·  safeTransferFrom   ·     142445  ·     148745  ·        144566  ·         12  ·            -  │
·····························|·····················|·············|·············|················|·············|················
|  Bleeps                    ·  setApprovalForAll  ·      24279  ·      46203  ·         42825  ·         26  ·            -  │
·····························|·····················|·············|·············|················|·············|················
|  Bleeps                    ·  setMinter          ·      47171  ·      47183  ·         47181  ·        120  ·            -  │
·····························|·····················|·············|·············|················|·············|················
|  ERC20                     ·  approve            ·      28982  ·      51429  ·         49699  ·         26  ·            -  │
·····························|·····················|·············|·············|················|·············|················
|  ERC20                     ·  transferFrom       ·     107631  ·     147110  ·        138121  ·         53  ·            -  │
·····························|·····················|·············|·············|················|·············|················
|  MeloBleeps                ·  mint               ·          -  ·          -  ·        120110  ·          2  ·            -  │
·····························|·····················|·············|·············|················|·············|················
|  MeloBleeps                ·  reserve            ·          -  ·          -  ·        139861  ·          2  ·            -  │
·····························|·····················|·············|·············|················|·············|················
|  MeloBleeps                ·  setMinter          ·          -  ·          -  ·         30061  ·          2  ·            -  │
·····························|·····················|·············|·············|················|·············|················
|  OpenSeaProxyRegistryMock  ·  setProxy           ·          -  ·          -  ·         43895  ·          1  ·            -  │
·----------------------------|---------------------|-------------|-------------|----------------|-------------|---------------·

  81 passing (2m)

The author seems aware of the limitation because there's actually a separate npm run void:deploy command that executes hardhat deploy --report-gas. For me it would be very convenient if the deployment cost was included in the main report though.

cameel · 2022-02-04T23:18:20Z

One more thing that may or may not be related. I noticed that I do not get the report if I run tests with mocha. I only get them with hardhat test directly. You can see that if you replace hardhat test in the above repro with npm run test. Is that something I should report as a bug or some known limitation?

cgewecke · 2022-02-05T00:00:41Z

@cameel Ah I've never tried that actually. 😅

If you run mocha --reporter eth-gas-reporter does anything happen?

I think it should find the purely mocha version of the reporter (which this plugin wraps) in the dependencies and connect to a client on localhost:8545. The options here are mostly mocha reporterOptions consumed by eth-gas-reporter.

cameel · 2022-02-05T00:17:27Z

Just tried adding this to the command from Bleeps (HARDHAT_DEPLOY_FIXTURE=true HARDHAT_COMPILE=true mocha --bail --recursive test) and looks like it invoked eth-gas-reporter but it had problems connecting:

ERROR: eth-gas-reporter was unable to resolve a client url from the provider available in your test context. Try setting the url as a mocha reporter option (ex: url='http://localhost:8545')

In any case I think this answers my question - mocha is just using its own reporter and has to be configured to use eth-gas-reporter - which is what Hardhat does by default. This unfortunately does not solve the problem I had with it - I was hoping I could somehow get an unmodified npm run test command to report gas - but it's also not like it's a huge problem. npm run test would just be a bit less likely to break in case a project changes the command to run tests in a different way. I just thought it might have been a bug in the reporter but looks like it's not :)

HetmanJones · 2022-02-05T16:03:56Z

I'm using hardhat-deploy and my deploy files don't use ethers, I use

import { DeployFunction } from 'hardhat-deploy/types';

const func: DeployFunction = async function (hre: HardhatRuntimeEnvironment) {
    const { deployments, getNamedAccounts } = hre;
    const { deploy } = deployments;
    const { deployer } = await getNamedAccounts();

    await deploy('Contract', {
        from: deployer,
        log: true,
    });
};

export default func;

func.tags = ['Contract'];

I think it would be very cool if the reporter can work with hardhat-deploy because having the deployment files and being able to reuse it with --deploy-fixture, makes things really easy. Otherwise we need to redeploy from each test suite?

nkuba · 2022-05-02T10:21:59Z

We experienced similar problems with missing gas reports for functions in many different use cases.

I was trying to reproduce the problem with a minimal configuration and I came to a conclusion that any usage of ethers in the deployment script along with the --deploy-fixture flag causes the gas report to be broken.

Please find an example in the repository https://github.com/nkuba/hardhat-gas-reporter-playground.

After running the yarn test you will get a following report:

·-----------------------|----------------------------|-------------|-----------------------------·
|  Solc version: 0.8.9  ·  Optimizer enabled: false  ·  Runs: 200  ·  Block limit: 30000000 gas  │
························|····························|·············|······························
|  Methods                                                                                       │
·············|··········|··············|·············|·············|·············|················
|  Contract  ·  Method  ·  Min         ·  Max        ·  Avg        ·  # calls    ·  eur (avg)    │
·------------|----------|--------------|-------------|-------------|-------------|---------------·

  1 passing (173ms)

It's enough to comment out the line referenced below to get a correct report.
ethers.utils.isAddress("0xFaKeAddrEss")

·-------------------------------|----------------------------|-------------|-----------------------------·
|      Solc version: 0.8.9      ·  Optimizer enabled: false  ·  Runs: 200  ·  Block limit: 30000000 gas  │
································|····························|·············|······························
|  Methods                                                                                               │
·················|··············|··············|·············|·············|···············|··············
|  Contract      ·  Method      ·  Min         ·  Max        ·  Avg        ·  # calls      ·  eur (avg)  │
·················|··············|··············|·············|·············|···············|··············
|  TestContract  ·  initialize  ·           -  ·          -  ·      68609  ·            1  ·          -  │
·················|··············|··············|·············|·············|···············|··············
|  TestContract  ·  setValue    ·           -  ·          -  ·      30190  ·            1  ·          -  │
·················|··············|··············|·············|·············|···············|··············
|  Deployments                  ·                                          ·  % of limit   ·             │
································|··············|·············|·············|···············|··············
|  TestContract                 ·           -  ·          -  ·     341398  ·        1.1 %  ·          -  │
·-------------------------------|--------------|-------------|-------------|---------------|-------------·

In deployment script we workaround a bug in hardhat-gas-reporter cgewecke/hardhat-gas-reporter#86. For some reason using `hre.ethers` makes gas report broken, the gas usage for functions is not included in the report. This is a problem that shows up in other projects using the deployment scripts of this project, e.g. @keep-network/ecdsa.

Workaround ethers import in deployment script for a bug in hardhat-ethers-plugin This PR is a workaround for an issue with hardhat-gas-reporter plugin (cgewecke/hardhat-gas-reporter#86). When hre.ethers is used in the deployment script it causes a gas report to be missing executed function's data. For tests executed in this repository, reports are fine, but when we use `@threshold-network/solidity-contracts` as a dependency in other projects (e.g. `@keep-network/ecdsa`) it breaks the reports there.

There is a but in hardhat-gas-reporter that we need to workaround in order to obtain gas report for tests: cgewecke/hardhat-gas-reporter#86 Long story short: we cannot use `hre.ethers` in the deployment script. Here we introduce an alternative deployment path just to get the report generated correctly. The alternative path usage is determined by `GAS_REPORTER_BUG_WORKAROUND` env property set to `true`. To run the tests with the alternative path to get the gas report execute `yarn test:gas-reporter-workaround`. It will execute the same test (upgrade tests were skipped as they are strictly related to the original deployment path). `yarn test` is still a preferred way to run the tests, as it does the deployment exactly the way we want it to test for mainnet. Use the alternative path only if you wish to get the gas report. Once a bug in the `hardhat-gas-reporter` is fixed, we should revert all the changes from this commit.

There is a bug in hardhat-gas-reporter that we need to workaround in order to obtain gas report for tests: cgewecke/hardhat-gas-reporter#86 Long story short: we cannot use `hre.ethers` in the deployment script. Here we introduce an alternative deployment path just to get the report generated correctly. The alternative path usage is determined by `GAS_REPORTER_BUG_WORKAROUND` env property set to `true`. To run the tests with the alternative path to get the gas report execute `yarn test:gas-reporter-workaround`. It will execute the same test (upgrade tests were skipped as they are strictly related to the original deployment path). `yarn test` is still a preferred way to run the tests, as it does the deployment exactly the way we want it to test for mainnet. Use the alternative path only if you wish to get the gas report. Once a bug in the `hardhat-gas-reporter` is fixed, we should revert all the changes from this commit.

ECDSA: Workaround hardhat-gas-reporter bug There is a bug in hardhat-gas-reporter that we need to work around in order to obtain a gas report for tests: cgewecke/hardhat-gas-reporter#86 Long story short: we cannot use hre.ethers in the deployment scripts. Here we introduce an alternative deployment path just to get the report generated correctly. The alternative path usage is determined by the GAS_REPORTER_BUG_WORKAROUND env property set to true. To run the tests with the alternative path to get the gas report execute yarn test:gas-reporter-workaround command. It will execute the same tests (upgrade tests were skipped as they are strictly related to the original deployment path). yarn test is still a preferred way to run the tests, as it does the deployment exactly the way we want it to test for mainnet. Use the alternative path only if you wish to get the gas report. Once a bug in the hardhat-gas-reporter is fixed, we should revert all the changes related to the workaround.

In #2970 we introduced a workaround for a bug in hardhat-gas-reporter plugin cgewecke/hardhat-gas-reporter#86. It turned out that there is a much simpler workaround, which is based on just removing `--deploy-fixture` flag. With the flag removed it is required that we load a global snapshot of the deployment (`deployments.fixture()`) instead of loading a specific tag (`deployments.fixture(<tag>)`. Which we do anyway after one of a recent code updates. We removed all the traces of the previous workaround.

…orkaround Workaround hardhat-gas-reporter bug with removing --deploy-fixture In this PR we replace a previous workaround (#2970) for a bug in the hardhat-gas-reporter plugin (cgewecke/hardhat-gas-reporter#86). It turned out that there is a much simpler workaround, which is based on just removing --deploy-fixture flag. With the flag removed it is required that we load a global snapshot of the deployment (deployments.fixture()) instead of loading a specific tag (deployments.fixture(<tag>). Which we do anyway after one of a recent code updates.

sherpya · 2023-03-07T15:45:14Z

I suspect it will only work if your contract is upgradable

cgewecke mentioned this issue Jan 20, 2022

enabled but does not report anything cgewecke/eth-gas-reporter#246

Closed

cgewecke changed the title ~~Empty gas report adding hardhat-ethers~~ Using --deploy-fixture (hardhat-deploy option) results in empty gas report Jan 28, 2022

cameel mentioned this issue Feb 4, 2022

Benchmarking external tests ethereum/solidity#12441

Merged

9 tasks

nkuba mentioned this issue May 2, 2022

Fix gas reporter for beacon and ecdsa keep-network/keep-core#2955

Closed

nkuba mentioned this issue May 2, 2022

Workaround ethers import in deployment script for a bug in hardhat-ethers-plugin threshold-network/solidity-contracts#100

Merged

nkuba mentioned this issue May 3, 2022

ECDSA: Workaround hardhat-gas-reporter bug keep-network/keep-core#2970

Merged

nkuba mentioned this issue May 11, 2022

Investigate possible another workaround for hardhat-gas-reporter bug keep-network/keep-core#2979

Closed

nkuba mentioned this issue May 11, 2022

Workaround hardhat-gas-reporter bug with removing --deploy-fixture keep-network/keep-core#2980

Merged

remedcu mentioned this issue Feb 7, 2024

Added Basic Gas Profiling safe-global/safe-locking#37

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Using --deploy-fixture (hardhat-deploy option) results in empty gas report #86

Using --deploy-fixture (hardhat-deploy option) results in empty gas report #86

rlogiacco commented Jan 16, 2022

cgewecke commented Jan 19, 2022

rlogiacco commented Jan 23, 2022

cgewecke commented Jan 23, 2022 •

edited

rlogiacco commented Jan 23, 2022

cgewecke commented Jan 23, 2022 •

edited

rlogiacco commented Jan 24, 2022

rlogiacco commented Jan 24, 2022

rlogiacco commented Jan 27, 2022

cgewecke commented Jan 28, 2022

rlogiacco commented Jan 28, 2022

cgewecke commented Jan 28, 2022

cameel commented Feb 4, 2022 •

edited

cameel commented Feb 4, 2022 •

edited

cgewecke commented Feb 5, 2022

cameel commented Feb 5, 2022 •

edited

HetmanJones commented Feb 5, 2022 •

edited

nkuba commented May 2, 2022

sherpya commented Mar 7, 2023

Using --deploy-fixture (hardhat-deploy option) results in empty gas report #86

Using --deploy-fixture (hardhat-deploy option) results in empty gas report #86

Comments

rlogiacco commented Jan 16, 2022

cgewecke commented Jan 19, 2022

rlogiacco commented Jan 23, 2022

cgewecke commented Jan 23, 2022 • edited

rlogiacco commented Jan 23, 2022

cgewecke commented Jan 23, 2022 • edited

rlogiacco commented Jan 24, 2022

rlogiacco commented Jan 24, 2022

rlogiacco commented Jan 27, 2022

cgewecke commented Jan 28, 2022

rlogiacco commented Jan 28, 2022

cgewecke commented Jan 28, 2022

cameel commented Feb 4, 2022 • edited

cameel commented Feb 4, 2022 • edited

cgewecke commented Feb 5, 2022

cameel commented Feb 5, 2022 • edited

HetmanJones commented Feb 5, 2022 • edited

nkuba commented May 2, 2022

sherpya commented Mar 7, 2023

cgewecke commented Jan 23, 2022 •

edited

cgewecke commented Jan 23, 2022 •

edited

cameel commented Feb 4, 2022 •

edited

cameel commented Feb 4, 2022 •

edited

cameel commented Feb 5, 2022 •

edited

HetmanJones commented Feb 5, 2022 •

edited