Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failed to deserialize response: expected value at line 1 column 1 #4865

Closed
2 tasks done
mpeyfuss opened this issue May 2, 2023 · 30 comments · Fixed by #6079
Closed
2 tasks done

Failed to deserialize response: expected value at line 1 column 1 #4865

mpeyfuss opened this issue May 2, 2023 · 30 comments · Fixed by #6079
Labels
T-bug Type: bug

Comments

@mpeyfuss
Copy link

mpeyfuss commented May 2, 2023

Component

Forge

Have you ensured that all of these are up to date?

  • Foundry
  • Foundryup

What version of Foundry are you on?

forge 0.2.0 (8307d6d 2023-04-20T00:05:51.425008000Z)

What command(s) is the bug in?

forge verify-contract

Operating System

Linux

Describe the bug

I am running forge in an AWS EC2 instance and on verification keep getting a 403 error from Etherscan which results in the Failed to deserialize response: expected value at line 1 column 1 error as it returns html with a captcha rather than json.

I've reached out to Etherscan and this was their response:

We are aware of our CloudFlare protections being particularly sensitive to unknown automated scripts, and returning 403 responses as a result.
We're looking into whitelisting this at the moment, however a quick workaround for this would be to make your request more identifiable by attaching a User Agent string

My understanding is that reqwest doesn't set a default User-Agent header and so CloudFlare decides to block my request.

Does anyone know of a way to add a User-Agent header to outgoing requests on forge?

This seems like it currently affects just a small subset of users, although it would future proof everyone if a default user agent was set by forge/foundry. For example, curl automatically applies a default user agent so it would make sense for forge/foundry to do the same.

@mpeyfuss mpeyfuss added the T-bug Type: bug label May 2, 2023
@mattsse
Copy link
Member

mattsse commented May 2, 2023

Does anyone know of a way to add a User-Agent header to outgoing requests on forge?

we already set this

ethers_etherscan::Client::builder()
.with_client(reqwest::Client::builder().user_agent(ETHERSCAN_USER_AGENT).build()?)

we check for various cloudflare errors, but this one we're missing apparently, do you know what kind of error is returned?

@mpeyfuss
Copy link
Author

mpeyfuss commented May 2, 2023

ah thanks for pointing me to that. I was on an older build but just now built again from source on the EC2. about to test

Not 100% sure the kind of error returned as it doesn't seem like a regular 403 error. But here is the html body that foundry couldn't deserialize.

<!DOCTYPE html>
<html lang="en-US">
<head>
    <title>Just a moment...</title>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
    <meta http-equiv="X-UA-Compatible" content="IE=Edge">
    <meta name="robots" content="noindex,nofollow">
    <meta name="viewport" content="width=device-width,initial-scale=1">
    <link href="/cdn-cgi/styles/challenges.css" rel="stylesheet">
    

</head>
<body class="no-js">
    <div class="main-wrapper" role="main">
    <div class="main-content">
        <noscript>
            <div id="challenge-error-title">
                <div class="h2">
                    <span class="icon-wrapper">
                        <div class="heading-icon warning-icon"></div>
                    </span>
                    <span id="challenge-error-text">
                        Enable JavaScript and cookies to continue
                    </span>
                </div>
            </div>
        </noscript>
        <div id="trk_jschal_js" style="display:none;background-image:url('/cdn-cgi/images/trace/captcha/nojs/transparent.gif?ray=7c08c85288277fb7')"></div>
        <form id="challenge-form" action="/api/?__cf_chl_f_tk=Q7F3hkX.VOk8KSj37RUZQAEw34ohIhGvc9K2pEp9d2c-1682952056-0-gaNycGzNMDs" method="POST" enctype="application/x-www-form-urlencoded">
            <input type="hidden" name="md" value="REAALLYY LONG VALUE OMITTED HERE">
        </form>
    </div>
</div>
<script>
    (function(){
        window._cf_chl_opt={
            cvId: '2',
            cZone: 'api-goerli.etherscan.io',
            cType: 'interactive',
            cNounce: '11195',
            cRay: '7c08c85288277fb7',
            cHash: '98433f73c9870d8',
            cUPMDTk: "\\/api\\/?__cf_chl_tk=Q7F3hkX.VOk8KSj37RUZQAEw34ohIhGvc9K2pEp9d2c-1682952056-0-gaNycGzNMDs",
            cFPWv: 'g',
            cTTimeMs: '1000',
            cMTimeMs: '0',
            cTplV: 5,
            cTplB: 'cf',
            cK: "",
            cRq: {
                ru: 'aHR0cHM6Ly9hcGktZ29lcmxpLmV0aGVyc2Nhbi5pby9hcGkv',
                ra: 'Tk9fVUE=',
                rm: 'UE9TVA==',
                d: 'HyOwClHdzKb0QWhXxeflQnZhLi9iZ7fHD9C+JBVClbvYfmbSMr0eTICiMY/XSloNj3BuVgnWI+iu9KVE7ISpl/t5XVS9v+jUUDQib5iMmaaTHcfWA/+cod0Jr95xHjnrv35LHfPkoZPgjKL2zuAlSahaeIRqYat5vw+1SQ9hoApJoTqZ77JRRqrrFodhQdFYRQPT2pl5PrbMb6YrksC26y6IDqUF/3pkVm+A0Qfs0u4Qaxltqm2RvbO7Ibx0MoYHxZyeZhTEa52lwtRNoDvN3vqJWYHlA3A0x8cOQ2YPE1YiuGtF/fjrqTSw9GOXasLgU0UTZXp2xY674bry0b/Cirzb3ElR612wVlOvyo7cfwsljp1lZf5I6OmhQXw/3lTG8lfcsyTYRk7Cvs0vigejlt6VsQXhYhgpXf0OwpPNIsEyjNg4X+7F6AA6zqPwfcSpHIdvmkOc6G+7rH49Ag2L0g9wWGpVGTmWvW9s+Zb/eDogqy3pJrkIVnHBi73V6r7BKMl/LgJoIqolCUlg0DoqdZRIju9ijDlPQYYaYKycLVT033HyzJdDWrJDGKwokRkGIwbZ7+P/eqDpXoUBrr3YzcKfUL5P6aMKrPg66gYrr99etaJ+xnoZ/GmS2/NjqwFZ',
                t: 'MTY4Mjk1MjA1Ni43MjkwMDA=',
                m: '43Q+P85nrusAlMAenGBEOgWlw5AZs3ny6hjIRkI0ew0=',
                i1: 'LNiT0NzLFeZ2oBCfHihVjQ==',
                i2: 'asxnmYxvJS85O7q/1ITnUQ==',
                zh: 'qb4aFuGlbJn/rUOkKXjUqElKDKE10jDqu5PE014OTwk=',
                uh: 'DV4j3Tmrbi5Rs1q3ahwVS6SgbPbI7np5884QO1u1Cgg=',
                hh: 'Ax949TKiHbaXasTISC7ryL1/i3VsF1So3LziNEbpSQM=',
            }
        };
        var trkjs = document.createElement('img');
        trkjs.setAttribute('src', '/cdn-cgi/images/trace/captcha/js/transparent.gif?ray=7c08c85288277fb7');
        trkjs.setAttribute('alt', '');
        trkjs.setAttribute('style', 'display: none');
        document.body.appendChild(trkjs);
        var cpo = document.createElement('script');
        cpo.src = '/cdn-cgi/challenge-platform/h/g/orchestrate/captcha/v1?ray=7c08c85288277fb7';
        window._cf_chl_opt.cOgUHash = location.hash === '' && location.href.indexOf('#') !== -1 ? '#' : location.hash;
        window._cf_chl_opt.cOgUQuery = location.search === '' && location.href.slice(0, location.href.length - window._cf_chl_opt.cOgUHash.length).indexOf('?') !== -1 ? '?' : location.search;
        if (window.history && window.history.replaceState) {
            var ogU = location.pathname + window._cf_chl_opt.cOgUQuery + window._cf_chl_opt.cOgUHash;
            history.replaceState(null, null, "\\/api\\/?__cf_chl_rt_tk=Q7F3hkX.VOk8KSj37RUZQAEw34ohIhGvc9K2pEp9d2c-1682952056-0-gaNycGzNMDs" + window._cf_chl_opt.cOgUHash);
            cpo.onload = function() {
                history.replaceState(null, null, ogU);
            };
        }
        document.getElementsByTagName('head')[0].appendChild(cpo);
    }());
</script>


</body>

</html>

@mpeyfuss
Copy link
Author

mpeyfuss commented May 2, 2023

Ah, unfortunately still getting the same response even with the latest build.

@Pythonberg1997
Copy link

Same issue

@ratankaliani
Copy link

ratankaliani commented Jun 9, 2023

Getting the same issue, any idea what it could be?

Is this a goerli specific issue?

@mpeyfuss
Copy link
Author

mpeyfuss commented Jun 9, 2023

Latest comms from the Etherscan API team made it seem like this is an issue they are aware of and their suggested User Agent fix doesn't quite work from my testing. No response back about that. Seems to also be affecting Aribtrum Goerli (which uses etherscan).

I have yet to try on mainnet fully but my initial testing went a lot better on mainnet so it seems restricted to goerli at least.

@kuncle
Copy link

kuncle commented Jun 13, 2023

I have been suspecting that there was an issue with my script or with my local environment until I spent nearly an hour and finally discovered this thread...

Latest comms from the Etherscan API team made it seem like this is an issue they are aware of and their suggested User Agent fix doesn't quite work from my testing. No response back about that. Seems to also be affecting Aribtrum Goerli (which uses etherscan).

I have yet to try on mainnet fully but my initial testing went a lot better on mainnet so it seems restricted to goerli at least.

I have been suspecting that there was an issue with my script or with my local environment until I spent nearly an hour and finally discovered this comment...

@ratankaliani
Copy link

Looks like it's broken on Gnosis as well - @mpeyfuss did you find a fix for Arbitrum Goerli?

@ratankaliani
Copy link

@mattsse Would it be possible to take a look at this? Contract verification via the CLI for chains outside of mainnet seems to be broken

@Evalir
Copy link
Member

Evalir commented Jun 13, 2023

Hey @ratankaliani — we have a fix for contract verification on ethers-rs, we just need to upstream. It'll be done in the upcoming days

@kuncle
Copy link

kuncle commented Jun 15, 2023

Hey @ratankaliani — we have a fix for contract verification on ethers-rs, we just need to upstream. It'll be done in the upcoming days

See you again. lol. any update on this issue?

@mpeyfuss
Copy link
Author

@Evalir can you link the upstream fix on ethers-rs so we can monitor and test?

@Evalir
Copy link
Member

Evalir commented Jun 20, 2023

Hmmm these fixes might have not circumvented this particular issue @mpeyfuss, but we do have the user agent string in place. We'll look into this to ensure we're not overlooking anything, even though it's particularly hard to test if this only happens from a remote instance—i cannot reproduce locally.

Is everyone who has commented on this issue running forge on a remote instance? Or is someone getting this locally? If so, it might be a different issue.

@ratankaliani
Copy link

Issue has resolved for me - I was able to deploy & verify contracts on Optimism Goerli earlier today.

@mpeyfuss
Copy link
Author

@Evalir yeah totally understood. It seems like the user agent string has not worked and I suspect this has to do with Cloudflare blocking the AWS IP range or something to that extent. The issue still persists with the latest build.

Definitely hard to test locally 😂

Is the foundry cloud build on ubuntu working with verification? Can't remember if contract verification to etherscan is tested in the github action off the top of my head.

@Evalir
Copy link
Member

Evalir commented Jun 21, 2023

Ah I think it was tested on the live tests, but we recently removed them, so not anymore :(

@ratankaliani
Copy link

@mpeyfuss Ah, working with Goerli today & yeah seems like Cloudflare is still blocking it even with the latest build.

@mpeyfuss
Copy link
Author

I can confirm this issue does not exist for mainnet eth, only testnets it seems. Not sure there is much we can do but we should keep this open until the Etherscan team gives an update.

@pedrobergamini
Copy link

Same issue here

@mpeyfuss
Copy link
Author

@Evalir just wanted to bump this thread again

@ShackAmeri
Copy link

Same error with forge script broadcast on Sepolia

@chen4903
Copy link

chen4903 commented Oct 9, 2023

the same here in goerli:

   [⠰] Compiling...
   [⠔] Compiling 3 files with 0.8.20
   [⠑] Solc 0.8.20 finished in 2.13s
   Compiler run successful with warnings:
   Warning (2072): Unused local variable.
     --> script/deploy.sol:11:9:
      |
   11 |         Core c = new Core();
      |         ^^^^^^
   
   Deployer: 0xd3E65149C212902749D49011B6ab24bba30D97c6
   Deployed to: 0xB43573c1d6E5A6c42A4FC1240Aa6d4D7bC5120C5
   Transaction hash: 0x81d80351e5f477c84bbef6143958e4d363e406405c2bfedd817543364c68d6f3
   Starting contract verification...
   Waiting for etherscan to detect contract deployment...
   Start verifying contract `0xb43573c1d6e5a6c42a4fc1240aa6d4d7bc5120c5` deployed on goerli
   2023-10-09T07:58:57.426391Z ERROR etherscan: Failed to deserialize response: expected value at line 1 column 1 res="              <!DOCTYPE html><html lang=\"en-US\"

@mattsse
Copy link
Member

mattsse commented Oct 9, 2023

2023-10-09T07:58:57.426391Z ERROR etherscan: Failed to deserialize response: expected value at line 1 column 1 res=" <html lang="en-US"

looks like the etherscan API response with an error or cloudflare page...

do you have the full response?

@chen4903
Copy link

chen4903 commented Oct 9, 2023

This is the full response:

[⠰] Compiling...
[⠒] Compiling 3 files with 0.8.20
[⠘] Solc 0.8.20 finished in 2.28s
Compiler run successful with warnings:
Warning (2072): Unused local variable.
--> script/deploy.sol:11:9:
|
11 |         Core c = new Core();
|         ^^^^^^

Deployer: 0xd3E65149C212902749D49011B6ab24bba30D97c6
Deployed to: 0x9C05899ed01E57Dad8BEca7e3137AD04BBf5f2B2
Transaction hash: 0x5bc0d34d2b1827d52e30cc8d1663a366e6d516ceffbe6ca65f54c705ae66150f
Starting contract verification...
Waiting for etherscan to detect contract deployment...
Start verifying contract `0x9c05899ed01e57dad8beca7e3137ad04bbf5f2b2` deployed on goerli
2023-10-09T08:12:46.795512Z ERROR etherscan: Failed to deserialize response: expected value at line 1 column 1 res="

<!DOCTYPE html>
<html lang="\&quot;en-US\&quot;">
<head>
<title>Just a moment...</title>
<meta http-equiv="\&quot;Content-Type\&quot;" content="\&quot;text/html;" charset="UTF-8\&quot;" />
<meta http-equiv="\&quot;X-UA-Compatible\&quot;" content="\&quot;IE=Edge\&quot;" />
<meta name="\&quot;robots\&quot;" content="\&quot;noindex,nofollow\&quot;" />
<meta name="\&quot;viewport\&quot;" content="\&quot;width=device-width,initial-scale=1\&quot;" />
<link href="\&quot;/cdn-cgi/styles/challenges.css\&quot;" rel="\&quot;stylesheet\&quot;" />
</head>
<body class="\&quot;no-js\&quot;">
<div class="\&quot;main-wrapper\&quot;" role="\&quot;main\&quot;">
<div class="\&quot;main-content\&quot;">
    <noscript>
    <div id="\&quot;challenge-error-title\&quot;">
    <div class="\&quot;h2\&quot;">
    <span class="\&quot;icon-wrapper\&quot;">
        <div class="\&quot;heading-icon" warning-icon\"=""></div></span>
    <span id="\&quot;challenge-error-text\&quot;">Enable JavaScript and cookies to continue</span>
    </div>
    </div>
    </noscript>
</div>
</div>
<script>(function(){window._cf_chl_opt={cvId: '2',cZone: \"api-goerli.etherscan.io\",cType: 'interactive',cNounce: '5086',cRay: '813529177d3bc09b',cHash: 'c548661e7180c05',cUPMDTk: \"\\/api\\/?apikey=CQWS36SV2WKNHU1RH7XY6QSWQK9MKJ54CY&module=contract&action=getabi&address=0x9c05899ed01e57dad8beca7e3137ad04bbf5f2b2&__cf_chl_tk=moDC8fLFgM9LR9i6AXJTnSiiNVrTWmrcW.qiQdQ05x0-1696839166-0-gaNycGzNCtA\",cFPWv: 'g',cTTimeMs: '1000',cMTimeMs: '0',cTplV: 5,cTplB: 'cf',cK: \"visitor-time\",fa: \"\\/api\\/?apikey=CQWS36SV2WKNHU1RH7XY6QSWQK9MKJ54CY&module=contract&action=getabi&address=0x9c05899ed01e57dad8beca7e3137ad04bbf5f2b2&__cf_chl_f_tk=moDC8fLFgM9LR9i6AXJTnSiiNVrTWmrcW.qiQdQ05x0-1696839166-0-gaNycGzNCtA\",md: \"bo20uCQgmDtvj7Ns.nEadYkobArkRYlH3Jof77jKKfs-1696839166-0-AUrlrDvdVzpROmpOcrQEir1Ts4Unha7FoPLyVc1n3V74R9MBSZmc3lE12vohpP2zcVU6y4Oi9Dp7DDkscd3oEoyOduA_s5zBFdfWKg2_Mk6EOzKmW2I1RRW50kbPnWoBFiJFNZaCoJh5jVI_zAjvPwbxz_TfggbKUHDYmSQ7E4ndzQvVNqYtWrnnoVDi48Q2DG9UjwHK3XOXEm8ZL0fL4BfI-9SJKCHcjXcTMgKWiHnej9VdIun73L-2rqCrdhPZMVletlWc8QP6mgfa-vz7KO9YcXzh3a-Pf8oGAHTepk8mS02Qz4hulrMsM1vpjkec3Fmt2TsUpV3oY4cdn-cgi/challenge-platform/h/g/orchestrate/chl_page/v1?ray=813529177d3bc09b';window._cf_chl_opt.cOgUHash = location.hash === '' && location.href.indexOf('#') !== -1 ? '#' : location.hash;window._cf_chl_opt.cOgUQuery = location.search === '' && location.href.slice(0, location.href.length - window._cf_chl_opt.cOgUHash.length).indexOf('?') !== -1 ? '?' : location.search;if (window.history && window.history.replaceState) {var ogU = location.pathname + window._cf_chl_opt.cOgUQuery + window._cf_chl_opt.cOgUHash;history.replaceState(null, null, \"\\/api\\/?apikey=CQWS36SV2WKNHU1RH7XY6QSWQK9MKJ54CY&module=contract&action=getabi&address=0x9c05899ed01e57dad8beca7e3137ad04bbf5f2b2&__cf_chl_rt_tk=moDC8fLFgM9LR9i6AXJTnSiiNVrTWmrcW.qiQdQ05x0-1696839166-0-gaNycGzNCtA\" + window._cf_chl_opt.cOgUHash);cpo.onload = function() {history.replaceState(null, null, ogU);}}document.getElementsByTagName('head')[0].appendChild(cpo);}());</script>
</body>
</html>
"
Error:
expected value at line 1 column 1

@ratankaliani
Copy link

ratankaliani commented Oct 10, 2023

Running into this same issue today as well on Goerli even after running foundryup:

Failed to deserialize response: expected value at line 1 column 1 res="<!DOCTYPE html><html lang=\"en-US\"><head><title>Just a moment...</title><meta http-equiv=\"Content-Type\" content=\"text/html; charset=UTF-8\"><meta http-equiv=\"X-UA-Compatible\" content=\"IE=Edge\"><meta name=\"robots\" content=\"noindex,nofollow\"><meta name=\"viewport\" content=\"width=device-width,initial-scale=1\"><link href=\"/cdn-cgi/styles/challenges.css\" rel=\"stylesheet\"></head><body class=\"no-js\"><div class=\"main-wrapper\" role=\"main\"><div class=\"main-content\"><noscript><div id=\"challenge-error-title\"><div class=\"h2\"><span class=\"icon-wrapper\"><div class=\"heading-icon warning-icon\"></div></span><span id=\"challenge-error-text\">Enable JavaScript and cookies to continue</span></div></div></noscript></div></div><script>(function(){window._cf_chl_opt={cvId: '2',cZone: \"api-goerli.etherscan.io\",cType: 'interactive',cNounce: '33797',cRay: '813aa27faa900822',cHash: '49d497c7c50c4a0',cUPMDTk: \"\\/api\\/

Forge scripts on Goerli are failing too.

@ratankaliani
Copy link

@mattsse Is there a release of foundry with the above PR merged?

@mattsse
Copy link
Member

mattsse commented Oct 10, 2023

next nightly

I also think we should consider calling a random etherscan API request first before deploying so we can check if there are issues

wdyt?

@ratankaliani
Copy link

ratankaliani commented Oct 10, 2023

Yeah I agree, would be good to add to these tests or a similar nightly suite to detect when there are Cloudflare errors.

tsite added a commit to tsite/foundry that referenced this issue Oct 22, 2023
Etherscan verification for non-mainnet chains requires a question
mark at the end of the verifier url in order to prevent a forward slash
from being added to the url which trips a cloudflare rule on requests
from ec2 boxes.

Verification on both Goerli and Sepolia both fails without this
character present.

Hardhat-verify does not add the extra forward slash to the verifier url
and has no issues verifying contracts on etherscan.

Fixes foundry-rs#4865, foundry-rs#5251, foundry-rs#5741
mattsse pushed a commit that referenced this issue Oct 22, 2023
* fix `forge verify-contract` etherscan cloudflare bug

Etherscan verification for non-mainnet chains requires a question
mark at the end of the verifier url in order to prevent a forward slash
from being added to the url which trips a cloudflare rule on requests
from ec2 boxes.

Verification on both Goerli and Sepolia both fails without this
character present.

Hardhat-verify does not add the extra forward slash to the verifier url
and has no issues verifying contracts on etherscan.

Fixes #4865, #5251, #5741

* run cargo +nightly fmt -- --check

* check if the api url already ends with ?

Also apply the fix to --verifier-url urls
@GuiFlam
Copy link

GuiFlam commented Mar 15, 2024

I am having a similar issue with Sepolia. Does anyone have a fix for this?

@magnetto90
Copy link

I am having a similar issue with Sepolia. Does anyone have a fix for this?

If you are using Infura change to Alchemy, it solved my issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
T-bug Type: bug
Projects
No open projects
Archived in project
Development

Successfully merging a pull request may close this issue.