Vercel's "Usage" tab shows much higher bandwidth use than is transferred to the client with Cloudflare

[kristjanmar]

I'm developing a NextJS site on Vercel and I've tested browsing my site myself and tracking the changes in Bandwidth in their "Usage" tab, which appears to update immediately after refreshing.

There is no one else visiting the site since it's just a development site, although I've added it to an actual domain on vercel. The bandwidth usage was zero before my first pageview.

The problem is that the bandwidth shown in the Usage tab is much higher than the bandwidth consumed by my browser according to Chrome Developer Tools. As in, 2-8 times higher.

Here's an example:

1. I have developer tools open on the "Network" tab. Cache is not disabled.
2. I go to a new page that is quite heavy, so the kB transferred goes up by 300KB. Prefetching of links is not enabled.
3. However, the bandwidth in the Usage tab goes up by 1.5MB.

I've already turned off the aggressive prefetching of links and taken steps to reduce my bundle size by 70%. But the bandwidth usage per session and pageview according to Vercel didn't budge.

It's as if there is something other than the traffic sent to the user that is contributing to a large part, or even most of, the bandwidth. I know the serverless functions have something to do with it, but I don't understand how that works.

Has anyone noticed this before and got an explanation, or found some solutions to reduce bandwidth consumed per user?

[fakenickels]

I'm experiencing the same problem. Usage hundreds of times bigger than what we have predicted and have measured by our own.

[kristjanmar]

Thank you, glad to hear that it isn't just me. I created an issue on this 15 days ago but I still haven't received a response: vercel/vercel#6468

[fakenickels]

We figured it out! It was actually a health check service that was causing the problem, we disabled it and the usage went back to normal

[kristjanmar]

I just enabled my site on Cloudflare 24 hours ago. They report 12GB data transferred in the past 24 hours, thereof oly 7GB from the origin -- which is my Vercel hosted site.

However, Vercel is reporting 35 GB of bandwidth used, which is 5 times more than the bandwidth transferred from Vercel's servers! What is going on here?

Is Vercel then going to force me off of the Pro plan unless if I fork up thousands of dollars a month because of a fraudulent bandwidth metric?

[kristjanmar]

Last 24 hours, Cloudflare reports 2.57GB served by the Origin (Vercel). At the same time, Vercel reports 15.3GB. Vercel's number is 6 times higher.

[abriginets]

Having the same issue with my website. It seems that Vercel counts bandwidth usage from resource size and not the amount of data being sent over the network. Here's how I found out:

• My page weight is 159 kilobytes, but Chrome shows that there's only 25 kilobytes of data sent over the network
• I've set up a health bot that makes simple HTTP/1.1 request from ~20 locations around the globe every 5 minutes.
• That means bot makes 240 requests per hour (60 minutes / 5 minutes = 12 times x 20 requests = 240 requests per hour)
• I've left my bot working for ~30 hours (240 requests x 30 hours = 7200 requests x 25 Kb = 180 000 kilobytes = 0,18 gigabytes

But my bandwidth meter shows 1.1 gigabyte:

Then I did my math again but with "resource size" this time:

• 240 requests x 30 hours = 7200 requests x 159 Kb = 1 144 800 kilobytes = 1.14 gigabytes.
• Eureka!

@mcsdevv @paulogdm @leerob @rauchg is that an expected behavior or a bug? For me this is very unfair to try as hard as possible to minimize bundle size and optimize as much things as possible and then get billed 6,5 times more for something I didn't even use. Especially considering the fact that every 100 GB above the limit costs 55$.

UPD: my production website that is hosted on VPS and has way more traffic (all sorts of crawlers and much higher rps from real users) consumes 1.01 gigabytes per day though it is not as much well optimized as NextJS could give you. This is nowhere near my bandwidth would be on Vercel these days.

[paulogdm]

Thank you very much for sharing the details with us. @mcsdevv have shared your post internally and we will reply soon.
The investigation is in progress.

[paulogdm]

@abriginets Can you give us the domain or deployment where you ran the tests? If you don't want to share this publicly, you can email me directly at paulo@vercel.com

[abriginets]

It's no problem, https://new.flaut.ru

[Timer]

Hello! We've investigated this report and looked at historical traffic for your website.

It appears that Check-Host (the uptime service you're using) does not request the asset with compression, so the accrued bandwidth will be the uncompressed size, which is what you're observing.

We setup an isolated test and requested the assets via Check-Host, and then again in Chrome, and can show the comparison clearly.

First, the records Vercel uses for billing (simplified to subset of columns):

And the Chrome Network tab for the "Mozilla ..." request seen in above table:

[abriginets]

Thank you for your fast reply! Yeah, I see now that my research was not representative since 100% of the traffic at my website was generated by client which doesn't support compression. I'll contact them and see if it's possible to enable gzip/brotli at their clients to save my bandwidth.

But now I'm concerned about the security of my website. How can I protect it from hackers who would make an attemp to exploit billing system and drain my bandwidth quota by sending let's say 60 reqs / minute for 30 days without compression headers? By my calculations it would consume around 400 gigabytes of bandwidth and it won't look like a DDoS attack for protection systems thus would not be mitigated.

[kristjanmar]

My site's usage tab is still reporting about 5-6X higher bandwidth than is transferred according to both Chrome Dev tools and CloudFlare. This is the URL: https://stockanalysis.com/

[joggienl]

@leerob thanks for the clarification. For me that actually helps.

I do think that the technical documentation on itself, on how isr works, is great and clear.

What imo needs to be improved is that isr can have a huge impact on the used bandwidth and function usage. The information is kind of in there but it is to implicit imo. I am having a hard time explaining higher costs this way to my clients.

I use it for an e-commerce project (on the pro plan) but did not anticipate to get up to 6TB bandwidth usage by using isr. I used 5 minutes revalidation time and did not disable preferching. So high bandwidth usage is kind of default that way.

For e-commerce you want to use low revalidation times, especially when products are involved. For example to make sure products are still in stock (it is not a great experience to have a product in stock for like 5 minutes because isr is serving it as in stock 😅).

Another thing that can help us understand is making the usage on the dashboard more insightful. It currently just shows “bandwidth usage” but i cannot relate the traffic to any destination. Same goes for function invocations. And with a price point of $55,- per 100GB, which is really costly, that adds up quickly.

Something not clear to me is, if isr cache is invalidated it wil rebuild. Are all nodes rebuilding the page or is the cache simply updated among the nodes? Either way, if i do understand correctly, the usage counted towards us is the traffic and function usage of all the nodes independently, which could mean 19x (based on 19 edge nodes) for a single page revalidation? I do feel like this internal traffic should only be counted once, we have no real influence on that process. Also, if nodes are added to the edge network, usage will immediately increase that way.. (which could potentially have high cost effects)

[leerob]

Good feedback - we should more clearly explain the relationship between ISR and bandwidth consumed. We will look into this, thank you.

if isr cache is invalidated it wil rebuild. Are all nodes rebuilding the page or is the cache simply updated among the nodes?

When a page is generated with ISR, it's actually persisted to static storage and pushed to every edge.

which could mean 19x (based on 19 edge nodes) for a single page revalidation?

The page is only being regenerated once per revalidation period, not 19 times 👍 You can verify this by putting a revalidation time of a minute. Even if 1,000,000 requests come in, only one revalidation (which triggers a function call to regenerate the page) happens.

if nodes are added to the edge network, usage will immediately increase that way

Usage is not tracked differently based on the edge location. The edge is choosing whichever location is closest the user, which will have the lowest latency for them.

[joggienl]

Good feedback - we should more clearly explain the relationship between ISR and bandwidth consumed. We will look into this, thank you.

awesome, thanks!

I mentioned 19x in my earlier response, if the revalidation only happens once per revalidation period, it also makes sense that the generated page is pushed via static storage. So thanks for that explainer!

Usage is not tracked differently based on the edge location. The edge is choosing whichever location is closest the user, which will have the lowest latency for them.

Also helpful, thanks!

Is the bandwidth that is caused by keeping all generated pages up to date across the edge (after revalidation is triggered and the page gets a rebuild) counted towards the "allowed" bandwidth usage?

A bit on a side note here, I've been tinkering with disabling prefetching on the tag a bit. I think it indeed is good advice to be cautious about prefetching and disable it for certain elements. A good remark here though is that by disabling prefetch with <Link prefetch={false} />, data is still being prefetched if you hover over a link (which is well documented).

[leerob]

Wanted to follow up here and mention it should now be much easier to see what files, routes, ISR functions, etc are incurring usage.

https://vercel.com/changelog/enhanced-observability-on-the-usage-tab

[joggienl]

That is actually an awesome feature. Using it for a while already (since released in July I believe). Very helpful. Thanks for sharing again :-)

[joggienl]

@kristjanmar are you using ISR?

[kristjanmar]

Yes, I am using ISR on most pages of my site.

[joggienl]

Me too, we had ISR and invalidated once per 5 minutes. We saw lot of traffic to the backend where next would get data from to build. Changing to client side fetching (parts) gave a massive drop in bandwidth usage.

I a way I got the feeling that perhaps every node on the edge (where stuff is being cached) is running builds and storing data. I have no data to back that feeling up though.

For reference, the difference is from up to 260GB per day (!!) to below 80GB a day. Which still is way to high imo but a bit better.

[abriginets]

I got the feeling that perhaps every node on the edge (where stuff is being cached) is running builds and storing data

It's good to know that someone mentioned that too. I've created a discussion thread vercel/vercel#6976 literally yesterday about something similar. The only difference there is that I'm using SSR and I'm seeing very long responses across all the regions except the one I'm near to.

[leerob]

Thank you for providing this information to help us further investigate the issue!

Our infrastructure team has used the information you provided to determine an improvement (working with the Cloudflare team). We have made a change to improve how we handle compression between Vercel and Cloudflare.

Our infrastructure now has more granular control over compression, and thus can decrease bandwidth between Vercel and Cloudflare.

Starting today (Nov 24th, 2021), you should start to see lower bandwidth on your Usage page from these changes.

[relferreira]

Thank you, but as @fprodo said, unfortunately, it didn't help

[CyberCyclone]

Thanks for the quick answer, but sadly, it does not help... what we need to know is what files are consumig this much bandwith: yesterday we migrated all of our images to a CDN, but it didn´t help, so we are still unable to know what is causing this.

Any chance to have some kind of log showing wich files are causing the most traffic in a project?

This is my problem. The dashboard shows high bandwidth usage with no evidence of its actual usage.

[samcx]

@CyberCyclone Can you email me (sam@vercel.com) your Vercel email so I can take a look?

[JuniorHoms]

Thanks for the response, but unfortunately it doesn't help... what we need to know is which files are consuming so much bandwidth: it's been less than a month since our site was grassed but over 400GB were consumed.
We have tried everything but still can't figure out what is causing this.
we want a solution , please

[leerob]

Please look at the top paths on the usage dashboard.

[luke-j]

I'm seeing this issue as well. With an almost exact doubling of bandwidth between what CloudFlare is reporting and what Vercel is reporting. eg. CloudFlare reports ~50GB where Vercel reports ~100GB.

I'm not using ISR and I'm not using serverless functions. A significant number of pages are using rewrites to paths within the same application, I wonder if Vercel is treating bandwidth passed through rewrites twice?

[samcx]

@luke-j Is it possible to send me your username on Vercel so I can take a look?

[luke-j]

Hey @samsisle My Vercel username is the same as my GitHub username. I sent you an email with some additional details I’m not able to share publicly.

[dsacramone]

Hey @samsisle My Vercel username is the same as my GitHub username. I sent you an email with some additional details I’m not able to share publicly.

Was this sorted out?

[samcx]

@dsacramone This was indeed sorted out. If you still have questions about usage and have information you wish to not share publicly, I recommend reaching out in a case → https://vercel.com/support/request.