Skip to content

Conversation

@arodland
Copy link
Contributor

ioutil.ReadAll uses a bytes.Buffer, which allocates memory by doubling.
Since we know exactly how muh data we expect to get, we can allocate it
in advance. This reduces the total amount of allocation, and ensures
that the slice stored in the item won't have excess capacity (which can
affect memory usage if the item is held for a long time, for instance in
a secondary cache).

ioutil.ReadAll uses a bytes.Buffer, which allocates memory by doubling.
Since we know exactly how muh data we expect to get, we can allocate it
in advance. This reduces the total amount of allocation, and ensures
that the slice stored in the item won't have excess capacity (which can
affect memory usage if the item is held for a long time, for instance in
a secondary cache).
@googlebot
Copy link

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed, please reply here (e.g. I signed it!) and we'll verify. Thanks.


  • If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
  • If your company signed a CLA, they designated a Point of Contact who decides which employees are authorized to participate. You may need to contact the Point of Contact for your company and ask to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the project maintainer to go/cla#troubleshoot.
  • In order to pass this check, please resolve this problem and have the pull request author add another comment and the bot will run again.

@arodland
Copy link
Contributor Author

This passes tests; it would be nice to have a benchmark to show the difference in allocs, but none of the existing benchmark tests exercise the network read path, and I wasn't ambitious enough to write one that does. The difference is clearly visible in the heap profile of a real app, though.

@arodland
Copy link
Contributor Author

CLA signed; hopefully the "alternate email" works as it should.

@googlebot
Copy link

CLAs look good, thanks!

@dwbuiten
Copy link

@bradfitz Ping on this, if you have time, of course; I'd love to not have to use an internal fork of this package in order to get highly reduced memory usage.

Sorry for the noise!

@bradfitz bradfitz merged commit f867c99 into bradfitz:master Jul 10, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants