Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

wiki: Would like a page to list community benchmarks #21664

Open
dr2chase opened this issue Aug 28, 2017 · 15 comments

Comments

@dr2chase
Copy link
Contributor

commented Aug 28, 2017

What did you expect to see?

A table listing important benchmarks from the open source community, so that Go compiler and runtime developers can get a better idea of which optimizations matter and which do not, and to avoid performance surprises late in the release cycle. The existing go1 benchmark suite is not that good.

Ideally these benchmarks would have the following properties:

  • they should matter; does it cost money if the benchmarked code runs slowly?
  • they should be go-gettable and not require customized steps for building the benchmark
  • they should run relatively quickly; ideally each individual benchmark iteration runs in less than a second (we'll probably want different sets of benchmarks for runtime and compiler)
  • ideally their timings are not noisy
  • they should run cleanly in a restricted environment, such as a Docker container
  • because only a few benchmarks from each suite can be run, they should not overlap other benchmarks in the list; we don't need a half-dozen benchmarks of the same transcendental functions

Information for each benchmark would include:

  • a short name for the benchmark
  • the path to the benchmark directory
  • a regexp for the benchmark specifying the one or two best benchmarks to run
  • a contact person for questions about the benchmarks

There may be more than one directory in a project containing benchmarks, and that's okay, these can be listed as separate benchmarks.

For example:

  Name = "gonum_blas_native"
  Repo = "github.com/gonum/blas/native"
  Benchmarks = "Benchmark(DasumMediumUnitaryInc|Dnrm2MediumPosInc)"
  Contact = "SomeoneOnTheGoNumTeam@example.com"

and

  Name = "gonum_matrix_mat64"
  Repo = "github.com/gonum/matrix/mat64"
  Benchmarks = "Benchmark(MulWorkspaceDense1000Hundredth|ScaleVec10000Inc20)"
  Contact = "SomeoneOnTheGoNumTeam@example.com"

I've done some work creating a preliminary list of benchmarks I think is likely to be interesting, and it includes code from the following github projects:

   ethereum_bitutil (repo=github.com/ethereum/go-ethereum/common/bitutil)
   ethereum_storage (repo=github.com/ethereum/go-ethereum/swarm/storage)
   ethereum_ethash (repo=github.com/ethereum/go-ethereum/consensus/ethash)
   ethereum_core (repo=github.com/ethereum/go-ethereum/core)
   ethereum_sha3 (repo=github.com/ethereum/go-ethereum/crypto/sha3)
   ethereum_ecies (repo=github.com/ethereum/go-ethereum/crypto/ecies)
   ethereum_corevm (repo=github.com/ethereum/go-ethereum/core/vm)
   ethereum_trie (repo=github.com/ethereum/go-ethereum/trie)
   eolian_dsp (repo=buddin.us/eolian/dsp)
   spexs2 (repo=github.com/egonelbre/spexs2/_benchmark)
   minio (repo=github.com/minio/minio/cmd)
   gonum_blas_native (repo=github.com/gonum/blas/native)
   gonum_lapack_native (repo=github.com/gonum/lapack/native)
   gonum_matrix_mat64 (repo=github.com/gonum/matrix/mat64)
   semver (repo=github.com/Masterminds/semver)
   hugo_helpers (repo=github.com/gohugoio/hugo/helpers)
   hugo_hugolib (repo=github.com/gohugoio/hugo/hugolib)
   k8s_api (repo=k8s.io/kubernetes/pkg/api)
   k8s_schedulercache (repo=k8s.io/kubernetes/plugin/pkg/scheduler/schedulercache)

In the list above, ethereum_whisperv5 does not appear because all the benchmarks were noisy.
I also don't have official contacts for these benchmarks, and I don't know if the benchmarks I selected are representative or not.

What did you see instead?

No such page.

@aclements

@dr2chase dr2chase self-assigned this Aug 28, 2017

@kevinburke

This comment has been minimized.

Copy link
Contributor

commented Aug 28, 2017

A good place to start would probably be the list of "Popular Packages" here, which I think measures how many times a given package is imported... if the packages in that list have benchmarks, we could add them.

https://godoc.org/

@odeke-em

This comment has been minimized.

Copy link
Member

commented Aug 28, 2017

@dr2chase awesome, thank you for starting this! It's great to see this up as a follow up of the team discussions at the Go contributor summit.

So in regards to

In the list above, ethereum_whisperv5 does not appear because all the benchmarks were noisy.
I also don't have official contacts for these benchmarks, and I don't know if the benchmarks I selected are representative or not.

For official contacts of the Ethereum benchmarks, kindly pinging @karalabe @fjl @gluk256: might y'all have some input on @dr2chase's questions?

@kevinburke

This comment has been minimized.

Copy link
Contributor

commented Aug 28, 2017

I'd appreciate at least one benchmark that looks at UUID serialization. There are a few different packages.

@dr2chase

This comment has been minimized.

Copy link
Contributor Author

commented Aug 28, 2017

If you can name an example or two of UUID serialization benchmarks, I'll give them a look.

@kevinburke

This comment has been minimized.

@kevinburke

This comment has been minimized.

Copy link
Contributor

commented Aug 28, 2017

for logging, uber/zap has a series of logging benchmarks for popular logging libraries

https://github.com/uber-go/zap#performance

helpfully compiled here

https://github.com/uber-go/zap/tree/master/benchmarks

@dlsniper

This comment has been minimized.

Copy link
Contributor

commented Aug 28, 2017

@dr2chase while I think this is awesome, and I can see the benefit / need for this, I think that this also needs to be tackled with care as people might misinterpret those projects as being reference projects and we already have a problem with the most frequent question being: what's the fastest X? (mainly, what's the fastest framework?).

@dr2chase

This comment has been minimized.

Copy link
Contributor Author

commented Aug 28, 2017

We could add a disclaimer. Wanting to be sure that a benchmark "matters" will tend to bias us towards popular projects, however.

@karalabe

This comment has been minimized.

Copy link
Contributor

commented Aug 29, 2017

I can help with getting benchmarks out of Ethereum if need be, though it will take a bit of looking around to see what's "meaningful" from Go's perspective and which ones aren't (some of our VM benchmarks are there to aid in pricing decisions, not so much in measuring raw performance).

@karalabe

This comment has been minimized.

Copy link
Contributor

commented Aug 29, 2017

Perhaps another aspect that would be nice to touch on is the performance on different platforms. In our specific use cases, some hashing packages are implemented in assembly for amd64, but not for others. It might be useful to see the discrepancies between architectures too thus.

@dlsniper

This comment has been minimized.

Copy link
Contributor

commented Aug 29, 2017

@dr2chase yes, disclaimer should be good enough imho (hopefully).

For some recommendations of popular packages:

Should I also list applications that contain benchmarks rather than just libraries?

@flanglet

This comment has been minimized.

Copy link

commented Aug 29, 2017

Maybe this data compression package would be useful as well:

https://github.com/flanglet/kanzi/tree/master/go/src/kanzi/benchmark

@dr2chase

This comment has been minimized.

Copy link
Contributor Author

commented Aug 29, 2017

Note that it has to be go-gettable. That is, there should be a one-line command like this:

go get -d -t -v github.com/flanglet/kanzi/tree/master/go/src/kanzi/benchmark

that succeeds. The intent is that this be input to a program that gets benchmarks, compiles them, and runs them in a container.

@kevinburke

This comment has been minimized.

Copy link
Contributor

commented Jan 18, 2018

A go-bindata benchmark would be helpful. How long does it take to e.g. compile JS, CSS, templates, images and SQL queries into a Go package.

https://github.com/jteeuwen/go-bindata
I have a fork here: https://github.com/kevinburke/go-bindata

@dr2chase

This comment has been minimized.

Copy link
Contributor Author

commented Jan 18, 2018

I went and looked, and there are tests, but no benchmarks. Does the proposed benchmark measure

  • the time to generate bindata,
  • the time to compile the generated bindata, or
  • the time to access the bindata?

The last one is the one that seems relevant to application performance.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
7 participants
You can’t perform that action at this time.