New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Crypto Hacking Contest! Implement optimized sha256() for supported CPUs #23

Open
avive opened this Issue Dec 10, 2018 · 12 comments

Comments

Projects
None yet
2 participants
@avive
Copy link
Member

avive commented Dec 10, 2018

Background

  • Intel accelerated crypto instructions define sha256 extension instructions - they have not been widely implemented on the last few generations of Intel CPUs
  • AMD ships both server and desktop CPUs with the sha extension instructions (EPYC and RYZEN)
  • Servers with EPYC cpus are now available on AWS, and are good candidates for running Spacemesh production POET for our MVP1 testnet milestone
  • Some benchmarks show ~x3-x5 pref over sha256 software implementations
  • There are already several open source libraries that use the SHA instructions to compute SHA256 for a small input buffer

Requirements

  1. Your dev box CPU is AMD RYZEN or AMD EPYC and it supports the sha256 instructions ( cpuid=sha256)
  2. You have knowledge of how to use assembly code sections in a go project. See https://blog.sgmansfield.com/2017/04/a-foray-into-go-assembly-programming/
  3. You are familiar with the concept of crypto hashes
  4. You are an intermediate or experienced go developer

The Task

  1. Implement and test sha256() using Intel sha256 instructions using assembly code published by open source libraries. Please try to use assembly directly from GO without resorting to C code. Your implemented type should implement the hash.Hash interface
  2. The hardware optimized sha256 function should be automatically used when the cpu support SHA (cpuid=SHA) instead of the current sha256 function
  3. You must write tests showing 100% binary compatibility of your implemented function with the existing sha256() function on a variety of hard-coded test vectors and random binary data. These tests must pass for this task to be complete
  4. Your solution must be as performant as possible. You may start by following the solution outlined here: randombit/botan@039b714 but further optimizations may be possible
  5. Update the POET code to test for cpuid=sha256 on app startup and use the new sha256 you wrote instead of the old one in this case. e.g. Your optimized hasher should be a drop-in replacement to the currently used one on supported systems
  6. Write some simple go benchmarks to compare your new sha256 function with the one currently used in the project (AVX2 instruction set optimized)
  7. Submit a PR for review in this repo when your code is ready for review and testing by our team!

Contest Terms

  • We are going to accept the first 3 developers who are interested to participate in this contest that introduce themselves on our Gitter dev talk channel
  • Each correct submission from the 3 accepted developers which passes the binary compatibility test will be tipped using a Gitcoin tip of 5 Eth
  • If a developer doesn't submit code in 2 weeks then we may accept other developers instead of him or her.
  • The fastest correct submission is going to be awarded additional 5 eth, a special custom Spacemesh honorary badge and some additional schwag
  • Good luck!

Additional resources

Contest Terms

By participating in the contest, you agree to these terms and conditions.

This contest is an experimental and discretionary rewards program to encourage and reward those who are helping to improve the platform. You should know that we can cancel or amend the terms of the program at any time, and we have the sole discretion to determine whether or not the criteria to qualifications for compensation hereunder, as described in the contest terms, are met in a satisfactory manner. When compensated in Ethereum cryptocoins, the amount will be determined according to the $ value of the cryptocoin on the date of payment.

We are unable to issue rewards to individuals who are on any US or Israeli sanctions lists, or who are in countries on any US or Israeli sanctions lists.

You are responsible for any tax implications depending on your country of residency and citizenship. There may be additional restrictions on your ability to receive the award depending upon your local law.

Your participation in the contest must not violate any law, or disrupt, compromise or make unauthorized use of any data or know-how that is not your own.

By participating in the program, you hereby: (i) represent that you are aware that the Spacemesh open source project is an MIT licensed free open source software and that your contribution is consistent with this fact, and (ii) grant us, our subsidiaries, affiliates and customers a non-exclusive, perpetual, irrevocable, worldwide, royalty-free, transferable, sub licensable (through multiple tiers) license to use, reproduce, adapt, modify, publish, distribute, publicly perform, create derivative work from, make, use, sell, offer for sale and import for any purpose any materials submitted by you in connection with any task performed in the framework of the contest.

@avive avive changed the title Implement optimized sha256() for supported CPUs Crypto Hacking Contest! Implement optimized sha256() for supported CPUs Dec 10, 2018

@gitcoinbot

This comment has been minimized.

Copy link

gitcoinbot commented Dec 10, 2018

Issue Status: 1. Open 2. Started 3. Submitted 4. Done


This issue now has a funding of 14.998 ETH (1341.01 USD @ $89.41/ETH) attached to it as part of the Spacemesh - https://spacemesh.io fund.

@avive

This comment has been minimized.

Copy link
Member

avive commented Dec 13, 2018

Don't have an AMD Zen CPU?
We will make available for you an AMD EPYC cpu dev server on the cloud to hack away.

@gitcoinbot

This comment has been minimized.

Copy link

gitcoinbot commented Dec 14, 2018

Issue Status: 1. Open 2. Started 3. Submitted 4. Done


Work has been started.

These users each claimed they can complete the work by 1 month, 3 weeks from now.
Please review their action plans below:

1) facundomedica has been approved to start work.

  1. Look at the available implementations
  2. Make an assembly version
  3. Replace the block function inside sha256
  4. Make tests

Learn more on the Gitcoin Issue Details page.

2) svenski123 has been approved to start work.

Hi, I'd like to work on this low level optimisation project over the holiday period.
I've expressed interest on your Gitter channel, happy to chat further though the holidays are nearly upon.
The task requirements and resource are quite from the bounty description and so can proceed independently.
It would be useful to have access to the AMD server on AWS mentioned in the bounty description.

Learn more on the Gitcoin Issue Details page.

3) coderrick has been approved to start work.

I will follow the challenge guidelines to implement an efficient solution for the AMD EPYC architecture. I will provide weekly updates to the submission which include (but is not limited to) the source, run instructions, and benchmark logs.

Learn more on the Gitcoin Issue Details page.

4) mortdeus has been approved to start work.

extend compiler and assembler to avoid hacky workaround

Learn more on the Gitcoin Issue Details page.

@avive

This comment has been minimized.

Copy link
Member

avive commented Dec 15, 2018

Here are some more concrete instruction regarding how to participate and benchmark:

  1. Fork the POET go project
  2. Add your new hash function implementation code as a package
  3. Add a new BenchmarkSha256Ex(b *testing.B) to hash_test.go that uses your new hash function instead of sha256-simd()
  4. Write a method that shows the improvement % of BenchmarkSha256Ex() over BenchmarkSha256Ex()
  5. Submit a PR for review - we will run it on supported hardware
@avive

This comment has been minimized.

Copy link
Member

avive commented Dec 16, 2018

Contest update. First entry achieves x2 improvement over sha256-simd running on AVX2 enabled cpu. Our test reference hardware is up and running with ZEN EPYC cpu on AWS. With the first entry, on this server, our POET code is performing 6.666 mhps for serial sha256() ops. That's about 15 seconds for 100M serial hashes.

@avive

This comment has been minimized.

Copy link
Member

avive commented Dec 17, 2018

Contest update: The contest is going to be open for submissions until Jan 15th 2019 to give more time to developers who may be offline due to the winter holiday season currently celebrated across the world.

@avive

This comment has been minimized.

Copy link
Member

avive commented Dec 17, 2018

@avive

This comment has been minimized.

Copy link
Member

avive commented Jan 6, 2019

Contest update - we have merged an amazing entry with very nice performance improvements. See spacemeshos/sha256-simd#2

@gitcoinbot

This comment has been minimized.

Copy link

gitcoinbot commented Jan 7, 2019

Issue Status: 1. Open 2. Started 3. Submitted 4. Done


Work for 14.998 ETH (2240.89 USD @ $149.41/ETH) has been submitted by:

  1. @svenski123

@avive please take a look at the submitted work:


@gitcoinbot

This comment has been minimized.

Copy link

gitcoinbot commented Jan 15, 2019

⚡️ A tip worth 5.00000 ETH (621.32 USD @ $124.26/ETH) has been granted to @facundomedica for this issue from @avive. ⚡️

Nice work @facundomedica! Your tip has automatically been deposited in the ETH address we have on file.

@gitcoinbot

This comment has been minimized.

Copy link

gitcoinbot commented Jan 15, 2019

⚡️ A tip worth 5.00000 ETH (621.32 USD @ $124.26/ETH) has been granted to @svenski123 for this issue from @avive. ⚡️

Nice work @svenski123! Your tip has automatically been deposited in the ETH address we have on file.

@avive

This comment has been minimized.

Copy link
Member

avive commented Jan 16, 2019

Thank you @svenski123 and @facundomedica for your submissions. As you were the first 2 devs who submitted entries with improvements - per the contest terms, a tip of 5 ETH was sent to each of you yesterday. We are going to announce first place winner(s) on 1/20 in few days and distribute the tip on that day. It is also very nice to see that minio lib merged @svenski123 contribution that resulted from this contest so all library users can benefit from the improvements.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment