Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Polkadot Runtime Environment: Alternative Implementation Grant #12

Open
EdwardAThomson opened this issue Jul 18, 2018 · 16 comments
Open

Polkadot Runtime Environment: Alternative Implementation Grant #12

EdwardAThomson opened this issue Jul 18, 2018 · 16 comments

Comments

@EdwardAThomson
Copy link
Contributor

@EdwardAThomson EdwardAThomson commented Jul 18, 2018

The Web3 Foundation is looking to attract teams for the development of an alternative implementation of Polkadot Runtime Environment (PRE), which is a core component of Polkadot.

The first step in writing a fully-fledged Polkadot implementation is to start by writing an implementation of the PRE. The idea behind the PRE is that it can be used to create standalone blockchains, while also allowing connectivity to the Polkadot network.

The Web3 Foundation contracted Parity Technologies to write the first implementation of Polkadot. Their implementation of the PRE is called ‘Substrate’. This product is currently a PRE but will have additional functionality in the future. For this grant we are concerned only with the PRE. The current Substrate code is sitting in the Polkadot GitHub repo and is implemented in Rust.

For more information please see the following document for further details on how to apply:

Polkadot Runtime Environment: Alternative Implementation Grant

(FYI, the following languages are preferred: Go (above all others), C++, and Java.)

@EdwardAThomson EdwardAThomson changed the title Alternative Substrate Implementation Grant Substrate: Alternative Implementation Grant Aug 3, 2018
@adrianbrink
Copy link
Contributor

@adrianbrink adrianbrink commented Aug 19, 2018

Is there a preferred language for the implementation?

An alternate implementation in Rust or Go would be very useful early on, since a lot of the current blockchain stack is written in one or the other.

Down the road, it would be interesting to get an implementation in JavaScript (if it ever supports compilation to wasm) or Java/C#.

@EdwardAThomson
Copy link
Contributor Author

@EdwardAThomson EdwardAThomson commented Aug 19, 2018

I'm due to release more information tomorrow (a little behind expected timeframe!).

The reference implementation is in Rust (i.e. Parity Tech), plus there is Jaco's JS implementation (a full Substrate client, but without block authoring).

The preferred implementations are: Java, Go, C++.

So, with some luck, I can release the full details tomorrow! :-)

@EdwardAThomson EdwardAThomson changed the title Substrate: Alternative Implementation Grant Polkadot Runtime Environment: Alternative Implementation Grant Aug 20, 2018
@gcsfred2
Copy link

@gcsfred2 gcsfred2 commented Aug 25, 2018

@EdwardAThomson Are there timelines for the grant applications?

@gcsfred2
Copy link

@gcsfred2 gcsfred2 commented Aug 27, 2018

I'm seeking to help a team in this endeavour, specially using Go. If there's an existing team that is looking for help, please contact me. gfrеⅾеrісο аt іģսаzս ḋοt ѕοⅼսtіоոѕ or on Twitter: ģсѕfrеḋ.

@EdwardAThomson
Copy link
Contributor Author

@EdwardAThomson EdwardAThomson commented Aug 29, 2018

@gcsfred2 The applications will be open for a while. At the moment there is a hope to have a couple more implementations, so the application process will remain open at least until we have a few interested teams working on this issue.

@ChainSafeSystems
Copy link

@ChainSafeSystems ChainSafeSystems commented Sep 16, 2018

Hello, our team was in Berlin during blockchain week and had an incredible time getting to know people from the Parity team. We showed interest in writing an implementation of the PRE in Go. I will be filling out the application over the next few days. Thank you!

@EdwardAThomson
Copy link
Contributor Author

@EdwardAThomson EdwardAThomson commented Sep 18, 2018

Hey ChainSafe team, that's great to hear! I see you have an email address on your profile. I will drop an email to say hello. Go is one of our preferred languages, so your application is of interest. We are starting to get quite a bit of interest now in writing alternative implementations.

@anjj123
Copy link
Contributor

@anjj123 anjj123 commented Sep 19, 2018

Keen to be involved in core development of Polkadot. Form submitted! :)

@EdwardAThomson
Copy link
Contributor Author

@EdwardAThomson EdwardAThomson commented Oct 5, 2018

Adding a list of relevant videos and blogs.

Videos:

Blogs etc:

Technical Chat channel:

We'd recommend jumping into the Substrate Technical channel and start asking questions. Some developers are on-hand and answering questions, plus there is also a handful of community members who are becoming experts too!

@EdwardAThomson
Copy link
Contributor Author

@EdwardAThomson EdwardAThomson commented Oct 5, 2018

Thanks for all of the interest so far. We have received a number of application forms, but the application process is still open in case any teams are wondering and were thinking of submitting a proposal. We are hoping to select 2 or 3 teams to develop alternative implementations, but we haven't selected anyone yet!

@EdwardAThomson
Copy link
Contributor Author

@EdwardAThomson EdwardAThomson commented Oct 8, 2018

A further update on this topic. Many of you have been asking about timescales and the amount of effort involved.

I had a quick chat with Jaco, a dev in Parity. He is the one who created the implementation in JavaScript. Hopefully this gives you a better idea. He worked pretty much solo too!

How long would you expect a dev / team of devs to create an implementation of Substrate?

"Difficult to say - the issue is that my implementation found a lot of bugs in the Rust libp2p code, so I was waiting for (what felt like forever), for fixes. Too early.

So first thing, what language - if not JS, Go or Rust, they need to write their own libp2p implementation. (Maybe there is a way to use the Go/Rust versions as libraries)"

"Then the tricky bits are:

  • implementation of Parity codec (2+ weeks to do properly) - needed for anything networking or RPC related
  • implementation of a Trie DB - they can learn from what is done in JS, Rust, Ethereum clients, however Substrate has their own version. (Once again, it may be possible to use the Rust version as a lib)
  • WASM environment, this stuff is not documented anywhere but in the code - took me around 1+ month here, but once again, implementations changed the whole time (too early)
    Then there are the bits and bobs - blockchain DB (over-and-above state trie), RPC calls, potential light client."

As a bare minimum, as he see it:

  • Trie
  • libp2p
  • WASM environment

"If in the implementation is in Go, the Trie they can actually take the Ethereum version and "just" drop in blake2 and the parity-codec."

For this grant, we are looking for the consensus part to get done too.

@Yuvasee
Copy link

@Yuvasee Yuvasee commented Oct 12, 2018

Hello, just filled out the form for C++ implementation.

@EdwardAThomson
Copy link
Contributor Author

@EdwardAThomson EdwardAThomson commented Oct 15, 2018

Hello, just filled out the form for C++ implementation.

Received! Thank you! @Yuvasee

The application process is still open. We will take a few more applications before making any decisions.

Did you know about our W3S conference? There are now reduced price and free tickets available:

https://medium.com/web3foundation/endowing-an-open-inclusive-community-at-web3-summit-3013456f4f84

@EdwardAThomson
Copy link
Contributor Author

@EdwardAThomson EdwardAThomson commented Oct 17, 2018

Hi everyone, we've had some questions about the development process and we thought it would be good to share the answers with everyone.

1. Understanding the motivation for an alternative implementation

  • To what extend is the present implementation a reference?

    • There is a specification is in a draft state. Hope to share it soon.
    • The current implementation is like a reference spec, but it isn't really.
    • RFP was started early, so we could start talking with teams.
    • We suggested that they get a proposal in early.
  • An independent implementation is sought? In that case, is it OK if the new implementation is based on the Rust implementation of Substrate?

    • As independent as possible. Done in an idiomatic way that suite that language.
    • Not a direct translation.
    • Empower an additional community of developers. Don't lock to a particularly set of developers.
    • Also can help to identify ongoing discrepancies between implementations.
    • As it is a decentralised network, we need alternative implementations. Don't want to have one team control the development of the network.
  • Better performance is sought? If so, what are the requirements/symptoms that we want to improve/cure?

    • Not really. Rust is fast enough. We really want to reach new developer communities and to improve decentralisation (network resiliency and governance).
  • Better portability for multiple platforms?

    • Expect it to work on Linux and Mac. (Unix like platforms)
    • Would be great on Windows.
  • Is there a rationale behind the preference for Go?

    • Want to include the existing Go community.
    • There is a Wasm interpreter in Go already. So it might be easier: https://github.com/perlin-network/life
    • We will probably fund at least 2 implementations. E.g. C++ and Go.
    • There is more to write in C++. Probably less reusable code. Possibly no libp2p in C++. Less Wasm support.

2. Understanding the implementation around WASM

  • Can we re-use an existing WASM implementation?

    • Can re-use an existing interpreter in the language of the implementation.
    • It may be possible to use the Rust implementation with a C++ PRE, as there are bindings, but would be better to have a C++ interpreter. If the C++ implementation isn't great, then either make pull requests or fork and improve.
  • Are we required to also implement native execution, or just the WASM interpreter?

    • Not necessary for spec compliance.
    • Can do an additional step.
    • There will be pieces of Wasm logic run frequently. Having the ability to run the logic as native code is important.

3. Understanding the “rules of engagement”

  • Do formal specifications or acceptance tests exist, or could they be developed as a first phase of the project?

    • We need these, but don't have them.
    • Would be willing to consider the tests as a separate work package.
  • Is there any formal deadline?

    • No deadline.
    • Ideally, an implementation by end of Q1 2019. Allows the implementation to start working with Polkadot.
  • In what form should we apply for a grant and to whom should the request be addressed?

    • Application form (as in original post).
    • Best to get a rough proposal in earlier. Even if it is not very well fleshed out.
  • Who will give the OK / decide at the end for the grant application?

    • Web3 Foundation. Peter will be the lead, but others will be involved (e.g. relevant people from Parity).
    • We may ask for additional information in order to clarify some things.
@keorn
Copy link

@keorn keorn commented Oct 18, 2018

@mnaamani
Copy link

@mnaamani mnaamani commented Dec 10, 2018

We have an interest in supporting this effort by doing a minimal port of the implementation of Parity codec, and any other rust modules with the idea to support writing a runtime in Go. So not a full alternative implementation.

This task of course will be taken on by the team selected to do the full Go implementation. So I was looking to see if there is possibility of collaboration or if no work has started at all, to perhaps apply for a grant for a narrower scope of work?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
9 participants
You can’t perform that action at this time.