Govern DX − 2

[bpierre]

Govern DX − 2

This is an iteration over the Govern DX plan, based on discussions I had with @sohkai and @nivida. The main idea is to focus our efforts towards a foundational set of pieces, while we identify and define the other pieces to build on top.

It eliminates Govern Server as it is, to replace it by three things:

• Govern Tx − the transactions service.
• Govern.js − the JS library taking over the server as a central piece.
• Govern Subgraph − the subgraph we already use.

The name Govern Server coult still refer to the different services together, or we could decide to bundle them all as a single executable in the future.

@0xGabi @Evalir @izqui @nivida @sohkai What do you guys think?

Related issues

• Govern Tx − Authentication Govern Tx − Authentication #224
• Govern Tx − Create transactions Govern Tx − Create transactions #225
• Govern Tx − Actions Govern Tx − Actions #226
• Govern.js − Fetch data from the subgraph Govern.js − Fetch data from the subgraph #227 (depends on Govern Subgraph − Implement dao / daos queries #228)
• Govern.js - Implement the REST client and the new actions from Govern Tx Govern.js: Implement the REST client and the new actions from Govern Tx #230 (depends on Govern Tx − Authentication #224, Govern Tx − Create transactions #225, Govern Tx − Actions #226)
• Govern Subgraph − Implement dao / daos queries Govern Subgraph − Implement dao / daos queries #228

New structure

The new Govern structure (simplified).

Govern Subgraph

A standard The Graph node that would expose on chain and IPFS data. The subgraph would stay close to the contracts, but also provide simpler abstractions and entities when necessary (e.g. query dao($name: string) {}).

Initial priority: add dao and daos queries.

Govern Tx

This service would allow to interact with Ethereum.

It would provide the following features:

• Per-account access using signature validation.
• Create DAOs.
• Schedule, challenge and execute actions.
• Gas spending rate limiting (gloabal + per account).

Initial priority: implement per-account access + DAO creation.

Govern.js

Govern.js would integrate the GraphQL queries of Govern Server to fetch the data from the subgraph and other sources directly.

It would mostly interact with:

• A Govern Tx node for paid transactions.
• A Govern Subgraph node to fetch the data from.
• The Snapshot Hub API.

It migth also interact with other sources:

• An Ethereum node to create direct transactions.
• Other subgraphs (e.g. Aragon Protocol).

We might decide to bundle all these services as a single executable (i.e. Govern Server) at a later point, but we want to initially keep the different pieces small and independent.

Initial priority: move to the Govern Subgraph as a data source.

Secondary / to explore

Govern Hooks

This service would allow to register web hooks and emit them.

It would provide the following features:

• Per-account access using signature validation.
• Register web hooks for specific events.
• Emit web hooks.

Status: implementing a hook mechanism using Govern.js and a Node.js server should be sufficient for the initial stages of the project. Deprioritizing it should allow us to focus more on the core pieces of the project.

Govern REST API

A read-only REST API to access Govern related data (read only). It would query the Subgraph directly. This is the HTTP REST API that was planned for Govern Server, as an independent service.

Status: its different use cases and features remain to be defined, and the GraphQL API should be enough in the initial stages of the project.

GraphQL Subscription Proxy

This would be a generic GraphQL server transforming a GraphQL endpoint without subscriptions (The Graph) into a GraphQL endpoint with subscriptions.

It would do so by doing HTTP polling internally and caching the results. Its goal would be to provide more optimized queries to users, and to remove some load from the subgraph node.

Status: this is an optimization piece that could be added once we reach a point where using a polling mechanism on the client side is not sufficient.

Previous structure (for reference)

The previous Govern structure (simplified).

[nivida]

I'm really happy to see the architecture going in the direction I asked for in the first week at Aragon! :-)

Govern Tx:

Important here is that schedule only can be called with the same address as defined in the submitter property of the passed container. I myself think we also could/should do this without any centralized service and with real meta transactions.

If we do it with a forwarder contract or just with a centralized version do we definitely have to extend our Queue contract to be able to handle the received signed messages accordingly (such implementations are well explained here; extending is not required if done with a per-user proxy contract)

We also just can replace the submitter property value with the address of the paying delegator. The disadvantage of it is it can confuse some users cause the correct containerHash will no longer be generated with the actual address of the submitter/the user.

Gas spending rate limiting (gloabal + per account).

If a remember correctly did Jorge also talk about a clear whitelist. This means if we have a "global" gas spending rate limit do we definitely have to add this mentioned whitelist. (Just saying because we probably have to discuss the business logic with the business team resp. Jorge & Product Team)

I agree with the priorities on this point because I think before we implement schedule, execute, and challenge do we have to implement the voting part with IPFS. I will talk later a bit more about the probably possible IPFS solution I thought about.

Govern.js

A Govern Subgraph node to fetch the data from.

Just to avoid any confusion should we probably call it internally TheGraph node. Because the current wording sounds as would we keep the current not really sense-making Govern server GraphQL API.

We might decide to bundle all these services as a single executable (i.e. Govern Server) at a later point, but we want to initially keep the different pieces small and independent.

I would always keep them as single Docker containers. Because it gives a DevOps more flexibility to just scale those parts of services which actually do need more power. Means this would in some cases save some resources. (Microservices FTW!)

Also here do I totally agree with the defined priorities. Overall does this sound way better than the first concept for Govern.js 💪

Govern Hooks

Deprioritizing it should allow us to focus more on the core pieces of the project.

Totally. This is something which is anyways possible to do in a nice way. We should better use those resources right now to kill the MySQL votes index DB Snapshot has.

Govern REST API

Status: its different use cases and features remain to be defined, and the GraphQL API should be enough in the initial stages of the project.

Totally. I think this is something we could easily add cause most of the required code is anyways already existing in the GIT history because of the current implementation of the server.

GraphQL Subscription Proxy

It would do so by doing HTTP polling internally and caching the results. Its goal would be to provide more optimized queries to users, and to remove some load from the subgraph node.

I think we could probably also do this on the client. If we use the "new" multi-threading features JS has will this I think run smoothly. The only argument against it could be the additional traffic it would require on the client instead of on the server-side.

IPFS & Votes

Currently, Snapshot is holding a MySQL DB on his server to have a votes index. This because with the infra concept they have is it not possible to get all the votes per proposal in another way.
I now thought about another solution with creating shared folders per proposal with IPFS/IPNS. The required setups to do so is something I do have to test any time soon and will the end decide if we actually can use it in our case.
The problem we still have here is that only one IPNS owner will exist and no direct way is existing to have this decentralized as we probably would like to have it. But let me see if I find a solution with also a good UX.

Edit: The IPFS idea after a quick read does obviously not work.. a decentralized IPFS indexer is what I in the end was looking for. Means we would have in the end to build a chain with a simple runtime and storage to index those votes/files.. which will I think have the same scalability problems we already have.

[sohkai]

@bpierre Agreed, I really like how this looks and it's much simpler :).

Govern REST API

Agree that we can keep this in our minds, but wait until we have further conviction on what a useful REST service would look like.

For example, we could consider bundling the snapshot API into this service in the future, or have a separate service altogether to abstract and aggregate multiple audit log sources.

GraphQL Subscription Proxy

This sounds relatively straightforward, and could be worth an experiment. But agreed that polling inside the client or on the server at this point will be a pretty minimal overhead especially as subgraphs are relatively cheap to query.

---

@bpierre @nivida

Govern Tx

I might need a quick up to speed on the specifics of this service and what we can and cannot do here. My (initial, loose) understanding is that we didn't need signature validation, and we can only submit schedule() and execute() transactions on behalf of users.

This is also one service that we can apply a lot of prior work and learnings that @promaty has already uncovered with the court/protocol backend, and a good candidate in the future to migrate to OpenZeppelin's Defender (at which point, our software essentially just acts as an auth layer on top of a Defender relayer).

IPFS & Votes

Really glad that you're thinking about this :).

creating of shared folders per proposal with IPFS/IPNS

The big headache I have around IPNS is that it seems like a PITA to set up and maintain, and as far as I've looked into it, it was much more complex than I liked for namespacing IPFS files—requiring crypto keys to maintain, etc.

It could be something worth a look into though; if Snapshot nodes become federated / can message each other, each node could build their own vote index on their IPNS domain. However, for a fraud proof to stay valid on IPNS, we would need some way to prove that the owner of the IPNS domain didn't update their index during the duration of the challenge.

[nivida]

I might need a quick up to speed on the specifics of this service and what we can and cannot do here. My (initial, loose) understanding is that we didn't need signature validation, and we can only submit schedule() and execute() transactions on behalf of users.

We can. But the actual submitter will no longer be the user means we would have to deposit the schedule collateral and the containerHash would be different cause the submitter is included see/see.

Really glad that you're thinking about this :).

I finally quickly checked the docs (lol) and it doesn't work or at least it doesn't really improve something.

[0xGabi]

Looking to this changes it makes a lot of sense after the first few iterations we went thought. I have a few comments:

Govern Subgraph

Initial priority: add dao and daos queries.

I think is key also here to be sure we all agree what a dao is in Govern context. I think we as a team still didn't reach a consensus.

Govern Tx

I agree with @sohkai that probably our end goal have to be migrate to a 3er party service like OpenZeppelin's Defender, the Relay service is quite align with what we are discussing.
So I consider this point have to shape our design principles. Probably one of the single items I'm seeing as particular to our use case is off-chain verification of witnesses.

Govern Hooks

I consider that this a great candidate to look for 3er party services as well. Tenderly for example allow us to leverage an existing alert functionality really interesting.

---

On a second note I'm a bit worried that we are not spending enough time to think about the Snapshot Hub API. In our current system this a black box that we expect to function as we expect but that may not be the case. I argue this maybe one of the most critical component of the system, specially looking a bit more into the future when Guardians require to have a verification scheme.

[nivida]

I think is key also here to be sure we all agree what a dao is in Govern context. I think we as a team still didn't reach a consensus.

For anyone looking for context.. here.

On a second note I'm a bit worried that we are not spending enough time to think about the Snapshot Hub API. In our current system this a black box that we expect to function as we expect but that may not be the case.

Totally. We probably should have a short call with Fabien where he elaborates us the API closer.