Overhaul of the KPI System

[bwhm]

Rationale

After 11 weeks of KPIs, we at Jsgenesis are of the impression it's not working the way we had hoped. Though there are many smaller issues that may have contributed to the failure of the current system, we think the biggest problem is simply that incentives for KPIs are not calibrated correctly.

As the KPIs reward all token holders based on their proportional share of the tJOY issuance, it would only makes financial sense for true "whales" to participate without a substantially larger fiat pool. Even then, the "free-rider problem" could come in to play.

Our hope was that these issues would be mitigated through funding proposals and competition for Council spots, but that has proved to be incorrect.

Our main objective for the KPI system continues to be attracting and training contributors to the project, while simulating the economic system we envision the mainnet platform will run on. At this stage, all improvements to the platform (in the form of code, marketing, hiring etc.) will need to be funded and executed by the platform independently.

It is to serve this ultimate goal that we have suggested some enhancements to the KPI system, which are outlined below as a proposal which we hope the Council will carefully consider.

Description

An executive summary of our proposal to improve the system are as follows:

• KPIs will be split in to "Council KPIs" and "Community KPIs".
• Council KPIs:
  • These KPIs will typically focus on short terms goals, like network stability, platform operations and allocation of resources.
  • They will always follow the Council term (i.e. they will last for seven days)
  • The rewards for achieved Council KPIs will be distributed to the Council and their backers (voters) only.
• Community KPIs:
  • These KPIs will typically be producing a deliverable or performing some task for the community or Jsgenesis.
  • These can be more long-term oriented, and require more work and some skills in a specific domain.
  • The reward structure will work similar to todays KPI system, but with a clear workflow, and with Jsgenesis providing a guarantee for the contributors.
• Both of these KPI types will require more involvement from the Council, thus requiring better incentives for them to be active.
• Jsgenesis will take a more involved in the election cycle, to avoid complacency and "free-riders".

More details below:

Definition of Terms:

• "Applicant" - A person that wants to to do the work for a KPI
• "Assignee" - A person that, after being an Applicant, gets approved, and may be given some privileges in line with agreed terms
• "CM" - Council Member
• "Deliverable" - Any item submitted either to the Council or Jsgenesis, with the intention of achieving the KPI.
• "Submitter" - A person that submits a KPI deliverable to the Council

Council KPIs

Recurring KPIs, such as "Block Production", "Proposal Clearance" and "Council Reporting" are good examples of potential Council KPIs. Alongside managing the working groups and the platform finances, this constitutes an important part of the Councils main responsibilities.

The common theme is that these KPIs either directly or indirectly require the Council to monitor the network and Working Groups' performances, the platform finances, and pay attention to Proposals and discussions.

Suggested Changes

The Council KPI rewards will go to the "successful" election (governance) participants, ie. elected CMs and those that voted for them, proportional to their stakes.

As perhaps the most critical role at this stage of our testnets,the CMs will now earn rewards in two ways. A fixed and predictable reward for getting elected in the first place, and a performance best bonus for doing a good job.

Finally, Jsgenesis will take a far more active role in the voting process, and scrutinize the voting history and activity level of each applicant. The main purpose of this is to counter the "free-rider problem", by voting out CMs that doesn't perform their tasks satisfactory. Note that Jsgenesis voting will not count as "stake" wrt. to the KPI reward distribution.

Community KPIs

KPIs such as Content Creation (e.g. KPI 2.3) and the recurring Telegram Bots are good examples of a Community KPI. These KPIs can address issues outside of the platforms day to day operations, and focus on longer term goals such as development, community building, creating useful tools, etc. In general, these are perhaps better understood as bounties and/or competitions, so a name change could be in the cards.

Our thinking here was always that community members would try to attack these sorts of KPIs themselves, then through a series of Text and Funding Proposals, perform the work, and claim a reward.

This has worked slightly better than for the Council-style KPIs, but it is still far from perfect. All of the factors below deserve some of the blame:

1. These KPIs have not been promoted sufficiently, neither inside nor outside of our community
2. The Council has had no incentive to perform tasks themselves, nor search for people to assign them
3. The funding structure we imagined have been poorly communicated - thus few know/understand it
4. There was no explicit guarantee that your work would be rewarded by the Council in the end
5. It requires some overhead, instead of just performing a task and claim the reward

The somewhat negative feedback loop meant that we were hesitant to put too much effort in creating these KPIs, which again made it less interesting for users.

Our suggestions below addresses most of these, but not the fifth. This is because we believe it's important to cultivate a system that mimics a workflow that is viable on mainnet.

Suggested Changes

In the new system, we rely on the increased incentives for the CMs to assist in setting the rules, creating a clear workflow and act as Project Managers. They are also the ones that decides what is submitted to Jsgenesis to be graded and rewarded. Note that Jsgenesis can overrule their decisions in situations if the "rules" are not adhered to. Especially in cases where a user spent time and resources in to something, without getting rewarded due to some mistake made by the Council.

Community KPIs may not have a deadline, but the prize and complexity may be adjusted if deemed necessary.

Rules

As the Community KPIs will vary in size, scope and reward, each time a new Community KPI is made, the Council must agree on the specific rules (this will be part of the Council's KPIs).

Reward Distribution

The Council decides how much of the total KPI reward will go to the Submitter, if the rewards should or can be split, and so forth.

Format

The format should try to optimize for the time, quality, risk and cost, associated with each KPI. The Closed, Free For All and First Come, First Served formats presented are just suggestions.

Closed

For a KPI that requires investing lots of time and/or other resources, it may be reasonable to guarantee one or more Applicants that gets Assigned some time to complete all, or some, of the work, without having someone come in and "snipe" the reward.

Free For All

For smaller, and perhaps more creative and subjective KPIs, it may make more sense to leave it as a "free for all". In this case, the Council sets a deadline, picks the best Deliverable(s), and rewards the Submitter(s) as per the rules.

First Come, First Served

For smaller, perhaps more time sensitive KPIs, one could choose a format where anyone can enter, but each Submitters Deliverable is reviewed by the chronological order they are submitted. The first acceptable Deliverable(s) is granted the reward(s).

Further

In addition the varieties outlined, other formats can be defined and chosen if they are more appropriate for a specific KPI.

A "new" Council must honor any agreements and rules set by their predecessors, for as long as the rules say so.

Workflow

The workflow will depend both on the Reward Distribution and the Format, and must be established beforehand.

• For "Closed" formats, an Applicant must present a bid why they should be assigned the given KPI. This should include detailed terms, such as time needed, costs, etc. If approved, this makes the terms valid.
• In some cases, it may make sense to break a KPI up in to milestones, with partial rewards at each stage. This builds trust as the Council can see the progress being made, and the Assignee can get chunks of the reward along the way.
• In other cases, the person may need some initial funding to get started.
• For "Closed" formats, the specifics of the workflow could be part of the Applicant's application for participation.

Steps

1. New KPIs Made

Although the Council decides on the rules and reward distribution, they listen for input from others in the platform forum and on Telegram. Within a reasonable time (as stated in the Council KPI), the rules for the KPI are presented in Text Proposal, voted through, and published on GitHub.

1.5. Work is Assigned

Depending on the rules chosen, there may be a step to assign the work to one or more Assignees.

This will require some back and forth through multiple Proposals, and should thus be avoided for less complex KPIs.

2. Work Happens

For a "Closed" format, it can mean a series of Text and Funding Proposals, waiting, and ongoing communication between the Assigned/Assignees, and the CMs.

For a "Free for All", it can be mean reviewing submitted Deliverables as they come in, or waiting for the deadline. How a Submitter should make the CMs aware of their Deliverable once ready (GitHub, Telegram, forum or Proposal) must be defined in the rules. A "First Come First Served" format will be similar to the "Free for All". Once one or more Deliverables are approved, the Submitter(s) should be considered as Assigned in Step 3.

3. The Work is Submitted to the Council

Regardless of format, once an Assignee, or otherwise qualified Submitter, considers their work done, they create a (final) Spending Proposal, which in total rewards them the agreed amount, links to all relevant discussion and rules, and a link to their work.

Once the Council considers the Deliverables complete, this final Spending Proposal is "approved" and successfully executed.

4. Jsgenesis Grading

After Step 3, the Submitter have received their reward, and their work now "belongs" to the platform.

Jsgenesis will then review and grade the Deliverable as such. This can result in a reward anywhere between nothing (failed), or everything (full score), and the fiat pool will be increased accordingly.

It may be that this reward is smaller than what was rewarded to the Submitter. This will cost the token holders, and one would expect the Council to be punished.

Councils Role

As seen in the workflow, the Councils role in the Community KPIs is substantial. They will work as the Project Managers, and are at the end held accountable for the quality of the Deliverable they submit for grading. These tasks may be a part of the Councils KPI directly, but the efficiency, creativity, rules, workflow, speed and outcome of the process will anyway be part of the Jsgenesis Council voting process.

[mochet]

I think a lot of the proposed changes could work great, particularly incentivizing those who put stake/vote behind council members.

A bit of the problem lays with the current Pioneer UI being a bit suboptimal (which I know is being worked on). I feel from the questions I've helped users with that the council role appears hugely technical to start with and the incentives for being a council member aren't easily explained.

Suggestions:

• For the first few weeks, Require council applicants to create a forum post introducing themselves, their interests and how often they're online before they are able to apply for council. Also require that they announce their application in the Telegram and link their forum intro post. (require could just mean incentivize in this case).
  • The current council UI is not ideal for this, but there is a forum thread where people can post this.
• Maybe starting with a slightly lower council size (say 6-8 members) could make a bit more competition for the spots (I discussed this a bit lower as well from the perspective of using a proposal to have a smaller council)
  • There would be increased competition for council spots
  • The advantage of a smaller council is that proposals have less friction and can be "pushed through" quicker. For now there hasn't been any instances of malicious proposals being submitted.
  • The disadvantages are probably too many to list (:
  • This can always be changed once the reward cycle for the role becomes clearer.
• Maybe starting to give the council a bit of predetermined budget areas to work with would make the possibilities of using their funds a bit clearer. I made an issue for this about multiple mints: Idea: multiple council mints joystream#1145
  • The council will eventually revolve a lot around accounting of funds as I understand so there needs to start being a system for this at some point.
  • If there are budgets, it is something that could be communicated on the website to new users so they immediately understand there is x amount of money available for content uploads or specific types of work.
• The current incentives of being a council member are unclear to average users and as far as I can tell are only known via chainstate, there is also a technical issue where full payment isn't recieved for the term.
  • When the new system is implemented it would be good to have the incentives (and their USD value equivelant) published on the blog/somewhere to show people very clearly what the role can possibly pay.
  • The current testnet page on the website could perhaps show what each role paid out over the past 7 days. It currently just lists token value + KPIs but doesn't really explain to a newcomer what each specific role pays out.
• I think that a lot of the deliverables assigned in the KPIs require a very significant amount of technical knowledge (like reports/tokenomics).
  • Having some more focus on less technical tasks (like video uploads, forum posts) may work to attract a more diverse userbase which can in turn help to attract people who have a desire to spend the time acquiring more specialist skills.

Council KPI ideas:

• Add an incentive for a largely automated council report. I've done these reports before and its a lot of copy + paste activity which could largely be taken from the chain directly. I could produce a template for this but the technical know-how on how to make the template come to life isn't something I'm capable of.

Community KPI ideas:

• I will think about these. There are several ideas I have but whether they're realistic right now has to be taken into account.

Other notes:

• I think the bounty system proposed in this issue (Idea: Crowdfunded Bounties joystream#1094) would be a really effective tool in getting people to start contributing to stuff. It would be good to know when it is expected to be deployed as it seems to cover some areas of KPIs slightly.
• Telegram is a bit limited as the central communication point, mainly because we can't easily have multiple channels. While we did have a council channel, but you can't guarantee people on the regular Telegram channel will join it. It also means users who are brand new to the Telegram channel have no idea there are other channels unless they are advertised regularly/better. Because everyone in crypto uses telegram it does mean there are a lot more users, but something like Gitter would make multiple channels a lot easier to manage. Gitter also has the advantage of users not having to sign up or join and still be able to see the chat history... Telegram, Discord and Slack do not have this feature I think.
  • This is a bit of a double edged sword as people should really be using on-chain stuff at some point.
  • With enough time and given there are distinct working groups a single Telegram channel as the main point of communication would probably be unworkable so if its an option to move to an alternative that allows for channels maybe that is something to consider.
• With the current userbase size the number of council members is probably a little high. I did have discussions with someone in DMs about potentially doing a proposal to reduce this number but it also has negatives associated with it:
  • ex. if 2 council members are AFK for a few days, then you can't pass any proposals
  • It has a 2 week grace period which means if there is a requirement to change these values again, it would take time for it to be implemented. This long grace period makes perfect sense on mainnet but makes it a bit of a challenge to sensibly propose a change to the voting cycle/council size.

[jimmy-tudeski]

Good to be here, I hope all of You are fine and safe.

I aslo think that most of proposed changes would be a great benefit to comunity in the future.

Regarding the point mentioned by mochet, I would like to give some of my comments.

1. For the first few weeks, Require council applicants to create a forum post introducing themselves, their interests and how often they're online before they are able to apply for council. Also require that they announce their application in the Telegram and link their forum intro post. (require could just mean incentivize in this case).
   The current council UI is not ideal for this, but there is a forum thread where people can post this

• I think that it's a great idea to ask aplicants to prepare some kind of introduction of themselves. Regarding the possibility to post such introductions and the fact that it should be available for all users i think that such candidate aplications with quick briefs should be linked anyhow and available to all users who like to vote for council members

Regarding the fact that telegram has limited functions and forum here is not so popular to comon users maybe we should think about launching a Discord server. There are lot of users familiar with discord and used to it, and discord gives lot more oprtunity for subchannels and some nice forum structure. I think that discord now gains much more atention than telegram.
If You would like to think about that I can be helpfull with that because I run some Polish comunity Discord group for traders and investors.

2. Regarding the Council size

• I do not think that there should be less than 10 people. I think that there are much more disadvantages in shirinking the council members. I even think of getting it wider in some time when there will be much bigger comunity but this one can be discussed later on proposals and forum

3. Comunity KPI Ideas - I also think that bounties campaigns would be great to activate the comunity. As I mentioned before Telegram has very limited functions as we all know and I think Discord in most popular now and I rally have some nice options there. Mochet mentioned that people would not have to be logged or signed in to see chat history but they only have to be users like on telegram so that is not a problem i think. Discord now is most popular social in crypto space in my opinion and we should think about also using it.

Those are some of my thoughs about certain points but in general I think the proposed changes are the directions we should follow.

I

[bwhm]

A bit of the problem lays with the current Pioneer UI being a bit suboptimal ...

No argument here :)

* For the first few weeks, Require council applicants to create a forum post introducing themselves, their interests and how often they're online before they are able to apply for council. Also require that they announce their application in the Telegram and link their forum intro post. (require could just mean incentivize in this case).

That would be a good idea!

* Maybe starting with a slightly lower council size (say 6-8 members) could make a bit more competition for the spots (I discussed this a bit lower as well from the perspective of using a proposal to have a smaller council)

...

Although not stated, this was indeed the "plan". As competition is needed, there needs to actually be a fight for the spots

  ...Information on mints and budgets...

We're going to deploy some resources on getting a "tokenomics overview" page, but it will take a couple of weeks probably...

  * When the new system is implemented it would be good to have the incentives (and their USD value equivelant) published on the blog/somewhere to show people very clearly what the role can possibly pay.
  * The current testnet page on the website could perhaps show what each role paid out over the past 7 days. It currently just lists token value + KPIs but doesn't really explain to a newcomer what each specific role pays out.

This might be possible to include in said page

* I think that a lot of the deliverables assigned in the KPIs require a very significant amount of technical knowledge (like reports/tokenomics).

These really shouldn't. I made a script for that (although it doesn't work after the upgrade :P ), so it could have been rather easy. Still, it's not trivial...

  * Having some more focus on less technical tasks (like video uploads, forum posts) may work to attract a more diverse userbase which can in turn help to attract people who have a desire to spend the time acquiring more specialist skills.

Yes. Although it's "easier" making these, it adds a barrier, and it makes grading more subjective. If the Council are doing some/all of it, that would be better for all!

Other notes:

* I think the bounty system proposed in this issue ([Joystream/joystream#1094](https://github.com/Joystream/joystream/issues/1094)) would be a really effective tool in getting people to start contributing to stuff. It would be good to know when it is expected to be deployed as it seems to cover some areas of KPIs slightly.

Yes, this system might be far better for bigger technical ones...

* Telegram is a bit limited as the central communication point, mainly because we can't easily have multiple channels....

This has been a long running topic of discussion for us. As long as the activity is relatively small, tlgrm works well. In the long term though, it has some serious limitations...

* With the current userbase size the number of council members is probably a little high. I did have discussions with someone in DMs about potentially doing a proposal to reduce this number but it also has negatives associated with it:
  
  * ex. if 2 council members are AFK for a few days, then you can't pass any proposals
  * It has a 2 week grace period which means if there is a requirement to change these values again, it would take time for it to be implemented. This long grace period makes perfect sense on mainnet but makes it a bit of a challenge to sensibly propose a change to the voting cycle/council size.

All of this is true, but the harm done should decrease if there is a non trivial reward in play. At least, that is our thinking...

[bwhm]

Good to be here, I hope all of You are fine and safe.

Glad to have you, and likewise :)

* I think that it's a great idea to ask aplicants to prepare some kind of introduction of themselves. Regarding the possibility to post such introductions and the fact that it should be available for all users i think that such candidate aplications with quick briefs should be linked anyhow and available to all users who like to vote for council members

Yes, an application approach with some similarities to how the storage and curator working groups is coming soon (tm). In the meantime, we could use the "about you" section in the membership, and display that for applicants, but as the whole membership and council election is currently being worked on, it may not be prudent if it requires too much work...

Regarding the fact that telegram has limited functions and forum here is not so popular to comon users maybe we should think about launching a Discord server. There are lot of users familiar with discord and used to it, and discord gives lot more oprtunity for subchannels and some nice forum structure. I think that discord now gains much more atention than telegram.

As stated in my previous reply, other communication platforms in general, and Discord specifically, has been discussed many times...

* I do not think that there should be less than 10 people. I think that there are much more disadvantages in shirinking the council members. I even think of getting it wider in some time when there will be much bigger comunity but this one can be discussed later on proposals and forum

This would not be a permanent reduction, but rather an initial one to "bootstrap" a good council culture...

1. Comunity KPI Ideas - I also think that bounties campaigns would be great to activate the comunity. As I mentioned before Telegram has very limited functions as we all know and I think Discord in most popular now and I rally have some nice options there. Mochet mentioned that people would not have to be logged or signed in to see chat history but they only have to be users like on telegram so that is not a problem i think. Discord now is most popular social in crypto space in my opinion and we should think about also using it.

Although we may stick with tlgrm for a while longer, I can't argue against that...

Those are some of my thoughs about certain points but in general I think the proposed changes are the directions we should follow.

Thanks to both of you :)

[freakstatic]

I think that the idea for a bonus for the council members about doing a good job is a good idea. Most of the members of the council seem to be a bit demotivated maybe that can help a bit.

The bounty system would be really great to see more engagement in the tasks that need to be done.

About the council size I shared the same opinion about reducing the size but now that we have more users maybe that would be necessary.

About automating the tokenomic report I have start to work in a script, based a bit in the work of @bwhm, which outputs a file with markdown format based on the template created by @mochet. This work is a bit stuck waiting for me to have free time 😁

The ideas presented would be a really good change to the current system.

About the telegram alternative would be good to have some channels and now even more with the bot posting messages. Like Gitter, Discord or Slack.

[mochet]

I shared this on the forum as well:
One part of the feedback loop that is a bit challenging at the moment is having a sufficient number of CMs engage with council reports in the form of providing meaningful feedback at the end of a council round. I think that any incentives provided for CMs should not just include the week where they perform as a council member, but also partly include an overlap of a few days post-council round that relate to them providing feedback for the council report.

By this I mean that perhaps 85% of incentives are related to the week where they are an actual council member, and 15% are related to providing feedback for council reports (which only occur after the council period has ended).

A new council should technically not be providing feedback on a previous council's report, but that's a bit of an issue with no obvious solution as they have to vote to approve it anyway. The only way to possibly change that is to complete the council report in the final stages of the council cycle, but that would possibly just lead to it being rushed and incomplete.

Especially in these early stages of council work this feedback is important information on how to better develop systems, so it should be incentivized at some level.

[blrhc]

I shared this on the forum as well:
One part of the feedback loop that is a bit challenging at the moment is having a sufficient number of CMs engage with council reports in the form of providing meaningful feedback at the end of a council round. I think that any incentives provided for CMs should not just include the week where they perform as a council member, but also partly include an overlap of a few days post-council round that relate to them providing feedback for the council report.

By this I mean that perhaps 85% of incentives are related to the week where they are an actual council member, and 15% are related to providing feedback for council reports (which only occur after the council period has ended).

A new council should technically not be providing feedback on a previous council's report, but that's a bit of an issue with no obvious solution as they have to vote to approve it anyway. The only way to possibly change that is to complete the council report in the final stages of the council cycle, but that would possibly just lead to it being rushed and incomplete.

Especially in these early stages of council work this feedback is important information on how to better develop systems, so it should be incentivized at some level.

This is a very interesting point - though I think once the council are paid well enough they should be incentivised to assist the next council with the report in the recognition that this is an important element of the council's responsibilities, even after their term has ended.

It should be made clear that previous council members who don't assist with preparing the report are much less likely to be voted for in subsequent rounds. Once councils are smaller and payments to councillors are larger, this will be a more significant incentive. Council members won't want to miss out on the opportunity to be on the council again once this role is a bit more exciting and lucrative.

[mochet]

The new KPI system has been implemented, you can find more about it here: https://github.com/Joystream/helpdesk/tree/master/roles/council-members#why-become-a-council-member