Theme always sucks

[Yngwarr]

Alright, it's not funny. I think that the reason why we have such a great amount of themes that are total crap is a complex site's usability issue. Here are the components that, I think, confuse a lot of newbies into suggesting crappy themes (the following is not a result of a professional UI analysis, but still seems reasonable):

1. First of all, three little fields for theme suggestions are worse than a Google Form because they makes user feel more obliged to fill fields with something even if it isn't relevant at all. The simplicity of the procedure doesn't help either. For me it's obvious that if one has nothing to say, one should stay silent, but the design doesn't even suggest silence as an option. I think, a single field with a "suggest another one" button would be a better UI.

2. Secondly, the list of old themes on a suggestions' page. I know that it's intended to show people what themes are considered good and to give some kind of inspiration, but it just doesn't work like this. Instead people who have less then 3 ideas of their own may very likely just see the old themes, change the word or two and submit (because they feel obliged, see p. 1).

3. Thirdly, the themes remain visible to submitters. Which leads to a lot of "see my sugestion, dudes!" posts with screenshots. Which may give people the kind of inspiration mentioned in p. 2 of this list. These results in a huge pile of quite similar themes.

All of these add up to a great amount of moderate-to-crap themes which are just dumped to the Slaughter stage without any pre-moderation (mostly because Slaughter is meant to be a pre-moderation phase). 4000 themes is a ridiculous number not many of us can review and the amount of nonsense, paraphrases and placeholders doesn't make this labour easier.

I know that a plenty of more in-depth reviews of the problem were written by the community in ldjam.com posts, but I struggle to find any here on GitHub. Just want you all to know that "theme always sucks" is a problem and it's getting worse.

Cheers,
Igor.

[mikekasprzak]

FWIW we actually addressed some of these issues, but had to roll-back the changes when it caused a bunch of other site-wide issues I didn't have time to investigate. Then this past year happened. I've been extremely busy working full time + overtime at a game studio (I needed an income, gotta eat), so my time for LD has been short. The game shipped a week ago, so once I clear my slate of my biz taxes and LD44, I will have time again. This is one of the reasons the schedule has changed, so I can make more time to work on LD between events.

…

On Tue., Apr. 23, 2019, 13:21 Igor Kislytsin, ***@***.***> wrote: Alright, it's not funny. I think that the reason why we have such a great amount of themes that are total crap is a complex *site's usability issue*. Here are the components that, I think, confuse a lot of newbies into suggesting crappy themes (the following is not a result of a professional UI analysis, but still seems reasonable): 1. First of all, three little fields for theme suggestions are worse than a Google Form because they makes user feel more obliged to fill fields with something even if it isn't relevant at all. The simplicity of the procedure doesn't help either. For me it's obvious that if one has nothing to say, one should stay silent, but the design doesn't even suggest silence as an option. I think, a single field with a "suggest another one" button would be a better UI. 2. Secondly, the list of old themes on a suggestions' page. I know that it's intended to show people what themes are considered good and to give some kind of inspiration, but it just doesn't work like this. Instead people who have less then 3 ideas of their own may very likely just see the old themes, change the word or two and submit (because they feel obliged, see p. 1). 3. Thirdly, the themes remain visible to submitters. Which leads to a lot of "see my sugestion, dudes!" posts with screenshots. Which may give people the kind of inspiration mentioned in p. 2 of this list. These results in a huge pile of quite similar themes. All of these add up to a great amount of moderate-to-crap themes which are just dumped to the Slaughter stage without any pre-moderation (mostly because Slaughter is meant to be a pre-moderation phase). 4000 themes is a ridiculous number not many of us can review and the amount of nonsense, paraphrases and placeholders doesn't make this labour easier. I know that a plenty of more in-depth reviews of the problem were written by the community in ldjam.com posts, but I struggle to find any here on GitHub. Just want you all to know that "theme always sucks" *is a problem* and it's *getting worse*. Cheers, Igor. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1797>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAL7MCJ4FSZTEMOGYTSNEJDPR5ASHANCNFSM4HH4GLLQ> .

[olivier-grech]

I think there are two courses of action that would solve the issue:

• Reduce the number of themes that are entering the Slaughter phase. There could be multiple solutions for this, for example we could automatically merge duplicates, reduce the number of suggestions per user from 3 to 2, reduce the maximum number of characters for a theme, etc.

• Manually filter themes. We could at least filter the non-sensical ones.

I believe some of those solutions are not too hard to implement. We could reach out to the community for the manual filtering.

[jasonleekennedy]

@olivier-grech, that is is my take on the slaughter round. Crowd sourcing the filtering of themes.

[egordorichev]

I'd still suggest this:

Or at least, ban themes, that were suggested too much/where before/are same as most of other/etc etc etc. I know, its hard, but I'd rather make one trusted person choose a theme, rather all of this... Like, I did not see a good/new theme since ldjam37 :/

[egordorichev]

Another suggestion would to not let first-time-participants to vote on themes, or make their votes count as less, because they didn't see old themes, and posts like this one show this really well.

Also, check out this post.

[jeroenpx]

What about the following idea?

When proposing a theme, you could force the person to also give five specific examples of totally different games you could make with the theme he/she wants to propose. All the examples would need to be in a different gerne. Maybe you could even force them to pick different gernes (from a dropdown) for each of the examples. E.g. a platformer, a dungeon crawler, ...

It would force people to really think about the theme they are submitting.

This could actually also be useful information when voting because it is sometimes not easy to immediately see the full potential of the theme... Maybe you didn't immediately think of a certain way you could interpret the theme...

I think, we would probably get less and hopefully higher quality themes this way.

[olivier-grech]

@jeroenpx I could probably find five different games for each of the themes suggested for Ludum Dare 44, and I have a terrible imagination. That doesn't make them any less bad.

[local-minimum]

I don't really think it is the old themes as suggestions that cause the same themes to always make it through slaughter. Because we had that issue before those were listed. But I agree in that the entire process is fundamentally broken because even on the final round almost all themes end up with a negative score. So it feels like "I'm participating in spite of how the themes work".

For me there is two good things about the theme process:

Participation & Last week suspense.

I find the idea of being able to improve on the first bit by requiring much more to submit by answering some questions like

A lot of participants find it very important that the theme allows for creating lots of very different types of games. How does your theme help with this?

Participants must be able to make games that are only text based, only uses audio, has no physics engine and so on. How does your theme help them?

A lot of veterans find it very annoying that the same themes just keep coming back every time. How does your theme break off from at least the past five LDJAMs?

And maybe some more questions. Cap at one theme per user. And in slaughter or whatever happens before finals, we get to see answers with the themes.

Then on the final week, I don't think -1 / 0 / 1 helps at all. Since most of us down votes the majority. After you voted it just feels like everything sucks. Even if there was one or two you might actually like. So I propose either every round you get to pick your three favorites or you get to sort all the themes by preference. I think both of these would give a much more positive hype. Slaughter could actually be out of this set of five pick the one you like. That would make it much more positive too

[local-minimum]

More controversial suggestion for who gets to suggest theme:

• Only those that end up top 3 in any category.
• Only those who has participated in at least X LDJAMs
• As a reward to those who graded the most and gave the most appreciated feedback on the previous event.
• By lottery among the members 1 month before the event, 100 invites to suggest a theme are sent out

[Yngwarr]

@jeroenpx, @local-minimum I see what is the point of your suggestions, but I don't think these would work because it relies on being reasonable and don't protect from nonsense. It would just feel like "I accept the license agreement" checkbox and, what's even worse, throw more worthless information in the faces of participants who really care about the theme, which is the exact opposite to what we should do.

I came up with, possibly, the most controversial idea of all. I know no-one would even think of implementing this exact idea, but just think about why it would almost certainly work out. The idea is: why not to charge $2 per a theme suggestion? This is why I think it would address the problem:

• It's not as simple as just fill the field, one must do several things to submit a theme (find a credit card, type the payment info, confirm the transaction).
• It stops one from submitting carelessly because it costs. Even if it's not so much money, one still may think that the irrelevant joke or moderate-to-crap idea is worth it. It may also provoke one to discuss a theme idea before submitting it which is a good thing.

Yes, it may leave some great ideas unspoken, but isn't it an acceptable trade-off for removing thousands of nonsense not-even-themes? I see the problem of having an enormous amount of crap in the simplicity of the process and lack of engagement from people submitting. I know the preferred way is to develop the community of developers who care by enlightening them about what makes the theme good and why it matters, but this alone just isn't enough on the scale this wide.

[local-minimum]

@Yngwarr a patreon survey to suggest themes would be a way to almost do that. And yes it would be controversial. But I think that today it didn't really matter if great ideas are spoken. They most surely are lost. I don't think there's a benefit to anyone in today's quantity, quite the opposite.

The most obvious way to make people take care and think hard about the theme suggestion would be to gamify it somehow. But I'm not sure of how that game should work.

[egordorichev]

@local-minimum it won't stop new guys or trolls from putting a 1$ pledge (or a bit more) voting, and then removing it after a few hours (they don't even get to pay anything). I like the point of suggesting only 1 theme and picking the top 3 fav ones. Answering about "why this theme is good" is a bit hard to implement, cause people can just dump random text there... I think, if we lower the number of themes, and we ban some themes with some simple js snippets (right when you suggest the theme, after pressing the button, it won't submit it, but say why this theme is not accepted). Tho, that might also turn to be pretty hard, cause themes evolve, but at the same time, stay the same (floating islands -> floating worlds xd).

[local-minimum]

Another way would be that Mike gives us ten themes and we get to vote on them.

[srakowski]

One thing we could try is to augment the slaughter to make it a non-binary affair. Good theme? Yes|No could be augmented to be: What do you think of this theme? BestThemeEver|LoveIt|LikeIt|Indifferent|WouldRatherNot|HateIt|NoWay|Snowman or numerically 8|5|2|0|-2|-5|-8|-99999 (we can leave that last one off, I just didn't like the snowman theme years ago :)). This would be a relatively simple change (I think) and could cause some new themes to shake out. We may end up in the same place, but adding some variability to the process may be all we need.

If it doesn't work we could augment things until it does (e.g. your scale grows the more times you participate giving veterans more weight). Thoughts?

[Yngwarr]

@srakowski, I think this would be an unnecessary over-complication. I mean, how do you decide if the theme is +2 or +5? Many would struggle with this and while people who care spend their precious time on this kind of decisions, the others will, most likely, rush through themes using only +max and -max buttons (because it's effective). So in the end we would find veterans more exhausted with slaughter and the pile of the worst themes from the pool.

[Yngwarr]

@local-minimum, the idea of charging per suggestion is to enforce the person to think twice before suggesting yet another theme. A patreon survey works a bit different. =)

[jcmonkey]

a video explaining on what themes are, how to make a theme, and how to choose good themes
that would go along way

[richard-sim]

I don't have a solution for theme slaughter, but for the actual voting rounds I think that Preferential Voting system may work wonders. A great explanation of how it works, and why it's good: https://www.reddit.com/r/coolguides/comments/bbxjj6/how_to_vote_in_australia/

[egordorichev]

Ok, the theme is just....

[tudddorrr]

So I'd like to pitch in a few ideas too:

• Get rid of the old pool of themes. Have a new pool of user suggestions for every event (opening up 1-3 months before).
• Have moderation on the themes that go in. Might not scale but at least cut out duplicates, themes that have been used for past events etc.
• Get rid of the rounds. They don't change much, they just add unnecessary complexity.
• Get rid of -1/0/1. 0 is equivalent to not exercising your right to vote in an election. -1 then becomes unnecessary as people just need to upvote the themes they like.

If all else fails @local-minimum's suggestion of voting on 10 themes is the most elegant approach the only problem being picking the 10 themes in the first place might be a nightmare.

[Jezzamonn]

One thing I'll point out is that at the end, almost all the themes end up negative. That means that no matter what theme gets picked from that list, a significant portion of people (possibly a majority of people!) will dislike the theme. People have pretty different ideas of what makes a good theme :)

Personally I don't think the voting system is too broken at the moment *ducks*

[local-minimum]

@Jezzamonn I interpret the same statistics inverse, but I think it might be a major point psychologically to remove negative votes and have voting by arranging first second third candidates (or something similar). Because marking all but one or two themes with -1 cements the negative feeling of everything sucks.

Maybe @mikekasprzak just for fun could see what themes would have made it out of slaughter if only upvotes counted.

I doubt we could make things so most people got happy, and that's not really the reason I'd like to have some changes implemented. I want the process to feel better even if we always get one bullet in thirty seconds on one screen with strange floating islands physics in the end

[mikekasprzak]

I don't have the tweet handy (see my reply to the results list), but the same theme won if downvotes were ignored. I've been pondering the idea myself. IMO the issue is more psychological than an actual problem, but yes if we were to remove down voting (like we made only liking allowed of posts/comments) it would feel better. Mirroring the other psychology wins, I'm thinking of changing it from upvote/meh/downvote to "pick your favourites", as many as you want, little hearts. Effectively the same thing, but psychologically it will make you find at least one you like, hopefully putting an end to some of the "I hate all the themes" posts. #positivity

…

On Wed., May 1, 2019, 15:47 Martin Zackrisson, ***@***.***> wrote: @Jezzamonn <https://github.com/Jezzamonn> I interpret the same statistics inverse, but I think it might be a major point psychologically to remove negative votes and have voting by arranging first second third candidates (or something similar). Because marking all but one or two themes with -1 cements the negative feeling of everything sucks. Maybe @mikekasprzak <https://github.com/mikekasprzak> just for fun could see what themes would have made it out of slaughter if only upvotes counted. I doubt we could make things so most people got happy, and that's not really the reason I'd like to have some changes implemented. I want the process to feel better even if we always get one bullet in thirty seconds on one screen with strange floating islands physics in the end — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1797 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAL7MCIXTS4N4XOGZ6QPNX3PTHXVLANCNFSM4HH4GLLQ> .

[a-t-richard]

After a few LDs, I started to "study" how the voting system of the LD works, especially round votes (in my lab we are a lot of people who work on voting theory).

As a preliminary result, it seems that the "LD round voting system" is similar to an approval voting system.
Unfortunately, this voting system can easily lead to "unfair" results.
(if you want, this video resume well how approval voting works: Voting Theory: Approval Voting)

I will perhaps study the round voting system more precisely but, as a first suggestion, you might like to look at the Majority Judgment voting system.

From my point of view, it seems to fit well with our voting case (and I think it's possible to keep an visual aspect like -1/0/1 with this system, even if its potentially better to use words)

[a-t-richard]

After discussing with colleagues in voting theory, i wanted to add a little erratum about what I said previously.

First, Majority Judgment might not be the most adapted voting system for a ludum dare (because of the condorcet principle).

And also, more generally and before trying to say which voting system is the best in our case, we should try to evaluate if the actual voting system is trully not good (How much participants found the final theme good or not? How much this theme inspired them during the ludum? etc.)

We can ask these questions during several ludum dare and if we found that generally people found that the theme sucks, there is a problem and we can discuss about another voting system.

And then we should discuss about which type of voting system we should use (or make experimentation in parallel ludum dare 🧑‍🔬 )

[mikekasprzak]

Since it's been a few years, I'm going to share my current thoughts on this.

TL;DR: The theme sucking is a feature. The process can be greatly improved, but "sucking" is unsolvable and doesn't need to be solved.

Saying or declaring that you're uncomfortable is one of the most uncomfortable things to do. With "The Theme Sucks" reaching meme status, everyone is comfortable saying "The Theme Sucks", which to me is a low key way of saying you're uncomfortable. Being uncomfortable isn't a bad thing. Some of the most innovative and original games seem to come out of Ludum Dare, so I don't necessarily think we're doing people wrong by making them uncomfortable.

Are some themes not ideal? Sure, but terrible themes don't win. That's why we have half a dozen rounds of selection and voting. Would it be better if the worst themes got eliminated earlier? Yes, and this is where we can improve.

I don't think giving anyone greater influence on theme selection is a good idea. Today I have great influence out of necessity, since I'm the one deciding if 2 themes are similar enough to eliminate one (cleaning up the later theme voting rounds). Behind the scenes I often have to remove 20-30 themes due to similarity checking not being a simple string compare (i.e. floating islands vs flying islands). I don't like this, it's a bottleneck, and it needs to change.

One partial solution is to not hide what others have suggested. We can use "smarts" to show what suggestions are some percentage similar to what you've written, asking the user to confirm they want to suggest this. Same goes for previous Ludum Dare themes. We've run too many events to list them all, but we can use the same logic on previous themes, discouraging folks from re-using them. This wont fix everything, but it should give us a more unique pool of ideas.

It would be nice if we could help people with their grammar and capitalization, or auto suggest more generalized versions of a theme. We had a suggestion back in LD41 "Combine 2 incompatible genres (e.g. turn based racing)" that made it to the final round that IMO could have been worded differently ("Combine two contrasting genres"). It does make me wonder if it's worth letting users include notes (i.e. examples) with each theme. 🤔

Providing more theme insight could mean we could lower the number of theme suggestions. Right now it's 3. I've heard cries for "only 1 suggestion", but I'm leaning towards 2.

I'm happy with the way the regular Theme Voting rounds + final round work right now.

That said The Slaughter could be better, could be more useful. My current solution is what I call Theme Fusion, which changes the theme dynamic from "do you like this Theme" to "which do you like better, Theme A or B", with an option to suggest the 2 themes are the same. The problem with Theme Fusion is that it requires more work to be sure (O^n), where as Theme Slaughter has just the list of themes (O^1). Lowering the suggestion pool will help, but this will require further experimentation to be sure we're getting enough coverage, and we may have to use smarts to begin discarding or lowering the priority of ideas. By design it wont make completionists happy, so we may need an artificial "you've done enough".

Ultimately the most important thing is the process. People have to like participating in selection. If it's a slog, a grind, and doesn't feel rewarding, people are not going to want to do it. Taking part in Theme Selection is how many people are introduced to Ludum Dare. We could do better.

[mikekasprzak]

We could generate stats. That said, rating thousands of themes, I don't think individual completionists can make a conscious impact. At some point all that voting becomes a blur. 😅

[richard-sim]

Time and time again, no matter what theme we end up with, it still works.

I find it very hard to be excited for a LD when there are great themes in the final voting rounds, only for a disliked theme to be chosen - especially after going through the slog of the slaughter. It's just so disappointing to feel like I've "worked" for trying to influence a good outcome, only for it to fail. I'm not young anymore (like many of us!), and have other things I'd rather be doing than working on a game for a theme that evokes the exact opposite feelings to what a theme should do: inspiration and excitement.

*edit: Ouch, that reads a lot harsher than I meant for it to. I love LD! But I did want to emphasise that there are real costs for "the theme sucks" that do hurt LD IMO, and I was speaking as someone who used to be a slaughter completionist (I haven't been for a while because the inevitable theme choice made it feel like wasted effort). However, I do agree that something IS definitely working here, as amazing and innovative games do come out of the themes. Perhaps @mikekasprzak is right that it's a Feature Not a Bug, and the bug that needs addressing is the pains around a poor theme (both due to feeling burnt by "working hard" during slaughter, and when it comes down to coming up with a game idea). On the other hand, with so many good themes that don't win, we can't conclude that LD is better off without them since we have no evidence! Maybe LD would be 10x what it is today if the theme was decided on by magical unicorns?

[local-minimum]

We could generate stats. That said, rating thousands of themes, I don't think individual completionists can make a conscious impact. At some point all that voting becomes a blur. 😅

No individually of course not, but maybe as a group. And perhaps it's fair because they are dedicated, but perhaps there are adjustments that would make sense. I mainly think it would be very interesting statistics.

[a-t-richard]

Well, I didn’t expected such a debate, it’s very interesting 😄

I’ll try to synthesize what I understood.

It seems that we all agree on what @local-minimum said:

I would be, because I do care about LD and I think it could be better at attracting more people and retaining those that have tried it once or twice. I view the theme-process as something that should build hype, be joyful and pull people into the event.

The theme selection process is a part of the LD’s folklore/tradition. It helps to build hype days after days.

However, we can still discuss how to improve each step of this process, to improve the involvement of peoples during the process and then potentially during the LD (Do we agree on that?).

First, it seems that we agree that the slaughtering phase can be a barrier to involvement of newcomers.

Secondly, as @richard-sim said, the involved ones can feel disappointed by the result.
(and it seems that we agree that one can feel inspired by a theme even if s/he "doesn’t like it")

It's just so disappointing to feel like I've "worked" for trying to influence a good outcome, only for it to fail.

But lets focus on slaughtering phase first.

What we want during this phase is to estimate which themes are interesting or not (or which are inappropriate), and those with best results are taken for the round phase.

However, slaughter all themes is time consuming and can be a barrier to newcomers.
Fortunately, we don’t need that everyone vote for every theme to obtain a representative result, if each theme got a representative sample of votes it’s ok. (for example, if there is 5000 participants, we only need ~350 votes on a specific theme to have a representative sample for this theme).
But, slaughtering (or not) all the themes is also a part of the LD’s tradition ^^

So, a "simple solution" could be sorting proposed theme for slaughtering and proposing ones with less votes.
And at the end, maybe check for proportional results, instead of total up vote.
(statistics of previous LDs could be interesting ^^)

And to avoid scaring newcomers, we can "hide" the total number of theme to slaughter and maybe gamify a little the process (ex. a swip system like "Reigns", or with achievements like "you slaughtered half of the themes !").

What do you think about all that?

(sorry for the long post 😅 )

[local-minimum]

I think for slaughter we might want to consider a wordle kind of scheme where we only allow a small number of themes (e.g. 10) every day (or couple of hours), but uniquely randomly drawn for each user.

It should decrease barrier of entry. It hopefully evens out how many themes each participant slaughters.

But maybe even more importantly, it encourages coming back to the slaughter to see what your slaughter options are right now.

It also encourages taking a little more time in deciding what is good or bad, which may or may not affect what themes get high slaughter scores.

[mikekasprzak]

Not to leave you guys hanging with slaughter suggestions, but the slaughter is a bottleneck on my end and it needs to be replaced. While we can improve the suggestion process, the English language is constantly changing, and it's able to communicate the same ideas multiple ways. This makes algorithmic "duplicate idea" removal unfeasible. The only way to get duplicate checking off my or any individual's plate is to make it part of the process.

Not to mince words, I will be replacing the Theme Slaughter with Theme Fusion once it is ready (sometime next year). Theme Fusion is not a perfect solution, and it has its own series of problems, primarily it creates more work. This added work can be reduced by using the data it generates, but our infrastructure can't currently handle doing work in the background. The processing we use to prioritize games is the one and only thing we do right now, because anything more would degrade overall website performance. Data processing needs to be isolated to its own servers: runners and database replicas.

---

In case anyone wants to talk about Theme Fusion, it works like this: rather than deciding if you like a theme, you decide which of 2 themes you like better: Theme A or Theme B. You can also choose a 3rd option: "these themes are the same to me".

Like the Theme Slaughter, you can also flag themes for being inappropriate.

---

Question: Should we still have you choose a favorite even when you say two themes are the same?

I'm leaning towards yes, since we still need to decide which presentation of a theme people like better. Currently I use the highest rated duplicate when deciding this, but I often prefer the wording of a slightly lower rated duplicate.

Question: If a theme is flagged, does that automatically mean you prefer the other theme, or should this data point be effectively discarded?

I'm leaning towards disregarding the comparison, so themes that unluckily end up paired with a bad one don't unfairly get data points suggesting they're better than something not in the running. Tangentially, if we keep the unflagged theme visible and give you a random 2nd theme to compare against in the flagged themes place, it wont feel wasted.

Question: Is it worth having an indecision option? An "I like both" or "I don't like either".

I'm leaning towards no, but I could see an argument for those moments you decide two themes are the same. That said I'm not sure indecision is a useful data point. If two themes are the same, we still need to pick which way it's presented resonates best with people.

Question: Which presentation is better?

• Two random themes every click
• The preferred themes stays, and you get a new random 2nd theme to compare against
• Combination of both: allow a theme to have a 5-10 theme streak, before replacing it

😉

Question: How do we decide what theme pairs to present?

With zero data it will have to be random, but I suspect behind the scenes we'll have to do something similar to game prioritization. The way that works is every game is assigned a score given the combination of ratings received, ratings given by the authors, as well as liked feedback received and given by the authors. The way the math works, scores eventually become negative (leading to panicked emails from folks that don't get it, lol), which has the side effect of highlighting games without ratings or feedback. The result is almost nobody goes without 1-2 rating or comments.

Similar scoring math should be used: themes with zero interactions have a prioritization score of zero. As themes receive interactions, they should become lower priority, to a point. Inappropriate themes could receive harsher negative prioritization, but as the theme pool gets coverage, they should get tested again to ensure that inappropriate flagging was justified. The problem is that this algorithm alone only ensures coverage and burying of inappropriate themes. We have more data available to us, and it should be factored in.

• A vs B
• A is B
• generally which themes are liked more (given the comparisons they were randomly placed in)
• generally which themes are liked less (given the comparisons they were randomly placed in)

This leads to the final question.

Question: How do we decide the top themes?

Some metric will need to be chosen. The easy answer is the most preferred themes, but even without using the collected data to change prioritization, some themes will naturally receive more attention than others. One option is assigning an average score from every interaction. That should yield similar (rough) results as the Slaughter, and the "A is B" data can be used to drop dupes.

This doesn't seem "end game" though. We need to remember that an average of "A vs B" data isn't the same thing as an average rating. We can improve Ludum Dare ratings by weighing the accuracy of a rating given the number of ratings: i.e. the score of a game with 50 ratings is likely more accurate than the score of a game with 15 ratings. New chin scratching required. 😉

[a-t-richard]

You could also compute sentence similarity to propose duels between the most "similar" themes in priority.

I propose huggingface’s api because it’s based on neural network and not on mathematical formula, its more a semantic/conceptual distance between sentences.

As @adfaure said, it could be interesting to have the previous theme list to make some tests.

[local-minimum]

@Avzgui Out of curiosity do you have a link to the api in particular you thought would be a good fit (note that our themes are partly single words and sentences).

[a-t-richard]

@local-minimum I thought about a model from Huggingface for sentence similarity: https://huggingface.co/models?pipeline_tag=sentence-similarity

Just select one model and you’ll have an API to test the model.

Edit: Here’s a little tutorial on how to use huggingface’s api: https://huggingface.co/tasks/sentence-similarity

[a-t-richard]

I wrote a little script to test theme similarity with a hugging face model with the themes of LD50 provided by @mikekasprzak .
(script and results: https://we.tl/t-6nv2gQJjs7 )

Example of result:

.
.
.
{
    "themeA": "It was just a dream",
    "themeB": "It was only a dream",
    "similarity": 0.9876455068588257,
     "levenshtein": 4
},
{
    "themeA": "Strength is weakness",
    "themeB": "Weakness is strength",
    "similarity": 0.9861432313919067,
    "levenshtein": 14
},
{
    "themeA": "Was just a dream",
    "themeB": "It was only a dream",
    "similarity": 0.9842526316642761,
    "levenshtein": 8
}
.
.
.

It tooks me ~30min to compute all combination with a not so good GPU.

By using HuggingFace’s REST API, computation should be really faster to compute similarities between a new theme and all themes already proposed, even in client side (to test).

Edit: I added levenshtein distance for comparison

[adfaure]

That looks great, thx.

How could we evaluate the results in a more "scientific" way ? It seems to work well on the subset you showed us, however we should evaluate the whole results.

@mikekasprzak Du you have records of the theme that you manually removed, so we could have some annotated data ?

[mikekasprzak]

Could we use the previous theme list to find the best approach to detect similar themes ?

Theme lists are available via the API. Here's LD50.

https://api.ldjam.com/vx/theme/idea/vote/get/276397

I wonder if sending the whole theme list to the clients won't be worst that doing the computation on the server.

We already send the complete list for the Theme Slaughter.

I propose huggingface’s api because it’s based on neural network and not on mathematical formula, its more a semantic/conceptual distance between sentences.

If it can be done entirely client side sure, but we don't have server-side compute resources for this.

[richard-sim]

Theme lists are available via the API.

Would it be possible to get the results of the slaughter with those? It'd be very valuable to see votes and what has been flagged.

If it can be done entirely client side sure, but we don't have server-side compute resources for this.

I presume it would be possible to run a one-off cleanup task pre-slaughter though? I'm imagining a simpler client-side implementation to catch the easy cases, and then a much more in-depth script gets run before the slaughter is launched.

[mikekasprzak]

Would it be possible to get the results of the slaughter with those?

At this time no. Only the list is exposed by the API. For what it's worth, flagged should already be removed.

I presume it would be possible to run a one-off cleanup task pre-slaughter though?

Yes, and we do exactly that. My current cleanup code removes everything with case, punctuation, or white-space differences. I believe someone did some further testing, eliminating plurals and suffixes. Unfortunately if I'm remembering correctly that only shaved off maybe 20 themes of 3000. 😅

[a-t-richard]

Question: How do we decide the top themes?

I just thought, maybe you can use something similar to the chess rating system to determine the top themes throught the theme fusion process 🤔
https://en.wikipedia.org/wiki/Chess_rating_system

(yeah, this is the formula they use in The Social Network)

[mikekasprzak]

Hmm, maybe Elo could work here. 🤔

Divide up the theme pool into levels, and let them compete at their level (or +/- one level), but we're not necessarily going to want anyone to retire... except maybe those that get pushed all the way down. I suppose that could be a thing: the active pool shrinks over time.

The problem is an individual (unless restricted) could potentially moon a favorite theme by always giving it preference. There might need to be a hard limit of say 10 interactions per individual theme per user. 🤔

[mikekasprzak]

Weird, GitHub seems to be glitching on me. Here's a screenshot of what the above reply should have said.

[a-t-richard]

Divide up the theme pool into levels, and let them compete at their level (or +/- one level), but we're not necessarily going to want anyone to retire... except maybe those that get pushed all the way down. I suppose that could be a thing: the active pool shrinks over time.

Maybe you can simplify Elo by just setting a score to each theme and propose random duels 🤔 .
It’s depend of what you want as a result for Theme Fusion.

Shrinking the active pool before the end of theme fusion can invisibilize situations where a theme A, with a low score, is prefered to a theme B with a high score.

The problem is an individual (unless restricted) could potentially moon a favorite theme by always giving it preference. There might need to be a hard limit of say 10 interactions per individual theme per user. 🤔

Strictly random duels and a high number of participants should avoid this kind of situation, but you can add a limit just to be sure.

[ausstein]

I think ELO is a great Idea. But only to calculate the ratings of the themes not to pair them. I would leave the pairing completely random. In the beginning assign a staring ELO of say 1000 to all themes the give the users random theme pairs and adjust their ELO based on the ELO formula. In the end the best themes should have the highest ELO score. I also think you could drop say the bottom 20% every day since they are very unlikely to suddenly gain all the ELO back

[a-t-richard]

Dropping themes with low ELO scores day after day, in addition to random duels, could work only if participation is stable day after day. No? 🤔

In humble my opinion, it’s enough to just make random duels and compute a score similar to ELO, and at the end of the week take those with the highest scores to the voting phase.

For more accurate scores, it’s possible propose in priority duels with low participation.
For example: the result of a duel A vs B with only 10 votes, is less accurate than a duel C vs D with 100 votes.