Unopenable challenges (and bad challenges in general) #108

Closed
urticadioica opened this Issue Mar 3, 2013 · 13 comments

4 participants

@urticadioica
Collaborator

Earlier today, user zulolosi submitted two challenges (currently on top of the main page) which are impossible to open in the browser, or start with vimgolf put. Using wget on the challenge URL still works, and reveals the problem: this user was trying to use the byte 0x80 to represent the Euro symbol. He's apparently never heard of UTF-8. The problem is probably caused by having a faulty UTF-8 character in the challenge.

EDIT: zulolosi has apparently learned how to UTF-8, and has deemed it wise to submit these two abominations again. The non-working versions are no longer at the top. Links to the non-opening versions (not that the links work, but that's the point): Extended Customers One, Cross-tab report.

Funny thing is, even though this is the most obvious case ever of where challenges should be deleted (even without the bad encoding, they're beyond awful), since you can't open the challenge in a browser, I don't believe the mods have access to the "Delete Challenge" button. Deleting this garbage may take direct access to the server. And some fix, so that users can't submit unopenable challenges, also seems to be in order.

This ties into the issue of bad challenges. Since anyone can submit any challenge, this has always been an issue to some degree, but it's been especially bad lately. I'll probably need to prepare a long rant to demonstrate how bad some of the challenges are. I'll certainly need to write up a list of guidelines that demonstrate what I mean by a "good" challenge and a "bad" one.

But first, I want to point out the source of the problem. The only editorial tool we've got to deal with bad challenges is having the mods delete them. I've seen @Gumnos say a few times he's been tempted to do just that. But he hasn't, and he's got good reason not to. That's a tough judgment call to make.

So as the only moderator who's an active player, he's got two choices: delete outright, or leave trash at the top of the homepage, which is effectively a recommendation for other players. I don't envy that choice.

Outright bad challenges aren't the whole story, anyway. There are lots of almost good challenges, which could be great if they didn't make a clumsy mistake, like stray whitespace, or non-tab indents without a modeline, or encouraging a help file copy strat. I'm more experienced at VimGolf than anyone, and I regret a few decisions I made in my own challenges. If we helped each other design new challenges, I think quality would go way up.

Another part of the problem is that, somehow, we seem to be encouraging new, inexperienced players to create most of our new challenges. That leads to more clumsy mistakes. It also leads to challenges that are designed more like the creator's homework problem than a game that would be fun to play. Too many challenges would be better on stackoverflow than VimGolf.

There's a lot more to say, but this is a discussion we need to have.

@igrigorik
Owner

a) Thanks for flagging this. Indeed, had to drop into the console to remove them.
b) (a) shouldn't have happened to begin with - need to fix the code on this end.

On the quality discussion: agreed, I'd love to see improvements in this department. The simplest strategy that comes to mind is to allow moderators to edit the challenge file itself -- this will require some work though, need to think through it. The other is to have a moderator approved queue.. but this imposes its own set of challenges (pun not intended).

@urticadioica
Collaborator

So I've been thinking about quality more myself, and I have a suggestion you might like.

A couple days ago, I got involved in a thread on reddit. Someone saw me mention VimGolf, and thought it looked great (I agree), but I felt I had to recommend some challenges first. I can't recommend the top of the main page right now.

When you look at the front page, there are three main ways we "endorse" a challenge (intentionally or not), in roughly this order of importance:

  1. Put it at the top of the page. Our primary endorsement. The newest challenges get first pick. This is a good thing. Effectively, we rally around the newest challenge and race to get first place, and that helps challenges get off to a good start. Newest on top gets ugly sometimes though, when the last few challenges aren't so good. (Right now is pretty bad, and it seems like around September it got pretty bad too).
  2. Put it at the bottom. Not as big an endorsement as the top, but it's the other end of the page. The quirk here is that the same challenges have been at the bottom the entire history of the site. Also, since the bottom is oldest, some users decide to play in that order. (Until they hit Flodder-challenge, and realize that might not be a good idea.)
  3. Number of entries. When I played through, I basically sorted by this. It works a lot better than playing from top or bottom. Basically, if lots of people try a challenge, it's probably somewhat accessible. If they then take several tries before finishing, it's probably pretty interesting.

There are other factors, like visible entries, best stroke count, or order in the RSS/Twitter feed, but those aren't visible on the front page, so they're far less important.

What we've got right now is basically like browsing reddit by "New". No one in their right mind does that. Maybe we could sort the front page by "Hot" instead? Having new challenges on top for a bit is great, for the rally effect, but if they fail to catch a good rally, they really need to sink, and let better challenges up to the top.

I ran some experiments with a super-simple formula: (entries + i) / age. i is a small number (try 1) that boosts new challenges a bit. (Finding the age of a challenge is easy, since the first 8 characters of the id happen to be Unix time.) I played with sorting by that order, and I thought the results were pretty good. Then I tried boosting age by 86400 (seconds in a day), and I liked how new challenges dropped (assuming they don't get more entries). Lots to tweak.

So, if bad challenges moved away from the top of the page faster, I'd be mostly satisfied. This method could be easier to implement and manage, and maybe more democratic than than making the mods do all the work.

@igrigorik
Owner

I like it - simple, and should work well.

Anyone against? :) /cc @Gumnos

@urticadioica
Collaborator

Here's a spreadsheet showing what the front page would look like with the formula I proposed.

Basically, for a new challenge to stay toward the top, it would need to average a new entry every 4-6 hours. Very modest (maybe too modest).

@jdelkins

Relating to this issue (the original one, not the challenge scoring idea)... When I open This revised challenge using ruby-1.9.3-p385, the .output file shows odd binary characters instead of the euro symbol, as below:

 Thomas    | C  |   1 930,00 â<82>¬ 

When I downgrade to ruby-1.8.7-p371, the output shows up properly:

 Thomas    | C  |   1 930,00 € 

This signifies either an incompatibility or bug either in vimgolf code, ruby, or some required module thereof.

@urticadioica
Collaborator

@jdelkins The issue you ran into is the same as #105. Thanks to your hint (where you found it only happened in 1.9), I posted #109 (hope I did that right), which should fix the problem.

@Gumnos

As @urticadioica mentions, I advocate for at least some measure by which rubbish challenges get demoted. It would be nice if they also played into the scoring calculation, as it seems a shame to have to jump through the hoops of lousy challenges just to keep a ranking. I too have considered deleting one or two of my early challenges as I learned over time what made for a better challenge. That's less grievous than deleting the lousy challenges of other people, but it's tempting at times. No objections here.

@urticadioica
Collaborator

So I had another thought. (Another one? Can't I make up my mind?) I was trying to think of even better ways of sorting the front page. I think my last idea would be a modest improvement, but this could be even better: sort by activity.

This is really simple, both in concept and hopefully implementation. A challenge gets posted? To the top. Someone posts a solution to the challenge? To the top. Someone posts a comment? To the top.

This idea should have the same basic advantages of my last idea. New challenges get a shot on the top. If they get popular, they'll stay there for a while. The most popular challenges will always be near the top, where they tend to attract even more players (a snowball effect). New challenges that are underperforming quickly sink, fall out of sight, and distract less attention.

But this method has a new advantage. The only reason long-term players like me visit the front page is to see what people are posting. I've already done basically every challenge I want to do, and I learn about new ones from Twitter or RSS. What I really want to see are new solutions and comments, but I effectively only see them on the newest challenges, since it's hard to check the entire page for activity.

Also, a big problem I found early on is that when old records are broken, no one ever notices. You'd think that should attract attention, but no one can see what's happened with a front page so big.

If we like this and decide to use it, I don't think the front page would need to be "prepped". It could start in its current order, and change over time.

I'd like to know what everyone thinks.

@jdelkins

@urticadioica, I'm just a newcomer to the site, but nevertheless feel compelled to comment because it seems I'm kind of hooked on it. I think you're on to something. For a low volume site the "to the top" policy with might work fine, and it is certainly a simple idea. I have three suggestions in reply, however:

First, if the site has (or grows to have) any significant traffic, then this simple system could result in a highly unstable front page, as the top items are constantly being pushed out. I think a slight refinement addressing this issue would be to order items by "actions per timeframe" where timeframe is something like "minimum of: time since the post was originally made or 24 hours." This would give a big but rapidly decaying bump to newly-posted challenges, while still providing the "trending" sort order you are looking for. This is really just a buffered version of your idea I think.

A second feature to address your particular (and vaild) use case would be to "star" or highlight challenges that have had activity since you last visited the site. The site already tracks users, so it should not be that hard to flag challenges according to whether they were "touched" since your last visit.

Finally, a great addition would be a "hall of fame" list. Since I have really enjoyed the challenges you recommended in your reddit post, I would even be so bold as to suggest that, by tradition, the hall of fame be maintained by the holder of the no. 1 leaderboard position. That person clearly has invested enough into the site and community to be able to judge and recommend challenges to the benefit of newcomers like myself.

Cheers and happy vimgolfing.

@igrigorik
Owner

@urticadioica re: activity. I think your original proposal could be augmented to take activity into account. It could be a simple linear formula with some weights for recency, popularity, etc.

Just for fun and reference, the HN algorithm: http://amix.dk/blog/post/19574

@urticadioica
Collaborator

I ran a lot of experiments with activity sorting. Didn't come up with anything I liked. I haven't come up with an objective measure that works better than (entries+1)/time. New users really need a better way of deciding which challenge to play, and short of personal recommendations, that formula is the best I've got.

As for the Hacker News algorithm, it's... virtually the same thing. It just adds a gravity term, which is a great idea for a news site, but probably not for VimGolf. On a news site, you want old content to fall quickly, but the ideal for VimGolf would be to put the best challenges of all time at the top, regardless of age.

Of course, one could always add more variables and potentially improve the recommendations. The best variable would likely be user experience, from voting, bumping, or a single player's "hall of fame", like @jdelkins mentioned. Popularity is positively correlated to challenge quality, but it's not perfect. (Some of its flaws are systematic biases, like preferring challenges that spent a long time on top or bottom, or overrating extremely easy "challenges".)

@igrigorik
Owner

Makes sense. I think we have a good enough solution to start -- let me take a run at it.

@igrigorik
Owner

Better late than never, right? :) ... New ranking live! Based on proposed entries / age heuristic.

@igrigorik igrigorik closed this Jun 23, 2013
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment