How do we grade questions? [Transcript]
Background: Five, four, three, two, one, zero. music starts playing All engines running! Liftoff -- we have a liftoff!
music continues playing
Jon: Well, hello! My name is Jon Ericson, and this is our... I think it is our third episode, and I'm here with Abby.
Jon: Aaaand, with Juan!
Juan: Howdy folks!
Jon: And we're just three community managers sitting in a room, talking about Stack Exchange and Stack Overflow. We are going to be talking about questions, and I had a question for both Abby and Juan. Have you heard of the expression: "There's no such thing as a bad question"?
Juan: Yes, all the time.
Abby: Yes, and I think so.
Jon: And what was the context when you heard that?
Abby: I think it was on a bumper sticker.
Jon and Abby laugh
Abby: Definitely outside the context. I mean, I've heard in the context of Stack Overflow and Stack Exchange as well, but it's one of those things that people tell ya; you know, you raise your hand in class, and "Hey! I have what's probably a stupid question", and someone says "There's no such thing as a stupid question, only stupid answers." or "bad answers" or whatever. That's definitely where I've heard it before. Jon or Juan laughs That's definitely what I've heard before.
Jon: Yeah; Juan, what about you?
Juan: Yeah, similar. I think mine has come from an H.R. presentation.
Jon: Ah, yes.
Juan: Where people are describing the benefits that you get if you sign this. This happens and ends in... It's okay if you ask questions because there is no such thing as a bad question.
Jon: Now, I have another question. Is that true? Like, in the H.R. presentation or in the classroom presentation? Can you think of some questions that were bad?
Juan: Yeah, definitely some bad ones. I think the bad ones are ones where people, obviously, weren't paying attention, and the presenter or the teacher just said that very same thing, like "write your name at the top of the test before you start", and someone says "where do we write our name?", well, that's not a good question. You obviously were not paying attention.
Abby: [laughs] Juan the former teacher over here seems to be speaking from experience.
Juan: Oh, yeah, man. Yeah. All the time. Never failed.
Abby: I think in the H.R. presentations you get, you know, people who were clearly just not listening and asking a question that was already addressed. But, for something like that, I think you can bet that probably a good proportion of the audience also wasn't listening, and so, repetition is actual beneficial to most other people there.
Jon: That's true. Uh, I was--I remember a class I took, a history class, and um, the class would always start off great. You know, the professor was really a great teacher, and then, about fifteen minutes in someone would raise their hand and ask a question and--I recognized this person after a few classes, and I just knew that this person would ask a question that was purposely controversial to, like, drive the professor off of his game so that he would talk about something else and, uh, it just was so aggravating. I'd roll my eyes and say "what does this have to do with the topic we're talking about today and why can't you let this professor just talk because he's awesome and he's not so much fun when he's talking about your boring question."
[Jon and Juan both laugh]
Abby: It's the student who kinda asks a question because they want to show off how smart they are and not because they want the answer?
Jon: Yeah, I almost think it was even like "if the professor doesn't lecture on something, then they can't test you on it"? [unintelligible] It's not true, but I think that's what they were thinking.
Juan: Exactly. You're still responsible for the material, even though they didn't particularly bring it up. There's still a book or some other research paper you've gotta read to get caught up.
Abby: Even if you manage to get the professor to just talk about a sandwich all day or whatever.
Abby: You will be tested on that sandwich, [and] you will be tested on the reading.
Jon: So, we're obviously talking about Stack Overflow and Stack Exchange.
Jon: And, that's sort of the lifeblood of our sites, are questions. You gotta have someone to ask the question; otherwise, there's no space for the answers to go. And, so, uh-- Abby, I'm going to pick on you because I know that you haven't thought as much but--
Jon: --what would you think would be a good way to measure the quality of a question?
[4:34] Abby: That's really really hard as we know because we've tried to figure it out for a long time: those upvotes and downvotes on questions the same way we do on answers, and I think the little hover text on the upvote/downvote button is this question is... "helpful and shows research effort" or the downvote says "this shows no research effort" and, um, the upvote... I don't know if those vary, or if that's just the Stack Overflow hovertext that I can remember. But to me that seems a very, uh, somewhat simplistic way of looking at the value of a question. Because as we talked about before that the value of a question is in the quality of the answers that it generates, that it spurs people to write, you know, there are really good anecdotes or nuggets of information that now exist on the internet because someone was inspired to write them based on this question. That to me makes the question valuable, and that makes it a better question, even though if, you know, if you look at it in the vaccum of the post by it's self that's not particularly interesting. So the answer to your question, I guess, the short answer to your question.. which I think is a good question by the way..
[5:43] All laugh
[5:45] Abby continues: Because I have no idea! Or rather I can make it as complicated as possible by thinking through it a little bit, because it's much more intricate, in my opinion, than just kind of "Oh this is useful" or "It seems like they tried alot", or something like that.
[5:59] Juan: Yeah I would chime in with, I think the measure of how good a question is, I think, also has to do with how well it has those listening or waiting for an answer. So if someone asks a question and the answer is not useful, then maybe the question wasn't a good question, because of the answer it generated. Again so it wasn't useful to those listening. Which is subjective, which is what makes it hard, right? [Jon: Yeah] "What's the best color in the world?" or "On a scale of one to ten, what's the best color of the alphabet?".
[6:39] All laugh
[6:42] Juan: I'll answer that!
[6:44] Jon: Yeah.. or even a question that's like, um, "I'm having this really difficult time with this code" and the answer is "You forgot a semicolon" and that's the problem. So the question might sound like it would be super helpful for future readers because it like "Oh I also have a problem with this thing" and then you get the answer and it's like "Well I didn't forget the semicolon, so that answer doesn't help me at all!". Heh.. [Abby: Yeah] so it's a waste of time.
[7:18] Abby: And that's where you might look at that question and think "Oh what a great question" and "They're really trying, it's well written" uh, they're showing their code, they're showing their work, it really looks like it's gonna be useful, but then the answer is "You forgot this semicolon" so that's another one where you kind of have to judge the question based the answer, which is hard to do until there's an answer. You know, you can't really do that in a vacuum and so you don't know maybe it's amazing, or it looks amazing, it's really just "Hey you changed the name of a variable and forgot to do it in this other place" or whatever, not actually that interesting an answer. So you never really know, I guess. [whisper] Can you really measure, the value of a question?
[7:50] All Laugh
[7:53] Juan: I don't know Abby.. How do we even know any of this is real?
[7:56] Abby: Exactly! Juan is on my level.
[8:01] Juan: Continue Jon, don't let us divert you.
[8:05] Jon: Well, that reminds me, on campus you can always tell when someone's taking a philosophy class, because they started talking about, you know, evil demons, and whether or not we were looking at a brain in a jar. [Abby laughs] Nobody really thinks about that until you take your first philosophy class and then it's like [Juan: very true] "I'm looking at a brain in a jar".
Abby: See, it's funny that you mention that because one of my side projects, Jon, and I've been kind of reading about philosophy [Jon: uh-huh!], and things along those lines sort of are casual (hobby) reading. That probably is where that comes from a lot, actually. Not in a class, but it may as well be.
Jon: laughs All right. I think both of you touched on something that I thought of, when I started using Stack Overflow, which is that: Really, when you get right down to it, the quality of the question depends on the quality of the answers. I have definitely found that you will go to a question, and someone will have just pasted their homework. They didn't do any research, and it doesn't look like it would be useful to anyone else in the world. Then, you get some really thoughtful, smart answer who says, "Maybe this is goo-- maybe not necessarily the best, or I can talk about this other thing that touches the question." You get this long, detailed, [and] wonderful answer; the question gets upvotes because the answer was so helpful that it's obviously (like) "the question was helpful too" [Abby: m-hm], and then people edit it to make it a better question; all that sort of thing.
Juan: I'm sure you are all in suspense about how I think. [Abby chuckles]. I noticed none of you asked me, so yeah [Jon and Abby laugh].
Abby: Hey, Juan, I've got a question for you. How do you think we can measure the quality of a question, just out of curiosity?
Juan: That is a great question. I'm going to make sure I answer your question, Abby. I didn't do this research but we have a wonderful data team who will answer your questions like this, and go and do some research. We want to come up with a very simple question grade, and the formula we ended up with was that we decided to rate questions as good, bad, [and] neutral. We are definitely saying that there are bad questions. This isn't going to be perfect because a lot of reasons. When we looked at good questions, we decided that question had a score > 0 or it had at least one answer and a score of zero, and it wasn't closed.
Which is a bit hard to say because there is a bunch of parenthesis there that you can't pick up on ... u.. the basic idea is if ... if people have taken time to upvote a question it probably is a good question. And .. uh .. right now, there aren't ... uh ... there are many question that don't get upvotes at all and so we say well if it doesn't get any vote you know, up or down ... uh .. if someone went and took the time to answer it then that is also a sign then that that is a good question.
Abby: And if it has a score of zero and an answer but that score of zero comes because of two people have upvoted it and two people down voted it, that still is a good question even though it has those two down votes?