If MDN cannot even seek feedback from its own maintainers before pushing major changes, how can the community trust them to ever have their feedback heard?

[clarfonthey]

Many have mentioned in mdn/yari#9208, the introduction of AI explain has seriously eroded MDN's trust with the community. Even one of your core maintainers commented in the issue stating that they did not know about the feature's planned inclusion until after the fact, and while they do take some of the blame for not proactively researching the matter, I think that this is shows an extremely dangerous status quo at MDN.

If MDN is not willing to actively bring its own maintainers into the loop when discussing features that can seriously affect its credibility, how can the community ever hope to feel included in these discussions?

It's incredibly clear that MDN is taking a massive L on this change. The thread includes a number of overwhelming arguments against this feature and its inclusion, which would have easily been noticed if MDN had taken the step to request feedback beforehand. In fact, it seems that this feature was planned for inclusion, developed, and released without regard for its community during any portion of the process, since at any one point during that process, the community would have likely shared these arguments. Note that I say arguments here because they are not merely statements of taste, but genuine reasons backed by evidence of why this feature is harmful.

The thread was closed for being "heated," but honestly, it was an extremely tame thread, all things considered. There was a lot of repeat arguments, but that was only because the community did not feel like their arguments were being taken seriously by those who decided to include this change. It's MDN's fault for completely failing to listen to the community here and to consider them when developing new features for MDN, and that's why so many people felt the need to express their concerns.

What steps are you taking right now to ensure that all future changes to MDN are done with the community's desires and needs kept in mind? How do you plan to incorporate community desires & feedback into the entire process, rather than damage control after the fact?

And, why was this done without community input in the first place? At this point, it's worth asking whether this change deliberately wasn't discussed before release because it would have been taken so negatively, and if that's the case, what do the people responsible have to say for that? Will any changes be made to rectify this, or will these people continue to make decisions at the detriment of the community?

There are a few active things MDN can do to truly fix this problem:

1. Actively foster community discussions on what people want out of MDN, and what could be improved.
2. Actively link roadmaps of planned changes to these discussions.
3. Actively seek feedback for possible solutions to presented problems, before development ever starts, and certainly before they're released.

How does MDN plan to execute these points?

[clarfonthey]

I noticed that the remark on actually answering this question on the call was simply hidden by a maintainer. Does MDN simply not wish to answer this question, and prefer to pretend they didn't see it?

[LeoMcA]

Thank you for your question and thanks for taking an interest in the community here on MDN. As we mentioned in the community call we tested the feature with users and have been actively receiving feedback from users since the feature launched. We are in the process of creating a clear strategy and communication channel for our community moving forward to be informed of and get involved in product work if they wish to.

If MDN cannot even seek feedback from its own maintainers before pushing major changes, how can the community trust them to ever have their feedback heard?

I’m not sure how the second part of this question relates to the first: seeking pre-emptive feedback from our content maintainers has nothing to do with responding to community feedback after the fact (which I hope you can see we have been doing, through the changes made to the feature, the community call we had, and now the answers to questions posed here).

The thread includes a number of overwhelming arguments against this feature and its inclusion, which would have easily been noticed if MDN had taken the step to request feedback beforehand.

We haven’t recently been sharing our product roadmap externally - we’re looking at ways in which we can share our plans for features more clearly in the future.

However, in this case, considering the bulk of negative feedback came from people (such as yourself) not actively involved in the MDN project, I’m not sure it would have made any difference - but that’s not a reason to not share our plans with the community, we received very constructive feedback from some of our most active community members over the past few weeks.

In fact, it seems that this feature was planned for inclusion, developed, and released without regard for its community during any portion of the process

We launched AI Help with a couple of feedback mechanisms: the thumbs up/down and survey link, intending to gather feedback from our community of both contributors and users.

We are actively doing this now: this is a beta feature, so we’re in a testing, evaluating and adjusting phase. The usage numbers and feedback we’ve received from logged in users has been very positive: out of all the features we’ve launched in the past couple of years, this one sees the second most usage after the Playground (which is especially impressive, considering the Playground is available to all users, while AI Help is only accessible by the vastly smaller number of MDN Plus users - around 0.5% of MDN’s audience).

And, after community feedback, we added an issue reporting link to allow us to more easily triage, improve upon poor answers, and keep track of them (rather than them getting lost in issue threads with hundreds of comments).

It's MDN's fault for completely failing to listen to the community here and to consider them when developing new features for MDN, and that's why so many people felt the need to express their concerns.

An extremely vocal small set of our community is not the entire MDN community. We thank you for your feedback, and concern, and we’re taking substantial portions of it on board.

Of course we considered the community when developing new features for MDN, that’s who we build features for. And this feature was built for a subset of our community not particularly represented on the issues discussing this feature, and whom many people commenting on the feature entirely forgot about: learners and those not yet capable of finding the correct information on MDN.

Some of the more useful feedback from the discussions is a fear that more junior developers might not be able to verify the information presented by AI Help, which is a question I’ve addressed here and here.

Just because you know how to use MDN, or you don’t find the feature useful, doesn’t mean nobody will. We have a duty to build features for our entire community. And from the people who used this feature and voted, more than 70% gave a positive review.

And, why was this done without community input in the first place? At this point, it's worth asking whether this change deliberately wasn't discussed before release because it would have been taken so negatively, and if that's the case, what do the people responsible have to say for that?

Because that’s how we’ve been approaching product decisions for a long time now: our recently launched Playground feature was done in the same way, and nobody’s been outraged about its development process. The same is true for Baseline, Offline, the improved sidebar filters, Collections, Updates, and the MDN Blog, just to name a few things we’ve launched over the past couple of years.

There are a few active things MDN can do to truly fix this problem:
Actively foster community discussions on what people want out of MDN, and what could be improved.

Our product team is currently reaching out to core community members inviting them to 1:1 interviews to conduct user research.

We’re also doing extensive user research with MDN’s users (beyond the community represented here) and capturing their feedback. And again, I remind you that we launched this feature in beta, to a limited subset of our audience, precisely to gather feedback from our users and our community.

You’re very welcome to open discussions here on GitHub around features you want added to MDN as well.

Actively seek feedback for possible solutions to presented problems, before development ever starts, and certainly before they're released.

Realistically, I’m not sure a product roadmap would be problem-oriented rather than solution-oriented, but you’re very welcome to give feedback once it’s public that you think a different solution would address the problem better.

I noticed that the remark on actually answering this question on the call was simply hidden by a maintainer. Does MDN simply not wish to answer this question, and prefer to pretend they didn't see it?

Hiding the comments was simply a matter of process: what number they had been recorded as doesn’t matter after the call. We’re answering them now.

[erisdev]

Just because you know how to use MDN, or you don’t find the feature useful, doesn’t mean nobody will. We have a duty to build features for our entire community.

Don't try to weaponize less experienced developers here. Multiple folks pointed out that the feature returns patently wrong answers. It's not a matter of me not finding it useful—it's matter of somebody I have never met and who does not have me to ask for clarification receiving wrong information and, having nothing else to go by, goes on to write incorrect code and then wonders why it isn't working. You are doing these folks a huge disservice and it's disgusting that you are trying to use them to support this asinine trend chasing.

Bad help is worse than no help. Bad help is what leads folks to decide they must just be bad at this whole computer thing and give up, because they followed the directions and nothing worked.

[clarfonthey]

If MDN cannot even seek feedback from its own maintainers before pushing major changes, how can the community trust them to ever have their feedback heard?

I’m not sure how the second part of this question relates to the first: seeking pre-emptive feedback from our content maintainers has nothing to do with responding to community feedback after the fact (which I hope you can see we have been doing, through the changes made to the feature, the community call we had, and now the answers to questions posed here).

You seem to have misunderstood my question. The goal is not to get feedback after the fact, but using community feedback before creating features to best tackle problems that the community would like to see solved. And so, these are both related: if you do not even give your own maintainers the option to provide feedback before creating features, how can the community trust to also have that option?

The thread includes a number of overwhelming arguments against this feature and its inclusion, which would have easily been noticed if MDN had taken the step to request feedback beforehand.

We haven’t recently been sharing our product roadmap externally - we’re looking at ways in which we can share our plans for features more clearly in the future.

Look no further than the original post here: I detailed a few key points, namely on fostering discussions with the community before making features. Sure, the technical details are unclear, but since MDN is more than willing to host community calls and other forums after creating features, surely they can simply host those same forums before.

However, in this case, considering the bulk of negative feedback came from people (such as yourself) not actively involved in the MDN project, I’m not sure it would have made any difference - but that’s not a reason to not share our plans with the community, we received very constructive feedback from some of our most active community members over the past few weeks.

This is a very sneaky argument. You just mentioned how MDN isn't actively sharing its roadmap externally, and so, I don't see any way people could get involved in development. I personally care a lot about MDN as a resource, but I don't see a way to get involved outside of GitHub discussions like the one that prompted this entire issue. That is exactly the point I'm arguing, and yet you're using it as a reason why there is no problem.

In fact, it seems that this feature was planned for inclusion, developed, and released without regard for its community during any portion of the process

We launched AI Help with a couple of feedback mechanisms: the thumbs up/down and survey link, intending to gather feedback from our community of both contributors and users.

Requesting feedback after launching something, even if you explicitly planned to launch the feature with such feedback mechanisms in place, is very different from incorporating feedback into development. Additionally, AI Help is a paid feature, and AI Explain is not, and a majority of MDN's users are not paying users.

We are actively doing this now: this is a beta feature, so we’re in a testing, evaluating and adjusting phase. The usage numbers and feedback we’ve received from logged in users has been very positive: out of all the features we’ve launched in the past couple of years, this one sees the second most usage after the Playground (which is especially impressive, considering the Playground is available to all users, while AI Help is only accessible by the vastly smaller number of MDN Plus users - around 0.5% of MDN’s audience).

And, after community feedback, we added an issue reporting link to allow us to more easily triage, improve upon poor answers, and keep track of them (rather than them getting lost in issue threads with hundreds of comments).

Again, we're talking about AI Explain too, which affects all users of MDN. Clearly you yourself know that the people who pay for MDN and those who don't are different people. In fact, the members of the community who can't afford any paid resources and use MDN for free are those most relevant to the features we're discussing, and that alone points toward AI Help being a horrible tool for gauging interest.

It's MDN's fault for completely failing to listen to the community here and to consider them when developing new features for MDN, and that's why so many people felt the need to express their concerns.

An extremely vocal small set of our community is not the entire MDN community. We thank you for your feedback, and concern, and we’re taking substantial portions of it on board.

Here's an analogy that might help. Imagine that Wikipedia launched an "AI Explain" feature that would request information from ChatGPT alongside articles. Over a thousand people write to them claiming that this feature is objectively harmful and propagating misinformation, including blatant lies and misleading (and wrong) explanations. It would be extremely misleading to imply that this 1000 people is a "vocal small set" of the users of Wikipedia, since so many people use the website. The only way you could claim this is if there were some proof that a substantially larger number of users both liked the feature and existed in a theoretical alternative reality where those arguments about it giving misinformation were somehow invalid, since again, people are not only arguing with their opinions on the matter, but objective facts.

You cannot pretend that you're taking in feedback when you send a majority of your time dismissing arguments instead of actually, seriously considering solutions. It is an indisputable fact, proven by evidence, that ChatGPT is incapable of doing what AI Explain sets out to do. And yet MDN are pretending like it's still a worthy cause and that those arguing this are simply a vocal minority.

Of course we considered the community when developing new features for MDN, that’s who we build features for. And this feature was built for a subset of our community not particularly represented on the issues discussing this feature, and whom many people commenting on the feature entirely forgot about: learners and those not yet capable of finding the correct information on MDN.

Of course? I don't think that you're allowed to say of course here, since I detailed pretty clearly why we have no reason to trust that MDN is actually considering the community when developing features. Of course MDN should be considering their features, for exactly the reasons you stated, but there is evidence that they are not, and you are supposed to be giving us a reason to believe those arguments. So far, I see none.

Some of the more useful feedback from the discussions is a fear that more junior developers might not be able to verify the information presented by AI Help, which is a question I’ve addressed here and here.

Just because you know how to use MDN, or you don’t find the feature useful, doesn’t mean nobody will. We have a duty to build features for our entire community. And from the people who used this feature and voted, more than 70% gave a positive review.

Again, you're using AI Help as a sample size which is clearly not a representative sample, as explained.

And, why was this done without community input in the first place? At this point, it's worth asking whether this change deliberately wasn't discussed before release because it would have been taken so negatively, and if that's the case, what do the people responsible have to say for that?

Because that’s how we’ve been approaching product decisions for a long time now: our recently launched Playground feature was done in the same way, and nobody’s been outraged about its development process. The same is true for Baseline, Offline, the improved sidebar filters, Collections, Updates, and the MDN Blog, just to name a few things we’ve launched over the past couple of years.

"That's how we've always done it" is such a horrendously bad argument to the point of being a common example of a bad argument. I don't doubt that MDN have released plenty of good features this way, since clearly, every feature has been done this way, for better or worse. Did it not strike anyone involved that working with the community more directly would be an important factor? Has anyone made any contributions toward ensuring this?

There are a few active things MDN can do to truly fix this problem:
Actively foster community discussions on what people want out of MDN, and what could be improved.

Our product team is currently reaching out to core community members inviting them to 1:1 interviews to conduct user research.

We’re also doing extensive user research with MDN’s users (beyond the community represented here) and capturing their feedback. And again, I remind you that we launched this feature in beta, to a limited subset of our audience, precisely to gather feedback from our users and our community.

Again, non-representative sample, etc. 1:1 interviews are also a powerful tool, but only if you're aware that those interviews are of a reprentative and diverse sample, which as shown, is unclear.

You’re very welcome to open discussions here on GitHub around features you want added to MDN as well.

I haven't proposed any features because I genuinely thought that MDN were doing a great job, before this. And I'm sure many people share that sentiment. People have only responded in such a strongly negative way because it's clear that MDN harbors contempt for those who have provided actual evidence for this feature being such a horrendously bad addition. We don't want an AI to explain issues. There are so many people invested in MDN that many of us would be willing to help write content ourselves. But MDN didn't ask for that, they simply implemented ChatGPT and did damage control when people found that it was lying to them.

Actively seek feedback for possible solutions to presented problems, before development ever starts, and certainly before they're released.

Realistically, I’m not sure a product roadmap would be problem-oriented rather than solution-oriented, but you’re very welcome to give feedback once it’s public that you think a different solution would address the problem better.

Certainly. As someone who has worked for several companies who had their own roadmaps, I understand first-hand how designing and presenting a roadmap comprises multiple full-time jobs. I would expect a roadmap to contain solutions, but I would also expect those solutions to be directly linked with problems or "user stories" if you like using synergetic business terms.

I noticed that the remark on actually answering this question on the call was simply hidden by a maintainer. Does MDN simply not wish to answer this question, and prefer to pretend they didn't see it?

Hiding the comments was simply a matter of process: what number they had been recorded as doesn’t matter after the call. We’re answering them now.

I will not fault you after the fact, but I would like to point out how terrible of a decision it was to hide the comment before answering the actual question. It makes sense to hide it given your current comment, since the original is useless, but without the presence of your commentary, it just looks like you weren't interested in answering it.

[clarfonthey]

Wow, and I thought I had problems being condescending.

I understand and share the frustration, but statements like this aren't really beneficial because they simply express frustration that has already clearly been expressed. It just bogs down the discussion and makes things harder to follow. Your mathematical analysis is very appreciated since it helps bolster arguments that I also made on representative samples, but I don't think that sprinkling in this extra stuff is very helpful.

And again, the analysis is very lovely. Thank you for that. <3

[nic-hartley]

Of course we considered the community when developing new features for MDN, that’s who we build features for. And this feature was built for a subset of our community not particularly represented on the issues discussing this feature, and whom many people commenting on the feature entirely forgot about: learners and those not yet capable of finding the correct information on MDN.

I'd like to harp on this for a moment with an anecdote.

Ask basically anyone who's been in webdev for a while and they'll tell you W3Schools is bad. It's unreliable, it's poorly written, it's not complete, etc. And yet, ask anyone to point to a single, specific, obvious error, and they usually can't. That's because for the longest time, W3Schools wasn't obviously wrong -- it was, say, 95% correct, with a few misleading or incorrect sentences in the rest of the page. It made small errors in minor facts, or slightly incorrectly documented some syntax. And yet despite each of these errors being relatively minor, it still garnered a reputation as absolutely awful, because the result of your documentation being 95% wrong is that on any complex enough project, you'll encounter a bug where your code does one thing, but the docs say it should do another.

As a learner, that's incredibly frustrating. Especially when you're not good at debugging, just understanding how to get to the problem can be frustrating, and when you trust a piece of documentation, realizing it's wrong can take a lot of work. We told learners to avoid W3Schools, and to use MDN, precisely because it's not slightly wrong. It can be overly technical, with each individual page making little sense on its own, especially for more obscure features, but it's at least correct. You might have to slog through five pages of wordy and dense explanation to understand flexboxes, but when you have a bug and refer back to the docs, you can simply trust that they're correct.

But then MDN introduced AI Help. And suddenly that's not a guarantee anymore. Suddenly, on the MDN site itself, there's a button promising to explain a piece of code, which often gets it wrong. Usually in minor ways, sure, but that's worse. When someone says something wrong, but they're totally confident about it, it's hard to notice their mistake, especially when it's a source you trust. Even if they're completely bass-ackwards about it, when you're learning, you don't know what the truth is, so you just take what they said as true, and then rely on it, and then tear your hair out for a week trying to figure out why these friggin' divs won't line up nicely.

The problem here isn't that "many people commenting on the feature entirely forgot about learners", it's precisely the opposite. This feature is actively harmful to learning, and is actively harmful to finding your way around MDN. It's misleading, unhelpful, or just plain outright wrong, far too much of the time. And worst of all, when it lies to your face, it does it in such confident English, on such a reputable website, that you think surely it must be right.

[DavidJCobb]

I take it Mozilla isn't ever going to give a response that isn't slimy and nakedly dishonest, then?

[erisdev]

Wild how many folks have brought up the fact that this misfeature doles out mistruths with absolute confidence, yet nobody in charge here is willing to actually address it. Are we all standing in a monumental blind spot, or is it simply cruel indifference? Or could it be time to start investigating potential connections between the folks at Mozilla and the charlatans who are hawking this crap? Smells like somebody in charge has a personal stake.