User confusion because warnings are hidden in gallery examples for stable docs

[ayshih]

No doubt there was a good reason to do this, but it can be confusing to users that warnings are hidden in the gallery examples for stable docs (was done in #5279):

sunpy/docs/conf.py

Lines 59 to 61 in bc8f08f

	# We want to ignore all warnings in a release version.
	if is_release:
	warnings.simplefilter("ignore")

That means when a user runs the example on their own machine, and they see warning messages that do not appear on the rendered example, the natural reaction is that the user thinks something is not set up properly on their end.

Can we stop hiding the warnings?

[alasdairwilson]

Do we need explanations in examples with expected warnings? callout boxes?

[nabobalis]

I think we should consider a topic guide (?) on this we would link to.

Kind of like the, how to fix metadata url we point to when map raises those.

[alasdairwilson]

Thinking about this more, what does a user think when they see lots of verbose warnings in example code, when I have seen stuff like that I have assumed that examples have not kept up with changes to the code and may not work correctly anymore.

[nabobalis]

This was our original intention behind hiding the warnings in the examples.

But it has caused confusion as to why the example output is different from the online build and a local run.

[ayshih]

We should be careful to not conflate different purposes for examples. If the example is a tutorial, the rendered example should show exactly what a user should see. If the example is an advertisement, we can feel free to hide whatever we want.

Of course, we should strive to minimize the number of warnings that show up in our tutorials. Having warnings show up in our tutorials and not supporting them with an explanation can contribute to warning fatigue.

[alasdairwilson]

I don't if it really is causing huge confusion.

If we have an example, we presumably are trying to teach the user some lesson. This is sidetracked if that example then emits a warning; either we have an unexplained warning in the example which makes the code look buggy or, we have to take time out of the example to say "oh by the way the reason why there is a warning of no observer passed into..." and suddenly the lesson the example was supposed to impart is lost.

I think the logic is that the warnings in our examples are "expected warnings", because the people that wrote them should understand why there is a warning, and so suppressing them makes sense to me.

[ayshih]

I disagree with basically everything you wrote.

I don't if it really is causing huge confusion.

It isn't causing huge confusion, but it causes significantly more confusion than the benefit it brings.

If we have an example, we presumably are trying to teach the user some lesson.

The purpose of a tutorial is to teach by doing. The users are intended to run the shown code and get the shown results. If they get warnings locally that are not shown in docs, it is natural for them to conclude that something is not correct in their setup.

This is sidetracked if that example then emits a warning; either we have an unexplained warning in the example which makes the code look buggy

If the warnings make the code look buggy, we need to fix the code!

we have to take time out of the example to say "oh by the way the reason why there is a warning of no observer passed into..." and suddenly the lesson the example was supposed to impart is lost.

Per diátaxis, we should refrain from going into such explanations in a tutorial anyway. It's sufficient and appropriate to say something like "(the warnings are expected)".

I think the logic is that the warnings in our examples are "expected warnings", because the people that wrote them should understand why there is a warning, and so suppressing them makes sense to me.

Er, the tutorial is for the benefit of the novice user, so it doesn't matter if the developer expected the warning. We need to make the user aware that the warning is expected.

---

I maintain that tutorial examples need to show exactly what a user should see in their own setup. There are multiple ways to address warnings, in order of increasing effort:

• Acknowledge them so that the user can move on unalarmed
• Explicitly filter out warnings in the example
• "Fix" the underlying code to stop the warning

The solution should not be to hide the warnings in our rendered docs. The user is going to see the warnings anyway when they actually run the code, so the facade of clean output will immediately crumble, resulting in user confusion.

[ayshih]

The only argument I can think of for hiding warnings is if the warnings were primarily coming from an external package. Since we don't continually re-render the docs for releases, there's the potential for externally generated warnings to become stale, and then there'd be a chance for confusion of a different kind. Most if not all of the warnings in our gallery are from the core library, so I don't think that's a good argument for ourselves, although it might be a good argument for an affiliated package.