Create checks to place violations on missing macros in templates

[stoyanK7]

From #13643 (comment)

Can we create a regexp module to place violations on tags in templates, or something similar to catch these?

We need a check to ensure that no examples are shown in templates without being tested. For that, we can place violations on source tags in templates.

[nrmancuso]

For now, it is probably better to get something in place, fix issues, and have manual suppressions.

[stoyanK7]

I'd say this check needs to target only source tags in the Examples section. There are numerous places where we have source tags in descriptions and notes.

[nrmancuso]

I'd say this check needs to target only source tags in the Examples section. There are numerous places where we have source tags in descriptions and notes.

I do not know of some simple way to do this, I think we can do #13682 (comment) and be in good shape for now. We can improve in the future. I am approving this issue simple implementation at #13682 (comment) in mind.

[nrmancuso]

Removing approved label after some research and consideration.

[nrmancuso]

We currently have no checks on xml template files at all.

1. Should we create a new checkstyle run/config for these?
2. What other checks should we have on template files?
3. Would it be easier/better to check for the presence of <macro name="example">?
4. Should we create checks for each macro in every template file?
   @rnveach @stoyanK7 @romani share your thoughts here, let's get this in place ASAP

[rnveach]

The cache files cache_non_main_files and cachefile (this last one has no templates listed) says those 2 run against all our xdocs. So if we create a new config, we should check that we aren't duplicating what those do already.

Since the xdocs are XML files, I don't see what else we can do besides relying on a bunch of regexps for anything extra which won't be fully reliable.

I am fine with whatever regexps to create.

[romani]

We can have all such validation, but I would rather focus on this at the end, it will be hard to send PR without macroses usage, so easy to spot, so hard to make because as there will be no examples on how to do this :).

[nrmancuso]

@romani macros will do no good if we do not update templates. We need to have validation in place for existence of macro placement in template where we expect them.

[romani]

I agree, but I would rather focus on migration of all and if any problems happens focus on issues to resolve. And the end catch leftovers by such validation.

[nrmancuso]

Then we will run into issues in student PRs when we are missing macros in templates, this will not be a good experience for them or maintainers.

[romani]

I just say, better to do this after we finish 80-90% migrations. But if somebody can do this earlier it is ok.

[nrmancuso]

I am on this