Limit the number of reported warnings

[hubertp]

Pull Request Description

Artifically limiting the number of reported warnings to 100. Also added benchmarks with random Ints to investigate perf issues when dealing with warnings (future task).
Ideally we would have a custom set-like collection that allows us internally to specify a maximal number of elements. But EnsoHashMap (and potentially EnsoSet) are still WIP when it comes to being PE-friendly.

The change also allows for checking if the limit for the number of reported warnings has been reached. It will visualize by adding an additional "Warnings limit reached." to the visualization.

The limit is configurable via --warnings-limit parameter to run.

Closes #6283.

Checklist

Please ensure that the following checklist has been satisfied before submitting the PR:

• All code follows the
  Scala,
  Java,
  and
  Rust
  style guides. In case you are using a language not listed above, follow the Rust style guide.
• All code has been tested:
  • Unit tests have been written where possible.

[radeusgd]

This seems like just a wrong solution to this problem.

We will be losing warnings that may be important through this. Imagine that a Vector.sort returns 1000 warnings about comparison but some other operation returns 1 very important warning about an encoding issue. With this, we will get the first 100 warnings all being the not so helpful Vector.sort issue and the encoding warning will become swallowed.

The idea of the ticket #6283 was to limit reported warnings for Vector.sort specifically. I think every function we have that runs risk of reporting too many warnings should have some way of its own to limit this number. An automatic solution is just very prone to data loss.

If we still want to limit the number of warnings globally, let's just ensure we attach an artificial warning which will indicate that additional warnings were hidden - so that the user will at least know that there are some hidden warnings - without it we are losing data that the user should see.

[radeusgd]

The idea of the ticket #6283 was to limit reported warnings for Vector.sort specifically. I think every function we have that runs risk of reporting too many warnings should have some way of its own to limit this number. An automatic solution is just very prone to data loss.

And indeed, it may be desirable to have some semi-automatic way to do this, as probably many methods on Vector could do this - like map or filter.

So maybe some global limit could be needed after all. But I still think the Vector.sort should have its specific limit (otherwise it will tend to uselessly saturate the global limit and any other warnings will always be swallowed - the issue is that Vector.sort can very often enter a state where it will emit LOTS of warnings, for stuff like map it should be a bit rarer).

But still, I think we absolutely have to have an indicator that 'there are more warnings which are hidden'.

[hubertp]

Yes, we can have "data loss" but do you imagine people scrolling through 100 warnings because 101th was "the important one" ?
It is similar to what compiler is doing; it is not reporting more than X warnings/errors because at some point it becomes just impossible to digest.

[radeusgd]

Yes, we can have "data loss" but do you imagine people scrolling through 100 warnings because 101th was "the important one" ? It is similar to what compiler is doing; it is not reporting more than X warnings/errors because at some point it becomes just impossible to digest.

Fair. But we should indicate that there are more warnings that got swallowed - so that the user know that there is something more.

And ideally, IMO we should limit the amount of warnings coming from the Vector.sort separately, to some lower value like even first 10. Otherwise a single Vector.sort will saturate all warnings and nothing else will show up in nodes that depend on that result.

[hubertp]

Fair. But we should indicate that there are more warnings that got swallowed - so that the user know that there is something more.

That can be added, yes.

And ideally, IMO we should limit the amount of warnings coming from the Vector.sort separately, to some lower value like even first 10. Otherwise a single Vector.sort will saturate all warnings and nothing else will show up in nodes that depend on that result.

Can be added as well.

[JaroslavTulach]

I'd like us to eliminate @TruffleBoundary one day...

[JaroslavTulach]

We will be losing warnings that may be important through this.

That's completely OK. Nobody will scan more than 100 warnings unless really, really interested in. javac also limits number of errors to 100 by default and I haven't yet see a developer to complain. Having an option to tune the number of kept warnings would allow users to re-run the application and see more warnings. Although the right approach is to eliminate the warnings, I guess.

we should indicate that there are more warnings that got swallowed

+1 one to that.

[Akirathan]

Is the warmup sufficient? I have a minor suggestion. You don't have to do this as I plan to do that in the future for all the benchmarks, but it would help with the stability of these benchmarks.

Modify the context to include an option for -D.engine.TraceCompilationDetails=true and check whether the output contains any "[engine]" logs, i.e., whether there are any Graal/Truffle compilations going on in the measurement phase. I guess it is enough to redirect the output to the stdout and just look when did the warmup phase ended and whether there are some logs from Graal. If you struggle with that too much, don't do that.

[Akirathan]

An inspiration how to do that easily is here: #6271 (comment)

[hubertp]

Sorry, I missed that comment.

[hubertp]