Reduce STDERR output during CI

[hangy]

Description: The default test output is reduced by only logging error or above to the TAP. If required for debugging, the TAP_LOG_FILTER environment variable can be set to a different level to override it.
Related issues and discussion: #3125

[teolemon]

fixed a little merge conflict

[stephanegigandet]

@hangy I'm not sure why, but all the traces for the script convert_auchan_data.pl now go to STDERR and it makes the test fail.

RangeError [ERR_CHILD_PROCESS_STDIO_MAXBUFFER]: stderr maxBuffer length exceeded
    at Socket.onChildStderr (child_process.js:386:14)
    at Socket.emit (events.js:311:20)
    at addChunk (_stream_readable.js:294:12)
    at readableAddChunk (_stream_readable.js:271:13)
    at Socket.Readable.push (_stream_readable.js:209:10)
    at Pipe.onStreamRead (internal/stream_base_commons.js:186:23) {
  code: 'ERR_CHILD_PROCESS_STDIO_MAXBUFFER',
  cmd: 'perl -c -CS -Ilib scripts/convert_auchan_data.pl',
  stdout: '',

[hangy]

@hangy I'm not sure why, but all the traces for the script convert_auchan_data.pl now go to STDERR and it makes the test fail.

RangeError [ERR_CHILD_PROCESS_STDIO_MAXBUFFER]: stderr maxBuffer length exceeded
    at Socket.onChildStderr (child_process.js:386:14)
    at Socket.emit (events.js:311:20)
    at addChunk (_stream_readable.js:294:12)
    at readableAddChunk (_stream_readable.js:271:13)
    at Socket.Readable.push (_stream_readable.js:209:10)
    at Pipe.onStreamRead (internal/stream_base_commons.js:186:23) {
  code: 'ERR_CHILD_PROCESS_STDIO_MAXBUFFER',
  cmd: 'perl -c -CS -Ilib scripts/convert_auchan_data.pl',
  stdout: '',

The script probably never worked before. $lc isn't defined in that line:

openfoodfacts-server/scripts/convert_auchan_data.pl

Line 59 in 7aaeb17

$lc = 'fr';

As to why it never showed an error during CI, I do not know.

[stephanegigandet]

@hangy : with those changes, will we be testing many more things than before? e.g. all the scripts in /scripts maybe ?

"Failing after 131m " --> it will be very problematic if tests take 2 hours.

[hangy]

@hangy : with those changes, will we be testing many more things than before? e.g. all the scripts in /scripts maybe ?

No, not really. We're compiling the scripts in master, too: https://github.com/openfoodfacts/openfoodfacts-server/blob/46b6657/.github/workflows/pull_request.yml#L84
There might be some hidden issue that makes it appear like everything is okay, but I did intent to ensure that scripts compile, as well.

"Failing after 131m " --> it will be very problematic if tests take 2 hours.

Agreed. That's much too long. Simply checking the Perl syntax/compiling the scripts really shouldn't take that long, though. So there is probably some hidden issue that causes too much data to be processed during compilation. I'll have a look ...

[stephanegigandet]

I'm not sure what's the status of this PR. Is it possible to break it in multiple parts?
There seem to be numerous improvements that woul be useful to merge.
But it also makes the tests fail because they are too long, so we can't merge it as is.

[hangy]

@stephanegigandet currently, I have no idea what might be causing the build to time out. 🤷‍♂️ I'll look into splitting it up, since some of the changes should make sense on their own.

[stephanegigandet]

Looks good. I had misunderstood how to use the TAP_LOG_FILTER variable from the comment line. I think I'll add that to a readme file in the t folder. :)

[VaiTon]

@hangy tests failing

[stephanegigandet]

t/producers.t (Wstat: 512 Tests: 0 Failed: 0)
Non-zero exit status: 2
Parse errors: No plan found in TAP output

-> that's weird, I don't understand why it fails.

[hangy]

-> that's weird, I don't understand why it fails.

Me neither. :) I didn't think I modified the files in that way...

[VaiTon]

@hangy do you still want to merge this?