avoid some redundant checks in TokenList::readfile()

[firewave]

calling isspace() is unnecessary since we handle all non-ASCII characters and already have special handling of all remaining whitespaces according to https://en.cppreference.com/w/cpp/string/byte/isspace.

Runinng with -q -D__GNUC__ -Ilib lib/valueflow.cpp

Clang 16 276,826,655 -> 270,642,840
GCC 13 283,762,095-> 280,179,302

[firewave]

I filed/updated several tickets about issues I discovered because of this:
llvm/llvm-project#67824
https://trac.cppcheck.net/ticket/7510
https://trac.cppcheck.net/ticket/12023
https://trac.cppcheck.net/ticket/12024
https://trac.cppcheck.net/ticket/12025

[danmar]

hmm I am not sure. The compiler knows the ascii codes. But if we assume the developer does not.. isn't it more explicit that '\t' '\n' '\r' are handled using std::isspace(ch) than ch <= ' '.

[firewave]

We basically separate the ASCII table is three parts with our code:

>= 0x80 - non-ASCII characters we bail out on

< ' ' - whitespaces and control characters which we treat as allowed and converted to ' '

... and the printable ASCII characters

So we only have either ' ' or a printable ASCII so the only character isspace() can match is ' '. So we get rid of the conversion step and simply exclude the lower part of the table.

[danmar]

but with these changes you assume that < ' ' will match \t and \r and \n ... technically it does. but if we pretend it didn't then your changes would break things.

on the other hand we did assume that < ' ' would match all other control characters already.

I don't have a strong negative opinion about this .. just wanted to point out that there might be a good reason to be explicit. It extends to the tickets you wrote.. technically there are some false negative but I am skeptic about Cppcheck warnings.

[firewave]

I don't have a strong negative opinion about this .. just wanted to point out that there might be a good reason to be explicit

I think the code is explicit enough since we are limiting us to the printable ASCII characters. If we weren't ìsspace() would have been necessary.