Fix the infinite loop when an input file lacks EOF

[Qining]

The input scanner can be trapped in an infinite loop if the given input
file does not have EOF (and is not ended with a 'whitespace').

The problem is caused by unget(), which keeps rolling back the scanner
pointer without hitting an EOF at the end of the file. This makes getch()
function keep returning the last character of the file and never ends,
and the effect of advance() is always counteracted by unget().

[johnkslang]

Doesn't this mean there was a get that returned EndOfInput, but that result was ignored? If so, this may be getting fixed in the wrong place.

[Qining]

Actually I'd say yes. In the switch/case statements of preprocessor/PpScanner.cpp, int TPpContext::tStringInput::scan(TPpToken* ppToken), the general pattern is like:
ch = getch();
loop if ch do not change state,
if ch can trigger state change, then ungetch().

The problem is the ungetch() call, which should check if (ch != EndOfFile) before calling it.

I was thinking whether I should add this if (ch !=...) to all the call sites of ungetch(). But in this initial commit, I plummeted the check to a lower layer. Maybe having the if statements in scan() is more legible?

Suggestion?

[Qining]

Hi @johnkslang , should I add if (ch != EndOfFile) before all caller sites of ungetch() or keep this CL in this way?

[Qining]

Hi @johnkslang ,

Is this PR ready to go?

Thanks

[dneto0]

The proposed solution seems quite reasonable to me. But I think we need to update the descriptions of the get(), peek(), ungetch() to clarify their end-of-input edge case behaviours.

[johnkslang]

I'm fine with it as long as it's truly a catchall for higher-level code that doesn't always, but eventually checks for end-of-input.

[Qining]

will work on the comments soon.