fgets() sometimes does not read a full line

[sboden]

Description

fgets() sometimes does not read a full line.

I'm reading old style fixed format ASCII files using PHP 8.0.26 (from the official docker image). The file only contains [a-zA-Z0-9], spaces and newlines. Every line in the file is exactly 559 characters long. The file gets downloaded to a local file system before the file is opened.

<?php
          ...
          if ($file = fopen($real_directory . '/' . $content, "r")) {
            $line_nr = 0;
            while (!feof($file)) {
              $line_nr++;
              $line = rtrim(fgets($file));
              // Get some fields from $line using substr().
            }
            fclose($file);
          }

Once every about 300K lines a line is not completely read into $line, and the substr() fails to get the data I expect: at a specific location in each line there's either the keyword 'IMPORTED' or 'REJECTED'. If that happens the code stops (by intention) as something would be wrong with the input file. Afterwards when I look at the "corrupted" file, the file is completely normal: no escape characters, nothing out of the ordinary. When I reprocess the file unchanged, it goes through without complaining (finding for the line with the latter problem the proper keyword).

I see it happening (although seldom), and I can't replicate it consistently.

PHP Version

PHP 8.0.26

Operating System

Debian GNU/Linux 11 (bullseye)

[kocoten1992]

Have you try it on a different computer? That sound like memory issue to me.

[sboden]

It's running in a docker container on AWS. 30 minutes after the last occurrence, I restarted it and then it passed without problems.

The program is not doing that much, it's a PHP cron job that closes the payments in its input files by setting a field in MySql: the primary key of the payment to close is in each line of the file as well as the IMPORTED/REJECTED status.

[Girgias]

How big is the file? It may be an issue with how the stream wrapper pulls data into the memory buffer.
Is the file also being written to when this processing happens?

[sboden]

It's an input file only, the file contents is not being changed while the reading occurs (the same job is sftp'ing the files to the local filesystem before the file is opened). Most of the input files are +-5Mb, up to 50Mb once a week.

The problem happens infrequently, but it does feel "dirty": I can manually verify that the file is correct but PHP somehow fails to read it correctly some of the time.

[Girgias]

Okay, let me see if I can mimic this situation.

[Girgias]

Are you sure there is not a trailling new line at the end of the file which gets fetched as an empty line?

[sboden]

Are you sure there is not a trailling new line at the end of the file which gets fetched as an empty line?

No empty lines, the instances I've seen are somewhere in the middle of a file. Hexdump does not show any invisible characters around the offending line (or anywhere else in the file for that matter). And the file is processed properly when the script is run again (without changing the input file).

It feels like some kind of memory corruption and I can't replicate it consistently. The script reads a file and updates a status field in a database using Symfony. The code is about 5 years old, pre PHP 8.0.x I've never seen the problem happen. The corruption could also be in other parts, but the effect I see is that I do an fgets() on a file expecting data in a certain place in the line and that data is not read, while visually inspecting the file the data is really there. And rerunning the script processes the data properly.

[Girgias]

Do you have opcache and/or the JIT engine enabled? Is it a long-running process or fired every time when a batch needs handling?

[sboden]

Do you have opcache and/or the JIT engine enabled? Is it a long-running process or fired every time when a batch needs handling?
No opcache, no JIT. Opcache interferes with AJAX in other parts of the application (but that's another problem). The job runs once every days at a set time, at which it will download and process the files it finds on an sftp server.

[Girgias]

Well, I'm out of ideas then as to why this might happen, @bukka would you maybe have an idea as to what could cause a stream to not load all of the content?