doxywizard freezes when large amounts of command line output is produced

[SamVanheer]

Describe the bug
When the doxygen command line program produces a lot of command line output the doxywizard program freezes.

This issue was mentioned in #9894 but that's closed so i'm reporting this separately with more information.

Expected behavior
The GUI stays responsive, showing command line output as it is produced until the program has finished.

To Reproduce
Running doxygen on a large project should produce sufficient output to cause the problem. The project where i encountered this problem has 500 source files with roughly 140000 lines of code. Alternatively modifying doxygen itself to produce more output should show the problem as well.

Version
1.9.7 (a2f34e1ed90f83713dc347b5134920f32c455d46)
The current Git master also has this problem.
Platform is Windows 10 64 bit.

Additional context
This happens because doxywizard uses an expensive method to format command line output:

void MainWindow::readStdout()
{
  if (m_running)
  {
    QByteArray data = m_runProcess->readAllStandardOutput();
    QString text = QString::fromUtf8(data);
    if (!text.isEmpty())
    {
      text1 += text;
      m_outputLog->clear();
      m_outputLog->append(APPQT(text1.toHtmlEscaped().trimmed()));
    }
  }
}

m_outputLog is a QTextEdit*. Calling clear and then adding the entire current log results in the wizard spending more and more time formatting the output. Eventually the program takes longer to format the text than it takes for more output to be produced, resulting in the wizard spending all of its time either formatting text or appending it to the text edit.

The symptoms first appear as the text edit becoming blank (due to calling clear) before the program freezes altogether.

The fix described in the issue linked above does not account for edge cases where the wizard reads part of a line resulting in additional line breaks being inserted.

This fix correctly handles output by appending it only if a whole line was read:

void MainWindow::readStdout()
{
  if (m_running)
  {
    QByteArray data = m_runProcess->readAllStandardOutput();
    QString text = QString::fromUtf8(data);
    if (!text.isEmpty())
    {
      text1 += text;
      //m_outputLog->clear();

      // Handle the case where we've read part of a line of text.
      // We'll append the line when we get the newline delimiting it.
      const int lastNewLine = text1.lastIndexOf(QChar::LineFeed);

      if (lastNewLine != -1)
      {
        if (lastNewLine != (text1.size() - 1))
        {
          m_outputLog->append(APPQT(text1.mid(0, lastNewLine).toHtmlEscaped().trimmed()));
          text1 = text1.mid(lastNewLine + 1);
        }
        else
        {
          m_outputLog->append(APPQT(text1.toHtmlEscaped().trimmed()));
          text1 = QString::fromLatin1("");
        }
      }
    }
  }
}

This fixes the freeze issues and keeps the UI responsive without noticeable slowdowns. It also reduces memory usage since text1 no longer contains all of the command line output.

Note that this code does not account for platform-specific line endings and does not handle edge cases where doxygen finishes before all output has been read.

There may be better ways to handle the logging of output from QProcess that don't involve using HTML which could simplify this solution to just appending the text that's been read.

[albert-github]

It looks like the entire output process dumps the information a little bit less "blocked" but I see one major problem i.e. that there are sometimes some extra newlines in the shown output like:

whilst this normally should look more like:

I ran the doxywizard on the internal documentation of doxygen

Edit the output obtained from "Save log" doesn't have these empty lines.

[SamVanheer]

QTextEdit puts the text for each append call into its own block, and there are newlines between blocks. The original code wraps all of the log output in a single set of tags so the newlines are preserved. Appending text line by line wraps each line with the tags.

Appending everything as plain text (like this) fixes the issue but changes the text formatting. You can probably configure it to use the same font settings to match the look and feel and wrap the saved log with the tags to get the same HTML log as before.

It might be a good idea to use QPlainTextEdit instead since it can handle large amounts of text better. Despite the name it does support HTML.

[albert-github]

Thanks for the link to the stack overflow answer (https://stackoverflow.com/questions/13559990/how-to-append-text-to-qplaintextedit-without-adding-newline-and-keep-scroll-at/18134824#18134824) this looks good.

I've just pushed a proposed patch, pull request #10118

[SamVanheer]

Thanks, i can confirm that those changes work as expected. Note that because the cursor is changed to the end it is not possible to scroll the output while doxygen is still working.

[albert-github]

Code has been integrated in master on GitHub (please don't close the issue as this will be done at the moment of an official release).

[albert-github]

Regarding the scrolling (like mentioned in #10115 (comment)) I didn't find a solution. so suggestions are welcome (and I also created https://stackoverflow.com/questions/76430485/qtextbrowser-scroll-through-text-whilst-new-text-is-still-being-added)

[albert-github]

Regarding the scroll problem, I got an answer and this looks like to fix the problem.
I've just pushed a proposed patch, pull request #10119

[SamVanheer]

I can confirm that the fix works. I haven't encountered any other problems.