Doxywizard truncates standard input passed to doxygen process.

[nitz]

Describe the bug
When using DOT_IMAGE_FORMAT = svg, DOT_TRANSPARENT = YES does not have any effect. dot outputs a svg with a solid object with a non-transparent background.

Expected behavior
The generated svg background object should either be transparent or not exist.

Screenshots

To Reproduce
dot-transparent-ineffective-svg.zip
Minimal example attached. To see affected output, run doxygen, then open docs/html/group__a.html. The generated docs use a custom css that colors the background of .contents divs purple, so you can see the white background on the generated SVG, as shown in the screenshot above.

Version
1.9.3 (c0b9eafbfb53286ce31e75e2b6c976ee4d345473), on x64 Windows 10 Pro 21H2 (19044.1526)

Stack trace
N/A

Additional context
Due to the way SVGs are included in the HTML output as an iframe, styling with CSS causes the non-transparent background to be ineffective. The dot file generated doesn't set a bgcolor attribute on the entire graph, though I do see it trying to in the code. As an example, here:

doxygen/src/dotgroupcollaboration.cpp

Lines 317 to 340 in e18f715

	void DotGroupCollaboration::writeGraphHeader(TextStream &t,const QCString &title) const
	{
	int fontSize = Config_getInt(DOT_FONTSIZE);
	QCString fontName = Config_getString(DOT_FONTNAME);
	t << "digraph ";
	if (title.isEmpty())
	{
	t << "\"Dot Graph\"";
	}
	else
	{
	t << "\"" << convertToXML(title) << "\"";
	}
	t << "\n";
	t << "{\n";
	if (Config_getBool(DOT_TRANSPARENT))
	{
	t << " bgcolor=\"transparent\";\n";
	}
	t << " edge [fontname=\"" << fontName << "\",fontsize=\"" << fontSize << "\","
	"labelfontname=\"" << fontName << "\",labelfontsize=\"" << fontSize << "\"];\n";
	t << " node [fontname=\"" << fontName << "\",fontsize=\"" << fontSize << "\",shape=box];\n";
	t << " rankdir=LR;\n";
	}

The SVG XML produced from the dot without the bgcolor attribute with the creates a polygon with a white fill:

<!--snipped-->
<polygon fill="white" stroke="transparent" points="-4,4 -4,-23 146,-23 146,4 -4,4"/>
<!--snipped-->

Manually removing that <polygon /> produces the desired outcome:

If I manually edit a dot file, to add the attribute:

--- group__a.dot        Thu Feb 17 08:59:12 2022
+++ group__a-modified.dot       Thu Feb 17 08:59:13 2022
@@ -1,5 +1,6 @@
 digraph "Group A"
 {
+  bgcolor="transparent"
   edge [fontname="Helvetica",fontsize="10",labelfontname="Helvetica",labelfontsize="10"];
   node [fontname="Helvetica",fontsize="10",shape=box];
   rankdir=LR;

The <polygon /> isn't created in the output.

The PR #8826 looks like it might be a potential fix for this, as the DOT_COMMON_ATTR it adds looks like it would be a place to specify bgcolor="transparent", but it is my feeling that the DOT_TRANSPARENT still should create that attribute itself, but for some reason isn't.

Cheers!

[albert-github]

Which version of dot do you use?

I tried with dot versions 2.38.0 and 2.40.1 and in both cases I see:

Regarding the purple backgroud of "Group B" this is another, different but fundamental, question. Probably this should be explicitly set to a color (white).

[nitz]

I'm using the dot that came with the latest stable win-x64 binaries from Graphviz:

C:\Users\cmd>dot -v
dot - graphviz version 2.50.0 (20211204.2007)
libdir = "C:\Program Files\Graphviz\bin"
Activated plugin library: gvplugin_dot_layout.dll
Using layout: dot:dot_layout
Activated plugin library: gvplugin_core.dll
Using render: dot:core
Using device: dot:dot:core
The plugin configuration file:
        C:\Program Files\Graphviz\bin\config6
                was successfully loaded.
    render      :  cairo dot dot_json fig gdiplus json json0 map mp pic pov ps svg tk vml xdot xdot_json
    layout      :  circo dot fdp neato nop nop1 nop2 osage patchwork sfdp twopi
    textlayout  :  textlayout
    device      :  bmp canon cmap cmapx cmapx_np dot dot_json emf emfplus eps fig gif gv imap imap_np ismap jpe jpeg jpg json json0 metafile mp pdf pic plain plain-ext png pov ps ps2 svg tif tiff tk vml xdot xdot1.2 xdot1.4 xdot_json
    loadimage   :  (lib) bmp eps gif jpe jpeg jpg png ps svg

Took a peek through Graphviz's changelog, but at a quick glance I don't see anything that looks like like a change that describes what we're seeing. I'm betting something must have changed somewhere along the line to produce the different behavior.

I'll see what an earlier version of dot/Graphviz does for me.

[nitz]

The earliest binaries directly from Graphviz's download page were for 2.44.1:

C:\Users\cmd>dot -v
dot - graphviz version 2.44.1 (20200629.0800)
libdir = "C:\Program Files\Graphviz\bin"
Activated plugin library: gvplugin_dot_layout.dll
Using layout: dot:dot_layout
Activated plugin library: gvplugin_core.dll
Using render: dot:core
Using device: dot:dot:core
The plugin configuration file:
        C:\Program Files\Graphviz\bin\config6
                was successfully loaded.
    render      :  cairo dot dot_json fig gd gdiplus json json0 map mp pic pov ps svg tk vml vrml xdot xdot_json
    layout      :  circo dot fdp neato nop nop1 nop2 osage patchwork sfdp twopi
    textlayout  :  textlayout
    device      :  bmp canon cmap cmapx cmapx_np dot dot_json emf emfplus eps fig gd gd2 gif gv imap imap_np ismap jpe jpeg jpg json json0 metafile mp pdf pic plain plain-ext png pov ps ps2 svg svgz tif tiff tk vml vmlz vrml wbmp xdot xdot1.2 xdot1.4 xdot_json
    loadimage   :  (lib) bmp eps gd gd2 gif jpe jpeg jpg png ps svg

And the output is the same:

I found a copy of 2.38.0 in their archive:

C:\Users\cmd>dot -v
dot - graphviz version 2.38.0 (20140413.2041)
libdir = "C:\Program Files\Graphviz\bin"
Activated plugin library: gvplugin_dot_layout.dll
Using layout: dot:dot_layout
Activated plugin library: gvplugin_core.dll
Using render: dot:core
Using device: dot:dot:core
The plugin configuration file:
        C:\Program Files\Graphviz\bin\config6
                was successfully loaded.
    render      :  cairo dot fig gd gdiplus map pic pov ps svg tk vml vrml xdot
    layout      :  circo dot fdp neato nop nop1 nop2 osage patchwork sfdp twopi
    textlayout  :  textlayout
    device      :  bmp canon cmap cmapx cmapx_np dot emf emfplus eps fig gd gd2 gif gv imap imap_np ismap jpe jpeg jpg metafile pdf pic plain plain-ext png pov ps ps2 svg svgz tif tiff tk vml vmlz vrml wbmp xdot xdot1.2 xdot1.4
    loadimage   :  (lib) bmp eps gd gd2 gif jpe jpeg jpg png ps svg

But I'm still seeing the same output.

Is perhaps the issue I'm running into here DOT_TRANSPARENT is perhaps not being correctly read here? Using DOT_CLEANUP = NO only seems to work if I introduce something that prevents dot from running successfully (I have just been sticking a " in the font name,) but when I do that I don't see the bgcolor="transparent line in any of the .dot files generated.

I don't have a development environment setup for doxygen, but if I can do that I'll run with a debugger attached and see if that line isn't executing as I think it should be.

[albert-github]

Did you, between runs, remove the html directory as doxygen caches the dot files (you can see whether or not doxygen runs dot with the command doxygen -d extcmd).

[nitz]

I was!

But in an attempt to do it all again (while using -d extcmd to make sure), I have perhaps stumbled upon where my actual issue may lie! I've been using doxywizard as I've been tinkering through different settings as I'm setting up my project. It seems running doxygen via the Run tab isn't the same as running doxygen from the command line:

Comparing the two outputs:

(this was with dot 2.50.0)

Pressing Show configuration in doxywizard correctly calls out that DOT_TRANSPARENT = YES, but perhaps there's something along the way causing that setting to not make it through to the actual doxygen invocation?

[albert-github]

After you changed your settings did you save the Doxyfile?

Did you try to use -d exctcmd in the run tab of the doxywizard as well?

[nitz]

I sure did. doxywizard is executing dot when we expect, but it's not getting the DOT_TRANSPARENT setting!

I replaced my doxygen.exe with a copy of cat.exe (Just a windows port of the gnu utility), and hit Run doxygen from doxywizard. ~2500 lines of the doxyfile it sends via stdin dumped out to the text box, and sure enough: DOT_TRANSPARENT isn't there.

I wonder why it's not coming through?

[nitz]

Oh interesting: while the output when I used cat was 2613 lines, it numbered the lines as it read them from stdin. It stopped after 2048. (The last setting visible is DOTFILE_DIRS, after which there is a few lines of comments and then an abrupt end before MSCFILE_DIRS is defined.)

2048 is way too "power of two" to be a random spot for it to stop! I'm looking at QTextStream to see if there's a line limit somewhere.

[albert-github]

Which version of the doxywizard are you using (see the about box)?

[nitz]

1.9.3 w/Qt 5.15.2

[albert-github]

Hmm, strange.

Very nice trick with cat!!

I remember we had problems in the past with something similar (see #8346, #8372) and that was solved.
Looks like something similar, and with the cat it is easy/ier to reproduce.
I have to dive into it.

[nitz]

The way I read the call tree from here:

doxygen/addon/doxywizard/doxywizard.cpp

Lines 568 to 577 in 26acf9d

	if (!m_runProcess->waitForStarted())
	{
	m_outputLog->append(APPQT(QString::fromLatin1("*** Failed to run doxygen\n")));
	return;
	}
	QTextStream t(m_runProcess);
	m_expert->writeConfig(t,false,false);
	m_runProcess->closeWriteChannel();
	if (m_runProcess->state() == QProcess::NotRunning)

After the process is started, the QTextStream is used to write each of the options to the process. I can't see anywhere if the QTextStream->QProcess is or isn't async, but I imagine it might be, due to buffering either in Qt, or in the pipe to the process itself (which Windows would own.)

After writeConfig() is called, closeWriteChannel() gets called immediately. The Qt documentation claims that will close the channel once "all data has been written to the process."

However. My somewhat wild guess is that the stream or pipe fills up to 2048 lines, then remaining lines get truncated, all before the doxygen process has opened or read from stdin. Then closeWriteChannel() gets called, which closes doxygen's stdin, letting the 2048 lines that are buffered (somewhere?) get read as input, and everything from line 2049+ gets lost.

I can't see anything in Qt's documentation that talks about a buffer size limit in any of the related classes. I also find it odd that it seems to be based on newlines rather than some character/byte limit. (Unless it actually is, and the 2048 lines is just an inconvenient red herring.) I do see that QIODevice (from which QProcess inherits,) can operate in a buffered or unbuffered mode, depending on it's implementations. The setReadChannel() call:

doxygen/addon/doxywizard/doxywizard.cpp

Line 548 in 26acf9d

m_runProcess->setReadChannel(QProcess::StandardOutput);

Doesn't specify the second argument, which defaults to QIODevice::Truncate. So if it is in fact buffered, then it would be using it.

Perhaps instead we could try skipping the buffer:

m_runProcess->setReadChannel(QProcess::StandardOutput, QIODevice::Truncate | QIODevice::Unbuffered);

Alternatively, perhaps before calling closeWriteChannel(), something like

    QTextStream t(m_runProcess);
    m_expert->writeConfig(t,false,false);

    while (m_runProcess->bytesToWrite() > 0)
    {
        m_runProcess->waitForBytesWritten(/*timeout is implicitly 30000 msecs*/);
    }

    m_runProcess->closeWriteChannel();

I'm not familiar enough with Qt to know exactly what might be the right way to toast this bread, but my hunch is that the issue is likely coming from this area!

And one final note:

I have confirmed that doxygen is able to handle the full input via stdin by using cat my-doxyfile | doxygen -!

[albert-github]

I did quite a few experiments, but no real solution.
In the documentation of QProcess I see:

Certain flags, such as Unbuffered and Truncate, are meaningless when used with some subclasses. Some of these restrictions are implied by the type of device that is represented by a subclass. In other cases, the restriction may be due to the implementation, or may be imposed by the underlying platform; for example, QTcpSocket does not support Unbuffered mode, and limitations in the native API prevent QFile from supporting Unbuffered on Windows.

Especially the last part ....
A solution like:

m_runProcess->start(doxygenPath + QString::fromLatin1("doxygen"), args, QIODevice::ReadWrite | QIODevice::Unbuffered);

instead of

m_runProcess->start(doxygenPath + QString::fromLatin1("doxygen"), args);

which is equivalent to:

m_runProcess->start(doxygenPath + QString::fromLatin1("doxygen"), args, QIODevice::ReadWrite);

doesn't work.

The only solution I can, at the moment, think of is to change:

m_expert->writeConfig(t,false,false);

into

m_expert->writeConfig(t,true,true);

though this is just moving the problem a bit as for big project we still night run into problems.

Another possibility might be to invoke doxygen heer not with - (stdout), but with the saved doxygen configuration setting.

@doxygen What is your opinion?