Obtain the complete command line and error message #852

Open
mcraveiro opened this Issue Jan 12, 2016 · 33 comments

Projects

None yet

4 participants

@mcraveiro

Hi,

Many thanks for flycheck, its really useful and I use on a daily basis. Today I just got stuck on something that is probably quite trivial but I can't quite figure out how to do. I can't see the complete error message from clang as it is very long and I can't see the command line that flycheck is using to run clang so I can't replicate the error on the shell. So my questions are:

  • is there some kind of log a-la flymake log level that shows all operations? Ideally the full compilation line so that I can re-run it from the shell.
  • is it possible to configure the size of the error messages so that it can be bumped up to cope with the larger-than-life c++ errors?

I am using the flycheck package with the following version:

Flycheck version: 0.26snapshot (package: 20160107.1218)

I am on Emacs 24.5.1 from Debian Testing.

Many thanks for your time.

Marco

@cpitclaudel
Member

Hey there,

Not a direct answer, but try C-c ! C-c (flycheck-compile); it will show you the full output of the checker, and the command line.

@mcraveiro

hi,

thanks very much for the prompt answer. This is exactly what I was looking for! I will keep the ticket open for a couple of days in case there is a way to increase the error message size and so on - but if not, this is sufficient for my use case.

Cheers

Marco

@cpitclaudel
Member

Glad to hear it :) Do you have an example of the issue you're running into (maybe from a project I could clone to look at it)? And does C-c ! l (flycheck-list-errors) show the information you're looking for?

@mcraveiro

Yes, well I haven't quite got to the bottom of it but I think I know the root cause.

A bit of context: I am using VTK, a C++ library which defines a whole load of macros. The project compiles fine from the makefiles. In terms of emacs, I am using ede-compdb, which provides a nice way to obtain all of the defines. Inspecting the defines (flycheck-clang-definitions) I get:

(note: line breaks on all text for readability)

(BOOST_ALL_DYN_LINK QT_CORE_LIB QT_GUI_LIB QT_NO_DEBUG QT_WIDGETS_LIB
vtkFiltersFlowPaths_AUTOINIT="1(vtkFiltersParallelFlowPaths)" 
vtkIOExodus_AUTOINIT="1(vtkIOParallelExodus)" vtkIOGeometry_AUTOINIT="1(vtkIOMPIParallel)" 
vtkIOImage_AUTOINIT="1(vtkIOMPIImage)" 
vtkIOSQL_AUTOINIT="2(vtkIOMySQL,vtkIOPostgreSQL)" 
vtkRenderingContext2D_AUTOINIT="1(vtkRenderingContextOpenGL)" 
vtkRenderingCore_AUTOINIT="4(vtkInteractionStyle,vtkRenderingFreeType,vtkRenderingFreeType
OpenGL,vtkRenderingOpenGL)" 
vtkRenderingFreeType_AUTOINIT="2(vtkRenderingFreeTypeFontConfig,vtkRenderingMatplotlib)" 
vtkRenderingLIC_AUTOINIT="1(vtkRenderingParallelLIC)" 
vtkRenderingVolume_AUTOINIT="1(vtkRenderingVolumeOpenGL)")

As you can see, there are a lot of brackets and so on, characters that appear to require escaping. And so that is what is done on the actual compilation line:

/usr/bin/clang\+\+-3.7 -fsyntax-only -fno-color-diagnostics -fno-caret-diagnostics -fno-diagnostics-
show-option -iquote /home/marco/Development/phd/neurite/projects/soma/src/ -std\=c\+\+14 -Wall 
-Wextra -DBOOST_ALL_DYN_LINK -DQT_CORE_LIB -DQT_GUI_LIB -DQT_NO_DEBUG 
-DQT_WIDGETS_LIB -DvtkFiltersFlowPaths_AUTOINIT\=\"1\(vtkFiltersParallelFlowPaths\)\" 
-DvtkIOExodus_AUTOINIT\=\"1\(vtkIOParallelExodus\)\" -DvtkIOGeometry_AUTOINIT\=\"1\(vtkIOMPIParallel\)\"
-DvtkIOImage_AUTOINIT\=\"1\(vtkIOMPIImage\)\" -DvtkIOSQL_AUTOINIT\=\"2\(vtkIOMySQL\,vtkIOPostgreSQL\)\"
-DvtkRenderingContext2D_AUTOINIT\=\"1\(vtkRenderingContextOpenGL\)\" -DvtkRenderingCore_AUTOINIT\=\"4\(vtkInteractionStyle\,vtkRenderingFreeType\,vtkRenderingFreeTypeOpenGL\,vtkRenderingOpenGL\)\"
-DvtkRenderingFreeType_AUTOINIT\=\"2\(vtkRenderingFreeTypeFontConfig\,vtkRenderingMatplotlib\)\"
-DvtkRenderingLIC_AUTOINIT\=\"1\(vtkRenderingParallelLIC\)\" -DvtkRenderingVolume_AUTOINIT\=\"1\(vtkRenderingVolumeOpenGL\)\"
-I /home/marco/Development/phd/neurite/build/output/clang-3.7/projects/soma/src/
-I /home/marco/Development/phd/neurite/projects/soma/src/ 
-I /home/marco/Development/local/include/ -I /usr/include/vtk-6.2/ -I /usr/include/tcl/
 -I /usr/include/jsoncpp/ -I /usr/include/x86_64-linux-gnu/ -I /usr/include/freetype2/ 
-I /usr/lib/openmpi/include/ -I /usr/lib/openmpi/include/openmpi/ -I /usr/include/python2.7/ 
-I /usr/include/hdf5/serial/ -I /usr/include/libxml2/ -I /usr/include/x86_64-linux-gnu/qt5/ 
-I /usr/include/x86_64-linux-gnu/qt5/QtCore/ -I /usr/lib/x86_64-linux-gnu/qt5/mkspecs/linux-g\+\+-64/ 
-I /usr/include/x86_64-linux-gnu/qt5/QtGui/ -I /home/marco/Development/phd/neurite/build/output/clang-3.7/stage/include/ 
-I /home/marco/Development/phd/neurite/projects/swc/include/ 
-I /home/marco/Development/phd/neurite/projects/soma/include/ 
-I /home/marco/Development/phd/neurite/projects/utility/include/ 
-I /usr/include/x86_64-linux-gnu/qt5/QtWidgets/ 
-I /home/marco/Development/phd/neurite/build/output/clang-3.7/ -x c\+\+ - 
< /home/marco/Development/phd/neurite/projects/soma/src/mainwindow.cpp

However this additional escaping is now resulting in invalid C++ preprocessor macros (I believe) which fail to compile:

In file included from <stdin>:2:
In file included from /usr/include/vtk-6.2/QVTKWidget.h:39:
In file included from /usr/include/vtk-6.2/vtkGUISupportQtModule.h:36:
In file included from(BOOST_ALL_DYN_LINK QT_CORE_LIB QT_GUI_LIB QT_NO_DEBUG QT_WIDGETS_LIB
vtkFiltersFlowPaths_AUTOINIT="1(vtkFiltersParallelFlowPaths)"
vtkIOExodus_AUTOINIT="1(vtkIOParallelExodus)" vtkIOGeometry_AUTOINIT="1(vtkIOMPIParallel)" 
vtkIOImage_AUTOINIT="1(vtkIOMPIImage)" 
vtkIOSQL_AUTOINIT="2(vtkIOMySQL,vtkIOPostgreSQL)"  /usr/include/vtk-6.2/vtkInteractionStyleModule.h:36:
/usr/include/vtk-6.2/vtkRenderingCoreModule.h:41:1: error: pasting formed
      'VTK_AUTOINIT_DECLARE_"4(vtkInteractionStyle,vtkRenderingFreeType,vtkRenderingFreeTypeOpenGL,vtkRenderingOpenGL)"',
      an invalid preprocessing token
/usr/include/vtk-6.2/vtkAutoInit.h:21:25: note: expanded from macro 'VTK_AUTOINIT'
/usr/include/vtk-6.2/vtkAutoInit.h:22:28: note: expanded from macro 'VTK_AUTOINIT0'
/usr/include/vtk-6.2/vtkAutoInit.h:25:24: note: expanded from macro 'VTK_AUTOINIT1'

This is the large error message I alluded to, of which I could only see the first four or five lines in the tool-tip.

Do you think the escaping is being done by flycheck? The variable looks ok out of ede-compdb...

Cheers

Marco

@mcraveiro

For good measure, I manually removed all escaping in emacs and then compiled the code on the shell and it worked - well, for certain definitions of "worked" :-)

/usr/bin/clang++-3.7 -fsyntax-only -fno-color-diagnostics -fno-caret-diagnostics -fno-diagnostics-show-option 
-iquote /home/marco/Development/phd/neurite/projects/soma/src/ -std=c++14 -Wall -Wextra 
-DBOOST_ALL_DYN_LINK -DQT_CORE_LIB -DQT_GUI_LIB -DQT_NO_DEBUG 
-DQT_WIDGETS_LIB -DvtkFiltersFlowPaths_AUTOINIT="1(vtkFiltersParallelFlowPaths)" 
-DvtkIOExodus_AUTOINIT="1(vtkIOParallelExodus)" 
-DvtkIOGeometry_AUTOINIT="1(vtkIOMPIParallel)" -DvtkIOImage_AUTOINIT="1(vtkIOMPIImage)" 
-DvtkIOSQL_AUTOINIT="2(vtkIOMySQL,vtkIOPostgreSQL)" 
-DvtkRenderingContext2D_AUTOINIT="1(vtkRenderingContextOpenGL)" 
-DvtkRenderingCore_AUTOINIT="4(vtkInteractionStyle,vtkRenderingFreeType,vtkRenderingFreeTypeOpenGL,vtkRenderingOpenGL)" 
-DvtkRenderingFreeType_AUTOINIT="2(vtkRenderingFreeTypeFontConfig,vtkRenderingMatplotlib)" 
-DvtkRenderingLIC_AUTOINIT="1(vtkRenderingParallelLIC)" 
-DvtkRenderingVolume_AUTOINIT="1(vtkRenderingVolumeOpenGL)" 
-I /home/marco/Development/phd/neurite/build/output/clang-3.7/projects/soma/src/ 
-I /home/marco/Development/phd/neurite/projects/soma/src/ 
-I /home/marco/Development/local/include/ -I /usr/include/vtk-6.2/ -I /usr/include/tcl/ 
-I /usr/include/jsoncpp/ -I /usr/include/x86_64-linux-gnu/ -I /usr/include/freetype2/ 
-I /usr/lib/openmpi/include/ -I /usr/lib/openmpi/include/openmpi/ -I /usr/include/python2.7/ 
-I /usr/include/hdf5/serial/ -I /usr/include/libxml2/ -I /usr/include/x86_64-linux-gnu/qt5/ 
-I /usr/include/x86_64-linux-gnu/qt5/QtCore/ -I /usr/lib/x86_64-linux-gnu/qt5/mkspecs/linux-g++-64/ 
-I /usr/include/x86_64-linux-gnu/qt5/QtGui/ 
-I /home/marco/Development/phd/neurite/build/output/clang-3.7/stage/include/ 
-I /home/marco/Development/phd/neurite/projects/swc/include/ 
-I /home/marco/Development/phd/neurite/projects/soma/include/ 
-I /home/marco/Development/phd/neurite/projects/utility/include/ 
-I /usr/include/x86_64-linux-gnu/qt5/QtWidgets/ 
-I /home/marco/Development/phd/neurite/build/output/clang-3.7/ -x c++ - 
< /home/marco/Development/phd/neurite/projects/soma/src/mainwindow.cpp

In file included from <stdin>:2:
In file included from /usr/include/vtk-6.2/QVTKWidget.h:40:
In file included from /usr/include/vtk-6.2/QVTKInteractor.h:43:
In file included from /usr/include/x86_64-linux-gnu/qt5/QtCore/QObject:1:
In file included from /usr/include/x86_64-linux-gnu/qt5/QtCore/qobject.h:40:
In file included from /usr/include/x86_64-linux-gnu/qt5/QtCore/qobjectdefs.h:41:
In file included from /usr/include/x86_64-linux-gnu/qt5/QtCore/qnamespace.h:37:
/usr/include/x86_64-linux-gnu/qt5/QtCore/qglobal.h:1067:4: error: "You must build your code with position independent
      code if Qt was built with -reduce-relocations. "         "Compile your code with -fPIC (-fPIE is not enough)."
In file included from <stdin>:15:
/home/marco/Development/phd/neurite/projects/utility/include/neurite/utility/test/logging.hpp:42:22: error: no member
      named 'framework' in namespace 'boost::unit_test'
<stdin>:16:10: fatal error: 'neurite/swc/io/hydrator_io.hpp' file not found
@lunaryorn
Member

@mcraveiro Hello. C-c ! C-c runs the command through your shell, hence escaping is necessary to prevent the shell from interpreting any special characters. Flycheck uses shell-quote-argument for this purpose, take a look at its docstring for more information.

You should be able to copy and paste the entire command to your shell terminal and run it, because essentially Flycheck does the same: Feed the escaped command to your shell :)

I'd like to take a step back though: What is the actual issue here? What are you trying to achieve and what's your problem? C-c ! C-c is not the proper solution, this command is just a debugging tool.

Please show how Flycheck behaves currently, and tell us what you'd like to see instead.

@mcraveiro

Hi @lunaryorn, if you look at the two comments above, this is exactly what I did; I took the command as provided by C-c ! C-c and ran it on the shell, reproducing the same errors I see in Flycheck. I then removed the escaping manually and re-ran the command and the errors went away. So I concluded that the problem is with how we are escaping quoted content:

-DvtkIOExodus_AUTOINIT\=\"1\(vtkIOParallelExodus\)\"

This should probably be:

-DvtkIOExodus_AUTOINIT\=\"1(vtkIOParallelExodus)\"

But I'm not an escaping expert unfortunately.

@lunaryorn
Member

@mcraveiro Ah sorry, I was lost in the amount of text, and didn't notice it. Probably escaping is flawed; I do not know. As I said, we do not escape ourselves, but rely on the built-in shell-quote-argument for this job.

However, that was not actually the point of my comment. I'd like to know why you are using flycheck-compile. As I said, it's just a debugging tool, and it should not be part of your regular Flycheck workflow. You should not need to use it.

So, why do you want to use flycheck-compile?


PS: I took the opportunity of this issue to file #854. We need to redesign flycheck-compile.

@mcraveiro

Well, I am using flycheck-compile because my code compiles fine outside of flycheck but fails with flycheck and I couldn't figure out why :-) Also, the output displayed on the screen only covers the first 4 or 5 lines, and thus is not enough to see the actual error message and I could not figure out how to increase the amount of output displayed on the tool-tip.

Even if I did see the whole error message, I still wouldn't be able to figure out that the problem is with the shell quoting when building the flycheck command line; for that I needed to see the flycheck-compile so that I could reproduce the problem in the command line.

I guess VTK is not going to be usable with flycheck, given the current shell-quote-argument behaviour and there are no easy fixes for this. Its not the end of the world.

Thanks for your time.

@lunaryorn
Member

@mcraveiro I think we should rather find out why the error is not properly displayed in Flycheck.

Did you take a look at the error list with C-c ! l? How does the error appear there? Could you share a screenshot of the error list?

@lunaryorn
Member

@mcraveiro Besides, as far as I can see for the second output you've shown, I think that issue is not VTK but Qt:

 error: "You must build your code with position independent
      code if Qt was built with -reduce-relocations. "         "Compile your code with -fPIC (-fPIE is not enough)."

Could you add -fPIC to flycheck-clang-args and see if that fixes the issue?

Furthermore I think you need to extend flycheck-clang-include-path to include the directory containing neurite/swc/io/hydrator_io.hpp.

Could you try whether these two things together make Flycheck work correctly?

@mcraveiro

Yes, please do not be confused with those, the Qt error is very simple and has been fixed already and so is the path. It is the VTK error before that I cannot resolve.

In terms of C-c ! l: I will look at this; but as I say, the error list at best would just show an ultra-complex clang error on macro expansion, which would give you no clues to the underlying problem.

To illustrate the problem with a minimum working example with just emacs (and excluding inner quotes to keep thing simple):

(message "%s" (shell-quote-argument "MACRO=1(ABC)"))
"MACRO\\=1\\(ABC\\)"

Then in shell:

$ echo "MACRO\\=1\\(ABC\\)"
MACRO\=1\(ABC\)

The escaped =, ( and ) are the problems, C++ is not expecting those. As you say this is not a flycheck error.

@mcraveiro

For good measure, an actual VTK macro is:

-DvtkIOExodus_AUTOINIT\=\"1\(vtkIOParallelExodus\)\" 

Ideally this should come out as:

-DvtkIOExodus_AUTOINIT\=\"1(vtkIOParallelExodus)\" 
@lunaryorn
Member

@mcraveiro I'm sorry but I see no VTK error in your second example.

I think we're talking past each other, and there's probably a misunderstanding somewhere, thus I'd like to summarise and clarify some points ere we continue:

  • The error in your first example was caused by a flaw in shell escaping of flycheck-compile, wasn't it?
  • Your second example where you manually corrected escaping shows no VTK errors anymore, does it?

Now I'd like to point out that escaping is specific to flycheck-compile. For its regular on-the-fly background syntax checks Flycheck does not use a shell at all, and instead directly invokes the corresponding executable, with no escaping involved. flycheck-compile is a special case.

As such, escaping as observed in flycheck-compile cannot be the cause of any flaw or error in the regular syntax checks, simply because there's no escaping in the latter.

Hence I'm particularly interested in an error list screenshot: It'll show us what Flycheck actually did during a regular syntax check.

@mcraveiro

@lunaryorn Ah yes thanks very much for the clarification! You are correct on all points above, so we are now in the same page. This is all very interesting because the error I got in flycheck is exactly the same one as I got when running the escaped code from the shell, so this is very puzzling indeed.

I will send you the full screenshot as soon as I am at the computer where the error is occurring,

Thanks for your help.

@lunaryorn
Member

@mcraveiro Thank you very much. I'd love to investigate this issue :)

@mcraveiro

Thanks! :-) so here we go. For some reason I can't actually do C-c ! l, I think this may be related to the prelude keymap. Running flycheck-list-errors I get:

    2     error           In include /usr/include/vtk-6.2/QVTKWidget.h... (c/c++-clang)
    2     error           In file included from... (c/c++-clang)
    2     error           In file included from... (c/c++-clang)
    2     error           In file included from... (c/c++-clang)
    2     error           In file included from... (c/c++-clang)
    2     error           In include /usr/include/vtk-6.2/QVTKWidget.h... (c/c++-clang)
    2     error           In file included from... (c/c++-clang)
    7     error           In file included from... (c/c++-clang)
    7     error           In file included from... (c/c++-clang)
    7     error           In file included from... (c/c++-clang)
    7     error           In file included from... (c/c++-clang)
    7     error           In file included from... (c/c++-clang)
    7     error           In file included from... (c/c++-clang)
    7     error           In file included from... (c/c++-clang)
    7     error           In file included from... (c/c++-clang)
    7     error           In file included from... (c/c++-clang)
    7     error           In file included from... (c/c++-clang)
    7     error           In file included from... (c/c++-clang)
    7     error           In file included from... (c/c++-clang)

This is in my *Flycheck errors* buffer.

@lunaryorn lunaryorn added K-bug and removed K-question labels Jan 13, 2016
@lunaryorn
Member

@mcraveiro Oh, that's bad 😢 That's definitely a bug in Flycheck parsing C++ output.

I'll try to fix it, but as I don't use C++, I'm afraid that it'll take a while. Please be patient.

@mcraveiro

@lunaryorn Nice one thanks.

One thing I'd say is, if the errors buffer contained both the full compiler invocation and the complete error messages, it would be easier to find out the problem - a bit like compilation does. Is this possible? Maybe via a "verbose" setting or some such. If I could see the command line and full error I probably could diagnose it very quickly, like I did with the quotes.

@mcraveiro

Ah, which I guess is #854 :-) cool, I'll wait for #854. To be honest I would't worry too much about this issue, once we have #854 I can provide more details and it perhaps would make it easier for you to zone in the problem...

@lunaryorn
Member

@mcraveiro I think we should not always show the command. If Flycheck works the command is at best redundant and at worst very distracting.

However, we definitely must improve the debugging capabilities for Flycheck, which we'll do in #854.

Thank you for your patience. I've given this issue low priority, if it's not at all important to you, so that we can wait for #854 to receive better debugging.

@lunaryorn lunaryorn added the P-low label Jan 13, 2016
@mcraveiro

Also, I just had a quick look at flycheck-checker-shell-command, and it seems we are escaping the command before execution?

         (command (mapconcat
                   #'shell-quote-argument
                   (funcall flycheck-command-wrapper-function
                            (cons (flycheck-checker-executable checker) args))
                   " ")))
@mcraveiro

yep low makes sense to me - thanks

@lunaryorn
Member

@mcraveiro Yes, of course. We must do so, otherwise the shell would interpret special characters and prevent the command from executing as intended.

@mcraveiro

Ah, sorry - but so what do you mean with this part:

Now I'd like to point out that escaping is specific to flycheck-compile. For its regular on-the-fly
background syntax checks Flycheck does not use a shell at all, and instead directly invokes the
corresponding executable, with no escaping involved.

@lunaryorn
Member

flycheck-checker-shell-command is only used by flycheck-compile. A normal syntax check uses flycheck-checker-substituted-arguments and passes the result directly to start-process, with no shell involved. See flycheck-start-command-checker.

@mcraveiro

Ah great, thanks. I'll play around with this until I hear from #854. Thanks a lot.

@hrehfeld

Any Progress? I also run into this with c++ and clang.

     1   9 warning         #pragma once in main file (c/c++-clang)
    13     warning         In include /home/hrehfeld/projects/mplan-repos3/mlib/src/mlib/render/Scene.hpp (c/c++-clang)
    13     error           In file included from (c/c++-clang)
    13     error           In file included from (c/c++-clang)

is there any sensible way to show the complete error msg?

@lunaryorn
Member

@hrehfeld I'm afraid that there's no progress, and won't be for quite some time. We'll improve debugging means at some point and try to fix error parsing with C/C++, but since none of us actually uses C/C++ the issue has somewhat low priority.

We would welcome pull request and patches that improve error parsing of C/C++.

@hrehfeld
hrehfeld commented Apr 27, 2016 edited

Can I help in some way? I'm not an expert elisp programmer, so it's rather time consuming for me to dig through the flycheck code, but I could provide logs etc.

@lunaryorn
Member

@hrehfeld I'm sorry but I think at this point we'll need code. Even if you provided Clang output we'd still have to go through the output and come up with good code to parse it.

@hrehfeld

I think #451 is related.

@lunaryorn
Member

Yes, I know, as is #864 and all related issues. The root cause is that Clang's and GCC's error reporting is a mess, with errors from different files intertwined and entangled, and we have to somehow try and aggregate errors upwards to the file that you're looking at in Emacs.

There've been issues every since Clang and GCC were added, and none has managed to come up with a good solution hitherto, and the whole issue is further complicated by different people having different preferences what should be reported and what detail show be revealed, and by the "mis-feature" that some errors in headers only appear when the header is included and the definition used somewhere.

Please don't expect us to come up with a satisfactory solution any time soon, if at all. I for my part must admit that I'm somewhat tired of the mess that is C/C++ includes and have little to no motivation to touch this part of Flycheck again.

I'm sorry and I understand that the situation is unfortunate for C/C++ programmers, but it is what it is.

@lunaryorn lunaryorn removed P-low bug labels Jun 29, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment