rewrite Units #60

masatake · 2014-08-06T09:49:08Z

help target is needed.

masatake · 2014-10-16T03:27:56Z

testing.mak is too comple. Rewrite is needed.
Subdirectories per parser is needed as suggested in @ffes in #88.
help target is needed.

ffes · 2014-10-16T08:13:12Z

When you are on it, could you add a target clean as well? Or maybe just rename clean-test to clean.

ffes · 2014-10-16T08:13:36Z

Another small thing I found:

Testing Units/xcmd-coffeetags...skipped (required feature coproc is not available)
Testing Units/xcmd-simple...skipped (required feature coproc is not available)

  Summary of "Units" test
  -------------------------
        #passed:  331
        #failed:  0
        #skipped(features):  0
        #skipped(languages):  0
        #known-bugs:  2

Note that the last two tests are skipped on Windows but not reported as such in the summary.
Or should I create a separate issue for this?

If checking them in subshells, their result are not reflected to the parent shell. Issue: #60 Reported-by: Frank Fesevur <ffes@users.sourceforge.net> Signed-off-by: Masatake YAMATO <yamato@redhat.com>

Issue: #60 Suggested-by: Frank Fesevur <ffes@users.sourceforge.net> Signed-off-by: Masatake YAMATO <yamato@redhat.com>

masatake · 2014-10-16T11:27:44Z

Thank you for reporting.
Could you try HEAD of master?

ffes · 2014-10-16T12:17:46Z

  Summary of "Units" test
  -------------------------
        #passed:  331
        #failed:  0
        #skipped(features):  3
        #skipped(languages):  0
        #known-bugs:  2

Solved

And thanks for the clean target

b4n · 2014-10-24T13:33:44Z

Not sure I should report this here or open a new issue, tell me.

Currently if I run make check right after cloning and building, I get 19 failed tests:

  Summary of "Units" test
  -------------------------
    #passed:  315
    #failed:  19
        Units/conflib-recursive
        Units/conflib-simple
        Units/ctags-simple
        Units/dosbatch_test.cmd
        Units/matlab_backtracking.m
        Units/matlab_test.m
        Units/matlab-tg-corpus
        Units/mib-simple
        Units/noext-tg-matlab
        Units/option-corpora-conflicts
        Units/option-disable-kind-in-builtin
        Units/option-lang-compatibility
        Units/option-same-kind-in-regex-and-builtin
        Units/regex-flag-long
        Units/regex-flag-simple
        Units/rust-simple
        Units/simple.cbl
        Units/simple.html
        Units/xcmd-simple
    #skipped(features):  0
    #skipped(languages):  1
    #known-bugs:  2

Apparently all outputs for those from my run are empty but matlab-tg-corpus and matlab_backtracking.m which are completely different (they have a lot more tags).

masatake · 2014-10-27T01:20:59Z

@b4n, thank you for reporting.
I'm suprised ato so many failed.
I'll take a look them.

masatake · 2014-10-29T18:20:46Z

Let's hunt them.
The most of all failed tests are something related to the features/functions I have worked on:-(

When a test is failed testing.mak may report a command line for the test like
Testing Units/dosbatch_test.cmd...FAILED
differences left in Units/dosbatch_test.cmd.t/DIFF.TMP
cmdline: ./ctags --options=NONE --libexec-dir=libexec --libexec-dir=+Units/dosbatch_...

testing.mak connects the stderr to /dev/null, so we cannot know what was happed when failed.
So I would like to see the stderr and stdout when you run the reported command line.

I would like to focus on dosbatch_test.cmd. May be the area tested by the test case is simpler than others.
If the output is so large or complex to put here, send it to me, yamato@redhat.com.

b4n · 2014-10-29T18:29:53Z

I'm afraid there's not much more information:

$ ./ctags --options=NONE --libexec-dir=libexec --libexec-dir=+Units/dosbatch_test.cmd.t --data-dir=data --data-dir=+Units/dosbatch_test.cmd.t -o - Units/dosbatch_test.cmd.t/input.cmd
No options will be read from files or environment

testing.mak connects the stderr to /dev/null, […]

I don't think it does, there is a STDERR.TMP in the unit dir, that contains it. Anyway, it's the same.

b4n · 2014-10-29T18:32:35Z

Hum, another issue suggested I don't have regex support builtin for some reason, I guess it might be it?

b4n · 2014-10-29T18:54:35Z

OK, with regex enabled, I only have one last failing test: Units/xcmd-coffeetags. This is kind of expected as I don't have coffeetags:

$ ./ctags --options=NONE --libexec-dir=libexec --libexec-dir=+Units/xcmd-coffeetags.d --data-dir=data --data-dir=+Units/xcmd-coffeetags.d -o - --options=Units/xcmd-coffeetags.d/args.ctags Units/xcmd-coffeetags.d/input.coffee
No options will be read from files or environment
libexec/drivers/coffeetags: 31: libexec/drivers/coffeetags: coffeetags: not found
ctags: Warning: xcmd exits abnormally status(32512): [libexec/drivers/coffeetags --list-kinds=coffee]

So all tests passes, but we should skip those if one of the requirement is missing (regexes or coffeetags).

masatake · 2014-10-29T20:04:01Z

About test cases failed when regex is not avaiable should be skipped.
"regex" must be added to features fie in each test case. I will do this.

@b4n

…ist-languages output If a unit test expects a language parser and target ctags executable doesn't support the language, the unit test should be skipped. languages file under each unit test case represents the language the test case expects ctags to have. However, matching code that compares languages file and the output of --list-languages was broken. As the result a unit test case is reported as failed though it should be skipped. This patch removes a space in output of --list-languages. Let's consider "coffee" language. If a coffeetags command is not available, the parser should be disabled. $ ./ctags --list-languages coffee [disabled] In testing.mak, this output is used in a for loop. The for loop picked up the output of "ctags --list-languages" word by word. So "coffee" and "[disabled]" are picked up separately. This is the reason why matching didn't work expectedly. With using sed command the for loop picked up the output of "ctags --list-languages" line by line("coffee[disabled]"). Reported by @b4n in #60. Signed-off-by: Masatake YAMATO <yamato@redhat.com>

masatake · 2014-10-29T20:14:51Z

Could you try the latest head?
testing.mak may skip xcmd-coffeetags automatically

b4n · 2014-10-29T20:36:45Z

Could you try the latest head?
testing.mak may skip xcmd-coffeetags automatically

Yes it's fixed, thanks :)

As reported in #60, regex is used in many areas. Test cases using regex internally should be skipped when testing ctags with no regex feature. Signed-off-by: Masatake YAMATO <yamato@redhat.com>

masatake · 2014-11-09T15:06:28Z

Could you try

$ make -f testing.mak units

This runs misc/units internally.

vhda · 2014-11-10T00:55:56Z

Nice update, specially when running valgrind! Thanks a lot.
Also like having a new bash script, instead of a multiline string within the makefile.

vhda · 2014-11-10T01:40:56Z

Something strange happens when a test run segfaults:
Testing verilog-sv-nocontext misc/units: line 203: 14080 Segmentation fault ${_CMDLINE} 2> "${ostderr}" > "${orawout}"
failed (unexpected exit status: 139)

@vhda

When ctags is crashed, the shell process running the ctags process reports the crash: Testing verilog-sv-nocontext misc/units: line 203: \ 14080 Segmentation fault ${_CMDLINE} 2> "${ostderr}" > "${orawout}" \ failed (unexpected exit status: 139) This report disturbs the well-formatted output of tests. With this patch, ctags is run in a subshell; and the subshell the stderr and stdout are redirected to SHSTDERR.tmp and SHSTDOUT.tmp. As the result the crash report doesn't disturbs the well-formatted output of tests but can be refereed as a separated file later if needed. Issue reporeted by @vhda in #60. Signed-off-by: Masatake YAMATO <yamato@redhat.com>

masatake · 2014-11-10T17:00:43Z

@vhda, could you try the HEAD of master?

@vhda

When ctags exits with abnormal status, its parent shell reports it to stderr. This is noise for our purpose. This patch makes the subshell close its stderr. This bug(noise) is reported by in #60 by @vhda. This patch reverts wrong fix (59cfb8d) for the same bug. Signed-off-by: Masatake YAMATO <yamato@redhat.com>

masatake · 2014-11-14T16:38:34Z

In addition to VG=1, you can specify UNITS_TIMEOUT=N and SHRINK=1. When a crash or timed-out is occured, units tries to find minimal "bad" input with the parameters.

masatake · 2014-11-15T19:45:39Z

Most of all things are overhauled. Please, open a new issue if you find a bug.

masatake self-assigned this Aug 6, 2014

masatake added Test facility labels Aug 6, 2014

masatake removed the Incompatibilities label Oct 16, 2014

masatake changed the title ~~help target for testing.mak~~ rewrite Units Oct 16, 2014

masatake added a commit that referenced this issue Oct 16, 2014

testing.mak: add clean target

e7f6f23

Issue: #60 Suggested-by: Frank Fesevur <ffes@users.sourceforge.net> Signed-off-by: Masatake YAMATO <yamato@redhat.com>

masatake added a commit that referenced this issue Nov 9, 2014

add regex to features files

688ba25

As reported in #60, regex is used in many areas. Test cases using regex internally should be skipped when testing ctags with no regex feature. Signed-off-by: Masatake YAMATO <yamato@redhat.com>

masatake closed this as completed Nov 15, 2014

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rewrite Units #60

rewrite Units #60

masatake commented Aug 6, 2014

masatake commented Oct 16, 2014

ffes commented Oct 16, 2014

ffes commented Oct 16, 2014

masatake commented Oct 16, 2014

ffes commented Oct 16, 2014

b4n commented Oct 24, 2014

masatake commented Oct 27, 2014

masatake commented Oct 29, 2014

b4n commented Oct 29, 2014

b4n commented Oct 29, 2014

b4n commented Oct 29, 2014

masatake commented Oct 29, 2014

masatake commented Oct 29, 2014

b4n commented Oct 29, 2014

masatake commented Nov 9, 2014

vhda commented Nov 10, 2014

vhda commented Nov 10, 2014

masatake commented Nov 10, 2014

masatake commented Nov 14, 2014

masatake commented Nov 15, 2014

rewrite Units #60

rewrite Units #60

Comments

masatake commented Aug 6, 2014

masatake commented Oct 16, 2014

ffes commented Oct 16, 2014

ffes commented Oct 16, 2014

masatake commented Oct 16, 2014

ffes commented Oct 16, 2014

b4n commented Oct 24, 2014

masatake commented Oct 27, 2014

masatake commented Oct 29, 2014

b4n commented Oct 29, 2014

b4n commented Oct 29, 2014

b4n commented Oct 29, 2014

masatake commented Oct 29, 2014

masatake commented Oct 29, 2014

b4n commented Oct 29, 2014

masatake commented Nov 9, 2014

vhda commented Nov 10, 2014

vhda commented Nov 10, 2014

masatake commented Nov 10, 2014

masatake commented Nov 14, 2014

masatake commented Nov 15, 2014