[RFC] Multi pass parsing over multi source files #1495

masatake · 2017-06-29T11:43:57Z

Close #80.

@vhda, how do you think about these changes? See the commits whose logs are started with [TEMPORARY].
I will update docs/internal.rst after getting your approval.

In my version, SystemVerilog emits the most of all tags in the first (-1) pass. However, SystemVerilog parser hands over some of them to the next mm pass via "barrel". In the seond (-2) pass, tags in the barrel are stored to the keyword table by setupMM method of SystemVerilog.

For testing I choose tags of typedef kind. In the -2 pass, SystemVerilog parser can recognize the next token of typedef'ed type. If such token is recognized in class context, the token can be tagged as "member" kind.
I didn't choose the kind name "variable" though 'v' was chosen. Because a member of class is not a variable.

This will not work well in interactive mode. I have to research the area more.

   $ cat a.sv
    class test;
       t_user t_user_memb;
    endclass
    $ cat b.sv
    typedef int t_user;
    $ ./ctags -o -  a.sv b.sv
    t_user	b.sv	/^typedef int t_user;$/;"	T
    t_user_memb	a.sv	/^   t_user t_user_memb;$/;"	v	class:test
    test	a.sv	/^class test;$/;"	C

…m singed to unsigned A negative value will be passed in the stages of multi pass parsing over multi source files Signed-off-by: Masatake YAMATO <yamato@redhat.com>

Signed-off-by: Masatake YAMATO <yamato@redhat.com>

…source files Signed-off-by: Masatake YAMATO <yamato@redhat.com>

Signed-off-by: Masatake YAMATO <yamato@redhat.com>

…rser typedefs gathered in "-1" mm pass is stored to barrel. At the first of the "-2" mm pass, typedefs are stored as keywords.

$ cat a.sv class test; t_user t_user_memb; endclass $ cat b.sv typedef int t_user; $ ./ctags -o - a.sv b.sv t_user b.sv /^typedef int t_user;$/;" T t_user_memb a.sv /^ t_user t_user_memb;$/;" v class:test test a.sv /^class test;$/;" C t_user_memb is captured as v.

coveralls · 2017-06-29T11:49:11Z

Coverage increased (+0.03%) to 85.144% when pulling ee912f3 on masatake:multi-pass-parsing-over-multi-source-files into 4ff09da on universal-ctags:master.

masatake · 2017-06-29T11:56:49Z

After thinking I found barrel is not needed here. Adding typedefed typename to the keyword table in the -1 pass will be enough...

masatake · 2017-06-29T13:48:01Z

Other interesting application of mm is "unknown" kind of python.
"unknown" kind objects tagged in -1st pass may be solved in -2nd pass...

Signed-off-by: Masatake YAMATO <yamato@redhat.com>

coveralls · 2017-06-29T15:04:06Z

Coverage increased (+0.04%) to 85.15% when pulling d997cb4 on masatake:multi-pass-parsing-over-multi-source-files into 4ff09da on universal-ctags:master.

masatake · 2017-06-30T09:46:28Z

What I wrote is something like linker. So I can borrow many concepts and ideas from linker.

vhda · 2017-06-30T13:33:33Z

I have the intention of dedicating part of my weekend to Universal Ctags.
I'll reply ASAP :)

Signed-off-by: Masatake YAMATO <yamato@redhat.com>

masatake · 2017-06-30T19:23:06Z

I updated the documents. I didn't take an example from SystemVerilog parser because I don't know it well.

coveralls · 2017-06-30T19:53:15Z

Coverage increased (+0.05%) to 85.159% when pulling 6ea7823 on masatake:multi-pass-parsing-over-multi-source-files into 4ff09da on universal-ctags:master.

vhda

Looking good and the number of added lines isn't as big as I would have expected, which is nice :)

I think some code code be made common. Please review my comments and analyze if it is possible.

I also remembered "libraries". Code that is central and used frequently in the project but that is rarely changed. It would make sense to be able to "seed" the second pass with such libraries... but this is beyond the scope of this change, I guess.

vhda · 2017-07-03T12:45:36Z

main/keyword.h

 extern void addKeyword (const char *const string, langType language, int value);
+
+/* addKeywordStrdup does strdup `string'.
+   Duplicated string is freed in  freeKeywordTable() */


Extra space in comment

Thank you. I will make a fixup! commit.

vhda · 2017-07-03T12:51:52Z

parsers/verilog.c

@@ -122,7 +124,8 @@ static kindDefinition SystemVerilogKinds [] = {
 { true, 'P', "program",   "programs" },
 { false,'Q', "prototype", "prototypes" },
 { true, 'R', "property",  "properties" },
- { true, 'T', "typedef",   "type declarations" }
+ { true, 'T', "typedef",   "type declarations" },
+ { true, 'v', "member",    "member elements" },


In the meanwhile I started thinking about 'o' and 'object'.
I'm raising this now, because it might make sense to maintain various "groups". E.g.: instances of classes are objects, instances of typedef are "custom types".

Verilog is an hardware description language that allows a special type of "object" called "instance". Design architecture is divided in "modules", which are instantiated in other top level modules. Having ctags parse these as special types would allow editors that support these tags to easily show the overall design structure.

Designing kinds is the most important task in ctags development.
I don't know Verilog. So if you are o.k., I'm o.k.
@RadekRR, this is the chance to reflect your idea to the software you are using:-).

vhda · 2017-07-03T12:56:04Z

parsers/verilog.c

+	else
+		return RESCAN_MM;
+}
+


Why make the parser aware of passes? In my mind I think we could modify the keyword table to indicate which types we are looking for in the first pass, and which are only valid in the second.

Core ctags could take care of the rest, by providing a smaller keyword table to the parser in the first pass and then the complete table in the second. The parser itself would be simplified and any other parser could easily use this new architecture.

Do you like the idea? Or am I missing something basic that will not allow this?

Why make the parser aware of passes?

The main part doesn't know which source file(s) should be rescanned.

vhda · 2017-07-03T13:14:28Z

docs/internal.rst

+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The main part of Universal-ctags provides APIs for Multi passes
+parsing over Multi source files (MM). The main part applies


Don't understand the meaning of "MM". First "M" is for multi, but what is the second for?
Typically you would do something like: "Multi source Files (MF)". That is, capitalize the words used in the abbreviation.

MM means "M"ulti pass parsing over "M"ulti source files.
Do I make sense?

masatake · 2017-07-03T14:21:42Z

@vhda, I feel gaps between what you want and what I wrote.

Could you look at

Please, look at the document explaining MM.
6ea7823

https://github.com/masatake/ctags/blob/6ea782371d7dc78b884a7882dd4dedd81a03f63c/docs/barrel.svg

Signed-off-by: Masatake YAMATO <yamato@redhat.com>

coveralls · 2017-07-03T15:41:47Z

Coverage increased (+0.06%) to 85.169% when pulling bded49b on masatake:multi-pass-parsing-over-multi-source-files into 4ff09da on universal-ctags:master.

masatake · 2017-07-06T07:58:54Z

I read again the original discussion.

My understanding of what you want.

pass 1. just making a hashtable and fill it with type names (or something names). You want to make no tag entries in this stage.

pass2. making tag entries and emiting them to tags file.

These flow can be implemented on MM. What you should is just ignoring Barrel. You can create SystemVerilog own type name table with functions declared in hashtable.h. SystemVerilog parser
must return RESCAN_MM.

However, you should emit some of tag entries in pass1 if possible. Because, newly introduced --interactive mode may expect per-source file processing. It means in the mode ctags runs only pass 1(-1). So the user of --interactive mode may recognize SystemVerilog parser does nothing.

wvandamm · 2019-08-12T09:50:47Z

Hi @masatake ,

Could I please inquire what the status is of this pull request? The functionality proposed seems highly interesting in any case!

Also a related question: the C++ parser appears to already be able to identify definitions for variables or objects of a type created with typedef or class. If I understand the discussion well that's something that this pull request was intending to add as well. For the SystemVerilog parser the same functionality does not appear to work yet though. So am I correct in assuming that the C++ parser has solved this problem in the parser itself? Or has some framework been added that other parsers could make use of to add similar functionality?

Thanks!

Wim

masatake · 2019-08-13T01:52:01Z

Could I please inquire what the status is of this pull request? The functionality proposed seems highly interesting in any case!

There is no progress other than changes I proposed here.

C++ parser may use heuristics. It is not perfect. e.g. about handling template variables.
See files under Units/parser-cxx.r/*.b.

Before implementing Multi-passes/Mult-files parsing, I have to improve the infrastructure for Multi-passes/Single-files parsing. See #2115.

If you want mm seriously, and want to implement, I will explain what kind of issues are.

masatake added 6 commits June 29, 2017 20:25

main: change the type of passCount parameter of createTagsForFile fro…

8c5acfd

…m singed to unsigned A negative value will be passed in the stages of multi pass parsing over multi source files Signed-off-by: Masatake YAMATO <yamato@redhat.com>

main: fix a typo in comment

a740929

Signed-off-by: Masatake YAMATO <yamato@redhat.com>

main: introduce mm, infrastructure for multi pass parsing over multi …

bd361cf

…source files Signed-off-by: Masatake YAMATO <yamato@redhat.com>

main: implement barrel API for MM

102a5d1

Signed-off-by: Masatake YAMATO <yamato@redhat.com>

[TEMPORARY] SystemVerilog: convert to multi-pass over multi-source pa…

77642fc

…rser typedefs gathered in "-1" mm pass is stored to barrel. At the first of the "-2" mm pass, typedefs are stored as keywords.

masatake changed the title ~~Multi pass parsing over multi source files~~ [RFC] Multi pass parsing over multi source files Jun 29, 2017

masatake added 3 commits June 29, 2017 23:37

main: introduce addKeywordStrdup

89e13b7

Signed-off-by: Masatake YAMATO <yamato@redhat.com>

[TEMPORARY] SystemVerilog: use addKeywordStrdup to avoid memory leak

0069603

Signed-off-by: Masatake YAMATO <yamato@redhat.com>

fixup! [TEMPORARY] SystemVerilog: add member(v) kind

d997cb4

Signed-off-by: Masatake YAMATO <yamato@redhat.com>

docs(web): write about mm/barrel API

6ea7823

Signed-off-by: Masatake YAMATO <yamato@redhat.com>

masatake mentioned this pull request Jul 1, 2017

byte offsets of symbols #1502

Closed

vhda suggested changes Jul 3, 2017

View reviewed changes

fixup! main: introduce addKeywordStrdup

bded49b

Signed-off-by: Masatake YAMATO <yamato@redhat.com>

masatake mentioned this pull request Feb 2, 2018

c/c++ struct declarations containing macro functions not handled correctly #1675

Closed

masatake mentioned this pull request Jul 15, 2018

C++: introduce "using" role to "namespace" kind #1783

Open

wvandamm mentioned this pull request Aug 12, 2019

Improve verilog analyzer oracle/opengrok#2892

Open

masatake added this to the Feature plan milestone Nov 10, 2019

masatake mentioned this pull request Apr 5, 2020

[systemverilog] parser mistakes typedef for a module's port #2413

Closed

masatake mentioned this pull request May 16, 2020

Cpreprocessor/C/C++/CUDA: infile macro expanding #2543

Merged

masatake mentioned this pull request Apr 29, 2022

When using stdout output, let's avoid any temporary files #3369

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[RFC] Multi pass parsing over multi source files #1495

[RFC] Multi pass parsing over multi source files #1495

masatake commented Jun 29, 2017 •

edited

coveralls commented Jun 29, 2017

masatake commented Jun 29, 2017

masatake commented Jun 29, 2017

coveralls commented Jun 29, 2017

masatake commented Jun 30, 2017

vhda commented Jun 30, 2017

masatake commented Jun 30, 2017 •

edited

coveralls commented Jun 30, 2017

vhda left a comment

vhda Jul 3, 2017

masatake Jul 3, 2017

vhda Jul 3, 2017

masatake Jul 3, 2017

vhda Jul 3, 2017

masatake Jul 3, 2017

vhda Jul 3, 2017

masatake Jul 3, 2017

masatake commented Jul 3, 2017

coveralls commented Jul 3, 2017

masatake commented Jul 6, 2017

wvandamm commented Aug 12, 2019

masatake commented Aug 13, 2019 •

edited

[RFC] Multi pass parsing over multi source files #1495

Are you sure you want to change the base?

[RFC] Multi pass parsing over multi source files #1495

Conversation

masatake commented Jun 29, 2017 • edited

coveralls commented Jun 29, 2017

masatake commented Jun 29, 2017

masatake commented Jun 29, 2017

coveralls commented Jun 29, 2017

masatake commented Jun 30, 2017

vhda commented Jun 30, 2017

masatake commented Jun 30, 2017 • edited

coveralls commented Jun 30, 2017

vhda left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

masatake commented Jul 3, 2017

coveralls commented Jul 3, 2017

masatake commented Jul 6, 2017

wvandamm commented Aug 12, 2019

masatake commented Aug 13, 2019 • edited

masatake commented Jun 29, 2017 •

edited

masatake commented Jun 30, 2017 •

edited

masatake commented Aug 13, 2019 •

edited