java doc support

[uujava]

This pull request provides optional JavaDoc Nodes in mirah AST and whitespace and comments tokenization in mirah lexer.

• JavaDoc Node could be used to generate java stub from mirah sources and process via standard javadoc tool. By default JavaDoc tokens skipped. To enable JavaDoc Node processing configure parser:

parser = MirahParser.new
parser.skip_javadoc false

• Whitespace and comments tokenization is needed for IDE Lexer support at least in Intellij IDEA(https://plugins.jetbrains.com/plugin/7951?pr=). By default MirahLexer skips white spaces and comments. Two optional parameters added to MirahLexer#lex(pos, skipWhiteSpaceAndComments, skipJavaDoc). Check test\test_lexer.rb for examples.

[uujava]

Note! Javadoc support implemented only for class, interface and method definitions.
There is an issue with macros for modifier keyword support (like abstract, protected etc).
Haven't found fast mmeta change for that and not sure how to handle that later in compiler anyway.
Introducing modifier keyword support would solve all the problems (#21)

[felixvf]

Why not make a doc comment just be syntactic sugar for a modifier method invocation?

/** some comment */
def foo
end

is equivalent to

doc(" some comment ",
  def foo
  end
)

Likewise

/** some comment */
protected def foo
end

is equivalent to

doc(" some comment ",
  protected(
    def foo
    end
  )
)

I like having doc-comments being part of the AST. This has advantages for automatically generated code by Mirah macros: The macros which autogenerate methods could also autogenerate appropriate doc code.

However, it is not clear why we should choose JavaDoc in particular. Other candidates are, for example, YARDoc and Doxygen.

For the syntax of doc comments: Should we use Java-style "/*.../"? Should we use Ruby-style "# ... \n"? Should we use a different style? Should it be possible to nest comments? Should Mirah have multiline-comments at all?

[uujava]

I agree, that it's questionable what should mirah use for doc comments. I prefer Java Doc for it's extended support in tools and IDE and plan to provide an integration in mirah compiler. I'm afraid it would be not that easy to integrate YARDoc as mirah have different than ruby syntax.

[uujava]

Had a quick look at Doxigen. It seems that doxygen integration could be implemented in the same way as java doc. Generate java stubs and pass it to doxigen.
My implementation does not restrict text content in "java doc" comments. It just adds an AST node with text. Any text content and doc tags could be used inside.

[uujava]

Implemented pull request in mirah mirah/mirah#381

[felixvf]

Once again:

Why not make a doc comment

/** foobar */ ...

just syntactic sugar for a macro invocation

doc(" foobar ", ...)

?

This solution solves all issues with macros which return methods or classes, such as protected, because – essentially – doc-comments are also just macros which return methods or classes.

[uujava]

1. IMHO, It is a usual way to separate documenting from code itself
2. What doc macro would actually do? To generate java stub it does not have information on inferred types. One need correct (generally compilable by javac) java stub for java doc processing using javadoc or doxigen tools.
3. Java docs are not usually one liners. It would not look that nice with doc(,) stuff :)
4. It's 3 additional characters to type :). Compare doc(" ",) <-> /** */
5. Macros are not easily integrates to IDE. How IDE support code should separate one macro from another?
6. Java doc processing could be optional, doc(" ",) macros is not
7. How would you pass options to the macro, say the folder to save javadocs or stubs?

That's why I do not like using macro for specifically for java doc.

As for modifiers - I still think that having them part of the language is more correct way than macros.

1. Afaik (could be mistaken) currently it's not supported macros chaining. Ex:

private abstract class X
 private synchronized def some_method
 end
# or
end

1. There are also an issue in typer:
   Compiler error when class having macros is abstract mirah#320
   strange behaviour for abstract macro called on a class mirah#289
   nested closures over abstract class compilation issue mirah#375

For me it was faster solution to have modifiers in AST to fix some of them :).
3. imho macros adds additional reinferring cycles that is compilation speed
3. IDE support again
4. I'd like to have

  synchronized attr_reader a:int
  private synchronized attr_writer a:int

Currently it's not supported with modifiers either, but could be implemented.

[felixvf]

I'm not suggesting against the /** ... */ syntax. I actually like multi-line comments, but we should carefully decide which syntax we should use for multi-line comments.

What I think is problematic is pushing Mirah features directly into the syntax. Pushing features into the syntax makes them inaccessible to meta-programming.

1. IMHO, It is a usual way to separate documenting from code itself

Sure

1. What doc macro would actually do? To generate java stub it does not have information on inferred types. One need correct (generally compilable by javac) java stub for java doc processing using javadoc or doxigen tools.

The default doc macro would just attach a DocComment node to the ClassDefinition or MethodDefinition. However, this could be overriden by the user (e.g. using import static or by defining a "doc" macro just for the class or for its superclass or by using the macro extension registration mechanism). The doc macro would not create java stubs for processing. This is left to a different tool (based on the mirah compiler), as in your suggestion.

1. Java docs are not usually one liners. It would not look that nice with doc(,) stuff :)
2. It's 3 additional characters to type :). Compare doc(" ",) <-> /** */

I would keep the syntactic sugar of multi-line comments, of course. doc("...", ...) is just the meaning underneath.

1. Macros are not easily integrates to IDE. How IDE support code should separate one macro from another?

It depends on the IDE. Most IDEs seem to be not meta-programming friendly. However, Mirah is a meta-programming language. That's why I'm intervening here. Having JavaDoc or something like this is a good thing, but not at the expense of making Mirah a non-meta-programming language. So the IDE has 2 options:

1. Just parse the Mirah source code according to the Mirah parser and use the resulting AST. This should allow for features such as syntax highlighting.
2. Parse the Mirah source code according to the Mirah parser and then apply all the macros and then use the resulting AST.

It is clear that the latter version takes more time. However, once we have recompilation support in the compiler (the compiler detects which files need to be recompiled and only compiles these, re-using the already compiled macros defined in other files), this shouldn't be a practical problem any more.

1. Java doc processing could be optional, doc(" ",) macros is not

Sure, but this makes no difference, Whether we have doc(" ... ", ...) macros or /** ... */ ... syntax or both, all of these are not optional. Also in case of doc(" ... ", ...) macros, JavaDoc processing is optional.

1. How would you pass options to the macro, say the folder to save javadocs or stubs?

That's why I do not like using macro for specifically for java doc.

Maybe there is a misunderstanding here: The doc(" ... ", ...) macro has just the purpose to be a hook to auto-generate and modify the meaning of /** ... */ .... It has not the purpose to actually generate JavaDoc input, this is left to a different tool (based on the mirah compiler), as in your suggestion.

As for modifiers - I still think that having them part of the language is more correct way than macros.

1. Afaik (could be mistaken) currently it's not supported macros chaining. Ex:

private abstract class X
 private synchronized def some_method
 end
# or
end

You are actually mistaken. Macro chaining works fine. For example,

protected abstract class Foo
    protected synchronized def foo
    end
end

is perfectly valid and compiling Mirah code, as is

class Foo
    protected attr_accessor foo:int
end

(see https://github.com/mirah/mirah/blob/0af6095bb85e63d4dd190b9eeb7e1c9dca6209e8/test/jvm/macros_test.rb#L411)

1. There are also an issues in typer:
   Compiler error when class having macros is abstract mirah#320
   strange behaviour for abstract macro called on a class mirah#289
   nested closures over abstract class compilation issue mirah#375

Yes, currently, both the type-inference and the macro expansion are done in the same phase, and the type-inference-macro-expansion superphase is applied twice, once before applying a macro on a ClassDefinition and once after applying the macro on a ClassDefinition. This should be fixed, but I haven't doing that so far, as this a quite far-reaching change and thus I expect something to break. But if the pain of not-fixing is bigger than the pain of fixing, then maybe we should decide whether the typer should run before or after applying macros, and then implement this fix.

For me it was faster solution to have modifiers in AST to fix some of them :).
3. imho macros adds additional reinferring cycles that is compilation speed

Sure, macros take more time when compiling. But they save time when typing. This is the tradeoff decision made for Mirah.

1. IDE support again

I do not understand this item, apart from that a macro-less Mirah would have easier IDE support. (However, a macro-full Mirah produces less need for having an IDE in the first place, because repeating code could be optimized away – using macros...)

1. I'd like to have

  synchronized attr_reader a:int
  private synchronized attr_writer a:int

Currently it's not supported with modifiers either, but could be implemented.

Oh, you could have it like it is done for the protected macro in https://github.com/mirah/mirah/blob/0af6095bb85e63d4dd190b9eeb7e1c9dca6209e8/src/org/mirah/builtins/object_extensions.mirah#L176 .
The reason this does not work out of the box is that attr_reader and friends actually returns a NodeList instead of a MethodDefinition, as things like

   attr_writer a:int, b:String

are currently allowed.

So the discussion seems to boil down to whether Mirah macros (and thus meta-programming) are first class features of Mirah or not, whether Mirah should evolve away from macros (and thus away from the ability of meta-programming) or not. IMHO, Mirah macros (and thus meta-programming) are the killer feature over Java, and thus should be employed generously.

Because we seem to have a stalemate here, I'd like to solicit other opinions. @baroquebobcat , @headius ?

[uujava]

Regarding macro chaining, you are right. It really works, sorry to be mistaken.