Permalink
Fetching contributors…
Cannot retrieve contributors at this time
1857 lines (1270 sloc) 49.9 KB

Perlito5 Java backend

Requirements

  • building the compiler requires perl and JDK (for javac). No extra modules are required.

  • minimum Java version is Java 8

    • Perlito5 runtime uses java.time.ZonedDateTime, which was introduced in Java 8.

    • Perlito5 runtime uses java.nio.file.Files, which was introduced in Java 7.

    • Java 7 is also required for named groups in regex, like: (?<name>X).

Build using make

  • type:

    $ make
  • alternately:

    • Update the Perl-based compiler perlito5.pl

      $ make build-5to5
    • Compile the compiler into a jar file perlito5.jar

      $ make build-5java
  • run a test or two:

    $ java -jar perlito5.jar -v
    This is Perlito5 9.021, an implementation of the Perl language.
    
    $ java -jar perlito5.jar -Isrc5/lib t5/unit/array.t
    ok 1 ...
    
    • Java specific tests are in t5/java-specific directory

Perlito5-Java platform differences

  • no timely destruction (DESTROY) (because we use Java memory management)

    • files don't "auto-close" at the end of a block
    • weaken() is a no-op
    • Try::Tiny finally doesn't work
    • Object::InsideOut will not cleanup unused objects
    • SelectSaver module doesn't work
  • no XS (because we use Java instead of C)

    • many CPAN modules which use C libraries don't work
    • some CPAN modules are already ported, see: src5/lib/Perlito5X/Java/
  • some system features are not readily available in Java, such as:

    • file permissions for setuid, setgid, and sticky bit are not implemented
    • some signals are not available in Java.

Perlito5-Java work-in-progress

  • goto

    • goto &code works, but it doesn't do a tail-call
    • goto LABEL - some use patterns work.
      • TODO - test Text::Balanced module
      • TODO - test File::Copy module
    • "computed goto" is not implemented
  • signals are partially implemented

    • $SIG{__WARN__} and $SIG{__DIE__} are implemented
    • other signals are not yet implemented
    • $SIG{ALRM} and alarm() not implemented
  • object system

    • method resolution order is not selectable
    • interaction between inheritance and overloading needs more tests
    • interaction between local and method cache needs more tests
  • tied variables

    • DESTROY not used, because we use Java memory management
  • overload is partially implemented

    • overload string, number, boolean work
    • binary operators work
    • mutators and assignment work
    • fallback needs more tests
    • dereferencing, iterators, filetest not implemented
    • . operator not implemented
    • no overloading not implemented
    • overload::constant not implemented
    • nomethod not implemented
    • some optimizations may cause a lower level overload to be called, for example: call 0+ instead of +, call "" instead of .
  • file handles are partially implemented

    • open() scalarref works
    • <DATA> works
    • open binary mode vs. open utf8 needs more tests
    • files don't auto-close, because we use Java memory management
    • $| works, but STDOUT->autoflush not implemented
  • tr() is partially implemented

    • modifier switches needs some tests
  • subroutines

    • my sub x {...} not implemented
  • lvalue $#a and other expressions: substr, ternary, keys, pos

    • needs more tests
  • local @_ doesn't work yet, because @_ is special

  • smartmatch ~~ operator partially implemented

    • when and given not implemented.
  • format() function partially implemented

    • line counter and page-break not implemented
    • sentence break algorithm not implemented
  • incomplete CORE function implementations

  • reset and m?? not implemented

  • clone() is work-in-progress

    • deep clone of references is not implemented
    • clone tied values is not implemented
  • Perl threads not implemented

    • Java threads can be used, with some limitations. See: misc/Java/TestConcurrent.pl and misc/Java/TestThread2.pl

Regex differences

  • regex modifiers /ismxgecr work the same as Perl

    • /xx works
    • /ee works
    • /r works
    • /g has problems in certain backtracking zero-length-matches
    • other modifiers are not yet implemented.
  • regex variables $1, $2, ... and $&, $', $` work; other variables are not yet implemented.

  • capturing in zero-length-match has problems. Failing tests:

    t5/unit/regex_zero_length_match_capture.t
    t5/unit/regex_zero_length_match_match.t
    t5/unit/regex_zero_length_replace.t
    
  • Perlito5 allows underscores in named captures. This is not allowed in Java regex.

  • TODO - check this error message, this may need to be implemented for compatibility:

    Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/\G{ <-- HERE / at (eval 2) line 20.

Eval-string

Limitations

  • eval compilation is slow; after compilation, the code runs at native speed.

  • eval bytecode is cached - this may leak memory

    • review the ClassLoader for leaks

    • add tests for leaks

Perlito5-Java extensions

The Perlito5 Java backend doesn't support Perl XS extensions.

Instead of XS, it can use Java classes directly, and there is also a Java package that connects Perl with Java.

Import a Java class as a Perl package at compile-time, and typed variables

Java classes can be added to a Perl script using a special package import declaration:

package Date   { import => "java.util.Date" };

The new package can be used as a variable type specification:

my Date $dt = Date->new();

variable types are optional, this also works:

my $dt = Date->new();
  • (experimental) as a special case, an empty package works for importing builtin types or primitives (String, Long, long)
package Integer {}

my Integer $i;

Using typed variables

package long {}
my long $j;             # Java variable
my $var;                # Perl variable
$var = $j;              # store Java value in Perl variable
$j = $var;              # get Java value from Perl variable; cast from scalar to long is automatic

Typed variables generate efficient, native Java. There are a few restrictions:

  • Only my variables can be typed.

  • Java variables are not captured by Perl closures. This means that a variable declared in a context will not be seen inside inner subroutine declarations (named or anonymous) and eval blocks. Loops and conditionals should work fine, because these are not usually implemented as closures.

    • workaround: store the Java value in a Perl variable

Inlining Java code with Java::inline

Java::inline can be used to add simple Java expressions to a Perl script

my @arr = Java::inline ' new String[]{ "a", "b" } ';
  • Java::inline works in eval-string and in ahead-of-time (pre-compilation) mode

  • Java::inline can be used inside expressions or in statement position

  • Java exceptions can be catched with a Perl eval-block

  • example: instantiate an Object with jrunscript

    $ jrunscript -cp . -l Perl5 
    perl> my $x = Java::inline " new Object() "; say ref($x); say $x; say ($x ? "true" : "false" );
    java.lang.Object
    java.lang.Object@6680f714
    true
    
  • example: retrieve a Java Class Object

    $ jrunscript -cp . -l Perl5 
    perl> eval { my $x = Java::inline q{ Class.forName("java.lang.Thread") }; say ref($x); say $x; }
    java.lang.Class
    class java.lang.Thread
    
    • only fully qualified class names are supported.

    • support for simple names can be added; the list of classes can be extracted from the ClassLoader

  • example: Java method override using Java::inline

    # See: https://github.com/bdevetak/perl2j/tree/master/examples/myapp
    package Date  { import => "java.util.Date" };
    my Date $dateJavaObject =
        Java::inline
            'new Date() {
                public String toString() {
                    return "Hello";    
                }    
            }';
    my $dateString_pObject = $dateJavaObject->toString();   # Hello

Java fields, methods and constructors

  • examples:

    $ jrunscript -cp . -l Perl5 
    perl> my $x = Java::inline q{ (Integer)123 }
    123
    perl> my $x = Java::inline q{ (char)90 }
    Z
    perl> my $x; eval { $x = Java::inline q{ Class.forName("java.lang.Math") } }; say $x->PI
    3.141592653589793
    perl> my $x; eval { $x = Java::inline q{ Class.forName("java.lang.Integer") } }; say $x->MAX_VALUE
    2147483647
    perl> my $x; eval { $x = Java::inline q{ Class.forName("java.lang.Thread") } }; say $x->currentThread()
    Thread[main,5,main]
    perl> $x[10] = eval { Java::inline q{ Class.forName("java.lang.Thread") } }; say $x[10]->currentThread()
    Thread[main,5,main]
    
    • TODO add tests for these examples
  • new invokes a constructor

  • Simple Java objects (String, Character, Integer, Long, Short, Byte, Double, Boolean) are converted to Perl values.

  • Other Java objects are seen by Perl as "blessed references"

  • Java exceptions can be catched with Perl eval-block

  • Note that explicitly returning a plain null from Java::inline is a syntax error, because Java doesn't know which class it belongs to:

    perl> my $x; eval { $x = Java::inline q{ null } }; say $x
    /PlEval3.java:23: error: reference to set is ambiguous
      both method set(String) in PlLvalue and method set(System) in PlLvalue match
    

    workaround: cast null to a specific class:

    perl> my $x; eval { $x = Java::inline q{ (String)null } }; say $x
    

Java Perl module

The Java Perl module is meant to emulate the Java global object in Nashorn.

$ jrunscript -cp .:perlito5.jar -l Perl5
perl> push @INC, "src5/lib";
1
perl> use Java

perl> Java->type("java.lang.Thread")
class java.lang.Thread
perl> $Thread = Java->type("java.lang.Thread"); $Thread->new();
Thread[Thread-0,5,main]
perl> $Thread->new( sub { say "123" } )->start()

perl> sub UUID () { Java->type("java.util.UUID") }
perl> UUID->randomUUID()->toString()
9e8aadb3-4d81-41c6-af85-7ab7213a9945

perl> $arr = Java->type("int[]")->new(10)
[I@2cc75074
perl> $arr->length
10
  • TODO compile Java->type() to static code if the type parameter is a constant

Java extensions in runtime (work in progress)

  • Syntax

    Jython, JRuby, Groovy and Nashorn provide some examples.

    Alternately, our own "pre-compile" mode introduced some syntax we could reuse.

    See also http://search.cpan.org/dist/Inline-Java/Java.pod

  • TODO add tests - Extensions are allowed in pre-compilation mode and in eval-string mode

  • native Java variables (typed variables)

    $ jrunscript -cp . -l Perl5 
    perl> package Java::Object { import => 'java.lang.Object' };    # import Java class
    import
    perl> my Java::Object $obj = Java::Object->new();
    java.lang.Object@4fe533ff
    
  • Java objects can be assigned to Perl scalar variables, array elements, or hash elements.

  • TODO - Assign from Perl scalar expression to typed variables requires explicit coercion (or unboxing).

    Example: use the Java package to create a boxed Java array; use a typed variable declaration to create an unboxed Java array:

    $ jrunscript -cp . -l Perl5 
    perl> push @INC, "src5/lib";
    perl> use Java
    perl> package Java::Array::Of::Int { import => "java.lang.Integer[]" }  
    perl> my Java::Array::Of::Int $aa; $aa = Java->type("Integer[]")->new(10);
    error: incompatible types: PlObject cannot be converted to int[]
    

    workaround: assign the expression to a variable, and then assign between variables:

    perl> package Java::Array::Of::Int { import => "java.lang.Integer[]" }
    perl> my Java::Array::Of::Int $aa; my $bb = Java->type("java.lang.Integer[]")->new(10); $aa = $bb;
    
  • TODO - type information is lost for Byte, Character values - these are converted to long, String

    • Note that casting an argument to Byte in a method call would not work, because the "argument list" object is a Perl list: it would convert the Byte to a Perl integer.
  • TODO - Array and Hash dereference syntax is not supported

  • TODO - assign to array, assign to hash

    my $v = Java->type("int[]")->new(10);
    $v->[0] = 123;      # TODO
    @$v = (1..3);       # TODO
    for (@$v) { ... }   # TODO
    for my Integer $x (@$v) { ... }     # TODO
    
  • TODO - test varargs call with 0 arguments

  • TODO - syntax for dereferencing scalars (Java objects are stored as references)

    my $v = Java->type("int[]")->new(10);
    my Integer::Array $i = $$v;     # TODO
    
  • TODO - syntax for Java method calls - typed argument lists are work in progress

    • TODO document method resolution order
  • Perl modules using extensions can be precompiled ahead-of-time in perlito5.jar, by adding an extra use statement in src5/util/jperl.pl

    • TODO add tests - interoperation of "ahead-of-time" compiler extensions and "eval-string" extensions

Calling a Perl subroutine from Java

  • "Java Scripting API"

    • javax.script API registers "Perl5" interpreter
    • See example at misc/Java/Script.java
    // $ cp misc/Java/Script.java .
    // $ javac -cp .:perlito5.jar Script.java 
    // $ java -cp .:perlito5.jar Script
    
    import javax.script.*;
    
    public class Script {
        public static void main(String[] args) throws Exception {
        
            ScriptEngineManager factory = new ScriptEngineManager();
            ScriptEngine engine = factory.getEngineByName("Perl5");
        
            Object o = engine.eval(" $x = 456; say 123 + $x; \"value was $x\" ");
            System.out.println("result: " + o);
        }
    }
  • "jrunscript" support

    $ jrunscript -cp .:perlito5.jar -l Perl5
    perl>
    
    perl> push @INC, "src5/lib";     # initialize @INC to load Perl modules (alternately, set PERL5LIB env variable)
    perl> use Java
    
    • enable debugging output: this is the same as setting -J DEBUG=1 in the command line

      perl> $Perlito5::Java::DEBUG=1
      
  • older API (deprecated)

    • using Main.apply()
    class MyJava {
        public static void main(String[] args) throws Exception {
            Main.init();
            PlObject[] res = Main.apply("main::test", "123");
            for (PlObject s: res) {
                System.out.println("Java result: " + s.toString());
            }
        }
    }

Importing a Java class into Perl

package Sample {
    import => "misc.Java.Sample"
};

package Array::Of::String {
    import => "java.util.ArrayList<String>",
}

my Array::Of::String $x = Array::Of::String->new();

Importing Java value types (Typed variables)

Java value types don't need to be imported, but there must exist a Perl package:

package String    { }
package Long      { }
package Integer   { }
package Boolean   { }
package Short     { }
package Byte      { }
package Double    { }
package Float     { }
package Character { }

my String $x  = "abc";
my Integer $y = 123;
# assigning a Java value to a Perl variable
my $v = Boolean->TRUE;
print "$v\n";   # 1

All primitive data type declarations are supported:

package long      { }
package int       { }
package short     { }
package byte      { }
package float     { }
package double    { }
package boolean   { }
package char      { }

Other value types can be imported:

package Java::Object { import => "java.lang.Object" };
package Java::Date   { import => "java.util.Date" };
package Java::Array::Of::String { import => "java.util.ArrayList<String>" }

my Java::Object $obj = Java::Object->new();

my Java::Array::Of::String $arr = Java::Array::Of::String->new();
$arr->add("123");
$arr->add($p->toString());
  • native arrays

    TODO tests - Perl arrays can be assigned a native array:

    package Byte { };
    @perl_array = JavaCall->toBytes();

    Native array variables can not be created directly. As a workaround, use "Java::inline" or "Java->type()".

    package String { };
    my @arr = Java::inline ' new String[]{ "a", "b" } ';
    print "arr[0] $arr[0], arr[1] $arr[1]\n";
    my $arr = Java->type("int[]")->new(10);
  • Typed values examples

    • Character

      Character values are numeric (not String)

      package Character { };
      my Character $b = 50;
      my Character $b = ord("a");
      my Character $b = $v;   # cast from scalar to Character is automatic (the scalar must have a numeric value)
      package Character { };
      package String { };
      my Character $b = String->new("a")->charAt(0);

      but:

      my Character $b = "a";  # wrong: numeric value is zero
    • Long

      package Long {};
      my Long $b = 100;
      my Long $b = Long->new(100);
      my Long $b = $v;    # cast from scalar to Long is automatic
    • Float

      package Float {};
      my Float $b = 100.0;
      my Float $b = Float->new(100.0);
      my Float $b = $v;   # cast from scalar to Float is automatic
  • Type propagation problems:

    $ java -jar perlito5.jar -Isrc5/lib  -e ' package float {}; my float $b = 100/3; $x = $b; say $x; '
    error: incompatible types: possible lossy conversion from double to float
            b_103 = (100D / 3D);
    ```sh
    $ java -jar perlito5.jar -Isrc5/lib  -e ' package int {}; my int $b = 100/3; $x = $b; say $x; '
    error: incompatible types: possible lossy conversion from double to int
            b_103 = (100D / 3D);
    • TODO add a casting operation to the native code emitter in Perlito5::Java
  • Java methods with type void should not be in the last line of a Perl block. This is because Perl blocks return the last value, and void is not acceptable as a value.

    perl> package System { import => "java.lang.System" }
    import
    perl> System->gc()
    /PlEval3.java:15: error: 'void' type not allowed here
                        return PerlOp.context(want, System.gc());
    

    workaround: add a plain-perl line, such as return, undef, or 1.

    perl> System->gc(); 1
    
  • TODO more tests: Java variables and expressions as Perl subroutine parameters.

    $ java -jar perlito5.jar -Isrc5/lib  -e ' package short {}; my short $b = ord("a"); say $b;'
    

Thread safety

Perl global variables are shared between threads. This includes for example: $_, $a, $b, $/, @INC, %SIG, $0, $1, $&, $". Perl variable @_ (the parameter list) is a special case, it behaves internally like a lexical and it may be captured by closures.

local stack is shared.

Perlito also has an internal boolean stack, which is shared.

Perl lexical variables are not shared between threads. Variables captured in closures running in different threads are shared.

Perl operators are not atomic.

  • TODO - Cloning the context

    • use the Context (javax.script.ScriptContext in Perlito5ScriptEngine) to store per-thread references to PlCx and PlV instances.

    • be aware of circular references when cloning PlV instances.

    • PlStringConstant instances and method caches also need to be cloned

      • TODO - check how 'local &sub' works in threaded perl

Perlito5 Java development tools

  • Execute a single-line quick test

    $ perl perlito5.pl -Isrc5/lib -Cjava -e ' sub x { return 123, 5 } my $x = x(); say "$x" ' > Main.java ; javac Main.java ; java Main
    5
  • Rebuild the compiler

    $ make build-5to5 

make rebuilds everything, including the java-eval-string and the nodejs-based compiler

  • Perl-Java test suite

    $ make test-5java   # tests the Java-precompile backend
    $ make test-5jar    # tests the Java-eval-string backend (perlito5.jar)
    $ make test         # tests the nodejs backend

    make test should pass everything.

    make test-5java passes a few tests

  • Syntax tree

    You may find useful when debugging,

    $ perl perlito5.pl -Isrc5/lib -Cast-json -e ''
    $ perl perlito5.pl -Isrc5/lib -Cast-perl5 -e ''

to see the internal representation

  • Other

use make clean to get rid of all those .class files

Perlito5 Java backend TODO list

CPAN distribution

This documentation should be copied to file Perlito5::Java, in the CPAN distribution.

Java-specific command line options

  • specify main Class name (currently "Main")

  • have a way to port a simple .pm to a .jar

  • have a way to port a simple .pm to a .java (without a main function)

    • specify input arguments

    • specify context (list, scalar, void)

    • specify what we want to return: PlObject vs. array of strings, etc

  • -J DEBUG=1 - enable debugging output

    $ java -jar perlito5.jar -I src5/lib -J DEBUG=1 -e ' say 123 '

Java distribution

  • upload the perlito5 jar to a Maven repository

  • add more options to Makefile

    • build stand-alone, precompiled "secure" script without eval-string / without "perlito5-lib"

      $ java -jar perlito5.jar -I src5/lib --nojava_eval -Cjava t5/unit/array.t > test.java
      $ javac test.java
      $ java Main
    • build android script

    • integrate with Perlito5 in CPAN

Precompile modules

  - TODO - transparently load `pmc` files in `require` and `use`.

  - `pmc` files could be plain `jar` files.
  • add src5/lib Perl files to perlito5.jar, instead of setting -I src5/lib or PERL5LIB=src5/lib.

    • precompile the modules in lib

BEGIN blocks

  - Loops containing: BEGIN blocks, `use` statements, or named subroutines.

      - lexical variables inside loops may not behave properly if they are captured at compile-time.

  - some data structures created by BEGIN need more work for proper serialization to AST:

      - some types of aliased values, like:  `*name2 = *name1{IO}`

      - lexical variables are not shared between closures created in BEGIN blocks

  - FIXED - bug capturing BEGIN variables in eval-string:

    ```
    $ java -jar perlito5.jar -I src5/lib -Cperl5 -e ' my @v; BEGIN { @v = (123); sub x { @v }; eval " sub yy { \@v }  " } x; yy; '
    *main::x = do {;
        sub {;
            @Perlito5::BEGIN::_100_v
        }
    };
    *main::yy = do {;
        sub {;
            @v      # <--- this should be @Perlito5::BEGIN::_100_v
        }
    };
    ```

  - bug capturing BEGIN variables in eval-string:

    Variables created during runtime are not captured by BEGIN blocks in eval-string.

    In this example, the named sub declaration is compiled at BEGIN time in eval:

    ```
    $ java -jar perlito5.jar -I src5/lib -e ' my @v = (123); sub yy; eval " sub yy { \@v }  "; print yy(); '
    [empty string; expected: 123]
    ```

    Anonymous subs are not affected by this bug, because anon subs are not BEGIN time.
  • runtime error messages sometimes do not include the line number in the Perl code

    • also caller() is only partially implemented

    • BEGIN line numbers show the line number at the time of eval - the line number is relative to the start of the BEGIN block

JVM and Java compiler related

  • Java 8 triggers this problem:

    - http://stackoverflow.com/questions/30707387/troubleshoot-slow-compilation
    - http://stackoverflow.com/questions/34223249/slow-compilation-with-jooq-3-6-plain-sql-and-the-javac-compiler
    
    - "The workaround is to compile at Java 7-compatibility level: javac -source 7, or just to use simpler constructions.
    - "the workaround is to introduce local variables when there are nested generic method calls that use generic type inference
    
  • detect and fix "unreachable code"

    - See: `t5/cmd/switch.t` line 65
    
  • Workaround JVM bytecode size limit

    According to the Java Virtual Machine specification, the bytecode of a method must not be bigger than 65536 bytes:

    • See: $Perlito5::CODE_TOO_LARGE in /src5

    • Test.java:2309: error: code too large

    • when compiling misc/Java/code_too_large.pl

    • current workaround: insert a closure every 100s of lines in a block:

      code...
      return (sub {
              code ...
              return (sub {
                      code ...
                  }->() )
          }->() )

XS support using libperl.so

Perlito5-Java extensibility in "pre-compile" mode

This documentation should be copied to file Perlito5::Java, in the CPAN distribution.

  • Java import

    package Sample {
        import => "misc.Java.Sample"
    };
    • the type specification can also contain a type argument
    package Array::Of::String {
        import => "java.util.ArrayList<java.lang.String>",
    }
    
    package Array::Of::String {
        import => "java.lang.String[]",
    }
    
    • TODO: what happens when a class is imported again - for example, import Int or Byte again

    • TODO: test that Perl modules can import Java classes - only tested in main program

  • Typed variables

    my TYPE VARLIST - See also: http://perldoc.perl.org/functions/my.html

    • no global typed variables (only my variables)

    • only scalar variables (no hash, array, code)

    • Note: - parameters to native calls are evaluated in scalar context - untyped variables are passed by reference - that is, v_x instead of v_x.get() - wantarray() context is not passed to native calls

      $ perl perlito5.pl -Isrc5/lib -I. -It -Cjava -e ' package Sample { import => "misc.Java.Sample" }; my $x = Sample->new(); ' > Test.java ; javac Test.java
      my $p_put = Sample->new();
      my $p_put = new Sample();
      
    • TODO - list assignment

      my Integer ($i, $j) = (123, 456);

      • creates a boxed Java variable (DONE)

    "$x" # Sample<0x1234567>

    $x is a Perl variable that contains a native Sample; it behaves like a Perl object

    my $x = $p_put;

    • puts the boxed object into a Perl scalar (DONE)

    my Sample $put = Sample->new();

    • creates a native Java variable (DONE)

    my Int $val = Sample->VAL;

    • "method call without parenthesis"

    read a class or instance variable

    my $x = $put;

    • puts the unboxed object into a Perl scalar (DONE)

    my $x = Sample->new()

    • stores the boxed pSample object in a Perl scalar (DONE)
    package Int { import => 'java.lang.Integer' };
    my Int $x = 3;          # $x is a Java Integer (not a PlScalar)  (TODO: test)
    
    • TODO: capture typed variables in closures

      maybe TODO: allow typed variables in parameter list; but they would probably lose the type information

    • TODO test: call Perl subroutines with native parameters

    • TODO test: call Java methods with Perl parameters

      Sample->new(10);                # native int
      Sample->new("abc");             # native String
      Sample->new($v);                # cast to the declared parameter type
      Sample->new(0 + $v);            # cast to int
      Sample->new(0.0 + $v);          # cast to double
      Sample->new("" . $v);           # cast to string
    • Method chaining:

      my $global_queue = ConcurrentLinkedQueue::Of::String->new();
      my ConcurrentLinkedQueue::Of::String $queue = $global_queue;
      my $x = $queue->poll();
      
      my $x = $global_queue->poll();    # uses reflection to find the method
      
    • TODO more tests: cast perl object to java object

      my Result $java_obj_result = $scan_result;
      
    • Array-deref:

      @$native will retrieve an iterator and produce a Perl list (TODO)

      and "keys %$native" can be written like:

      @{ $native->entrySet() }
      @{ $native->keySet() }
      @{ $native->values() }
      (TODO)
      
    • Array-deref in boolean context:

      Automatic call to ->hasNext on iterator reference in boolean context. The idea is that instead

      while ($iterator->hasNext()) { ... }

      we can type:

      for (@$bar) { ... }

      First, the @$bar dereferencing of native java list would obtain an iterator reference and then due to having iterator reference in boolean context, the call to hasNext() would autmatically happen. That way we will be able to write idiomatic Perl loops on native java Lists

      TODO - this is partially implemented; add tests.

    • Native values in Perl expressions:

      in this case, we can just assign the value to PlLvalue because PlLvalue knows what to do with each type this also takes care of $x and 0+$x (TODO)

      PlLvalue.set() is overloaded - Array, Hash, int, string, boolean, ...

      src5/lib/Perlito5/Java/Runtime.pm 1463

      (TODO) comparison and logical operators with Java variables:

      if ($typed == $typed) ...
      if ($typed eq $typed) ...
      if ($typed || $typed) ...
      if ($typed && $typed) ...
      
  • interoperation between native expressions and perl expressions

    method calls on typed variables call Java methods, method calls on untyped variables call Perl methods

    subroutines are always Perl

    Class method calls on imported classes are Java, all other Class method calls are Perl.

    Parameters to Java method calls should be converted to Java automatically. (TODO: test, examples)

    storing a Java object into a Perl scalar converts the value to a Perl object (blessed reference); Java int, String, double are converted to Perl values.

    using Java objects in other types of expressions than scalar assignment is not well defined. (TODO: test, examples)

    using Perl objects in Java code is not well defined. (TODO: test, examples)

    storing a Java object into a typed variable keeps the Java object as-is.

    • test case:

      $ perl perlito5.pl -Isrc5/lib -I. -It -Cjava -e ' package my::Sample { import => "misc.Java.Sample" }; my $x = my::Sample->new(); say "ref: ", ref($x), " x ", $x; my @arr = (1,2,5); say ref(\@arr); $x = \@arr; say ref($x); my my::Sample $z = my::Sample->new(); $x = $z; 
    • (maybe deprecated) everything at the right side of ...->to_JavaThing()->... is native-call

  • native expressions

    $x++;         # autoincrement
    $x = $x + 1;  # assignment
    print $x;     # print
    • Note that native variables must be initialized:

      java -jar perlito5.jar -Isrc5/lib  -e ' package Integer {}; my Integer $x; $x ++ ' 
      error: variable x_103 might not have been initialized
              x_103++;
      
    • test case:

      package Integer {}
      my Integer $count;
      my Integer $i = 0;
      while ( $i < 400 ) {
          my Integer $j = 0;
          while ( $j < 400 ) {
              my Integer $k = 0;
              while ( $k < 400 ) {
                  $k = $k + 1;
                  $count = $count + 1;
              }
              $j = $j + 1;
          }
          $i = $i + 1;
      }
      my $c = $count;
      print "done $c\n";

Value types

  • Native array variables:

    # import a native array Class
    #
    #  package Java::Array::Of::String {
    #       import => "java.lang.String[]",
    #  }
    • not implemented:

      • Conversion from Perl scalar to native array is not implemented.
      # Possible implementation for creating native array variables
      #
      # my byte @bytes;
      # $#bytes = 9;      # indexes 0..9 (10 elements)
      # Not implemented: assign native array to Perl scalar
      #   (assign to Perl array works, see above)
      #
      # my $arr3 = Java::inline ' new String[]{ "a", "b" } ';
      # print "arr3[1] $arr3->[0]\n";
      # print "arr3[1] $arr3->[1]\n";
    • (this syntax is maybe deprecated) Conversion from Perl scalar to native array is not implemented.

      # TODO - Not implemented: Perl scalar to native array
      # using auto-generated methods
      #
      # package String { };
      # $arrayref->toStringArray()
      #
      # package int { };
      # $arrayref->to_intArray()
      #
      # package Java::Date   { import => "java.util.Date" };
      # $arrayref->to_JavaDateArray()
    • See: toArray(T[] a) in https://docs.oracle.com/javase/7/docs/api/java/util/ArrayList.html

      in Perl:

      char::Array, boolean::Array, float::Array, double::Array, long::Array, short::Array, int::Array, byte::Array
      
      # alternate implementation:
      #
      # package String { };
      # my String::Array = String::Array->new(@array)
      $ perl perlito5.pl -Isrc5/lib -I. -It -Cjava -e ' package byte::Array { type => 'byte[]' } my byte::Array $x = byte::Array->new("A","B","C");'

      this should return:

      byte[] v_x = new byte[3]; // alocate 3 bytes - the native arrays in java are not dynamic
      byte[0] ='A';
      byte[1] = 'B';
      byte[2] = 'C';

      or:

      byte[] v_x = new byte[] { "A", "B", "C" };
    • alternately:

      package my_chars { type => 'char[]' }
    • Investigate adding support for plain "Object" arguments. See http://docs.oracle.com/javase/7/docs/api/java/util/Formatter.html arguments

  • mixing Java containers, Perl, and Java::inline See: misc/Java/TypedIterator.pl

    package Iterator::Of::String {
        import => "java.util.Iterator<String>",
    };
    package ArrayList::Of::String {
       import => "java.util.ArrayList<String>",
    }
    
    sub foo {
        my $x;
        Java::inline '
            ArrayList<String> listA = new ArrayList<String>();
            listA.add("element 1");
            listA.add("element 2");
            listA.add("element 3");
        ';
        $x = Java::inline "listA";
        return $x;
    }
    
    my $bar = foo();
    my ArrayList::Of::String $arr = $bar;
    my Iterator::Of::String $iterator = $arr->iterator();
    while($iterator->hasNext()) {
      my $element = $iterator->next();
      say $element;
    }
  • autobox as-needed

    • add tests

    • runtime methods should accept String, int, double, boolean types and maybe other types of number (byte, ...)

    • Array and Hash should accept other types of containers

      • String accepts char in constructor (DONE)
      • Hash accepts String for index (DONE)
      • Array accepts int for index (DONE)

(experimental) Extending a Java class with Perl

Extending a Java class with Perl is very experimental, the API is going to change.

These extensions are experimental and may be deprecated:

  • (experimental) an extends specification works for adding methods to an existing class

    • TODO - (experimental) test syntax for creating new Java subclass ("extends" and "implements")
  • (experimental) an implements specification works for adding methods to an existing interface

  • (experimental) a header specification works for creating a Java package

# create a Java package
package header { java_path => 'org.perlito.udfs' };

# import the original Java class
package Java::Date   { import => "java.util.Date" };

# create and import the extended class
package My::Date {
    extends => 'Java::Date',
    decl => [ "public", "final" ],              # public final class
    'Java::inline' => " // ... Java code ... \n",
    methods => [
        toString => {
            decl => [ "public" ],               # public method
            args => [],                         # no arguments
            return => "String",                 # returns String
            code => "main::my_date_string",     # implemented in Perl, see below
        },
    ],
}

package main;

# Perl implementation for My::Date->toString()
sub my_date_string {
    my $self = shift;
    print "date_string: self is $self\n";
    return "Hello";
}

my Java::Date $j_date = Java::Date->new();
my $s1 = $j_date->toString();   # original class
my My::Date $date = My::Date->new();
my $s2 = $date->toString();     # extended class

print $s1, " ", $s2, "\n";   # prints date and "Hello"

Autovivification of aliased parameters

  • (DONE)

    $ perl -e ' use Data::Dumper; my %z; my $s; my @w; sub x {$_[0] = 3} x($z{d}, $s, $w[3]); print Dumper [\%z, $s, \@w] '
    $VAR1 = [
          {
            'd' => 3
          },
          undef,
          []
        ];
    $ perl -e ' use Data::Dumper; my %z; my $s; my @w; sub x {$_[1] = 3} x($z{d}, $s, $w[3]); print Dumper [\%z, $s, \@w] '
    $VAR1 = [
          {},
          3,
          []
        ];
    $ perl -e ' use Data::Dumper; my %z; my $s; my @w; sub x {$_[2] = 3} x($z{d}, $s, $w[3]); print Dumper [\%z, $s, \@w] '
    $VAR1 = [
          {},
          undef,
          [
            undef,
            undef,
            undef,
            3
          ]
        ];

Slices

DONE $scalar = delete @hash{qw(foo bar)}; # $scalar is 22
DONE @array  = delete @hash{qw(foo baz)}; # @array  is (undef,33)
url: http://perldoc.perl.org/functions/delete.html

DONE $scalar = @hash{qw(foo bar)}; # $scalar is 22
DONE @array  = @hash{qw(foo baz)}; # @array  is (undef,33)

DONE print "@hash{qw(foo bar)}" # should print: 11 22

DONE @hash{qw(foo bar)} = $scalar; # %hash{foo} is $scalar && %hash{bar} is undef
DONE @hash{qw(foo baz)} = @array; # %hash{foo} is @array[0] && %hash{bar} is @array[1]
  • TEST

    %hash = (foo => 11, bar => 22, baz => 33); $scalar = delete @hash{qw(foo bar)}; print "$scalar\n"
    %hash = (foo => 11, bar => 22, baz => 33); @array = delete @hash{qw(foo bar)}; print "@array\n"
    
    %hash = (foo => 11, bar => 22, baz => 33); $scalar = @hash{qw(foo bar)}; use feature 'say'; say $scalar
    %hash = (foo => 11, bar => 22, baz => 33); @array = @hash{qw(foo bar)}; use feature 'say'; say "@array"
    
    %hash = (foo => 11, bar => 22, baz => 33); print "@hash{qw(foo bar)}"
    
    my $scalar = 11; @hash{qw(foo bar)} = $scalar; use feature "say"; say $hash{foo}
    my @array = (11, 22); @hash{qw(foo bar)} = @array; use feature "say"; say "@array"

Variables

  • delete local EXPR

  • subroutine lookups could also be "our"-like (also method lookups)

Overflow from long to double

  • partially implemented - needs more work, tests

  • Note: integer operations may have problems with type erasure

  • example:

    $i + subr();  # subr() returns pObject instead of pInt
    
  • this needs more tests

Tail-call

caller sends a parameter to enable tail-call
callee pushes the call to a stack
caller pops from the stack and call the closures in order
  • tailcalls

    same-subroutine tailcalls could execute a redo in the current subroutine.

Missing features, or partially implemented, or untested

  • Object-related

    • UNIVERSAL::

      can
      isa
      DOES
      
    • (DONE) Scalar::blessed

    • TODO tests - unit tests (work in progress)

    • TODO tests - tests for modules in src5/lib/Perlito5X and src5/lib/Perlito5X/Java

    • TODO tests - invalidate method cache when subroutine changes or @INC changes

  • Perl features

    • overload

      • TODO - remove optimizations when one of the operators is overloaded
    • tie()

      • TODO tie filehandle

        $ java -jar perlito5.jar -Isrc5/lib  -e ' tie *FH, "NewHandle";'
        Can't locate object method "TIEHANDLE" via package "NewHandle" (perhaps you forgot to load "NewHandle"?)
        
      • how does Perl behave when tieing a SCALAR to a Handle?

      • lazy lookup: possibly incomplete impl for proxy objects, this needs more tests

      • TODO test - tie() in BEGIN

    • exceptions

      `goto`, `last`, `redo`
      
      Note: remember to cleanup the stacks (boolean, local).
      
    • continue blocks

    • dualvars

      for string-to-number optimization
      
    • unicode-strings vs. byte-strings

    • subroutine signatures

    • return (list)

    • assignment to splice

    • (DONE) sprintf

      http://docs.oracle.com/javase/7/docs/api/java/lang/String.html#format(java.lang.String,%20java.lang.Object...)
      
      http://docs.oracle.com/javase/7/docs/api/java/util/Formatter.html
      
    • pack

    • unpack

    • typeglob and symbol table operations

      • TODO - add tests (See: Symbol.pm)

        $ java -jar perlito5.jar -I src5/lib  -e ' use Data::Dumper; %This::xyz = (); $This::X::y = 0; $This::vvv = 234; %x = %This::; print Dumper (\%x); print Dumper \%This::X::; print Dumper \%::; '
        $VAR1 = {
                'X::' => *This::X::,
                'vvv' => *This::vvv,
                'xyz' => *This::xyz,
            };
        $VAR1 = {
                'y' => *This::X::y,
            };
      • TODO - < $This::X::y > is not creating a "namespace"; but < $This::X::y = 0 > does work.

    • caller

      • TODO - add tests (See: perldoc -f caller)

        $ perl -e ' sub x { print "in " . __PACKAGE__ . "\n"; v() }  x(); { package X; *x = \&main::x; x() } sub v { $v = caller(); print "called from $v\n" } '
        in main
        called from main
        in main
        called from main
      • TODO - line number is off by 1

      • TODO - anonymous subroutines and eval string are not shown in the call stack

        $ java -jar perlito5.jar -I src5/lib -e ' sub x { print "@{[ caller($_) ]}\n" for 0..3; } sub yy { eval "x()"; }; ( sub { yy() } )->(); '
        main -e 1 main::x
        main -e 1 main::yy
        
        $ perl -e ' sub x { print "@{[ caller($_) ]}\n" for 0..3; } sub yy { eval "x()"; }; ( sub { yy() } )->(); '
        main (eval 1) 1 main::x 1    0
        main -e 1 (eval) 0  x()  256
        main -e 1 main::yy 1    0
        main -e 1 main::__ANON__ 1    256
      • namespaces need more work:

        $ perl -e ' package AAA; sub x { my $v = caller; print "caller: $v\n"; print "$_: @{[ caller($_) ]}\n" for 0..3; }; package BBB; sub yy {AAA::x(); }; package CCC; sub cc { BBB::yy() }; cc(); '
        caller: BBB
        0: BBB -e 1 AAA::x 1    0  
        1: CCC -e 1 BBB::yy 1    0  
        2: CCC -e 1 CCC::cc 1    256  
        3: 
        
        $ java -jar perlito5.jar -I src5/lib -I . -e ' package AAA; sub x { my $v = caller; print "caller: $v\n"; print "$_: @{[ caller($_) ]}\n" for 0..3; }; package BBB; sub yy {AAA::x(); }; package CCC; sub cc { BBB::yy() }; cc(); '
        caller: BBB
        0: BBB -e 1 AAA::x
        1: CCC -e 1 BBB::yy
        2: main -e 1 CCC::cc
        3:    
    • __DATA__ sections

      `%Perlito5::DATA_SECTION` contains the `__DATA__` for each package
      
    • add test:

      `print " ${__PACKAGE__} \n"`
      
  • BUG - auto modification is not lazy:

    $ perl -e ' $c{$_} //= scalar keys %c for qw(a b c a b d); use Data::Dumper; print Dumper \%c '
    $VAR1 = {
              'b' => 2,
              'a' => 1,
              'd' => 4,
              'c' => 3
            };
    
    $ jrunscript -cp .:perlito5.jar -l Perl5
    perl> $c{$_} //= scalar keys %c for qw(a b c a b d); use Data::Dumper; print Dumper \%c
    $VAR1 = {
            'a' => 0,
            'b' => 1,
            'c' => 2,
            'd' => 3,
        };
    
  • Add tests

    • NaN, Inf, -0

    • @perl_array = java_native[]

      supported types: byte[], int[], and imported_class[]
      
      not implemented: long[], String[], Double[], char[]
      

Regex

  • Quotemeta: \Q

  • Modifiers are not serialized yet (DONE - add tests)

    $ perl -e ' my $user_agent_regexp = "123";  my $regexp = qr/$user_agent_regexp/x; print $regexp; '
    (?^x:123)
  • but a qr// on an existing qr// behaves differently:

    $ perl -le ' my $x = qr/ (\w) /; my $z = qr/$x/x; print $z '
    (?^: (\w) )
    
    $ perl -le ' my $x = qr/ (\w) /; my $z = qr/before $x after/x; print $z '
    (?^x:before (?^: (\w) ) after)

See also:

Storable

See:

Memory management

Threads

  • See: http://perldoc.perl.org/perlmod.html#Making-your-module-threadsafe CLONE, CLONE_SKIP

  • global variables should be cloned in threads, use this to get the thread id: Thread.currentThread().getId()

  • Examples:

    misc/Java/TestConcurrent.pl
    misc/Java/TestThread.pl
    misc/Java/TestThread2.pl    # atomic
    
  • pass the threadid as a parameter to all closures;

    closure can use that thread id to get/set the value from globals hash.

  • TODO - per-thread: "local" stack, "boolean" stack, regex_result

Optimizations

  • inline the block in map, grep, sort

  • do-block and eval-block in void-context don't need a subroutine wrapper

  • don't pre-expand ranges in loops (DONE)

  • replace regex with index

  • use continue and break when possible (in place of Perl next, last)

  • identify variables that don't need a true lvalue container; store in a PerlObject instead of PerlLvalue, this is one less level of indirection.

  • investigate performance of "proxy" lvalues; when taking an lvalue out of an array or hash, return a proxy with a reference to the container and index.

  • investigate using Lambdas: https://stackoverflow.com/questions/26257266/are-java-8-lambdas-compiled-as-inner-classes-methods-or-something-else

  • investigate bundling together the calls to Perlito5::Grammar::Block::eval_begin_block() to reduce the java compiler initialization overhead.

    eval_begin_block() is currently called for each named subroutine definition.

Possible workarounds for slow compilation:

  • Java ASM

    • TODO: prototype eval-string with ASM
    • See: misc/Java_asm/
  • write a tiny interpreter for simple expressions

  • preload modules in src5/util/jperl.pl

Bootstrapping using perlito5-lib.jar

  • bootstrapping is not possible with perlito5.jar, because it is built without the grammar modules.

    $ perl perlito5.pl --bootstrapping -Isrc5/lib -Cjava src5/util/perlito5.pl > Main.java
    
    $ javac -J-Xms2000m -J-Xmx2000m -J-Xss1000m -source 7 Main.java
    # errors - TODO - fixme
    #   this is because the compiler uses eval-string
    #   try: --nojava_eval
    
    # test the bootstrapping
    $ java Main --bootstrapping -Isrc5/lib -Cjava src5/util/perlito5.pl > Main2.java
    $ diff Main.java Main2.java
    [ no differences ]
  • Using the perlito5-lib.jar file

    • TODO - explain this

      import org.perlito.Perlito5.*;

Modules

  • ported modules are in src5/lib/Perlito5X/Java and src5/lib/Perlito5X

  • there is a port of JSON.pm - it is pure-perl and slow. It would be nice to have a native-java version.

    update: src5/lib/Perlito5X/Java/JSON.pm uses javax.json.stream.JsonParser but this JsonParser class doesn't seem to be installed by default.

  • TODO: DBI.pm

Attributes

$ perl -e ' sub MODIFY_CODE_ATTRIBUTES { print "called with (@_)\n"; return }  sub xxx :Integer { print 123 } '
called with (main CODE(0x7fb99a02db18) Integer)