New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[rebase of #2206] SI-874 JSR-223 compliance for the interpreter #2238

Merged
merged 2 commits into from Mar 12, 2013

Conversation

Projects
None yet
10 participants
@adriaanm
Member

adriaanm commented Mar 12, 2013

review by @rjolly to make sure I didn't butcher anything during the rebase

@rjolly

This comment has been minimized.

Contributor

rjolly commented Mar 12, 2013

LGTM
(now that the repl is modularized, perhaps I could add jarlister again ?)

@ghost ghost assigned adriaanm Mar 12, 2013

@adriaanm

This comment has been minimized.

Member

adriaanm commented Mar 12, 2013

The modularisation isn't done yet. We won't stop until these things are in separate github projects, with their own artifacts. We want to be able to accept and publish improvements to them independently of the compiler's schedule.

adriaanm added a commit that referenced this pull request Mar 12, 2013

Merge pull request #2238 from adriaanm/pr-2206-rebased
[rebase of #2206] l JSR-223 compliance for the interpreter

@adriaanm adriaanm merged commit 5262cd5 into scala:master Mar 12, 2013

1 check passed

default pr-checkin-per-commit Took 92 min.
Details
defineClass(name, bytes, 0, bytes.length, protectionDomain)
}
lazy val protectionDomain = {

This comment has been minimized.

@paulp

paulp Mar 14, 2013

Contributor

It's my bad for letting adriaan handle these delicate matters and not managing any review myself, but there is a lot wrong with this method, among which is that it breaks running against anything which isn't a jar because regular paths don't have a ! in them.

This comment has been minimized.

@rjolly

rjolly Mar 14, 2013

Contributor

My proposed fix : rjolly@97e9d56

This comment has been minimized.

@rjolly

rjolly Mar 15, 2013

Contributor

What about rjolly@be6f97c ?

@retronym

This comment has been minimized.

@takawitter

This comment has been minimized.

takawitter commented Apr 29, 2013

Here is the code to invoke scala-2.11.0-M2 interpreter through JSR223.
https://gist.github.com/takawitter/5479445

Writing the code, I found two problems.
Two are critical:

  1. We need to turn usejavacp option.
  2. Use native bind method to bind variables.
    These are appeared at the code as the problem that must be solved using Scala native interface.

Former problem is more serious when the Scala interpreter invoked in webapp.
When application executed as webapp, the java.class.path has paths needed to run webapp container
and don't have paths needed to run each webapp.
So when Scala interpreter executed in webapp, this webapp must setup java.class.path for scala
to be able to find the classes in WEB-INF/lib/*.jar and WEB-INF/classes.

@NicolasRouquette

This comment has been minimized.

NicolasRouquette commented Jan 10, 2014

Thanks to all who worked on building support for JSR-233 scala scripting!
I managed to use Scala 2.11.0M7 in the Mule ESB runtime!
See: https://www.mulesoft.org/jira/browse/STUDIO-4330

  • Nicolas.
@adriaanm

This comment has been minimized.

Member

adriaanm commented Jan 10, 2014

Cool! Thanks for letting us know (and for sharing workarounds!).

Always nice to hear about successes -- makes for a nice change from bug reports :-)

@atomixnmc

This comment has been minimized.

atomixnmc commented Apr 15, 2014

It will be nice if any news in the progress making Scala integrate with JSR-223 going to the scala-lang.org asap! Thanks and keep rocking!

@som-snytt

This comment has been minimized.

Contributor

som-snytt commented Apr 15, 2014

I have a commit that refactors the JSR-223 with improvements, under issue 8441, once 2.11 snaps out of its funk.

mpociecha added a commit to mpociecha/scala that referenced this pull request Nov 21, 2014

Finish flat classpath implementation
Finished implementations for various classpath types - also using ManifestResources.
In accordance with the idea and the initial implementation of @gkossakowski they
tries to make the best use of the specificity of a given file type instead of using
AbstractFile everywhere. From now flat classpath really works and can be turned on.

There are following types of flat classpath:

for both source and class files:
- for directories - it uses the real directory having representation on a disk
- for zip and jar files - it uses FileZipArchive - Grzegorz was doing some
experiments with different ways of handling such files (efficiency etc.) and
stated that this one is the best.

- aggregate - equivalent of MergedClassPath from the old implementation. It groups
classpath instances created for different files. It ensures the distinction of cp
entries and merges corresponding entries for class and source files into one entry
as SymbolLoaders class makes use of that.

only for class files:

- for ManifestResources - it's closely related to the support for JSR-223 and to
classes listed in the jar's manifest. In general right now it's not possible to use
it in Scala out of the box (without using additional tools) as this support is
postponed. The old classpath has been properly prepared in the PR created by
@rjolly scala#2238 so the new one also got this feature.
ManifestResources is a ZipArchive without a real underlying file placed on a disk and
in addition with some unsupported methods from AbstractFile so the implementation has
to use the iterator provided by AbstractFile.

There's added caching for classpath for zip and jar files and for ManifestResources
which can be turned off using a flag -YdisableFlatCpCaching. The cache is just a map
AbstractFile -> FlatClassPath what should reduce the number of created cp and file
instances e.g. in the case of many ScalaPresentationCompilers in Scala IDE. There's
no added cache for loaded symbols etc. (honestly I didn't look at that - I'm even not
sure how much sense would have such things).

In SymbolLoaders there's added PackageLoaderUsingFlatClassPath that was needed as
the new implementation uses relatively few cp objects which can return entries from
many packages. In the case of old implementation there's the structure of nested
classpath objects, where each such an object can return only entries from one level of
hierarchy but it returns also another classpath objects for nested levels included in it.

I removed sbt compiler interface hack - instead of it there's introduced the dynamic
dispatch in Global's classPath method. The type of classPath changed from
ClassPath[AbstractFile] to ClassFileLookup[AbstractFile] but it should be okay as
ClassFileLookup defines needed methods. I tested it with sbt and everything seemed
to work correctly. But it could be problematic if someone uses classPath from Global
and requires exactly the old implementation. I'm not aware of such cases. In the case
of compatibility problems with some libraries using compiler's internals we could
think about bringing back some hacks. But it would be better to avoid this.

I created PathResolverFactory which is used instead of a concrete implementation of
PathResolver - it dispatches to the proper classpath implementation so, in combination
with matches in some other places, all places using classpath create/get the proper type
of it.

I changed scalap's Main.scala to be able to use both implementations and also to use
flags related to the classpath implementation. By the way the structure of its methods
has been refactored - there's no change in a way in which scalap works.

The classpath invalidation is modified to work properly for the old cp implementation
after changes made in Global. In the case of the attempt to use the invalidation for
the flat cp right now it just throws exception with a message that the flat one
currently doesn't support the invalidation. And also that's why the partest's test for
the invalidation has been changed to use (always) the old implementation. There's added
proper comment with TODO to this file.

As I always do - I removed unused private methods, unused imports and commented lines which
I saw by the way and were obvious for me. In the case of less obvious commented lines
I added TODOs as someone should look at such places some day and clean them up.

mpociecha added a commit to mpociecha/scala that referenced this pull request Nov 24, 2014

Finish flat classpath implementation
Finished implementations for various classpath types - also using ManifestResources.
In accordance with the idea and the initial implementation of @gkossakowski they
try to make the best use of the specificity of a given file type instead of using
AbstractFile everywhere. From now flat classpath really works and can be turned on.

There are following types of flat classpath:

for both source and class files:
- for directories - it uses the real directory having representation on a disk
- for zip and jar files - it uses FileZipArchive - Grzegorz was doing some
experiments with different ways of handling such files (efficiency etc.) and
stated that this one is the best.

- aggregate - equivalent of MergedClassPath from the old implementation. It groups
classpath instances created for different files. It ensures the distinction of cp
entries and merges corresponding entries for class and source files into one entry
as SymbolLoaders class makes use of that.

only for class files:

- for ManifestResources - it's closely related to the support for JSR-223 and to
classes listed in the manifest file placed in the JAR. In general right now it's
not possible to use it in Scala out of the box (without using additional tools
such as jarlister) as this support is postponed. The old classpath has been properly
prepared in the PR created by @rjolly scala#2238 so the new one also got this feature.
ManifestResources is a ZipArchive without a real underlying file placed on a disk and
in addition implementing some methods declared in AbstractFile as unsupported
operations. Therefore the implementation has to use the iterator provided by AbstractFile.

There's added caching for classpath for zip and jar files and also for ManifestResources.
It can be turned off using a flag -YdisableFlatCpCaching. The cache is just a map
AbstractFile -> FlatClassPath what should reduce the number of created cp and file
instances e.g. in the case of many ScalaPresentationCompilers in Scala IDE. There's
no added cache for loaded symbols etc. (honestly I didn't look at that - I'm even not
sure how much sense would have such things).

In SymbolLoaders there's added PackageLoaderUsingFlatClassPath that was needed as
the new implementation uses relatively few cp objects which can return entries from
many packages. In the case of old implementation there's the structure of nested
classpath objects, where each such an object can return only entries from one level of
hierarchy but it returns also another classpath objects for nested levels included in it.

The sbt compiler interface hack is removed - instead of it there's introduced the dynamic
dispatch in a Global's classPath method. The type of a classPath is changed from
ClassPath[AbstractFile] to ClassFileLookup[AbstractFile] but it should be okay as
ClassFileLookup defines needed methods. I tested it with sbt and everything seemed
to work correctly. But it could be problematic if someone uses a classPath from a Global
and requires exactly the type of the old implementation. I'm not aware of such cases.
In the case of compatibility problems with some libraries using compiler's internals we
could think about bringing back some hacks. But it would be better to avoid this.

There's added PathResolverFactory which is used instead of a concrete implementation of
a PathResolver - it dispatches to the proper classpath implementation so, in combination
with matches in some other places, in all places using classpath we create/get the proper
type.

Scalap's Main.scala is changed to be able to use both implementations and also to use
flags related to the classpath implementation. By the way the structure of its methods
has been refactored - there's no change in a way scalap works.

The classpath invalidation is modified to work properly for the old cp implementation
after changes made in a Global. In the case of the attempt to use the invalidation for
the flat cp it just throws exception with a message that the flat one currently doesn't
support the invalidation. And also that's why the partest's test for the invalidation
has been changed to use (always) the old implementation. There's added an adequate
comment with TODO to this file.

As I always do - I removed unused private methods, unused imports and commented out lines
which I saw by the way and were obvious for me. In the case of less obvious commented out
lines I added TODOs as someone should look at such places some day and clean them up.

mpociecha added a commit to mpociecha/scala that referenced this pull request Dec 1, 2014

Add flat classpath implementation using ManifestResources
There's added the flat classpath type using ManifestResources,
closely related to the support for JSR-223 (Scripting for the Java
Platform). It uses classes listed in the manifest file placed in the
JAR. It's related to jar files so it's created using
ZipAndJarFlatClassPathFactory and is cached.

In general currently it's not possible to use it in Scala out of the
box (without using additional tools such as jarlister) as this
support is postponed. The old classpath has been properly prepared in
the PR created by @rjolly scala#2238
so the new one also got this feature.

ManifestResources is a ZipArchive without a real underlying file
placed on a disk and in addition implementing some methods declared
in AbstractFile as unsupported operations. Therefore the
implementation has to use the iterator. I wanted to have the similar
behaviour as in the case of directories and zip/jar files - be able
to get a directory entry for a package without iterating all entries.
This is achieved by iterating all entries only once and caching
packages.

This flat classpath type was the last needed one.
@duffqiu

This comment has been minimized.

duffqiu commented Mar 2, 2015

Anyone can give us an example how to use scripting in scala standard app not in REPL?
Thanks

@rjolly

This comment has been minimized.

Contributor

rjolly commented Mar 2, 2015

@duffqiu does the @takawitter gist above not fit your need ?

@duffqiu

This comment has been minimized.

duffqiu commented Mar 3, 2015

No. It misses something I figure out now.
Here is my example:

  val m = new ScriptEngineManager()
  val engine = m.getEngineByName("scala")

  val settings = engine.asInstanceOf[scala.tools.nsc.interpreter.IMain].settings
  settings.usejavacp.value = true  //here is the critical stuff


  engine.put("m", 10)
  engine.eval("1 to m.asInstanceOf[Int] foreach println")

if we miss settings.usejavacp.value = true we always get the "not found xxxx" error or null point exception.

The optional ways is to use the -Xbootclasspath/a:/<xxx.jar> way to include all the scala jar files when call java -jar

@rjolly

This comment has been minimized.

Contributor

rjolly commented Mar 3, 2015

Yes, it's in @takawitter 's gist. Note that, as I said in the gist comments, this is not the right way to do it, which is to "jarlist" scala-library.jar . I am going to write a FAQ.

@duffqiu

This comment has been minimized.

duffqiu commented Mar 3, 2015

please do that. If you could push the scala-lang is more better as the tutorial

BTW, jarlist is a standard tool?

@mpociecha

This comment has been minimized.

Member

mpociecha commented Mar 3, 2015

Hi. No, jarlister is not the standard Scala tool. You can get it from rjolly's repository (https://github.com/rjolly/jarlister) and then use it to modify your jar files. @rjolly will provide more details, I think.

@duffqiu

This comment has been minimized.

duffqiu commented Mar 3, 2015

I think scala office needs to provide a standard way how to use scala as script. I saw @rjolly 's jarlister, but it is 2 years ago. So I am afraid to use it. -:)
I think at lease we need to update the scala lib version in jarlister command.

@duffqiu

This comment has been minimized.

duffqiu commented Mar 9, 2015

Hi @takawitter,

Any update on the FAQ?

@rjolly

This comment has been minimized.

Contributor

rjolly commented Mar 9, 2015

@duffqiu I have updated the scala version in the jarlister build file.

@takawitter

This comment has been minimized.

takawitter commented Mar 9, 2015

@duffqiu I don't figure out what's the problem.
Setting usejavacp option is needed by design (I guess that one reason may be a security).
Currently we have three choices to control class path that scala interpreter uses

  1. Use jarlister to listup classes into manifest.
  2. Turn usejavacp option on. (But webapps or some apps executed on the container such like webapp server can't adopt this way)
  3. Forget about JSR223, use scala native classes and build classpath yourself.
@takawitter

This comment has been minimized.

takawitter commented Mar 10, 2015

I found that usemanifestcp is valid only for jars (invalid for directories in classpath) because of
this linehttps://github.com/scala/scala/blob/124cf2f62f559caf37a5d5df7e15db7ba5958bcf/src/compiler/scala/tools/nsc/util/ClassPath.scala#L119
.
Is that expected behavior?
Is there any document that describes the concepts and specs of usemanifestcp?

@rjolly

This comment has been minimized.

Contributor

rjolly commented Mar 10, 2015

Yes it is on purpose. Manifest is only meaningful for jars. Hence classes in directories cannot be made available through this mechanism. I don't know what it would imply to implement it, but I don't think it is indispensable (one can always make a jar file). For further info regarding this and other questions see http://jscl-meditor.sourceforge.net/scalaint.pdf http://jscl-meditor.sourceforge.net/scalaint-slides.pdf

rjolly added a commit to rjolly/scala that referenced this pull request Apr 19, 2015

Jarlister utility with ant task and man page
To take full advantage of JSR-223 compatibility, added in 2.11, one must
list a jar's classes into its manifest, to make the library visible from
the interpreter when run with jrunscript. This is necessary for
scala-library.jar, but also any third party library. This commit provides
a jarlister utility, together with ant task and documentation. I made an
independent jarlister project available on github, but users are afraid of
using it, as it is not official, see the comments in
scala#2238

rjolly added a commit to rjolly/scala that referenced this pull request Jun 15, 2015

Jarlister utility with ant task and man page
To take full advantage of JSR-223 compatibility, added in 2.11, one must
list a jar's classes into its manifest, to make the library visible from
the interpreter when run with jrunscript. This is necessary for
scala-library.jar, but also any third party library. This commit provides
a jarlister utility, together with ant task and documentation. I made an
independent jarlister project available on github, but users are afraid of
using it, as it is not official, see the comments in
scala#2238

rjolly added a commit to rjolly/scala that referenced this pull request Jul 14, 2015

Jarlister utility with ant task and man page
To take full advantage of JSR-223 compatibility, added in 2.11, one must
list a jar's classes into its manifest, to make the library visible from
the interpreter when run with jrunscript. This is necessary for
scala-library.jar, but also any third party library. This commit provides
a jarlister utility, together with ant task and documentation. I made an
independent jarlister project available on github, but users are afraid of
using it, as it is not official, see the comments in
scala#2238
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment