Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build and publish against current Scala 2.12 milestone #245

Closed
JoshRosen opened this issue Apr 4, 2016 · 26 comments

Comments

@JoshRosen
Copy link
Contributor

commented Apr 4, 2016

It would be great if jackson-module-scala could build and publish artifacts against the current Scala 2.12 milestone (currently 2.12.0-M4). This is a blocker for building Apache Spark against Scala 2.12 (see SPARK-14222).

@JoshRosen

This comment has been minimized.

Copy link
Contributor Author

commented Apr 6, 2016

Note: now that 2.12.0-M4 is out we should compile against that and not M3 because apparently there are some binary incompatibilities between the two milestones.

@cowtowncoder

This comment has been minimized.

Copy link
Member

commented Apr 6, 2016

@JoshRosen contributions would be most welcome -- Scala module has lots of work, but limited developer resources. I don't know how much work new backend would be, although I have hard 2.12 is supposed to be easier to migrate to than earlier versions.

@JoshRosen

This comment has been minimized.

Copy link
Contributor Author

commented Apr 6, 2016

I think that Spark's use of jackson-module-scala is fairly shallow, so from my perspective it might be less effort for us to just remove this dependency from Spark. I'm exploring this option at apache/spark#12213 and if that works then I'll close this request.

@cowtowncoder

This comment has been minimized.

Copy link
Member

commented Apr 6, 2016

@JoshRosen yes, it may make sense to remove jackson scala module if so (fwtw basic jackson-databind may work well enough even from Scala). Either way, let's keep this open: I am sure 2.12 version will be needed by some users so it's good to have placeholder.

@nbauernfeind

This comment has been minimized.

Copy link
Member

commented Apr 14, 2016

I will release an M4 version of 2.7.3 in the very near future. Is 2.7.x sufficient or do you need older versions as well? To keep this up to date, can you please remember to ping us as scala 2.12 releases new milestones? New releases of the scala compiler are unfortunately no longer a day-to-day requirement for me.

@nbauernfeind

This comment has been minimized.

Copy link
Member

commented Apr 14, 2016

Hmm, I'm having trouble compiling actually. Have you seen this before?

[info] 'compiler-interface' not yet compiled for Scala 2.12.0-M4. Compiling...
java.lang.VerifyError: Uninitialized object exists on backward branch 209
Exception Details:
  Location:
    scala/collection/immutable/HashMap$HashTrieMap.split()Lscala/collection/immutable/Seq; @249: goto
  Reason:
    Error exists in the bytecode
  Bytecode:
    0x0000000: 2ab6 005c 04a0 001e b200 b4b2 00b9 04bd
    0x0000010: 0002 5903 2a53 c000 bbb6 00bf b600 c3c0
    0x0000020: 00c5 b02a b600 36b8 0040 3c1b 04a4 015e
    0x0000030: 1b05 6c3d 2a1b 056c 2ab6 0036 b700 c73e
    0x0000040: 2ab6 0036 021d 787e 3604 2ab6 0036 0210
    0x0000050: 201d 647c 7e36 05bb 0019 59b2 00b9 2ab6
    0x0000060: 0038 c000 bbb6 00cb b700 ce1c b600 d23a
    0x0000070: 0619 06c6 001a 1906 b600 d6c0 0082 3a07
    0x0000080: 1906 b600 d9c0 0082 3a08 a700 0dbb 00db
    0x0000090: 5919 06b7 00de bf19 073a 0919 083a 0abb
    0x00000a0: 0002 5915 0419 09bb 0019 59b2 00b9 1909
    0x00000b0: c000 bbb6 00cb b700 ce03 b800 e43a 0e3a
    0x00000c0: 0d03 190d b900 e801 0019 0e3a 1136 1036
    0x00000d0: 0f15 0f15 109f 0027 150f 0460 1510 190d
    0x00000e0: 150f b900 eb02 00c0 0005 3a17 1911 1917
    0x00000f0: b800 ef3a 1136 1036 0fa7 ffd8 1911 b800
    0x0000100: f3b7 0061 3a0b bb00 0259 1505 190a bb00
    0x0000110: 1959 b200 b919 0ac0 00bb b600 cbb7 00ce
    0x0000120: 03b8 00e4 3a13 3a12 0319 12b9 00e8 0100
    0x0000130: 1913 3a16 3615 3614 1514 1515 9f00 2715
    0x0000140: 1404 6015 1519 1215 14b9 00eb 0200 c000
    0x0000150: 053a 1819 1619 18b8 00f6 3a16 3615 3614
    0x0000160: a7ff d819 16b8 00f3 b700 613a 0cb2 00fb
    0x0000170: b200 b905 bd00 0259 0319 0b53 5904 190c
    0x0000180: 53c0 00bb b600 bfb6 00fe b02a b600 3803
    0x0000190: 32b6 0100 b0                           
  Stackmap Table:
    same_frame(@35)
    full_frame(@141,{Object[#2],Integer,Integer,Integer,Integer,Integer,Object[#110]},{})
    append_frame(@151,Object[#130],Object[#130])
    full_frame(@209,{Object[#2],Integer,Integer,Integer,Integer,Integer,Object[#110],Object[#130],Object[#130],Object[#130],Object[#130],Top,Top,Object[#25],Object[#60],Integer,Integer,Object[#108]},{Uninitialized[#159],Uninitialized[#159],Integer,Object[#130]})
    full_frame(@252,{Object[#2],Integer,Integer,Integer,Integer,Integer,Object[#110],Object[#130],Object[#130],Object[#130],Object[#130],Top,Top,Object[#25],Object[#60],Integer,Integer,Object[#108]},{Uninitialized[#159],Uninitialized[#159],Integer,Object[#130]})
    full_frame(@312,{Object[#2],Integer,Integer,Integer,Integer,Integer,Object[#110],Object[#130],Object[#130],Object[#130],Object[#130],Object[#2],Top,Object[#25],Object[#60],Integer,Integer,Object[#108],Object[#25],Object[#60],Integer,Integer,Object[#108]},{Uninitialized[#262],Uninitialized[#262],Integer,Object[#130]})
    full_frame(@355,{Object[#2],Integer,Integer,Integer,Integer,Integer,Object[#110],Object[#130],Object[#130],Object[#130],Object[#130],Object[#2],Top,Object[#25],Object[#60],Integer,Integer,Object[#108],Object[#25],Object[#60],Integer,Integer,Object[#108]},{Uninitialized[#262],Uninitialized[#262],Integer,Object[#130]})
    full_frame(@395,{Object[#2],Integer},{})

    at scala.collection.immutable.HashMap$.scala$collection$immutable$HashMap$$makeHashTrieMap(HashMap.scala:179)
    at scala.collection.immutable.HashMap$.scala$collection$immutable$HashMap$$makeHashTrieMap(HashMap.scala:183)
    at scala.collection.immutable.HashMap$HashMap1.updated0(HashMap.scala:211)
    at scala.collection.immutable.HashMap.$plus(HashMap.scala:59)
    at scala.collection.immutable.HashMap.$plus(HashMap.scala:62)
    at scala.collection.immutable.Map$Map4.updated(Map.scala:201)
    at scala.collection.immutable.Map$Map4.$plus(Map.scala:202)
    at scala.collection.immutable.Map$Map4.$plus(Map.scala:180)
    at scala.collection.mutable.MapBuilder.$plus$eq(MapBuilder.scala:29)
    at scala.collection.mutable.MapBuilder.$plus$eq(MapBuilder.scala:25)
    at scala.collection.generic.Growable.scala$collection$generic$Growable$$$anonfun$1(Growable.scala:59)
    at scala.collection.generic.Growable$$Lambda$19/1633498752.apply(Unknown Source)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
    at scala.collection.generic.Growable.$plus$plus$eq(Growable.scala:59)
    at scala.collection.mutable.MapBuilder.$plus$plus$eq(MapBuilder.scala:25)
    at scala.collection.generic.GenMapFactory.apply(GenMapFactory.scala:48)
    at scala.sys.package$.env(package.scala:61)
    at scala.tools.nsc.settings.ScalaSettings.defaultClasspath(ScalaSettings.scala:29)
    at scala.tools.nsc.settings.MutableSettings.defaultClasspath(MutableSettings.scala:19)
    at scala.tools.nsc.settings.ScalaSettings.$init$(ScalaSettings.scala:59)
    at scala.tools.nsc.settings.MutableSettings.<init>(MutableSettings.scala:20)
    at scala.tools.nsc.Settings.<init>(Settings.scala:12)
    at scala.tools.nsc.Driver.process(Driver.scala:41)
    at scala.tools.nsc.Main.process(Main.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:483)
    at sbt.compiler.RawCompiler.apply(RawCompiler.scala:26)
    at sbt.compiler.AnalyzingCompiler$$anonfun$compileSources$1$$anonfun$apply$2.apply(AnalyzingCompiler.scala:146)
    at sbt.compiler.AnalyzingCompiler$$anonfun$compileSources$1$$anonfun$apply$2.apply(AnalyzingCompiler.scala:142)
    at sbt.IO$.withTemporaryDirectory(IO.scala:291)
    at sbt.compiler.AnalyzingCompiler$$anonfun$compileSources$1.apply(AnalyzingCompiler.scala:142)
    at sbt.compiler.AnalyzingCompiler$$anonfun$compileSources$1.apply(AnalyzingCompiler.scala:139)
    at sbt.IO$.withTemporaryDirectory(IO.scala:291)
    at sbt.compiler.AnalyzingCompiler$.compileSources(AnalyzingCompiler.scala:139)
    at sbt.compiler.ComponentCompiler$$anonfun$compileAndInstall$1.apply(ComponentCompiler.scala:63)
    at sbt.compiler.ComponentCompiler$$anonfun$compileAndInstall$1.apply(ComponentCompiler.scala:60)
    at sbt.IO$.withTemporaryDirectory(IO.scala:291)
    at sbt.compiler.ComponentCompiler.compileAndInstall(ComponentCompiler.scala:60)
    at sbt.compiler.ComponentCompiler$$anonfun$getLocallyCompiled$1.apply$mcV$sp(ComponentCompiler.scala:50)
    at sbt.IfMissing$Define.apply(ComponentManager.scala:75)
    at sbt.ComponentManager.sbt$ComponentManager$$createAndCache$1(ComponentManager.scala:39)
    at sbt.ComponentManager$$anonfun$sbt$ComponentManager$$fromGlobal$1$1.apply(ComponentManager.scala:27)
    at sbt.ComponentManager$$anonfun$sbt$ComponentManager$$fromGlobal$1$1.apply(ComponentManager.scala:26)
    at sbt.ComponentManager$$anon$1.call(ComponentManager.scala:50)
    at xsbt.boot.Locks$GlobalLock.withChannel$1(Locks.scala:93)
    at xsbt.boot.Locks$GlobalLock.xsbt$boot$Locks$GlobalLock$$withChannelRetries$1(Locks.scala:78)
    at xsbt.boot.Locks$GlobalLock$$anonfun$withFileLock$1.apply(Locks.scala:97)
    at xsbt.boot.Using$.withResource(Using.scala:10)
    at xsbt.boot.Using$.apply(Using.scala:9)
    at xsbt.boot.Locks$GlobalLock.ignoringDeadlockAvoided(Locks.scala:58)
    at xsbt.boot.Locks$GlobalLock.withLock(Locks.scala:48)
    at xsbt.boot.Locks$.apply0(Locks.scala:31)
    at xsbt.boot.Locks$.apply(Locks.scala:28)
    at sbt.ComponentManager.lock(ComponentManager.scala:50)
    at sbt.ComponentManager.lockGlobalCache(ComponentManager.scala:49)
    at sbt.ComponentManager.sbt$ComponentManager$$fromGlobal$1(ComponentManager.scala:25)
    at sbt.ComponentManager$$anonfun$files$1$$anonfun$apply$2.apply(ComponentManager.scala:44)
    at sbt.ComponentManager$$anonfun$files$1$$anonfun$apply$2.apply(ComponentManager.scala:44)
    at sbt.ComponentManager.sbt$ComponentManager$$getOrElse$1(ComponentManager.scala:32)
    at sbt.ComponentManager$$anonfun$files$1.apply(ComponentManager.scala:44)
    at sbt.ComponentManager$$anonfun$files$1.apply(ComponentManager.scala:44)
    at sbt.ComponentManager$$anon$1.call(ComponentManager.scala:50)
    at xsbt.boot.Locks$GlobalLock.withChannel$1(Locks.scala:93)
    at xsbt.boot.Locks$GlobalLock.xsbt$boot$Locks$GlobalLock$$withChannelRetries$1(Locks.scala:78)
    at xsbt.boot.Locks$GlobalLock$$anonfun$withFileLock$1.apply(Locks.scala:97)
    at xsbt.boot.Using$.withResource(Using.scala:10)
    at xsbt.boot.Using$.apply(Using.scala:9)
    at xsbt.boot.Locks$GlobalLock.ignoringDeadlockAvoided(Locks.scala:58)
    at xsbt.boot.Locks$GlobalLock.withLock(Locks.scala:48)
    at xsbt.boot.Locks$.apply0(Locks.scala:31)
    at xsbt.boot.Locks$.apply(Locks.scala:28)
... etc

@nbauernfeind nbauernfeind reopened this Apr 14, 2016

@JoshRosen

This comment has been minimized.

Copy link
Contributor Author

commented Apr 14, 2016

@nbauernfeind, are you using a Java 8 JDK? Scala 2.12 requires JDK 8+.

@nbauernfeind

This comment has been minimized.

Copy link
Member

commented Apr 14, 2016

I believe so... but maybe I'm missing something more obvious:

$ java -version
java version "1.8.0_40"
Java(TM) SE Runtime Environment (build 1.8.0_40-b27)
Java HotSpot(TM) 64-Bit Server VM (build 25.40-b25, mixed mode)
$ echo $JAVA_HOME
/Library/Java/JavaVirtualMachines/jdk1.8.0_40.jdk/Contents/Home
@JoshRosen

This comment has been minimized.

Copy link
Contributor Author

commented Apr 14, 2016

Which SBT version are you using? I've noticed that this project doesn't seem to have a project/build.properties file for fixing the SBT version, so there's a chance we're using different versions.

@cowtowncoder

This comment has been minimized.

Copy link
Member

commented Apr 14, 2016

@nbauernfeind As to 2.7.3 vs older versions: the only one missing is 2.6.6 which would be nice to have, but not absolutely essential. It is the last patch release for 2.6, so there won't be full releases for anything but 2.7 and later. I doubt there are any fixes in 2.6 branch for scala module.

@nbauernfeind

This comment has been minimized.

Copy link
Member

commented Apr 14, 2016

I'm using sbt 0.13.11, are you? Are you sure you're compiling the 2.12 branch? That can be done by either changing the version := line or by running sbt +compile to make sure you cross compile them all.

@JoshRosen

This comment has been minimized.

Copy link
Contributor Author

commented Apr 14, 2016

@nbauernfeind, I'm also using 0.13.11. sbt clean +compile seems to work fine for me.

@nbauernfeind

This comment has been minimized.

Copy link
Member

commented Apr 15, 2016

Appears to have been some issue upstream of sbt (maybe the specific package of scala 2.12..?). It is now working for me once again. I'm going to see if I can get Option implemented properly first, but will otherwise cut a release tomorrow morning either way.

@JoshRosen

This comment has been minimized.

Copy link
Contributor Author

commented Apr 15, 2016

I have my Spark branch configured to build off of the Sonatype snapshots of this repo, so there's no huge rush to publish a new release yet in case you'd like to take extra time to fully address the Option patches.

@nbauernfeind

This comment has been minimized.

Copy link
Member

commented Apr 15, 2016

Ahh I figured out what it was. It wasn't sbt at all, but rather the version of the 1.8 jdk. One of my computers was updated enough to compile (which is why one worked). I upgraded to 1.8.0_77 and now it (basically) compiles. I get a different error on this version of the jdk, but it is an error I think I can work with.

Thanks for letting me know that you are currently not blocked! I did find what I need to do to finish the Option code which I'm working through now.

(The Error)

[error] /Users/nate/code/jackson-module-scala/src/main/java/com/fasterxml/jackson/module/scala/ScalaModule.java:7: class com.fasterxml.jackson.module.scala.ScalaModule inherits unrelated defaults for $init$() from types com.fasterxml.jackson.module.scala.deser.ScalaNumberDeserializersModule and com.fasterxml.jackson.module.scala.introspect.ScalaAnnotationIntrospectorModule
[error] public class ScalaModule extends DefaultScalaModule
@cowtowncoder

This comment has been minimized.

Copy link
Member

commented Apr 15, 2016

@nbauernfeind Ok good. I am doing additional work for Optional-like things for 2.8, to reduce boilerplate code. I know that won't be useful with 2.7 yet, but perhaps looking at ReferenceTypeSerializer and ReferenceTypeDeserializer (and subtypes for AtomicReference that extend them) can also help. There are couple of points wrt polymorphic type handling, @JsonValue, unwrapped values that are tricky to get right if starting from scratch.

@nbauernfeind

This comment has been minimized.

Copy link
Member

commented Apr 15, 2016

I took a look at both of those, but decided to follow GuavaOptionalSerializer just in case there's something special about the Atomic implementations. I'm almost there.

@cowtowncoder

This comment has been minimized.

Copy link
Member

commented Apr 15, 2016

@nbauernfeind sounds good. 2.8 can then refactor things to use common base, to unify. Jdk8 optional is similar to Scala one as well.

@nbauernfeind

This comment has been minimized.

Copy link
Member

commented Apr 18, 2016

Due to FasterXML/jackson-databind#1203 I won't be able to commit the Option as TypeReference change until 2.7.4. So, I've gone ahead and cut a 2.7.3 release including a scala 2.12-M4 jar. They should be in maven central in a few hours. I will retroactively publish jars for 2.12-M5 and 2.12.0 when the are available in the upcoming months.

I will also cut the 2.6.6 release with an artifact for scala 2.12-M4.

For jackson 2.8, I would like to stop publishing scala 2.9 artifacts so that we can start moving forward with features that can only be supported using scala reflection. What do you think the best way is to communicate, and gain support for, this with the community?

@nbauernfeind

This comment has been minimized.

Copy link
Member

commented Apr 18, 2016

Just kidding; I am not going to publish a scala 2.12.0-M4 artifact for 2.6.6. There are a few non-trivial test cases that fail. I'd rather spend my time improving 2.7.x and 2.8.x until I hear that someone desperately needs a 2.6.x artifact ;).

@JoshRosen

This comment has been minimized.

Copy link
Contributor Author

commented Apr 18, 2016

For jackson 2.8, I would like to stop publishing scala 2.9 artifacts so that we can start moving forward with features that can only be supported using scala reflection. What do you think the best way is to communicate, and gain support for, this with the community?

I wonder how many people are stuck on Scala 2.9.x but nevertheless want to use newer versions of Jackson. I suspect that this is a very small, niche group at this point.

@nbauernfeind

This comment has been minimized.

Copy link
Member

commented Apr 18, 2016

Whoops did I say Scala 2.9? I meant Scala 2.10. We deprecated 2.9 a long time ago now.

@JoshRosen

This comment has been minimized.

Copy link
Contributor Author

commented Apr 18, 2016

Ah, 2.10 is a different story. At least for now, I think that Spark is planning to keep Scala 2.10 support for our upcoming Spark 2.0.0 release (cite) (it's possible that we'll drop 2.10.x support mid-way through the Spark 2.x lifecycle but we'll keep it initially for now).

@nbauernfeind

This comment has been minimized.

Copy link
Member

commented Apr 18, 2016

Ok, in that case I'll branch a 2.10 build off of master that won't support 2.11 goodness.

@cowtowncoder

This comment has been minimized.

Copy link
Member

commented Apr 19, 2016

I think I would ask about Scala 2.10 on jackson-user list. From statistics perspective it seems to be (too?) widely used, but this may not be same as "how many would need Scala 2.10 for NEW versions of Jackson".
But at least maybe it's good to get discussion started; so perhaps keep 2.10 support for Jackson 2.8.
There is parallel "When do we require Java 8" question, which could be at earliest Jackson 2.9 I think.

@zhranklin

This comment has been minimized.

Copy link

commented Jul 14, 2016

@nbauernfeind 2.12-M5 has been released for a while. Could you compile this for it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.