Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Problem in compiling on Ubuntu 11.10 #7

Closed
hgavert opened this issue Feb 9, 2012 · 8 comments
Closed

Problem in compiling on Ubuntu 11.10 #7

hgavert opened this issue Feb 9, 2012 · 8 comments

Comments

@hgavert
Copy link

hgavert commented Feb 9, 2012

Firstly, I'm not familiar with ruby, sbt or scalding and only starting scala... so I might be doing something obviously wrong. So, here is what I just tried to do. This is Ubuntu 11.10 - and system fully up to date with apt-get. So, check the scala version:

me@ulap:~/dev/scala$ scalac -version
Scala compiler version 2.9.0.1 -- Copyright 2002-2011, LAMP/EPFL

Then get SBT

# https://github.com/harrah/xsbt/wiki/Getting-Started-Setup
me@ulap:~/bin$ wget http://typesafe.artifactoryonline.com/typesafe/ivy-releases/org.scala-tools.sbt/sbt-launch/0.11.2/sbt-launch.jar
# echo java -Xmx512M -jar `dirname $0`/sbt-launch.jar "$@" > sbt

Then get scalding:

me@ulap:~/dev/scala$ git clone http://github.com/twitter/scalding.git
Cloning into scalding...
remote: Counting objects: 134, done.
remote: Compressing objects: 100% (97/97), done.
remote: Total 134 (delta 40), reused 110 (delta 16)
Receiving objects: 100% (134/134), 78.49 KiB, done.
Resolving deltas: 100% (40/40), done.

Next change the scala version & compiler

# edit built.sbt
# scalaVersion := "2.9.0.1"

# edit scripts/scald.rb
# COMPILE_CMD="scalac"

However running sbt update with this version fails:

me@ulap:~/dev/scala/scalding$ ~/bin/sbt update
[info] Loading project definition from /home/me/dev/scala/scalding/project
[info] Updating {file:/home/me/dev/scala/scalding/project/}default-b80ea6...
[info] Resolving com.eed3si9n#sbt-assembly;0.7.3 ...
[info] Resolving org.scala-tools.sbt#sbt_2.9.1;0.11.2 ...
[info] Resolving org.scala-tools.sbt#main_2.9.1;0.11.2 ...
[info] Resolving org.scala-tools.sbt#actions_2.9.1;0.11.2 ...
[info] Resolving org.scala-tools.sbt#classfile_2.9.1;0.11.2 ...
[info] Resolving org.scala-tools.sbt#io_2.9.1;0.11.2 ...
[info] Resolving org.scala-tools.sbt#control_2.9.1;0.11.2 ...
[info] Resolving org.scala-lang#scala-library;2.9.1 ...
[info] Resolving org.scala-tools.sbt#interface;0.11.2 ...
[info] Resolving org.scala-tools.sbt#logging_2.9.1;0.11.2 ...
[info] Resolving org.scala-tools.sbt#process_2.9.1;0.11.2 ...
[info] Resolving org.scala-tools.sbt#classpath_2.9.1;0.11.2 ...
[info] Resolving org.scala-tools.sbt#launcher-interface_2.9.1;0.11.2 ...
[info] Resolving org.scala-lang#scala-compiler;2.9.1 ...
[info] Resolving org.scala-tools.sbt#incremental-compiler_2.9.1;0.11.2 ...
[info] Resolving org.scala-tools.sbt#collections_2.9.1;0.11.2 ...
[info] Resolving org.scala-tools.sbt#api_2.9.1;0.11.2 ...
[info] Resolving org.scala-tools.sbt#persist_2.9.1;0.11.2 ...
[info] Resolving org.scala-tools.sbinary#sbinary_2.9.0;0.4.0 ...
[info] Resolving org.scala-tools.sbt#compile_2.9.1;0.11.2 ...
[info] Resolving org.scala-tools.sbt#ivy_2.9.1;0.11.2 ...
[info] Resolving org.apache.ivy#ivy;2.2.0 ...
[info] Resolving com.jcraft#jsch;0.1.31 ...
[info] Resolving commons-httpclient#commons-httpclient;3.1 ...
[info] Resolving commons-logging#commons-logging;1.0.4 ...
[info] Resolving commons-codec#commons-codec;1.2 ...
[info] Resolving org.scala-tools.sbt#completion_2.9.1;0.11.2 ...
[info] Resolving jline#jline;0.9.94 ...
[info] Resolving org.scala-tools.sbt#run_2.9.1;0.11.2 ...
[info] Resolving org.scala-tools.sbt#task-system_2.9.1;0.11.2 ...
[info] Resolving org.scala-tools.sbt#tasks_2.9.1;0.11.2 ...
[info] Resolving org.scala-tools.sbt#tracking_2.9.1;0.11.2 ...
[info] Resolving org.scala-tools.sbt#cache_2.9.1;0.11.2 ...
[info] Resolving org.scala-tools.sbt#testing_2.9.1;0.11.2 ...
[info] Resolving org.scala-tools.testing#test-interface;0.5 ...
[info] Resolving org.scala-tools.sbt#compiler-interface;0.11.2 ...
[info] Resolving org.scala-tools.sbt#precompiled-2_8_1;0.11.2 ...
[info] Resolving org.scala-tools.sbt#precompiled-2_8_0;0.11.2 ...
[info] Resolving org.scala-tools.sbt#precompiled-2_9_0;0.11.2 ...
[info] Done updating.
[info] Set current project to scalding (in build file:/home/me/dev/scala/scalding/)
Getting Scala 2.9.0.1 ...

:: problems summary ::
:::: WARNINGS
                module not found: org.scala-lang#scala-compiler;2.9.0.1

        ==== local: tried

          /home/me/.ivy2/local/org.scala-lang/scala-compiler/2.9.0.1/ivys/ivy.xml

        ==== typesafe-ivy-releases: tried

          http://repo.typesafe.com/typesafe/ivy-releases/org.scala-lang/scala-compiler/2.9.0.1/ivys/ivy.xml

        ==== Maven Central: tried

          http://repo1.maven.org/maven2/org/scala-lang/scala-compiler/2.9.0.1/scala-compiler-2.9.0.1.pom

        ==== Scala-Tools Maven2 Repository: tried

          http://scala-tools.org/repo-releases/org/scala-lang/scala-compiler/2.9.0.1/scala-compiler-2.9.0.1.pom

        ==== Scala-Tools Maven2 Snapshots Repository: tried

          http://scala-tools.org/repo-snapshots/org/scala-lang/scala-compiler/2.9.0.1/scala-compiler-2.9.0.1.pom

                module not found: org.scala-lang#scala-library;2.9.0.1

        ==== local: tried

          /home/me/.ivy2/local/org.scala-lang/scala-library/2.9.0.1/ivys/ivy.xml

        ==== typesafe-ivy-releases: tried

          http://repo.typesafe.com/typesafe/ivy-releases/org.scala-lang/scala-library/2.9.0.1/ivys/ivy.xml

        ==== Maven Central: tried

          http://repo1.maven.org/maven2/org/scala-lang/scala-library/2.9.0.1/scala-library-2.9.0.1.pom

        ==== Scala-Tools Maven2 Repository: tried

          http://scala-tools.org/repo-releases/org/scala-lang/scala-library/2.9.0.1/scala-library-2.9.0.1.pom

        ==== Scala-Tools Maven2 Snapshots Repository: tried

          http://scala-tools.org/repo-snapshots/org/scala-lang/scala-library/2.9.0.1/scala-library-2.9.0.1.pom

                ::::::::::::::::::::::::::::::::::::::::::::::

                ::          UNRESOLVED DEPENDENCIES         ::

                ::::::::::::::::::::::::::::::::::::::::::::::

                :: org.scala-lang#scala-compiler;2.9.0.1: not found

                :: org.scala-lang#scala-library;2.9.0.1: not found

                ::::::::::::::::::::::::::::::::::::::::::::::



:: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
unresolved dependency: org.scala-lang#scala-compiler;2.9.0.1: not found
unresolved dependency: org.scala-lang#scala-library;2.9.0.1: not found
Error during sbt execution: Error retrieving required libraries
  (see /home/me/.sbt/boot/update.log for complete log)
[error] {file:/home/me/dev/scala/scalding/}default-bc2517/*:scala-instance: xsbti.RetrieveException: Could not retrieve Scala 2.9.0.1
[error] Total time: 3 s, completed Feb 9, 2012 9:22:34 AM
me@ulap:~/dev/scala/scalding$

So, try with 2.9.0

# so, chance the version to 2.9.0
# edit built.sbt
# scalaVersion := "2.9.0"

sbt update works:

me@ulap:~/dev/scala/scalding$ ~/bin/sbt update
...
[success] Total time: 3 s, completed Feb 9, 2012 9:27:08 AM

Then sbt test - which fails like this:

me@ulap:~/dev/scala/scalding$ ~/bin/sbt test
[info] Loading project definition from /home/me/dev/scala/scalding/project
[info] Set current project to scalding (in build file:/home/me/dev/scala/scalding/)
[info] Compiling 18 Scala sources to /home/me/dev/scala/scalding/target/scala-2.9.0/classes...
[error] /home/me/dev/scala/scalding/src/main/scala/com/twitter/scalding/Source.scala:145: type mismatch;
[error]  found   : Seq[cascading.tap.Tap[_ <: cascading.flow.FlowProcess[_], _, _, _]]
[error]  required: Seq[cascading.tap.Tap[?>: Nothing <: cascading.flow.FlowProcess, ?>: Nothing <: Any, ?>: Nothing <: Any, ?>: Nothing <: Any]]
[error]           case _ => new MultiSourceTap(taps.toSeq : _*)
[error]                                             ^
[error] /home/me/dev/scala/scalding/src/main/scala/com/twitter/scalding/Source.scala:173: type mismatch;
[error]  found   : Seq[cascading.tap.Tap[_ <: cascading.flow.FlowProcess[_], _, _, _]]
[error]  required: Seq[cascading.tap.Tap[?>: Nothing <: cascading.flow.FlowProcess, ?>: Nothing <: Any, ?>: Nothing <: Any, ?>: Nothing <: Any]]
[error]           JobConf, RecordReader[_,_], OutputCollector[_,_]](taps.toSeq : _*)
[error]                                                                  ^
[error] two errors found
[error] {file:/home/me/dev/scala/scalding/}default-bc2517/compile:compile: Compilation failed
[error] Total time: 31 s, completed Feb 9, 2012 9:29:53 AM

It would be really helpful to have good documentation on the installation parameters. What to change for different versions of scala or hadoop (I didn't even get that far, but I know it will come) as well as host name settings and whatever you have there.

@azymnis
Copy link
Contributor

azymnis commented Feb 9, 2012

A couple of questions:

What version of sbt are you running?
Can you try it by changing scala to 2.8.2 and see what happens?

@johnynek
Copy link
Collaborator

Note, sbt will download the correct version of scala. It does not need to match your system. It will keep it inside the project, and if you don't change the version number of scala, it should work.

That said, 2.9.0.1 might have some issue. I have only tested successfully with 2.8.1 and 2.9.1. Can you try with one of those?

Running with 2.9.1:

sbt clean
sbt update
sbt test

works fine for me. Also with 2.8.1 (2.8.2 doesn't work for me currently, but version fragility is not an uncommon issue with scala, the best I can offer immediately is try to use a version known to work). I'll wrestle with 2.8.2 and try to make it compile also. These are not real bugs, but type issues due to weakly typed (but type parameterized half the time) Cascading.

PS: I am sorry about the deprecation warnings, but hadoop deprecated then undeprecated some files, so the version of cascading uses the code that was, for a a short time, deprecated. It's not pretty, but not much we can easily do from our position.

@hgavert
Copy link
Author

hgavert commented Feb 10, 2012

Hi!

I'm not sure if I'm doing the "running with 2.9.1" correctly, so here are the steps that I did:

$ git clone http://github.com/twitter/scalding.git
$ emacs -nw build.sbt # change 
$ ~/bin/sbt clean

$ ~/bin/sbt update
[info] Loading project definition from /home/me/dev/scala/scalding/project
[info] Set current project to scalding (in build file:/home/me/dev/scala/scalding/)
[info] Updating {file:/home/me/dev/scala/scalding/}default-bc2517...
[info] Resolving org.scala-lang#scala-library;2.9.1 ...
[info] Resolving cascading#cascading-core;2.0.0-wip-165 ...
[info] Resolving riffle#riffle;0.1-dev ...
[info] Resolving org.slf4j#slf4j-api;1.6.1 ...
[info] Resolving thirdparty#jgrapht-jdk1.6;0.8.1 ...
[info] Resolving org.codehaus.janino#janino;2.5.16 ...
[info] Resolving org.slf4j#slf4j-log4j12;1.6.1 ...
[info] Resolving log4j#log4j;1.2.16 ...
[info] Resolving cascading#cascading-local;2.0.0-wip-165 ...
[info] Resolving com.google.guava#guava;10.0.1 ...
[info] Resolving com.google.code.findbugs#jsr305;1.3.9 ...
[info] Resolving cascading#cascading-hadoop;2.0.0-wip-165 ...
[info] Resolving org.apache.hadoop#hadoop-core;0.20.2 ...
[info] Resolving commons-cli#commons-cli;1.2 ...
[info] Resolving xmlenc#xmlenc;0.52 ...
[info] Resolving commons-httpclient#commons-httpclient;3.0.1 ...
[info] Resolving commons-logging#commons-logging;1.0.3 ...
[info] Resolving commons-codec#commons-codec;1.3 ...
[info] Resolving commons-net#commons-net;1.4.1 ...
[info] Resolving oro#oro;2.0.8 ...
[info] Resolving org.mortbay.jetty#jetty;6.1.14 ...
[info] Resolving org.mortbay.jetty#jetty-util;6.1.14 ...
[info] Resolving org.mortbay.jetty#servlet-api-2.5;6.1.14 ...
[info] Resolving tomcat#jasper-runtime;5.5.12 ...
[info] Resolving tomcat#jasper-compiler;5.5.12 ...
[info] Resolving org.mortbay.jetty#jsp-api-2.1;6.1.14 ...
[info] Resolving org.mortbay.jetty#jsp-2.1;6.1.14 ...
[info] Resolving org.eclipse.jdt#core;3.1.1 ...
[info] Resolving ant#ant;1.6.5 ...
[info] Resolving commons-el#commons-el;1.0 ...
[info] Resolving net.java.dev.jets3t#jets3t;0.7.1 ...
[info] Resolving commons-logging#commons-logging;1.1.1 ...
[info] Resolving net.sf.kosmosfs#kfs;0.3 ...
[info] Resolving junit#junit;4.5 ...
[info] Resolving hsqldb#hsqldb;1.8.0.10 ...
[info] Resolving commons-lang#commons-lang;2.4 ...
[info] Resolving de.javakaffee#kryo-serializers;0.9 ...
[info] Resolving com.googlecode#kryo;1.04 ...
[info] Resolving com.googlecode#reflectasm;1.01 ...
[info] Resolving asm#asm;3.2 ...
[info] Resolving com.googlecode#minlog;1.2 ...
[info] Resolving org.scala-tools.testing#specs_2.8.0;1.6.5 ...
[info] Done updating.
[success] Total time: 5 s, completed Feb 10, 2012 9:20:21 AM
$ ~/bin/sbt test
[info] Loading project definition from /home/me/dev/scala/scalding/project
[info] Set current project to scalding (in build file:/home/me/dev/scala/scalding/)
[info] Compiling 18 Scala sources to /home/me/dev/scala/scalding/target/scala-2.9.1/classes...
[warn] there were 20 deprecation warnings; re-run with -deprecation for details
[warn] one warning found
[info] Compiling 8 Scala sources to /home/me/dev/scala/scalding/target/scala-2.9.1/test-classes...
[error] File name too long
[error] one error found
[error] {file:/home/me/dev/scala/scalding/}default-bc2517/test:compile: Compilation failed
[error] Total time: 77 s, completed Feb 10, 2012 9:21:54 AM

BTW, the sbt version is quite likely 0.11.2 (as that is part of the URL for the latest version)

@johnynek
Copy link
Collaborator

Are you running on linux with an encrypted home?

https://groups.google.com/forum/?fromgroups#!topic/simple-build-tool/wtD6vgdiy6g

This seems like an issue others have had.

"For anyone following this thread, another solution is:

ln -s /tmp project/plugins/target

Not the most ideal, but it seems to be working."

@hgavert
Copy link
Author

hgavert commented Feb 13, 2012

Laptop, so of course I'm running encrypted home. Company laptop, so I would not even have a choice :-(
Reading that makes me wonder about the linux encryption... I'll try that later on and report what happens.

@johnynek
Copy link
Collaborator

Just to document: you shouldn't need to change the sbt file to build. sbt will download the scala compiler jar you need to compile. So, please just try to run the tests with the existing sbt build file.

If that does not work, please report back. After that we can address the issue of using scald.rb, which does use a scala compiler and needs to match the version used to compile scalding.

@johnynek
Copy link
Collaborator

johnynek commented Mar 2, 2012

I'm closing this issue since it seems to be a problem with Ubuntu encrypted home + sbt.

@johnynek johnynek closed this as completed Mar 2, 2012
@genspire
Copy link

To save some people's time, I wanted to add a comment about my experience with this issue.

With the solution to this problem listed above i.e. using ln -s /tmp project/plugins/target

I received the following error when running sbt update:
/home/prj/build.sbt:1: error: not found: object AssemblyKeys
import AssemblyKeys._
^
[error] Type error in expression

When I used Sullivan's solution (from the same thread) it worked without issue:
mkdir /tmp/myproject-target
cd ~/workspace/myproject
rm -rf target
ln -s /tmp/myproject-target target

Hope this helps

rubanm added a commit that referenced this issue Apr 16, 2015
Merge from twitter/scalding
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants