Drastically simplify distribution building for 2.11. #120

Merged
merged 15 commits into from Jan 16, 2014

Conversation

Projects
None yet
3 participants
Owner

adriaanm commented Jan 9, 2014

See README and individual commits for info.

The main idea is that these scripts should do nothing more than repackaging org.scala-lang % scala-dist and its transitive dependencies. No duplication allowed. The process is reduced to pushing a tag to scala/scala-dist by the same name as the one from scala/scala (whose artifacts have been published to maven), and running sbt s3-upload on windows (for the msi) and unix (everything else).

We no longer package source code ourselves. GitHub provides source archives for every tag.

Review by @jsuereth, /cc @retronym, @gkossakowski.

This has all been tested locally. I'll do a dry run on jenkins tomorrow and link to the results.

adriaanm added some commits Dec 14, 2013

@adriaanm adriaanm Delete cruft, simplify directory structure. c21c096
@adriaanm adriaanm Remove duplication in license/copyright. a93b89a
@adriaanm adriaanm Generic.settings: mappings for bin/, lib/, doc/ and man/
A Scala distribution is fully determined by the contents and dependencies of
the maven artifact "org.scala-lang" % "scala-dist" % version.value.

The contents of scala-dist itself is extracted to populate bin/, doc/ and man/.
The (transitive) dependencies populate the lib/ folder.

This configures sbt native packager's Universal config.

The next commit will take the jars under the javadoc classifier and extract/move
them to api/ to provide settings for the packager's UniversalDocs config.
e258c79
@adriaanm adriaanm Docs.settings: mappings for the api/ directory.
All dependencies of scala-dist under the javadoc classifier are included as
"api/jars/$artifactId-$version-javadoc.jar".

The core jars are also expanded under api/scala-{library|reflect|compiler}.

Also make constructing mappings reusable.

Needed to use updateClassifiers to get -javadoc artifacts,
don't seem to need to include the javadoc classifier in the scala-dist config.
e9c59b7
@adriaanm adriaanm Commented stuff carried over from old version. ab7922a
@adriaanm adriaanm Wix.settings: mappings for creating the MSI.
Takes the result of staging universal and universal-docs,
and creates a WIX xml description for `packageBin in Windows`
to be able to create an MSI.

We no longer distributing examples, and link to GitHub for sources.

As .bat files are published to maven using the windows line endings,
no need to convert.

PS: Would've called in Windows.settings, but that clashes with the
packager's Windows config object.
c1edd83
@adriaanm adriaanm Derive version from git tag.
The version of this build determines the Scala version to package.
We look at the closest git tag that matches v[0-9].* to derive it.
For testing, the version may be overridden with -Dproject.version=...
ebd0b81
@adriaanm adriaanm Create debian & rpm packages.
`debian:packageBin` and `rpm:packageBin` create `scala-$version.[deb|rpm]` in `target/`

- subdirs not requiring special treatment are rooted under `installTargetUnix.value`
- docs go to `installTargetUnixDocs.value`
- create symlinks under /usr/bin/ for all scripts in `installTargetUnix.value / "bin"`
- make scripts in `installTargetUnix.value / "bin"` executable, drop ".bat" files
- map man to /usr/share/man/
- map doc to `installTargetUnixDocs.value`
- map api to `installTargetUnixDocs.value / "api"`
35a6693
@adriaanm adriaanm Standardize msi's name. c55646b
@adriaanm adriaanm Discontinue packaging of tool-support.
Hopefully someone will take over maintenance, or we will when demand arises.
6e54357
@adriaanm adriaanm s3-upload support, and cleanup
Only load Wix settings when on Windows,
do the rest of the work when not running on windows
(packageBin in universal, universal-docs, debian, rpm)
9c88063
@adriaanm adriaanm Update README. f37d7ad
Owner

adriaanm commented Jan 9, 2014

This depends on scala/scala#3277 -- UPDATE: this has been merged, and jobs are successful, so full steam ahead!

@adriaanm adriaanm Split out versioning.
The windows version is used for the unix versions,
so can't have its computation only run when running on windows.
969f16c
Owner

adriaanm commented Jan 10, 2014

Works on unix: https://scala-webapps.epfl.ch/jenkins/view/scala-release-2.11.x/job/scala-release-2.11.x-unix/10/console
Works on windows: https://scala-webapps.epfl.ch/jenkins/view/scala-release-2.11.x/job/scala-release-2.11.x-windows/6/console (the failure is an S3 failure also sometimes seen in the old jobs because of VirtualBox issues -- retrying as job 7)

Owner

adriaanm commented Jan 10, 2014

The last thing between us and one-click releases for both maven and native packages is integrating sonatype staging repo closing (https://github.com/adriaanm/binfu/blob/master/sonafu.sh) into https://scala-webapps.epfl.ch/jenkins/view/scala-release-2.11.x/job/scala-release-2.11.x/.

For tying the binary compatibility knot (https://github.com/scala/scala-modules-build/blob/master/build.sh) I'll need to implement something very similar, so I'm hopeful we'll actually get there!

Of course, there's always more to be done. Then, we'll need automation for generating scala/scala-lang PRs (release notes, download pages,...) and a big red RELEASE button (which would release the closed sonatype repo).

@jsuereth jsuereth commented on the diff Jan 10, 2014

README.md
- * scala/scala must be in the user home directory
- * Please run `ant dist-opt` in in the `scala/scala` project.
- * zip the `<scala/scala>/dist/latest` directory into a file called `scala-dist.zip`
- * Copy `scala-dist.zip` into `<scala/scala-dist>/target/tmp/scala-dist.zip`
- * Start [sbt](https://github.com/harrah/xsbt) in the `<scala/scala-dist>` directory.
- * Run one of the build commands.
+Due to limited resources, the native packages are quite rudimental,
+and tool-support isn't packaged at all.
@jsuereth

jsuereth Jan 10, 2014

Member

That's probably ok. maybe we can get some community contributions after this whole process is dramatically simplified.

@jsuereth jsuereth commented on the diff Jan 10, 2014

build.sbt
@@ -0,0 +1,28 @@
+import com.typesafe.sbt.SbtGit._
+import S3._
+
+// so we don't require a native git install
+useJGit
+
+// The version of this build determines the Scala version to package.
+// We look at the closest git tag that matches v[0-9].* to derive it.
+// For testing, the version may be overridden with -Dproject.version=...
+versionWithGit
@jsuereth

jsuereth Jan 10, 2014

Member

I always smile when I see people using this.

@adriaanm

adriaanm Jan 10, 2014

Owner

:-) git ftw! I think we'll see tag-driven scala releases for 2.11.x!

@jsuereth jsuereth commented on the diff Jan 10, 2014

project/Docs.scala
+/** Create mappings for UniversalDocs under the api/ directory.
+ *
+ * All dependencies of scala-dist under the javadoc classifier are included as
+ * "api/jars/$artifactId-$version-javadoc.jar".
+ *
+ * The core jars are also expanded under api/scala-{library|reflect|compiler}
+ *
+ */
+object Docs {
+ import ScalaDist._
+
+ def settings: Seq[Setting[_]] = Seq(
+ name in UniversalDocs := s"scala-docs-${version.value}",
+ // libraryDependencies += scalaDistDep(version.value, "javadoc"), // seems not to be necessary
+ // need updateClassifiers to get javadoc jars
+ mappings in UniversalDocs ++= createMappingsWith(updateClassifiers.value.toSeq, universalDocsMappings)

@jsuereth jsuereth commented on the diff Jan 10, 2014

project/ScalaDist.scala
+// compile a file that exercises each module (xml, parsers, akka-actor,...)
+// run the whole test suite against the scala instance we're shipping?
+
+// can't call it Universal -- that's taken by the packager
+object ScalaDist {
+ def createMappingsWith(deps: Seq[(String, ModuleID, Artifact, File)],
+ distMappingGen: (ModuleID, Artifact, File) => Seq[(File, String)]): Seq[(File, String)] =
+ deps flatMap {
+ case d@(ScalaDistConfig, id, artifact, file) => distMappingGen(id, artifact, file)
+ case _ => Seq()
+ }
+
+ // used to make s3-upload upload the file produced by fileTask to the path scala/$version/${file.name}
+ private def uploadMapping(fileTask: TaskKey[File]) = Def.task {
+ val file = fileTask.value
+ file -> s"scala/${version.value}/${file.getName}"
@jsuereth

jsuereth Jan 10, 2014

Member

nice simplification!

@jsuereth jsuereth and 1 other commented on an outdated diff Jan 10, 2014

project/Unix.scala
+ * docs go to `installTargetUnixDocs.value`
+ * create symlinks under /usr/bin/ for all scripts in `installTargetUnix.value / "bin"`
+ * make scripts in `installTargetUnix.value / "bin"` executable, drop ".bat" files
+ * map man to /usr/share/man/
+ * map doc to `installTargetUnixDocs.value`
+ * map api to `installTargetUnixDocs.value / "api"`
+ */
+object Unix {
+ val installTargetUnix = SettingKey[File]("install-target-unix", "The location where we will install Scala.")
+ val installTargetUnixDocs = SettingKey[File]("install-target-docs-unix", "The location where we will install the Scala docs.")
+
+ def settings = Seq (
+ installTargetUnix := file("/usr/share/scala"),
+ installTargetUnixDocs := file("/usr/share/doc/scala"),
+
+ (packageBin in Rpm) <<= ((packageBin in Rpm)
@jsuereth

jsuereth Jan 10, 2014

Member

Do you need it it to be staged? I thought if you just et up the mappings it should work.

@adriaanm

adriaanm Jan 10, 2014

Owner

good point -- remnant from when we were still patching things; trying without and seems to going through
I'll add a commit

@jsuereth jsuereth and 1 other commented on an outdated diff Jan 10, 2014

project/Unix.scala
+ installTargetUnix := file("/usr/share/scala"),
+ installTargetUnixDocs := file("/usr/share/doc/scala"),
+
+ (packageBin in Rpm) <<= ((packageBin in Rpm)
+ dependsOn (stage in Universal)
+ dependsOn (stage in UniversalDocs)),
+
+ (packageBin in Debian) <<= ((packageBin in Debian)
+ dependsOn (stage in Universal)
+ dependsOn (stage in UniversalDocs)),
+
+ // symlinks for s"/usr/bin/$script" --> s"${installTargetUnix.value}/bin/$script"
+ linuxPackageSymlinks ++= (
+ (mappings in Universal).value collect {
+ case (file, name) if (name startsWith "bin/") && !(name endsWith ".bat") =>
+ LinuxSymlink("/usr/" + name, (installTargetUnix.value / name).getAbsolutePath)
@jsuereth

jsuereth Jan 10, 2014

Member

nice. Is this borrowed from native packager? Did the "GenericMappingsToLinux" settings not work?

@adriaanm

adriaanm Jan 10, 2014

Owner

It is borrowed -- sorry, hacked the last bits in anger, so didn't reuse all the bits I could've reused. Let's gradually refine from here.

@jsuereth jsuereth and 1 other commented on an outdated diff Jan 10, 2014

project/Unix.scala
+ },
+
+ // RPM Specific
+ name in Rpm := "scala",
+ rpmVendor := "typesafe",
+ rpmUrl := Some("http://github.com/scala/scala"),
+ rpmLicense := Some("BSD"),
+ rpmGroup := Some("Development/Languages"),
+
+ // This hack lets us ignore the RPM specific versioning junks.
+ packageBin in Rpm := {
+ val simplified = target.value / s"${(name in Rpm).value}-${version.value}.rpm"
+
+ val rpm = (packageBin in Rpm).value match {
+ case reported if reported.exists => reported
+ case _ => // hack on top of hack because RpmHelper.buildRpm is broken -- `spec.meta.arch` doesn't necessarily match the arch `rpmbuild` decided on
@jsuereth

jsuereth Jan 10, 2014

Member

This seems to be a MacOSX issue. IF you think you know the issue, let me know. I'd love to fix it, but wasn't able to isolate the problem. It isn't borked on RedHat/Ubuntu.

@adriaanm

adriaanm Jan 10, 2014

Owner

clarified mac issue in comment

@jsuereth jsuereth commented on an outdated diff Jan 10, 2014

project/Unix.scala
+
+ val rpm = (packageBin in Rpm).value match {
+ case reported if reported.exists => reported
+ case _ => // hack on top of hack because RpmHelper.buildRpm is broken -- `spec.meta.arch` doesn't necessarily match the arch `rpmbuild` decided on
+ (PathFinder(IO.listFiles((target in Rpm).value)) ** "*.rpm").get.find(file =>
+ file.getName contains (name in Rpm).value).get
+ }
+
+ IO.copyFile(rpm, simplified)
+ simplified
+ },
+
+ // Debian Specific
+ name in Debian := "scala",
+ debianPackageDependencies += "openjdk-6-jre | java6-runtime",
+ debianPackageDependencies += "libjansi-java",
@jsuereth

jsuereth Jan 10, 2014

Member

not really needed anymore.

@jsuereth jsuereth commented on an outdated diff Jan 10, 2014

project/Versioning.scala
+ }
+
+ // This is a complicated means to convert maven version numbers into monotonically increasing windows versions.
+ private def makeWindowsVersion(version: String): String = {
+ val Majors = new scala.util.matching.Regex("(\\d+).(\\d+).(\\d+)(-.*)?")
+ val Rcs = new scala.util.matching.Regex("(\\-\\d+)?\\-RC(\\d+)")
+ val Milestones = new scala.util.matching.Regex("(\\-\\d+)?\\-M(\\d+)")
+ val BuildNum = new scala.util.matching.Regex("\\-(\\d+)")
+
+ def calculateNumberFour(buildNum: Int = 0, rc: Int = 0, milestone: Int = 0) =
+ if(rc > 0 || milestone > 0) (buildNum)*400 + rc*20 + milestone
+ else (buildNum+1)*400 + rc*20 + milestone
+
+ version match {
+ case Majors(major, minor, bugfix, rest) => Option(rest) getOrElse "" match {
+ case Milestones(null, num) => major + "." + minor + "." + bugfix + "." + calculateNumberFour(0,0,num.toInt)
@jsuereth

jsuereth Jan 10, 2014

Member

Yay for ensuring windows upgrades work correctly.

@jsuereth jsuereth commented on the diff Jan 10, 2014

project/Wix.scala
+
+import com.typesafe.sbt.SbtNativePackager._
+import com.typesafe.sbt.packager.Keys._
+
+import com.typesafe.sbt.packager.windows._
+import WixHelper.{generateComponentsAndDirectoryXml, cleanFileName}
+
+// can't call it Windows, that's a config name
+object Wix {
+ // Windows installer configuration
+ def settings: Seq[Setting[_]] = Seq(
+ mappings in Windows := (mappings in Universal).value,
+ // distributionFiles in Windows += (packageMsi in Windows).value,
+
+ wixProductId := "7606e6da-e168-42b5-8345-b08bf774cb30",
+ wixProductUpgradeId := "6061c134-67c7-4fb2-aff5-32b01a186968",
@jsuereth

jsuereth Jan 10, 2014

Member

These are the same as before, right?

Member

jsuereth commented Jan 10, 2014

This looks very very good man! I have one comment, one question, and otherwise looks great from me.

Owner

adriaanm commented Jan 10, 2014

Thanks!

@adriaanm adriaanm Review feedback. Unix mappings don't depend on stage.
Because we no longer need to patch the scripts (as we gave Scala proper home,
rather than plopping all the jars in /usr/share/java).
08b6232
Owner

adriaanm commented Jan 11, 2014

@jsuereth, I believe I've addressed your feedback!

I know its existing code, but this could be more easily expressed as """"(\d+).(\d+).(\d+)(-.*)?""".r

Owner

adriaanm replied Jan 16, 2014

Thanks, cleaned up: f525856

adriaanm merged commit b930927 into scala:master Jan 16, 2014

@liancheng liancheng pushed a commit to liancheng/scala-dist that referenced this pull request Jul 20, 2014

@sryza @pwendell sryza + pwendell SPARK-1183. Don't use "worker" to mean executor
Author: Sandy Ryza <sandy@cloudera.com>

Closes #120 from sryza/sandy-spark-1183 and squashes the following commits:

5066a4a [Sandy Ryza] Remove "worker" in a couple comments
0bd1e46 [Sandy Ryza] Remove --am-class from usage
bfc8fe0 [Sandy Ryza] Remove am-class from doc and fix yarn-alpha
607539f [Sandy Ryza] Address review comments
74d087a [Sandy Ryza] SPARK-1183. Don't use "worker" to mean executor
6983732

scabug referenced this pull request in scala/bug Apr 7, 2017

Closed

publish runner scripts as a maven artifact #7508

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment