Prep for release to maven #13

lbustelo · 2016-03-14T17:27:10Z

This fixes https://issues.apache.org/jira/browse/TOREE-245.

This PR contains modifications to build to release JAR artifacts of Toree.

lbustelo · 2016-04-05T15:43:04Z

I'm adjusting the checklist above based on http://incubator.apache.org/guides/releasemanagement.html#check-list

Lull3rSkat3r · 2016-04-05T16:56:03Z

I'll look into the following:
Uploading files for distribution: http://www.apache.org/dev/release.html#upload-scp
Signing releases: http://www.apache.org/dev/release-signing.html#basic-facts

lbustelo · 2016-04-05T17:06:23Z

I'm going to work on Rename bin package to contain the name 'incubating'

lbustelo · 2016-04-05T17:13:42Z

Adding item to list to Rename referent to toree kernel in notebook to Apache Toree based on http://incubator.apache.org/guides/branding.htm

lbustelo · 2016-04-05T19:47:29Z

I'm going to work on Package Notice/Licence file in pip distro

lbustelo · 2016-04-05T19:55:58Z

Based on http://incubator.apache.org/guides/releasemanagement.html#best-practice-license I'm creating an item to list the License of other bundled dependencies in our package. i.e. LGPL.

lresende · 2016-04-05T22:17:54Z

etc/pip_install/setup.py

@@ -27,12 +27,22 @@

 setup_args = dict(
    name='toree',
-    author='Toree Development Team',
+    author='Apache Toree Development Team',
    author_email='user@toree.incubator.apache.org',


Toree does not have an user list, right ?

Good catch... Will change to dev.

lbustelo · 2016-04-06T21:39:46Z

@lresende What is the best way to create the aggregate LICENSE and NOTICE file? We are using sbt-assembly, but it does not seem to be doing all the dependencies. For example, our LGPL dependency does not ship with a LICENSE in the jar, so it does not get merge.

Questions are then?
How do you get a list of what packages we need to aggregate into the LICENSE and NOTICE files.

lresende · 2016-04-06T23:13:03Z

There is no magic, I guess it's a manual process, by looking into all dependencies being packaged on the distribution and create the LICENSE and NOTICE file. Some projects does it a little different, where they add the LICENSE files into a folder and either point to them or aggregate them.

lbustelo · 2016-04-07T13:57:56Z

It seems that spark has all of it already compiled in their top level LICENSE and NOTICE files https://github.com/apache/spark/blob/branch-1.0/LICENSE

Lull3rSkat3r · 2016-04-07T19:02:40Z

I have compiled this list of license information for our direct and transitive dependencies. @lbustelo, you should be able to find information in most of those links for the NOTICES file.

BSD-like

Direct dependencies

org.clapper:classutil_2.10:1.0.3 -> http://software.clapper.org/javautil/#copyright-and-license
org.scala-lang:scala-reflect:2.10.4 -> http://www.scala-lang.org/license.html
org.scalamacros:quasiquotes_2.10:2.0.1 -> http://www.scala-lang.org/license.html

Transitive Dependencies

jline:jline:2.6 -> https://github.com/jline/jline2/blob/master/LICENSE.txt
org.clapper:grizzled-scala_2.10:1.1.2 -> http://software.clapper.org/grizzled-scala/#copyright-and-license
org.clapper:grizzled-slf4j_2.10:1.0.1 -> http://software.clapper.org/grizzled-slf4j/#copyright-and-license
org.scala-stm:scala-stm_2.10:0.7 -> https://nbronson.github.io/scala-stm/license.html
org.scalaz:scalaz-concurrent_2.10:7.1.2 -> https://github.com/scalaz/scalaz/blob/series/7.1.x/etc/LICENCE
org.scalaz:scalaz-core_2.10:7.1.2 -> https://github.com/scalaz/scalaz/blob/series/7.1.x/etc/LICENCE
org.scalaz:scalaz-effect_2.10:7.1.2 -> https://github.com/scalaz/scalaz/blob/series/7.1.x/etc/LICENCE
asm:asm-commons:3.3 -> http://asm.ow2.org/license.html
asm:asm-tree:3.3 -> http://asm.ow2.org/license.html
asm:asm:3.3 -> http://asm.ow2.org/license.html

MIT license

Direct dependencies

net.sf.jopt-simple:jopt-simple:4.6 -> https://pholser.github.io/jopt-simple/license.html
org.slf4j:slf4j-api:1.7.5 -> http://www.slf4j.org/license.html
org.slf4j:slf4j-api:1.7.1 -> http://www.slf4j.org/license.html

Transitive Dependencies

com.lihaoyi:ammonite-terminal_2.10:0.5.0 -> https://github.com/lihaoyi/Ammonite/blob/master/LICENSE

GNU General Lesser Public License (LGPL) version 3.0

Direct dependencies

org.zeromq:jeromq:0.3.4 -> https://github.com/zeromq/jeromq/blob/master/COPYING

lbustelo · 2016-04-08T15:12:36Z

a couple of links guiding NOTICE file content

http://www.apache.org/dev/licensing-howto.html#mod-notice

Copyright notifications which have been relocated from source files (rather than removed) must be preserved in NOTICE. However, elements such as the copyright notifications embedded within BSD and MIT licenses need not be duplicated in NOTICE -- it suffices to leave those notices in their original locations.

It is important to keep NOTICE as brief and simple as possible, as each addition places a burden on downstream consumers.

Do not add anything to NOTICE which is not legally required.

http://apache.org/legal/resolved.html#required-third-party-notices

When a release contains third party works, the licenses covering those works may ask that consumers are informed in certain specific fashions. These third party notices vary from license to license. Apache releases should contain a copy of each license, usually contained in the LICENSE document. For many licenses this is a sufficient notice. For some licenses some additional notice is required. In many cases, this will be included within the dependent artifact.

A required third-party notice is any third party notice which isn't covered by the above cases.

See Bundling Other ASF Products for a note on required notices when a release contains another Apache product.

lbustelo · 2016-04-08T16:05:16Z

based on https://github.com/zeromq/jeromq/blob/master/COPYING.LESSER#L91... I'm adding the license to the uber packages (pip, bin, and assembly jar)

lbustelo · 2016-04-08T17:17:35Z

Thread regarding single release with LGPL http://apache-incubator-general.996316.n3.nabble.com/Release-dependant-on-LGPL-td48424.html

Lull3rSkat3r · 2016-04-08T20:04:08Z

In case we need it, here is a list of all the Apache Dependencies:

commons-logging:commons-logging:1.1.3
org.apache.ivy:ivy:2.4.0-rc1
com.google.guava:guava:14.0.1
org.apache.commons:commons-exec:1.3
org.joda:joda-convert:1.6
joda-time:joda-time:2.3
com.github.alexarchambault:coursier-cache_2.10:1.0.0-M9
com.github.alexarchambault:coursier_2.10:1.0.0-M9
com.typesafe.akka:akka-slf4j_2.10:2.3.11
com.typesafe.akka:akka-actor_2.10:2.3.11
com.typesafe:config:1.2.1
com.typesafe.play:play-json_2.10:2.3.6
com.typesafe.play:play-datacommons_2.10:2.3.6
com.typesafe.play:play-functional_2.10:2.3.6
com.typesafe.play:play-iteratees_2.10:2.3.6
org.springframework:spring-core:4.1.1.RELEASE

Lull3rSkat3r · 2016-04-11T14:33:58Z

I have a PR for removing the systems tests. I created a jira for adding them back with jupyter kernel test.

Added scripts to generate signatures and checksums for artifacts Added scripts to verify signatures and checksums during release audit

* Updated sbt-dependency-graph plugin to latest version * Added GPG signing for published jars * Added environment variable for GPG password and binary * Added extra legal files to rat-excludes

lbustelo · 2016-04-21T21:05:43Z

@chipsenkbeil Please take a look at this. I think it should be ready for merge. We still do not have a way to automate some of this, but I think we can tackle that separately. There are quite a bit of changes here that it's better to merge now and than having to keep rebasing this PR.

chipsenkbeil · 2016-04-22T13:37:28Z

@lbustelo @Lull3rSkat3r, read through it all and looks good. It looks like the system tests have been removed (along with using a consistent Akka timeout). Do we have plans in the near future to use the Jupyter kernel test project?

chipsenkbeil · 2016-04-22T13:38:03Z

kernel-api/src/test/scala/org/apache/toree/utils/TaskManagerSpec.scala

@@ -34,7 +34,7 @@ class TaskManagerSpec extends FunSpec with Matchers with MockitoSugar
  with Eventually with Timeouts
 {
  implicit override val patienceConfig = PatienceConfig(
-    timeout = scaled(Span(200, Milliseconds)),
+    timeout = scaled(Span(30, Seconds)),


Should this have a constant under test.utils similar to the Akka one?

Maybe... my conversation with @Lull3rSkat3r was that I feel these tests are trying to be units, but are actually more of an integration test. Either we find a different way to test the functionality without involving Akka, or we moved them as part of overall SystemTest.

lbustelo · 2016-04-22T13:42:57Z

@chipsenkbeil There is a separate issue for that.

chipsenkbeil · 2016-04-22T13:47:42Z

Cool, then LGTM. Want me to merge?

lbustelo · 2016-04-22T13:58:15Z

@chipsenkbeil yes... merge it

lbustelo force-pushed the ReleaseToMaven branch 5 times, most recently from 65cd813 to c675333 Compare April 4, 2016 21:59

lbustelo force-pushed the ReleaseToMaven branch from 4274907 to d9fd9be Compare April 5, 2016 20:37

lresende reviewed Apr 5, 2016
View reviewed changes

lbustelo force-pushed the ReleaseToMaven branch from a5277b7 to 474f046 Compare April 6, 2016 14:17

lbustelo added 3 commits April 21, 2016 12:22

Added a target to expose calling sbt tasks

af462b9

Removed unecessary task

71cc85e

Adding release requirements to sbt

cace425

lbustelo force-pushed the ReleaseToMaven branch from 20c676a to 85fa298 Compare April 21, 2016 17:27

Lull3rSkat3r and others added 22 commits April 21, 2016 14:44

Removed duplicate check-licenses file

6be30f4

Fix run script due to assembly name change

0964d0e

Adding LICENSE/NOTICE/DISCLAIMER to release packages

9bf45fc

Fixed pip reference to Apache Toree and added Disclaimer in discription

269e51d

Pip install fixes to use name Apache Toree and print DISCLAIMER

79d163e

Building source package

beb2b15

Fixed mailing list address used in pip package

2e4b647

Add signatures and checksums

3b2cad1

Added scripts to generate signatures and checksums for artifacts Added scripts to verify signatures and checksums during release audit

Add ensime files to ingore list for src distro

498f173

Update rat-excludes and add license headers

e4fc7d8

Fix make target to only sign once

63d59b5

Amended the DISCLAIMER with LGPL statement

13e68fb

Fixed requirements for LICENSE and NOTICE

580f57b

Adding and pointing to each of the licenses bundled in bin packages

297914a

Add GPG signing of jars

23c3abb

* Updated sbt-dependency-graph plugin to latest version * Added GPG signing for published jars * Added environment variable for GPG password and binary * Added extra legal files to rat-excludes

Removed system tests

a7f5003

Added common test timeouts for akka tests.

9157a5a

Add pip signing and uploading.

09d769a

Enabled publishing jars to maven

09bda85

Removed oracle jdk from travis build

92abb20

Added Release notes

c23fe1c

Increase timeout for test in ExecuteRequestHandlerSpec

d08b117

lbustelo force-pushed the ReleaseToMaven branch from f469993 to d08b117 Compare April 21, 2016 20:02

chipsenkbeil reviewed Apr 22, 2016
View reviewed changes

asfgit merged commit d08b117 into apache:master Apr 22, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Prep for release to maven #13

Prep for release to maven #13

lbustelo commented Mar 14, 2016 •

edited

lbustelo commented Apr 5, 2016

Lull3rSkat3r commented Apr 5, 2016

lbustelo commented Apr 5, 2016

lbustelo commented Apr 5, 2016

lbustelo commented Apr 5, 2016

lbustelo commented Apr 5, 2016

lresende Apr 5, 2016

lbustelo Apr 5, 2016

lbustelo commented Apr 6, 2016

lresende commented Apr 6, 2016

lbustelo commented Apr 7, 2016

Lull3rSkat3r commented Apr 7, 2016

lbustelo commented Apr 8, 2016

lbustelo commented Apr 8, 2016

lbustelo commented Apr 8, 2016

Lull3rSkat3r commented Apr 8, 2016

Lull3rSkat3r commented Apr 11, 2016

lbustelo commented Apr 21, 2016

chipsenkbeil commented Apr 22, 2016

chipsenkbeil Apr 22, 2016

lbustelo Apr 22, 2016

lbustelo commented Apr 22, 2016

chipsenkbeil commented Apr 22, 2016

lbustelo commented Apr 22, 2016

Prep for release to maven #13

Prep for release to maven #13

Conversation

lbustelo commented Mar 14, 2016 • edited

lbustelo commented Apr 5, 2016

Lull3rSkat3r commented Apr 5, 2016

lbustelo commented Apr 5, 2016

lbustelo commented Apr 5, 2016

lbustelo commented Apr 5, 2016

lbustelo commented Apr 5, 2016

lresende Apr 5, 2016

Choose a reason for hiding this comment

lbustelo Apr 5, 2016

Choose a reason for hiding this comment

lbustelo commented Apr 6, 2016

lresende commented Apr 6, 2016

lbustelo commented Apr 7, 2016

Lull3rSkat3r commented Apr 7, 2016

BSD-like

Direct dependencies

Transitive Dependencies

MIT license

Direct dependencies

Transitive Dependencies

GNU General Lesser Public License (LGPL) version 3.0

Direct dependencies

lbustelo commented Apr 8, 2016

lbustelo commented Apr 8, 2016

lbustelo commented Apr 8, 2016

Lull3rSkat3r commented Apr 8, 2016

Lull3rSkat3r commented Apr 11, 2016

lbustelo commented Apr 21, 2016

chipsenkbeil commented Apr 22, 2016

chipsenkbeil Apr 22, 2016

Choose a reason for hiding this comment

lbustelo Apr 22, 2016

Choose a reason for hiding this comment

lbustelo commented Apr 22, 2016

chipsenkbeil commented Apr 22, 2016

lbustelo commented Apr 22, 2016

lbustelo commented Mar 14, 2016 •

edited