[SPARK-1997] update breeze to version 0.8.1 #940

witgo · 2014-06-02T14:49:25Z

breeze 0.8.1 dependent on scala-logging-slf4j 2.1.1 The relevant code on #1369

AmplabJenkins · 2014-06-02T14:52:58Z

Merged build triggered.

AmplabJenkins · 2014-06-02T14:53:08Z

Merged build started.

AmplabJenkins · 2014-06-02T15:33:24Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-06-02T15:33:24Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15348/

AmplabJenkins · 2014-06-02T16:17:59Z

Merged build triggered.

AmplabJenkins · 2014-06-02T16:18:04Z

Merged build started.

markhamstra · 2014-06-02T16:44:04Z

What is the reason for this change, and how does it affect our intention to maintain binary compatibility?

AmplabJenkins · 2014-06-02T16:58:48Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-06-02T16:58:49Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15350/

rxin · 2014-06-02T20:29:10Z

I don't think we ever expose Breeze in public facing API, do we?

markhamstra · 2014-06-02T20:49:27Z

We shouldn't, so I think we maintain source compatibility without any trouble. Are the MIMA checks good enough to catch binary incompatibility when we make significant changes to library dependencies? Not that I am asserting that this Breeze change is particularly significant, since I haven't look at the diff from 0.7. I'm more interested in the general case of what needs to be checked or done when we want to update dependencies while staying within the bounds of 1.x.

witgo · 2014-06-03T01:50:29Z

@markhamstra , breeze 0.7 does not support scala 2.11 .

markhamstra · 2014-06-03T02:55:08Z

Neither does spark 1.0.0.

We've offered no guarantee that any spark 1.x will work with scala 2.11. If it turns out that we can't cross-compile for scala 2.10 and 2.11 with essentially the same source code and dependencies and preserving our intention to preserve binary compatibility, then that may be the occasion that forces the beginning of spark 2.0 development.

In any event, the breaking of our intent to preserve binary compatibility is no small matter and shouldn't slip into a PR without considerable discussion. I'm not certain that that is what is happening with this PR, but I do think that it is critical that we understand how strong our checks for binary compatibility are and what are the bounds on dependency upgrades that we can make while preserving that compatibility.

witgo · 2014-06-03T13:39:53Z

Speaking on spark, breeze no big change from 0.7 to 0.8.1. Of course, this conclusion has not been a lot of testing

mengxr · 2014-06-04T08:56:22Z

Let's put this PR on hold until Spark moves to Scala 2.11. Good to see that the breeze functions we used did not change in breeze-0.8.1.

ktham · 2014-06-04T22:45:12Z

Just looked at the release notes for Breeze 0.8 and initial glance at the commits... is there any particular reason why we are holding back this PR? If we want to cross compile with Scala 2.11, then we have to upgrade to breeze 0.8 because there does not exist a 2.11 build of breeze 0.7

We'll need to upgrade eventually.

mengxr · 2014-06-05T04:39:09Z

@ktham I think @markhamstra 's comments addressed the concerns very well. Unless there are critical bugs in breeze 0.7, we should wait until Spark core starts to move. Breeze has a fast release cycle. So we don't need to rush to 0.8.1.

nevillelyh · 2014-06-19T03:56:13Z

We just hit a bug when DenseVector in 0.7 triggers StackOverflowError in KryoSerializer.

mengxr · 2014-06-19T06:56:21Z

@nevillelyh Is there a JIRA for it? Is it fixed in 0.8.1?

nevillelyh · 2014-06-19T15:26:32Z

@mengxr https://issues.apache.org/jira/browse/SPARK-2200

SparkQA · 2014-07-11T02:42:35Z

QA tests have started for PR 940. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16545/consoleFull

SparkQA · 2014-07-11T04:21:19Z

QA results for PR 940:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16545/consoleFull

SparkQA · 2014-07-19T12:28:03Z

QA tests have started for PR 940. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16848/consoleFull

mengxr · 2014-07-21T07:44:56Z

Jenkins, retest this please.

mengxr · 2014-07-28T00:51:15Z

Jenkins, retest this please.

SparkQA · 2014-07-28T00:53:45Z

QA tests have started for PR 940. This patch DID NOT merge cleanly!
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17265/consoleFull

avati · 2014-08-01T15:01:54Z

@mengxr Based on the dependency graph, I am guessing we will now have jar hell problem with scalalogging-slf4j 2.1.2 (needed by breeze 0.8.1) vs 1.0.1 (used by sql/). Breeze 0.7 was depending on 0.7 too. But 2.1.2 and 1.0.1 are not compatible.

So #1701 (or equivalent) needs to be merged soon to resolve the issue.

srowen · 2014-08-01T15:19:02Z

Is that a problem? just update sql/catalyst to use 2.1.2? Do we know if that would be any problem? @witgo has a PR to standardize use of scalalogging-slf4j anyway (#1369) which bumps to 2.1.2 and seems to have worked. Maybe that should go in with this to ensure all is compatible.

mengxr · 2014-08-01T15:19:26Z

Ah, I see. Tests were against individual build instead of the assembly jar. We should have integration tests in the future.

avati · 2014-08-01T15:23:32Z

Yes, either #1701 or #1369. We are already broken till they are committed.

On Fri, Aug 1, 2014 at 8:19 AM, Xiangrui Meng notifications@github.com
wrote:

Ah, I see. Tests were against individual build instead of the assembly
jar. We should have integration tests in the future.

—
Reply to this email directly or view it on GitHub
#940 (comment).

mengxr · 2014-08-01T15:28:43Z

I think Jenkins should be fine but the assembly jar is broken. Is it right?

avati · 2014-08-01T15:41:07Z

I think Jenkins should be fine but the assembly jar is broken. Is it right?

I think so, just like commons-math3

witgo · 2014-08-01T16:36:52Z

How do we resolve this issue?

mengxr · 2014-08-01T17:01:45Z

I reverted the change in #1718 and asked @marmbrus to take look at the dependency issues caused by scala-logging.

srowen · 2014-08-01T17:07:16Z

I think @witgo's PR strongly suggests it's OK to upgrade to 2.1.2?

breeze-0.8.1 causes dependency issues, as discussed in #940 . Author: Xiangrui Meng <meng@databricks.com> Closes #1718 from mengxr/revert-breeze and squashes the following commits: 99c4681 [Xiangrui Meng] downgrade breeze version to 0.7

avati · 2014-08-01T17:19:51Z

I reverted the change in #1718 #1718
and asked @marmbrus https://github.com/marmbrus to take look at the
dependency issues caused by scala-logging.

The other option could have been to keep #940 merged and merge also #1701
(if #1369 is too invasive/requires more review), that combination seems to
work at least for me.

marmbrus · 2014-08-01T17:33:54Z

I'm okay with upgrading the version of scala logging that we use as @avati suggests.

`breeze 0.8.1` dependent on `scala-logging-slf4j 2.1.1` The relevant code on apache#1369 Author: witgo <witgo@qq.com> Closes apache#940 from witgo/breeze-8.0.1 and squashes the following commits: 65cc65e [witgo] update breeze to version 0.8.1

breeze-0.8.1 causes dependency issues, as discussed in apache#940 . Author: Xiangrui Meng <meng@databricks.com> Closes apache#1718 from mengxr/revert-breeze and squashes the following commits: 99c4681 [Xiangrui Meng] downgrade breeze version to 0.7

…he#940) Co-authored-by: Egor Krivokon <>

…& Sum which handle by themself internally (#940) * Fix Number of partitions (0) must be positive * [SPARK-34037][SQL] Remove unnecessary upcasting for Avg & Sum which handle by themself internally ### What changes were proposed in this pull request? The type-coercion for numeric types of average and sum is not necessary at all, as the resultType and sumType can prevent the overflow. ### Why are the changes needed? rm unnecessary logic which may cause potential performance regressions ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? tpcds tests for plan Closes #31079 from yaooqinn/SPARK-34037. Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Liang-Chi Hsieh <viirya@gmail.com> (cherry picked from commit a235c3b) * [SPARK-34037][SQL] Remove unnecessary upcasting for Avg & Sum which handle by themself internally ### What changes were proposed in this pull request? The type-coercion for numeric types of average and sum is not necessary at all, as the resultType and sumType can prevent the overflow. ### Why are the changes needed? rm unnecessary logic which may cause potential performance regressions ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? tpcds tests for plan Closes #31079 from yaooqinn/SPARK-34037. Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Liang-Chi Hsieh <viirya@gmail.com> (cherry picked from commit a235c3b) * fix Co-authored-by: Kent Yao <yao@apache.org>

…he#940) Co-authored-by: Egor Krivokon <>

witgo changed the title ~~update breeze to version 0.8.1~~ [WIP] update breeze to version 0.8.1 Jun 2, 2014

witgo changed the title ~~[WIP] update breeze to version 0.8.1~~ [SPARK-1997] update breeze to version 0.8.1 Jun 3, 2014

witgo changed the title ~~[SPARK-1997] update breeze to version 0.8.1~~ [WIP][SPARK-1997] update breeze to version 0.8.1 Jun 3, 2014

witgo changed the title ~~[WIP][SPARK-1997] update breeze to version 0.8.1~~ [SPARK-1997] update breeze to version 0.8.1 Jun 6, 2014

asfgit closed this in 0dacb1a Aug 1, 2014

mengxr mentioned this pull request Aug 1, 2014

[HOTFIX] downgrade breeze version to 0.7 #1718

Closed

witgo deleted the breeze-8.0.1 branch August 1, 2014 17:17

mengxr mentioned this pull request Aug 3, 2014

remove scalalogging from dependencies scalanlp/breeze#288

Merged

Agirish pushed a commit to HPEEzmeral/apache-spark that referenced this pull request May 5, 2022

MapR [SPARK-921] Replace sudo command with maprexecute in Spark (apac…

0c07103

…he#940) Co-authored-by: Egor Krivokon <>

udaynpusa pushed a commit to mapr/spark that referenced this pull request Jan 30, 2024

MapR [SPARK-921] Replace sudo command with maprexecute in Spark (apac…

390753b

…he#940) Co-authored-by: Egor Krivokon <>

[SPARK-1997] update breeze to version 0.8.1 #940

[SPARK-1997] update breeze to version 0.8.1 #940

Conversation

witgo commented Jun 2, 2014

AmplabJenkins commented Jun 2, 2014

AmplabJenkins commented Jun 2, 2014

AmplabJenkins commented Jun 2, 2014

AmplabJenkins commented Jun 2, 2014

AmplabJenkins commented Jun 2, 2014

AmplabJenkins commented Jun 2, 2014

markhamstra commented Jun 2, 2014

AmplabJenkins commented Jun 2, 2014

AmplabJenkins commented Jun 2, 2014

rxin commented Jun 2, 2014

markhamstra commented Jun 2, 2014

witgo commented Jun 3, 2014

markhamstra commented Jun 3, 2014

witgo commented Jun 3, 2014

mengxr commented Jun 4, 2014

ktham commented Jun 4, 2014

mengxr commented Jun 5, 2014

nevillelyh commented Jun 19, 2014

mengxr commented Jun 19, 2014

nevillelyh commented Jun 19, 2014

SparkQA commented Jul 11, 2014

SparkQA commented Jul 11, 2014

SparkQA commented Jul 19, 2014

mengxr commented Jul 21, 2014

mengxr commented Jul 28, 2014

SparkQA commented Jul 28, 2014

avati commented Aug 1, 2014

srowen commented Aug 1, 2014

mengxr commented Aug 1, 2014

avati commented Aug 1, 2014

mengxr commented Aug 1, 2014

avati commented Aug 1, 2014

witgo commented Aug 1, 2014

mengxr commented Aug 1, 2014

srowen commented Aug 1, 2014

avati commented Aug 1, 2014

marmbrus commented Aug 1, 2014