Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-6546][Build] Using the wrong code that will make spark compile failed!! #5198

Closed
wants to merge 9 commits into from
Closed

[SPARK-6546][Build] Using the wrong code that will make spark compile failed!! #5198

wants to merge 9 commits into from

Conversation

pzzs
Copy link
Contributor

@pzzs pzzs commented Mar 26, 2015

wrong code : val tmpDir = Files.createTempDir()
not Files should Utils

@AmplabJenkins
Copy link

Can one of the admins verify this patch?

@pzzs pzzs changed the title A little spell wrong, but this will make spark compile failed!! [SPARK-6300][Build] Using the wrong code that will make spark compile failed!! Mar 26, 2015
@pzzs pzzs changed the title [SPARK-6300][Build] Using the wrong code that will make spark compile failed!! [SPARK-6546][Build] Using the wrong code that will make spark compile failed!! Mar 26, 2015
@pzzs
Copy link
Contributor Author

pzzs commented Mar 26, 2015

i think this is pressing @marmbrus

@JoshRosen
Copy link
Contributor

Jenkins, this is ok to test.

@JoshRosen
Copy link
Contributor

Do you know which PR broke this?

@liancheng
Copy link
Contributor

@JoshRosen It should be #4289.

@SparkQA
Copy link

SparkQA commented Mar 26, 2015

Test build #29206 has started for PR 5198 at commit 6e0140d.

  • This patch merges cleanly.

@liancheng
Copy link
Contributor

�This LGTM pending Jenkins. #4289 somehow passed Jenkins, however the build history has been cleaned now.

@liancheng
Copy link
Contributor

@JoshRosen @marmbrus #4289 was using Guava's com.google.common.io.Files according to the first commit of that PR, see here. However, #5029 was merged earlier, and deprecated Guava Files by Utils.Files. These two combined caused this build failure. (There're no conflicts in the eyes of Git, but there do exist semantic conflicts.)

@SparkQA
Copy link

SparkQA commented Mar 26, 2015

Test build #29206 has finished for PR 5198 at commit 6e0140d.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • case class UnregisterApplication(appId: String)
    • class RegexTokenizer extends UnaryTransformer[String, Seq[String], RegexTokenizer]
    • case class Data(weight: Double, mu: Vector, sigma: Matrix)
    • implicit class DslSymbol(sym: Symbol) extends ImplicitAttribute
    • case class Alias(child: Expression, name: String)(
    • class Column(protected[sql] val expr: Expression) extends Logging

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29206/
Test PASSed.

@pzzs
Copy link
Contributor Author

pzzs commented Mar 26, 2015

you are right @liancheng

@zzcclp zzcclp mentioned this pull request Mar 26, 2015
Closed
zzcclp referenced this pull request Mar 26, 2015
@jeanlyn @marmbrus
[SPARK-5498][SQL]fix query exception when partition schema does not m…
e6d1406 
…atch table schema

In hive,the schema of partition may be difference from  the table schema.When we use spark-sql to query the data of partition which schema is difference from the table schema,we will get the exceptions as the description of the [jira](https://issues.apache.org/jira/browse/SPARK-5498) .For example:
* We take a look of the schema for the partition and the table

```sql
DESCRIBE partition_test PARTITION (dt='1');
id                  	int              	None
name                	string              	None
dt                  	string              	None

# Partition Information
# col_name            	data_type           	comment

dt                  	string              	None
```
```
DESCRIBE partition_test;
OK
id                  	bigint              	None
name                	string              	None
dt                  	string              	None

# Partition Information
# col_name            	data_type           	comment

dt                  	string              	None
```
*  run the sql
```sql
SELECT * FROM partition_test where dt='1';
```
we will get the cast exception `java.lang.ClassCastException: org.apache.spark.sql.catalyst.expressions.MutableLong cannot be cast to org.apache.spark.sql.catalyst.expressions.MutableInt`

Author: jeanlyn <jeanlyn92@gmail.com>

Closes #4289 from jeanlyn/schema and squashes the following commits:

9c8da74 [jeanlyn] fix style
b41d6b9 [jeanlyn] fix compile errors
07d84b6 [jeanlyn] Merge branch 'master' into schema
535b0b6 [jeanlyn] reduce conflicts
d6c93c5 [jeanlyn] fix bug
1e8b30c [jeanlyn] fix code style
0549759 [jeanlyn] fix code style
c879aa1 [jeanlyn] clean the code
2a91a87 [jeanlyn] add more test case and clean the code
12d800d [jeanlyn] fix code style
63d170a [jeanlyn] fix compile problem
7470901 [jeanlyn] reduce conflicts
afc7da5 [jeanlyn] make getConvertedOI compatible between 0.12.0 and 0.13.1
b1527d5 [jeanlyn] fix type mismatch
10744ca [jeanlyn] Insert a space after the start of the comment
3b27af3 [jeanlyn] SPARK-5498:fix bug when query the data when partition schema does not match table schema
@WangTaoTheTonic WangTaoTheTonic mentioned this pull request Mar 26, 2015
Closed
@liancheng
Copy link
Contributor

@DoingDone9 Would you mind to add your name to JIRA and GitHub, so that we can include your name in the credit list of the next release? Also, please add your name in your local Git configuration:

$ git config --global user.name "Your Name"

@liancheng
Copy link
Contributor

Merged into master, thanks!

@asfgit asfgit closed this in 855cba8 Mar 26, 2015
@pzzs
Copy link
Contributor Author

pzzs commented Mar 26, 2015

ok i will change @liancheng

@zzcclp zzcclp mentioned this pull request Mar 26, 2015
Closed
@viirya viirya mentioned this pull request Mar 26, 2015
Closed
@jeanlyn jeanlyn mentioned this pull request Mar 27, 2015
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
5 participants
@pzzs @AmplabJenkins @JoshRosen @liancheng @SparkQA