Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update deprecated hash function #259

Merged
merged 1 commit into from Sep 19, 2017

Conversation

esmioley
Copy link
Contributor

Guava deprecated hashString(String) in v15, and removed it in v16.
Replace call with hashUnencodedString(String), which replace it, to
be compatible with newer versions of Guava.

Guava deprecated hashString(String) in v15, and removed it in v16.
Replace call with hashUnencodedString(String), which replace it, to
be compatible with newer versions of Guava.
@CLAassistant
Copy link

CLAassistant commented Sep 18, 2017

CLA assistant check
All committers have signed the CLA.

@ovj
Copy link
Contributor

ovj commented Sep 18, 2017

@vinothchandar , @n3nash FYI.

@vinothchandar
Copy link
Member

Happy to pull this.. but fyi

<!-- we have to stay at <= 16.0, due to issues with HBase client -->
            <dependency>
                <groupId>com.google.guava</groupId>
                <artifactId>guava</artifactId>
                <version>15.0</version>
            </dependency>

@vinothchandar vinothchandar merged commit 6230e15 into apache:master Sep 19, 2017
@esmioley esmioley deleted the remove_guava_deprecated_hash branch September 19, 2017 20:40
@vinothchandar
Copy link
Member

I am afraid we have a problem. Spark is still on 14.0 and a job fails as below.

java.lang.NoSuchMethodError: com.google.common.hash.HashFunction.hashUnencodedChars(Ljava/lang/CharSequence;)Lcom/google/common/hash/HashCode;
	at com.uber.hoodie.table.HoodieCopyOnWriteTable$UpsertPartitioner.getPartition(HoodieCopyOnWriteTable.java:363)
	at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:152)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
	at org.apache.spark.scheduler.Task.run(Task.scala:99)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)

I am going to revert this change. Can you comment on how you got this tested? I am using Spark 2.1

@vinothchandar
Copy link
Member

@esmioley ^^

vinothchandar pushed a commit to vinothchandar/hudi that referenced this pull request Sep 28, 2017
vinothchandar pushed a commit that referenced this pull request Sep 28, 2017
@esmioley
Copy link
Contributor Author

We're using a spark build running with guava 16 (which is why it was failing in the first place, requiring a change)

@vinothchandar
Copy link
Member

when you say spark build, is this our own spark library build or final jar given to spark-submit has guava 16.

vinishjail97 pushed a commit to vinishjail97/hudi that referenced this pull request Dec 15, 2023
Co-authored-by: Y Ethan Guo <ethan.guoyihua@gmail.com>
Co-authored-by: 苏承祥 <1142819049@qq.com>
Co-authored-by: 苏承祥 <sucx@tuya.com>
Co-authored-by: Danny Chan <yuzhao.cyz@gmail.com>
Co-authored-by: Alexey Kudinkin <alexey.kudinkin@gmail.com>
Co-authored-by: harshal <harshal.j.patil@gmail.com>
Co-authored-by: Pramod Biligiri <pramodbiligiri@gmail.com>
Co-authored-by: huangxiaoping <35296098+huangxiaopingRD@users.noreply.github.com>
Co-authored-by: Teng <teng_huo@outlook.com>
Co-authored-by: zhuanshenbsj1 <34104400+zhuanshenbsj1@users.noreply.github.com>
Co-authored-by: Rahil C <32500120+rahil-c@users.noreply.github.com>
Co-authored-by: Rahil Chertara <rchertar@amazon.com>
Co-authored-by: Balaji Varadarajan <balaji.varadarajan@robinhood.com>
Co-authored-by: Nicholas Jiang <programgeek@163.com>
Co-authored-by: Sagar Sumit <sagarsumit09@gmail.com>
Co-authored-by: sandyfog <154525105@qq.com>
Co-authored-by: Shiyan Xu <2701446+xushiyan@users.noreply.github.com>
Co-authored-by: Manu <36392121+xicm@users.noreply.github.com>
Co-authored-by: Bingeng Huang <304979636@qq.com>
Co-authored-by: hbg <bingeng.huang@shopee.com>
Co-authored-by: Tim Brown <tim@onehouse.ai>
Co-authored-by: Lokesh Jain <ljain@apache.org>
Co-authored-by: rfyu <39233058+rfyu@users.noreply.github.com>
Co-authored-by: Raymond Xu <xu.shiyan.raymond@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants