Kyspark 3.2.x 4.x qa merge#42650

Closed

zheniantoushipashi wants to merge 696 commits into

apache:masterfrom

zheniantoushipashi:kyspark-3.2.x-4.x-qa-merge

This pull request is big! We're only showing the most recent 250 commits

Commits on Sep 14, 2021

[SPARK-36705][FOLLOW-UP] Fix unnecessary logWarning when PUSH_BASED_SHUFFLE_ENABLED is set to false

Minchu Yang
authored and
Mridul Muralidharan
committed
[SPARK-36702][SQL] ArrayUnion handle duplicated Double.NaN and Float.Nan

AngersZhuuuu
authored and
cloud-fan
committed
[SPARK-36712][BUILD][FOLLOWUP] Improve the regex to avoid breaking pom.xml
dongjoon-hyun
committed

Commits on Sep 15, 2021

Commits on Sep 16, 2021

Commits on Sep 17, 2021

Commits on Sep 20, 2021

Commits on Sep 21, 2021

Commits on Sep 22, 2021

[SPARK-36771][PYTHON][3.2] Fix `pop` of Categorical Series

xinrong-meng
authored and
ueshin
committed
[SPARK-36820][3.2][SQL] Disable tests related to LZ4 for Hadoop 2.7 profile

sunchao
authored and
viirya
committed
[SPARK-36803][SQL] Fix ArrayType conversion when reading Parquet files written in legacy mode

Ivan Sadikov
authored and
cloud-fan
committed
[SPARK-36753][SQL] ArrayExcept handle duplicated Double.NaN and Float.NaN

AngersZhuuuu
authored and
cloud-fan
committed
[SPARK-36818][PYTHON] Fix filtering a Series by a boolean Series

xinrong-meng
authored and
ueshin
committed

Commits on Sep 23, 2021

[SPARK-36791][DOCS] Fix spelling mistakes in running-on-yarn.md file where JHS_POST should be JHS_HOST

jiaoqb
authored and
HyukjinKwon
committed
[SPARK-36782][CORE] Avoid blocking dispatcher-BlockManagerMaster during UpdateBlockInfo

authored and
gengliangwang
committed
[MINOR][SQL][DOCS] Correct the 'options' description on UnresolvedRelation

HyukjinKwon
authored and
huaxingao
committed
[SPARK-36795][SQL] Explain Formatted has Duplicate Node IDs

Michael Chen
authored and
HyukjinKwon
committed
[SPARK-36782][CORE][FOLLOW-UP] Only handle shuffle block in separate thread pool

Ngone51
authored and
gengliangwang
committed
Preparing Spark release v3.2.0-rc4
gengliangwang
committed
Preparing development version 3.2.1-SNAPSHOT
gengliangwang
committed

Commits on Sep 24, 2021

Commits on Sep 27, 2021

Commits on Sep 28, 2021

Commits on Sep 29, 2021

[MINOR][DOCS] Mention other Python dependency tools in documentation
HyukjinKwon
committed

Commits on Sep 30, 2021

[SPARK-36865][PYTHON][DOCS] Add PySpark API document of session_window

sarutak
authored and
HeartSaVioR
committed

Commits on Oct 3, 2021

[SPARK-36907][PYTHON] Fix DataFrameGroupBy.apply without shortcut

ueshin
authored and
HyukjinKwon
committed

Commits on Oct 5, 2021

[SPARK-36874][SQL] DeduplicateRelations should copy dataset_id tag to avoid ambiguous self join

sarutak
authored and
cloud-fan
committed
[SPARK-36705][FOLLOW-UP] Support the case when user's classes need to register for Kryo serialization

authored and
Mridul Muralidharan
committed

Commits on Oct 6, 2021

Commits on Oct 20, 2021

#11 SQLConf support thread local prop

hn5092
authored and
chenzhx
committed
add classForName to Utils

zheniantoushipashi
authored and
chenzhx
committed
export spark ugi

zheniantoushipashi
authored and
chenzhx
committed
Change repository kyligence

Mingming Ge
authored and
chenzhx
committed
rebase spark 3 rc3

Mingming Ge
authored and
chenzhx
committed
support cdh-5.7 hive version 1.2.1.spark2

7mming7
authored and
chenzhx
committed
add read bytes metric in FileSourceScanExec and HiveTableScanExec

7mming7
authored and
chenzhx
committed
fix EventLoggingListener throws Direct self-reference leading to cycle.

7mming7
authored and
chenzhx
committed
filters add new filter Trivial

7mming7
authored and
chenzhx
committed
performance issue when gc blocks

7mming7
authored and
chenzhx
committed
bug fix taskmetrics

7mming7
authored and
chenzhx
committed
KE-13182 support similar to

7mming7
authored and
chenzhx
committed
cast varchar/char as string when no precision

7mming7
authored and
chenzhx
committed
Add the alias function week of weekofyear

7mming7
authored and
chenzhx
committed
write parquet check cell size

7mming7
authored and
chenzhx
committed
add config to allow user hadoop classpath first

7mming7
authored and
chenzhx
committed
add sparkConf spark.am_filter.enabled

7mming7
authored and
chenzhx
committed
define spark default db

7mming7
authored and
chenzhx
committed
add SJF scheduler

7mming7
authored and
chenzhx
committed

Commits on Oct 21, 2021

support limit offset

7mming7
authored and
chenzhx
committed
ignore ut SPARK-31646

7mming7
authored and
chenzhx
committed
performance issue when gc blocks

7mming7
authored and
chenzhx
committed
support limit offset

7mming7
authored and
chenzhx
committed
ignore ut

7mming7
authored and
chenzhx
committed
support ceil timestamp

7mming7
authored and
chenzhx
committed
support ceil timestamp func

7mming7
authored and
chenzhx
committed
KE-19992 s3 read retry before accessing FileStatus properties (#191)

authored and
chenzhx
committed
support SparkExecutorHdfsLogAppender

7mming7
authored and
chenzhx
committed
KE-19998, add more metrics to applicationinfo

shaoxiong.zhan
authored and
chenzhx
committed
KE-20353 # avoid cartesian partition oom (#198)

JhusiJeremy
authored and
chenzhx
committed
limit mem usage during collect

7mming7
authored and
chenzhx
committed
KE-19990, [spark3] Support skip.header.line.count option in Hive table

shaoxiong.zhan
authored and
chenzhx
committed
release 3.1.x-kylin-4.x-r1

7mming7
authored and
chenzhx
committed
rebase spark branch 3.1

7mming7
authored and
chenzhx
committed
rebase spark branch 3.1

7mming7
authored and
chenzhx
committed
update spark version

zheniantoushipashi
authored and
chenzhx
committed
update spark version

zheniantoushipashi
authored and
chenzhx
committed
KE-19948, introduce func of kill application

microbearz
authored and
chenzhx
committed
KE-19989 spark eventlog roll up

zheniantoushipashi
authored and
chenzhx
committed
update spark version

zheniantoushipashi
authored and
chenzhx
committed
release r19 by spark 3.1.2 rc3

7mming7
authored and
chenzhx
committed

Commits on Oct 22, 2021

release r20

7mming7
authored and
chenzhx
committed
release r20

7mming7
authored and
chenzhx
committed
SPARK-34800 use fine-grained lock in SessionCatalog.tableExists (#222)

woyumen4597
authored and
chenzhx
committed
upgrade parquet version

huangfeng1993
authored and
chenzhx
committed
KC-7417: support skip csv header in hive partition table

zheniantoushipashi
authored and
chenzhx
committed
KE-26764 fix loophole in spark (#253)

woyumen4597
authored and
chenzhx
committed
upgrade to r22 (#254)

woyumen4597
authored and
chenzhx
committed
release r23 add streaming jars
chenzhx
committed
update kafka client version
chenzhx
committed
KE-15603 upgrade scala to 2.12.13 (#257)

Enwei Jiao
authored and
chenzhx
committed
KE-27870, fix default db bug

leozhiyu
authored and
chenzhx
committed
KE-27048, query scan bytes is incorrect.

leozhiyu
authored and
chenzhx
committed
KE-26107: The canCast method of type of char/varchar is modified to be consistent with StringType

Kyligence
authored and
chenzhx
committed
KE-26862 dataset - toIterator (#298)

lxian
authored and
chenzhx
committed
KE-29738 minor

whuGuo
authored and
chenzhx
committed
KE-29932 Support SoftAffinity Scheduler to FileScanRDD for Spark3 (#308)

zzcclp
authored and
chenzhx
committed
upgrade to r31

whuGuo
authored and
chenzhx
committed
KE-29177 fix roll up

no-valid
authored and
chenzhx
committed
KE-29976: spark.sql.orc.filterPushdown not working with Spark 3.1.1 for tables with varchar data type

biaobiao.sun
authored and
chenzhx
committed
KE-30503, add pom with MultiDelimitSerDe which causes sampling error problems

gleonSun
authored and
chenzhx
committed
KE-30503, upgrade spark version

gleonSun
authored and
chenzhx
committed
KE-27088 cherry-pick KE-14216 Avoid sparder driver OOM for decoding data (#326)

authored and
chenzhx
committed
KE-30726 stop task result getter correctly (#321)

lxian
authored and
chenzhx
committed
AL-3815 fix two adjacent limit cause offset error

7mming7
authored and
chenzhx
committed
release r37

7mming7
authored and
chenzhx
committed
KE-31310 Fix Memory leak of ExecutionListenerBus

jiale.he
authored and
chenzhx
committed
Reduce the scope of synchronized to prevent potential deadlock
chenzhx
committed
upgrade spark version r38
chenzhx
committed

Commits on Jan 2, 2022

upgrade spark version to 3.2.0
chenzhx
committed
KE-31895 replace specific nameservice hacluster

7mming7
authored and
chenzhx
committed
release 3.1.1-kylin-4.x-r42

7mming7
authored and
chenzhx
committed
AL-4964 add log4j-over-sl4j (#361)
chenzhx
committed
AL-4217 skip view cache when lookupRelation (#365)
chenzhx
committed
KE-33181 update parquet version to 1.2.2-kylin-r1 (#368)

xifeng
authored and
chenzhx
committed
[SPARK-35561][SQL] Remove leading zeros from empty static number type partition

dchvn
authored and
chenzhx
committed
release 3.2.0-kylin-4.x-r45
chenzhx
committed

Commits on Jan 6, 2022

fix ut
chenzhx
committed

Commits on Jan 17, 2022

update version to v46
chenzhx
committed

Commits on Jan 20, 2022

KE-33545 update mortbay jetty version to 7.0.0-pre5
chenzhx
committed

Commits on Feb 22, 2022

KE-34130 delete log4j-over-slf4

leejaywei
authored and
chenzhx
committed
KE-34191 replace partition Table path (#388)

leesanQAQ
authored and
chenzhx
committed
KE-34191 fix replace partition table storage path

7mming7
authored and
chenzhx
committed
release r49

7mming7
authored and
chenzhx
committed
revert KE-34130 delete log4j-over-sl4j and update version to r50 (#394)

leejaywei
authored and
chenzhx
committed
update to 3.1.1-kylin-4.x-r51-beta

ygjia
authored and
chenzhx
committed
update tomcat-embed-core

ygjia
authored and
chenzhx
committed
update tomcat-annotations-api

ygjia
authored and
chenzhx
committed
Update to 3.1.1-kylin-4.x-r51

ygjia
authored and
chenzhx
committed
KE-34191 fix replace partition table storage path

7mming7
authored and
chenzhx
committed
KE-34191 fix replace partition table storage path

7mming7
authored and
chenzhx
committed
KE-34191 fix replace partition table storage path

7mming7
authored and
chenzhx
committed
AL-5357 fix api-util in profile hadoop2.7 and update to 3.1.1-kylin-4.x-r52

ygjia
authored and
chenzhx
committed

Commits on Feb 23, 2022

upgrade spark version to 3.2
chenzhx
committed

Commits on Mar 2, 2022

Wait to resolve missing attributes before applying DeduplicateRelations
chenzhx
committed
KE-33600 Add Parquet Column Name Check Switch (#406)

authored and
chenzhx
committed
revert KE-33600 Add Parquet Column Name Check Switch (#407)

jlfsdtc
and
longfei.jiang
authored

Commits on Mar 3, 2022

KE-34130 delete log4j-over-sl4j and update version to r55 (#409)
leejaywei
authored

Commits on Mar 4, 2022

Commits on Mar 7, 2022

Commits on Mar 11, 2022

KE-31638 add parameter in request head to proxy spark ui (#424)
leejaywei
authored

Commits on Mar 24, 2022

KE-31356 kyspark-3.2.x -4.x fix ut (#421)
leejaywei
authored

Commits on Mar 27, 2022

Commits on Mar 29, 2022

Commits on Apr 13, 2022

KE-29526 SparkUI fix amIpFilter (#445)
ygjia
authored

Commits on Apr 14, 2022

Fix release and delete log4j2 (#447)

leejaywei
and
Enwei Jiao
authored

Commits on May 5, 2022

KE-35827 new second storage sql pushdown (#453)

authored

Commits on May 6, 2022

AL-5705 when taskId is not empty, add taskId subStr(0,8) in local mode's eventlog name (#449)

jlfsdtc
and
longfei.jiang
authored
AL-5705 update version (#454)

jlfsdtc
and
longfei.jiang
authored

Commits on May 11, 2022

KE-36162 fix AnalysisException bug roundBase decimalType when scale < 0 (#456)
songzhxlh-max
authored

Commits on May 16, 2022

fix ut (#462)
chenzhx
authored

Commits on May 18, 2022

Commits on May 19, 2022

Commits on May 24, 2022

KE-36161 fix replace function (#471)
songzhxlh-max
authored

Commits on May 31, 2022

KE-36478 support string function in DS V2 (#472)

chenzhx
and
beliefer
authored

Commits on Jun 1, 2022

KE-36478 revert offset code (#473)
chenzhx
authored

Commits on Jun 2, 2022

[SPARK-38768][SQL] Remove Limit from plan if complete push down limit to data source (#474)

chenzhx
and
beliefer
authored

Commits on Jun 7, 2022

KE-37052 translate boolean column to V2Predicate (#476)
chenzhx
authored

Commits on Jun 21, 2022

KE-37170 fix broadcastManager NPE (#480)
yabola
committed
[SPARK-39316][SQL] Merge PromotePrecision and CheckOverflow into decimal binary arithmetic (#481)

authored and
yabola
committed
KE-35544 fall back to fetching all partition metadata when filtering partitions not supported

Yu Gan
authored and
yabola
committed

Commits on Jul 1, 2022

KE-37693 user-specified repartition shouldn't be optimized with CoalesceShufflePartitions

Yu Gan
authored and
yugan95
committed

Commits on Jul 5, 2022

KE-35596 Update jackson & tomcat version for vulnerability issue
yhcast0
committed

Commits on Jul 6, 2022

Merge pull request #492 from yhcast0/KE-35596
yhcast0
authored

Commits on Jul 20, 2022

Commits on Aug 8, 2022

Commits on Aug 23, 2022

KE-36175 update private token

Yu Gan
authored and
yugan95
committed
KE-36175 [FOLLOWUP] fix hadoop 2.7.4 not compatible

Yu Gan
authored and
yugan95
committed
KE-36175 [FOLLOWUP] fix read-write separation mode

Yu Gan
authored and
yugan95
committed
KE-36175 [FOLLOWUP] fix read-write separation mode when application restart

Yu Gan
authored and
yugan95
committed

Commits on Sep 13, 2022

Commits on Sep 15, 2022

AL-6643 Fix security problem (#516)

yhcast0
authored and
chenzhx
committed
KE-38231 Revert upgrade guava

RolatZhang
authored and
chenzhx
committed
[SPARK-39819][SQL] DS V2 aggregate push down can work with Top N or Paging (Sort with expressions) (#525)

committed

Commits on Oct 13, 2022

kyspark-3.2.x-4.x-qa cherrypick into kyspark-3.2.x-4.x (#543)

authored

Commits on Nov 1, 2022

KE-39406 Due to a security breach, upgrade commons-text from 1.9 to 1.10.0
Mrhs121
committed

Commits on Nov 14, 2022

KE-40047 fix snyk vulnerabilities, upgrade ivy
Mrhs121
committed

Commits on Nov 24, 2022

[SPARK-40211][CORE][SQL] Allow customize initial partitions number in take() behavior

liuzqt
authored and
yabola
committed
KE-40007 Fix vulnerability, upgrade jackson-databind

Jiale He
authored and
yabola
committed
[SPARK-40320][Core] Executor should exit when initialization failed for fatal error
yabola
committed

Commits on Dec 19, 2022

AL-4757 when refresh InMemoryFileIndex, can recursiveDirChildrenFiles (#566)

jlf
authored and
yabola
committed
AL-7433 Fix vulnerabilities (#574)

yhcast0
authored and
yabola
committed
AL-7518 fix snyk vulnerabilities, upgrade jetty from 9.4.43.v20210629… (#573)

Mrhs121
authored and
yabola
committed
AL-7626 Fix vulnerabilities

yhcast0
authored and
yabola
committed
KE-40038 Support Logical View
yabola
committed

Commits on Feb 3, 2023

SP-35241 Reduce the output partition of output stage to avoid producing small files
7mming7
committed

Commits on Feb 10, 2023

Revert "SP-35241 Reduce the output partition of output stage to avoid producing small files"
zheniantoushipashi
committed

Commits on Feb 28, 2023

KE-35241 Reduce the output partition of output stage to avoid producing small files
7mming7
committed

Commits on Mar 24, 2023

AL-8065 [SPARK-39927][BUILD] Upgrade to Avro 1.11.1 (#599)

leejaywei
and
iemejia
committed
KE-40948 Add rowGroup filters info (#596)

yabola
authored and
leejaywei
committed
KE-40636 Fix spark history server leaves inprogress files forever (#601)

authored and
leejaywei
committed
AL-8113 Upgrade netty-all to 4.1.89.Final & protobuf-java to 3.16.3 (#605)

yihuawu1986
authored and
leejaywei
committed

Commits on Apr 10, 2023

cherry-pick 3.3 commits (#616)

authored

Commits on Apr 14, 2023

Ke 4.6.8.0 newten cherrypick (#617)

Mrhs121
and
jlf
authored

Commits on May 11, 2023

KE-40433 add page index filter log (#619) (#624)
ygjia
authored

Commits on May 27, 2023

KE-41399 spark-42388 Avoid parquet footer reads twice in vectorized reader (#623) (#629)
yabola
authored

Commits on Jun 15, 2023

AL-7861 simplify conditionals in filter in case of large CaseWhen expr

Yu Gan
authored and
yugan95
committed

Commits on Jul 7, 2023

Commits on Jul 27, 2023

[SPARK-38631][CORE] Uses Java-based implementation for un-tarring at Utils.unpack (#643) (#648)

Mrhs121
and
HyukjinKwon
authored

Commits on Aug 18, 2023

Commits on Aug 19, 2023

Revert "KE-42055 parquet footer read cache (#641)" (#664)
ygjia
authored