[SPARK-42395][K8S]The code logic of the configmap max size validation lacks extra content #39967
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The default size of the CONFIG_MAP_MAXSIZE should not be greater than 1048576
|
Could you please review this pr? @dongjoon-hyun @LuciferYang |
|
Can I take your free time to help me to review this pr please？@dongjoon-hyun @HyukjinKwon @srowen @LuciferYang |
|
I don't know enough about K8S to review this. Seems harmless |
|
Thanks for review! |
|
Can you help recommend others to the review？ @srowen |
|
Can anyone else help review this PR? @viirya |
|
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
[SPARK-42395][K8S]The code logic of the configmap max size validation lacks extra content
What changes were proposed in this pull request?
This PR aims to add the extra content of the configmap into the configmap max size validation
In each configmap, Spark adds an extra content in a fixed format,this extra content of the configmap is as belows:
spark.kubernetes.namespace: default
spark.properties: |
#Java properties built from Kubernetes config map with name: spark-exec-b47b438630eec12d-conf-map
#Wed Feb 08 20:10:19 CST 2023
spark.kubernetes.namespace=default
The example specific content comes from the command output:
kubectl get cm spark-exec-b47b438630eec12d-conf-map -o yaml
apiVersion: v1
data:
spark.kubernetes.namespace: default
spark.properties: |
#Java properties built from Kubernetes config map with name: spark-exec-b47b438630eec12d-conf-map
#Wed Feb 08 20:10:19 CST 2023
spark.kubernetes.namespace=default
immutable: true
kind: ConfigMap
metadata:
creationTimestamp: "2023-02-08T12:14:13Z"
labels:
spark-app-selector: spark-9c4a3cc1d752491cba899135d8fbf492
spark-role: executor
name: spark-exec-b47b438630eec12d-conf-map
namespace: default
resourceVersion: "62638833"
uid: 28f7bbd1-38aa-4e8e-b715-9cc8eb047074
Why are the changes needed?
If no chanages,there may be a problem like jira SPARK-42344
Does this PR introduce any user-facing change?
The major of this pr is the same to the jira SPARK-42344.
How was this patch tested?
Local test
TEST CASE 1:
1.mock a file named log4j.properties in spark home directory
2.the file name length is 16,and the extra content size is 163.
set the size of the log4j.properties to:
1048576 - 163 -16 = 1048397
3.truncate the file to 1048397
truncate -s 1048397 log4j.properties
ls -l
-rwxr-xr-x 1 root root 1048397 Feb 9 16:13 log4j.properties
4.test with the commond spark-submit with arg :
--conf
spark.home=/home/work/code/apache/spark-github/dist
5.Test expectations:
The configmap is not include the content log4j.properties.Because the size is just equal to 1048576.
There is only one data in configmap spark-exec-d09ef186354c6ebe-conf-map
kubectl get cm
NAME DATA AGE
ingress-traefik 1 203d
kube-root-ca.crt 1 227d
spark-exec-d09ef186354c6ebe-conf-map 1 33s
spark-standalone-exporter 1 203d
spark-standalone-graphite-mapping 1 203d
kubectl get cm spark-exec-e33068863475c9ee-conf-map -o yaml
apiVersion: v1
data:
spark.properties: |
#Java properties built from Kubernetes config map with name: spark-exec-e33068863475c9ee-conf-map
#Thu Feb 09 12:36:20 CST 2023
spark.kubernetes.namespace=default
immutable: true
kind: ConfigMap
metadata:
creationTimestamp: "2023-02-09T04:36:20Z"
labels:
spark-app-selector: spark-8e68465119bd411ba0b0cc797abe294c
spark-role: executor
name: spark-exec-e33068863475c9ee-conf-map
namespace: default
resourceVersion: "62889868"
uid: ba8d7f31-a4c5-42fd-9aba-a02b73eb37bc
TEST CASE 2:
1.the file name length is 16,and the extra content size is 163.
set the size of the log4j.properties to:
1048576 - 163 - 16 - 1= 1048396
2.truncate the file to 1048396
truncate -s 1048396 log4j.properties
ls -l
-rwxr-xr-x 1 root root 1048396 Feb 9 16:32 log4j.properties
3.test with the commond spark-submit with arg :
--conf
spark.home=/home/work/code/apache/spark-github/dist
4.Test expectations:
The configmap is include the content log4j.properties.
There is 2 datas in the configmap spark-exec-1d90d186354e63ee-conf-map
kubectl get cm
NAME DATA AGE
ingress-traefik 1 203d
kube-root-ca.crt 1 227d
spark-exec-1d90d186354e63ee-conf-map 2 52s
spark-standalone-exporter 1 203d
spark-standalone-graphite-mapping 1 203d