Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Spark: Don't daemonize, don't pretend to tail logs we don't use #16607

Merged
merged 2 commits into from
Nov 6, 2015
Merged

Spark: Don't daemonize, don't pretend to tail logs we don't use #16607

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
436839f
Spark: Don't daemonize, don't pretend to tail logs we don't use
zmerlynn Oct 30, 2015
de92fb6
Remove unneeded pieces
zmerlynn Nov 5, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
2 changes: 1 addition & 1 deletion examples/spark/images/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ all: push
# this tag and reset to v1. You should also double check the native
# Hadoop libs at that point (we grab the 2.6.1 libs, which are
# appropriate for 1.5.1-with-2.6).
TAG = 1.5.1_v1
TAG = 1.5.1_v2

containers:
docker build -t gcr.io/google_containers/spark-base base
Expand Down
4 changes: 0 additions & 4 deletions examples/spark/images/base/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,5 @@
FROM java:latest

RUN apt-get update -y
RUN apt-get install -y scala

ENV hadoop_ver 2.6.1
ENV spark_ver 1.5.1

Expand Down Expand Up @@ -30,7 +27,6 @@ RUN mkdir -p /opt && \
RUN wget -O /opt/spark/lib/gcs-connector-latest-hadoop2.jar https://storage.googleapis.com/hadoop-lib/gcs/gcs-connector-latest-hadoop2.jar

ADD log4j.properties /opt/spark/conf/log4j.properties
ADD setup_client.sh /
ADD start-common.sh /
ADD core-site.xml /opt/spark/conf/core-site.xml
ADD spark-defaults.conf /opt/spark/conf/spark-defaults.conf
Expand Down
24 changes: 0 additions & 24 deletions examples/spark/images/base/setup_client.sh
View file
Open in desktop

This file was deleted.

6 changes: 2 additions & 4 deletions examples/spark/images/master/start.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,6 @@
. /start-common.sh

echo "$(hostname -i) spark-master" >> /etc/hosts
export SPARK_LOCAL_HOSTNAME=spark-master
export SPARK_MASTER_IP=spark-master

/opt/spark/sbin/start-master.sh
tail -F /opt/spark/logs/*
# Run spark-class directly so that when it exits (or crashes), the pod restarts.
/opt/spark/bin/spark-class org.apache.spark.deploy.master.Master --ip spark-master --port 7077 --webui-port 8080
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you verify the output works.. (docker logs = spark output) I tried building your images but they ubuntu based :-(
| used apt-get.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup. There's a pile of private iterations hidden behind that one tag bump.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What failed? Blech, I thought this was supposed to be agnostic.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

apt-get failed on the base image.

Step 1 : RUN apt-get update -y
---> Running in 6d09b8abfdef
unexpected EOF
Makefile:10: recipe for target 'containers' failed

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Come to think of it why are you installing scala?

You should only need the jvm for compiled binaries.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a bit of legacy I haven't even touched. I can try to take it out, gimme a bit.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. I took out setup_client.sh as well, because it's no longer how we get to the master. (It used to shim in the address for the driver, if I'm understanding it, now DNS just works.)

2 changes: 1 addition & 1 deletion examples/spark/images/worker/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,6 @@ FROM gcr.io/google_containers/spark-base:latest

ADD start.sh /
ADD log4j.properties /opt/spark/conf/log4j.properties
EXPOSE 8080
EXPOSE 8081

ENTRYPOINT ["/start.sh"]
5 changes: 2 additions & 3 deletions examples/spark/images/worker/start.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,5 @@

. /start-common.sh

/opt/spark/sbin/start-slave.sh spark://spark-master:7077

tail -F /opt/spark/logs/*
# Run spark-class directly so that when it exits (or crashes), the pod restarts.
/opt/spark/bin/spark-class org.apache.spark.deploy.worker.Worker spark://spark-master:7077 --webui-port 8081
2 changes: 1 addition & 1 deletion examples/spark/spark-driver-controller.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ spec:
spec:
containers:
- name: spark-driver
image: gcr.io/google_containers/spark-driver:1.5.1_v1
image: gcr.io/google_containers/spark-driver:1.5.1_v2
resources:
requests:
cpu: 100m
2 changes: 1 addition & 1 deletion examples/spark/spark-gluster/spark-master-controller.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ spec:
spec:
containers:
- name: spark-master
image: gcr.io/google_containers/spark-master:1.5.1_v1
image: gcr.io/google_containers/spark-master:1.5.1_v2
ports:
- containerPort: 7077
volumeMounts:
Expand Down
2 changes: 1 addition & 1 deletion examples/spark/spark-gluster/spark-worker-controller.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ spec:
spec:
containers:
- name: spark-worker
image: gcr.io/google_containers/spark-worker:1.5.1_v1
image: gcr.io/google_containers/spark-worker:1.5.1_v2
ports:
- containerPort: 8888
volumeMounts:
Expand Down
11 changes: 1 addition & 10 deletions examples/spark/spark-master-controller.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,19 +13,10 @@ spec:
spec:
containers:
- name: spark-master
image: gcr.io/google_containers/spark-master:1.5.1_v1
image: gcr.io/google_containers/spark-master:1.5.1_v2
ports:
- containerPort: 7077
- containerPort: 8080
livenessProbe:
exec:
command:
- /opt/spark/sbin/spark-daemon.sh
- status
- org.apache.spark.deploy.master.Master
- '1'
initialDelaySeconds: 30
timeoutSeconds: 1
resources:
requests:
cpu: 100m
13 changes: 2 additions & 11 deletions examples/spark/spark-worker-controller.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,18 +13,9 @@ spec:
spec:
containers:
- name: spark-worker
image: gcr.io/google_containers/spark-worker:1.5.1_v1
image: gcr.io/google_containers/spark-worker:1.5.1_v2
ports:
- containerPort: 8888
livenessProbe:
exec:
command:
- /opt/spark/sbin/spark-daemon.sh
- status
- org.apache.spark.deploy.worker.Worker
- '1'
initialDelaySeconds: 30
timeoutSeconds: 1
- containerPort: 8081
resources:
requests:
cpu: 100m