/bigtop

BIGTOP-2561: add juju bundle for hadoop-spark #166

Closed
wants to merge 15 commits into
from

Conversation

Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
3 participants
@kwmonroe @petevg @ktsakalozos
@kwmonroe
Member

kwmonroe commented Dec 2, 2016

This combines hadoop-processing and spark-processing into a single bundle:

https://jujucharms.com/hadoop-spark/

Test results are available here across multiple clouds:

http://data.vapour.ws/cwr-tests/results/bundle_hadoop_spark/index.html

We are working through infrastructure issues atm (hence a lot of yellow triangles), but we haven't seen any failures with the charms included in this bundle for the last couple weeks. Hence I'm putting it up for review / inclusion upstream.

kwmonroe and others added some commits Sep 22, 2016

@kwmonroe
initial hadoop-spark bundle
35b8d13
@kwmonroe
colocate bundle applications 
To save resources, colocate some of the applications when it makes sense.
Put the namenode and resourcemgr on the same unit; spark and zeppelin;
ganglia and rsyslog.

Update readme formatting and callout these colocated apps. Also tweak
the restricted network section and add a section for connecting to zepp.

Note, machine spec in the bundle yaml start with machine '1' instead of
'0'. This is because someone may deploy this on juju-1.x, where machine
'0' is the bootstrap node.
d1387ed
@ktsakalozos @kwmonroe
Be explicit on series for CWR to deploy the right series
b52a7c5
@kwmonroe
refresh charms
040bab2
@kwmonroe
refresh hadoop-spark bundle 
- update readme to be consistent with other bigtop-based bundles
- bump all charms to latest revisions
- update amulet test to call charm smoke tests and work around machine spec
a29bac8
@kwmonroe
skip slave smoke tests and update smoke failure message
ef3d705
@kwmonroe
bump spark revno for spark-bench fixes
506ba32
@kwmonroe
refresh spark charm and readme for new sparkbench suite
4c0f812
@kwmonroe
charm revision bump 
- new ganglia charms report app version and fix '-broken' hook failures
- new spark charm correctly runs sparkbench regardless of spark exe mode
ff19096
@kwmonroe
bump spark rev for updated test case and sparkbench composite is now … 
…seconds
5a961a6
@kwmonroe
bump zk revno to pick up test rename and app version
71c5731
@kwmonroe
bump rsyslog for lint and zk for an amulet test enhancement (ease loc… 
…al testing)
745deec
@kwmonroe
add ppa, charm rev bump, new constraints 
- ensure tests use juju/stable ppa
- bump zk to latest (consistent bigtop_ prefix, readme tweaks)
- bump core hadoop charms to latest (new swap handler, detect failed service
start, add spark user/group to nn/rm)
- aws defaults to 8gb root filesystem; bump this to 20gb
638221b
@kwmonroe
rev charms to pick up new swap handler
34ffbe4
@kwmonroe
workaround azure root-disk: https://bugs.launchpad.net/juju/+bug/1645408
3fc9934
@petevg
Contributor

petevg commented Dec 2, 2016

LGTM/+1 (I was just able to deploy this bundle to aws without any hitches, too)

@asfgit asfgit closed this in f4d023b Dec 2, 2016

@kwmonroe kwmonroe deleted the juju-solutions:feature/hadoop-spark-bundle branch Dec 2, 2016

kulikovav pushed a commit to kulikovav/bigtop that referenced this pull request Feb 8, 2017

@kwmonroe
BIGTOP-2561: add juju bundle for hadoop-spark (closes #166) 
Signed-off-by: Kevin W Monroe <kevin.monroe@canonical.com>
5a54329
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment