Support for highly available nimbus upcoming feature in storm 1.0

[drewrobb]

Jumping the gun here, but storm 0.11 1.0 supports HA nimbus when released: apache/storm#354. It would be extremely nice if the mesos storm framework supported this directly. I see two options:

A) Just allow the user to run multiple storm-mesos nimbus processes directly, and configure the HA options separately if desired. Make sure that the framework leader and associated state follows the elected nimbus master.

B) Split the framework scheduler and the actual nimbus process, so that the scheduler will launch N nimbus instances as mesos tasks.

Seems like choice A) would be a lot simpler to implement, but I'm not really sure. A) would certainly not be any more difficult to deploy assuming that the user is already using a framework like marathon that could provision multiple instances of nimbus (and would not re-implement this responsibility).

[erikdw]

@drewrobb : I'm not yet familiar with the HA Nimbus implementation upcoming in Storm 0.11 1.0. I found reference to the design doc though so will need to read through that to be able to answer you.

[salimane]

@drewrobb it seems 0.11.0 is now storm 1.0.0.
Storm 1.0.0 is going to be released the next weeks or so, so we should start checking support for 1.0.0 .

I'm really excited for Storm HA support.

@drewrobb @erikdw had you have time to check on this ?
Thanks

[DarinJ]

Allong those lines we'll need to just ensure 1.0.0 will run as there may be breaking changes (as in 0.10.0). Fun as I haven't fully migrated to 0.10.x yet...

[salimane]

@DarinJ you might want to start migrating to 1.0.0 :D

[erikdw]

@salimane : ah, I missed that change from 0.11.0 to 1.0.0 (I was assuming that both still existed, which would have been ultra-confusing with 2.0.0 also in the works!).

I haven't looked at 1.0.0 at all, except reading some random code reviews and emails on the storm user and dev lists.

I'm in a similar boat to @DarinJ, except my boat might be further from shore:

• We have over a dozen teams that we'd need to verify their stuff running on the new storm version in our staging environment (herding cats is easy in comparison)
• I'm 99% sure that storm's change in 0.10.0 from "logback" to "log4j 2" is going to require us to revamp the logging library that our users leverage for logging in a consistent manner to the correct location, and that means getting everyone to upgrade. That will not be fun.
• I'm frankly terrified of upgrading storm. There have been so many random gotchas whenever we have done our upgrades within the 0.9.x train that I'm not confident in these huge rewrites being seamless.
• We're still running a private forked version of this project right now! We are currently focused on getting off of this fork onto the mesos/storm public repo, hence our increased activity in this repo.

[salimane]

@erikdw I was also bitten during testing by the migration from "logback" to "log4j 2" but, luckily, we are using puppet for server provisioning https://github.com/meltwater/puppet-storm. So there would be only one place to update.

[salimane]

this just became real as storm version 1.0.0 was just released today https://storm.apache.org/2016/04/12/storm100-released.html :)

[erikdw]

@salimane : ohh no. :) Thanks for pointing this out.

[erikdw]

Via PR #177 we have added support for storm 1.0+. We haven't done anything specifically for supporting HA Nimbus in this project yet, but now we have the support for the version of storm that includes it.