Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Storm-166: Nimbus HA design doc and implementation. #354

Merged
merged 96 commits into from
Aug 24, 2015

Commits on Sep 12, 2014

  1. upgraded to logback 1.0.13

    Midpoint Applications authored and Parth-Brahmbhatt committed Sep 12, 2014
    Configuration menu
    Copy the full SHA
    2b8dade View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    4e1f474 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    670ad01 View commit details
    Browse the repository at this point in the history

Commits on Sep 14, 2014

  1. Configuration menu
    Copy the full SHA
    9280afb View commit details
    Browse the repository at this point in the history

Commits on Sep 22, 2014

  1. Configuration menu
    Copy the full SHA
    d0aa8ff View commit details
    Browse the repository at this point in the history

Commits on Sep 23, 2014

  1. Configuration menu
    Copy the full SHA
    4cb2eee View commit details
    Browse the repository at this point in the history

Commits on Sep 24, 2014

  1. Configuration menu
    Copy the full SHA
    4d0c650 View commit details
    Browse the repository at this point in the history

Commits on Oct 6, 2014

  1. Configuration menu
    Copy the full SHA
    50c4c34 View commit details
    Browse the repository at this point in the history

Commits on Oct 7, 2014

  1. Configuration menu
    Copy the full SHA
    f5ac420 View commit details
    Browse the repository at this point in the history
  2. Merge remote-tracking branch 'upstream/master' into STORM-166

    Conflicts:
    	storm-core/test/clj/backtype/storm/supervisor_test.clj
    Parth-Brahmbhatt committed Oct 7, 2014
    Configuration menu
    Copy the full SHA
    b074843 View commit details
    Browse the repository at this point in the history

Commits on Oct 8, 2014

  1. Configuration menu
    Copy the full SHA
    0e24a43 View commit details
    Browse the repository at this point in the history
  2. BugFix: bittorrent code was downloaded in wrong folder. Fixed in heck…

    …y way. All tests pass now and was able to run wordcount and excalmation topologies.
    Parth-Brahmbhatt committed Oct 8, 2014
    Configuration menu
    Copy the full SHA
    050e8fd View commit details
    Browse the repository at this point in the history

Commits on Oct 9, 2014

  1. Making the code distributor injectable via strom configs. Changes to …

    …HDFSCodeDistributor. Working version of HDFSCodeDistributor.
    Parth-Brahmbhatt committed Oct 9, 2014
    Configuration menu
    Copy the full SHA
    932cecf View commit details
    Browse the repository at this point in the history

Commits on Oct 22, 2014

  1. Adding the auto discovery of nimbuses for all clients. Modified UI to…

    … display list of nimbus hosts and current leader.
    Parth-Brahmbhatt committed Oct 22, 2014
    Configuration menu
    Copy the full SHA
    16a3ce4 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    fbb326b View commit details
    Browse the repository at this point in the history
  3. Removing NIMBUS_HOST config values, updated unit tests so they can su…

    …cceed in absence of the NIMBUS-CONF value.
    Parth-Brahmbhatt committed Oct 22, 2014
    Configuration menu
    Copy the full SHA
    f0701b3 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    046262a View commit details
    Browse the repository at this point in the history

Commits on Dec 1, 2014

  1. Configuration menu
    Copy the full SHA
    11aadf5 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    728b35a View commit details
    Browse the repository at this point in the history
  3. Implemented leader latch listener that relinquishes lock if it does n…

    …ot have all the active topology code locally, keeps the lock if it can verify all active topology code exists locally.
    Parth-Brahmbhatt committed Dec 1, 2014
    Configuration menu
    Copy the full SHA
    771f6a9 View commit details
    Browse the repository at this point in the history

Commits on Dec 2, 2014

  1. Configuration menu
    Copy the full SHA
    609f569 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    50a33ac View commit details
    Browse the repository at this point in the history
  3. Adding wait loop in master nimbus to allow for desired code replicati…

    …on to be achieved before the topology is activated.
    Parth-Brahmbhatt committed Dec 2, 2014
    Configuration menu
    Copy the full SHA
    522696c View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    e037d3b View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    e5c14e2 View commit details
    Browse the repository at this point in the history

Commits on Dec 3, 2014

  1. Configuration menu
    Copy the full SHA
    c684939 View commit details
    Browse the repository at this point in the history

Commits on Dec 5, 2014

  1. Configuration menu
    Copy the full SHA
    32d7838 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    dda2ee1 View commit details
    Browse the repository at this point in the history

Commits on Dec 8, 2014

  1. Configuration menu
    Copy the full SHA
    4e35c1a View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    d198242 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    62ee10b View commit details
    Browse the repository at this point in the history
  4. Added security login for HDFSCodeDistributor so it would work with se…

    …cure and non secure HDFS.
    
    Removed couple of TODOs from BitTorrentCodeDistributor.
    Parth-Brahmbhatt committed Dec 8, 2014
    Configuration menu
    Copy the full SHA
    635c9b2 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    6468058 View commit details
    Browse the repository at this point in the history

Commits on Dec 9, 2014

  1. Configuration menu
    Copy the full SHA
    7f9c7db View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    2b63019 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    49680fd View commit details
    Browse the repository at this point in the history

Commits on Dec 11, 2014

  1. Configuration menu
    Copy the full SHA
    00300ae View commit details
    Browse the repository at this point in the history

Commits on Dec 12, 2014

  1. adding todo.

    Parth-Brahmbhatt committed Dec 12, 2014
    Configuration menu
    Copy the full SHA
    678afed View commit details
    Browse the repository at this point in the history

Commits on Dec 16, 2014

  1. minor bug fixes.

    Parth-Brahmbhatt committed Dec 16, 2014
    Configuration menu
    Copy the full SHA
    fa69f2a View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    02c27a9 View commit details
    Browse the repository at this point in the history

Commits on Dec 17, 2014

  1. Configuration menu
    Copy the full SHA
    88e70a8 View commit details
    Browse the repository at this point in the history
  2. Merge remote-tracking branch 'upstream/master' into STORM-166

    Conflicts:
    	conf/defaults.yaml
    	pom.xml
    	storm-core/src/clj/backtype/storm/cluster.clj
    	storm-core/src/clj/backtype/storm/daemon/nimbus.clj
    	storm-core/src/clj/backtype/storm/thrift.clj
    	storm-core/src/clj/backtype/storm/ui/core.clj
    	storm-core/src/jvm/backtype/storm/Config.java
    	storm-core/src/jvm/backtype/storm/utils/NimbusClient.java
    	storm-core/test/clj/backtype/storm/nimbus_test.clj
    	storm-core/test/clj/backtype/storm/security/auth/auth_test.clj
    	storm-core/test/clj/backtype/storm/utils_test.clj
    Parth-Brahmbhatt committed Dec 17, 2014
    Configuration menu
    Copy the full SHA
    6b0da16 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    1dccc1b View commit details
    Browse the repository at this point in the history
  4. Fixing merge issues.

    Parth-Brahmbhatt committed Dec 17, 2014
    Configuration menu
    Copy the full SHA
    8241146 View commit details
    Browse the repository at this point in the history

Commits on Dec 18, 2014

  1. Configuration menu
    Copy the full SHA
    df4d8c2 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    366130e View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    256c35a View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    e5a96f4 View commit details
    Browse the repository at this point in the history

Commits on Dec 19, 2014

  1. defaulting to LocalFileSystemCodeDistributor, moving all code distrib…

    …utors to their own packages.
    Parth-Brahmbhatt committed Dec 19, 2014
    Configuration menu
    Copy the full SHA
    547ed49 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    50fdcee View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    d0bb383 View commit details
    Browse the repository at this point in the history
  4. STORM-166: fixing a bug where the leader lock was not queued due to i…

    …ncorrect latch state being used
    Parth-Brahmbhatt committed Dec 19, 2014
    Configuration menu
    Copy the full SHA
    dd991e5 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    07b69b7 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    e05d666 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    9cd52c8 View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    c82482d View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    2954eae View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    a92a1e9 View commit details
    Browse the repository at this point in the history
  11. Configuration menu
    Copy the full SHA
    a75c72c View commit details
    Browse the repository at this point in the history
  12. STORM-166: modifed some config key names to match the convention, add…

    …ed the documentation for configuration.
    Parth-Brahmbhatt committed Dec 19, 2014
    Configuration menu
    Copy the full SHA
    27d6b4c View commit details
    Browse the repository at this point in the history
  13. STORM-166: deleting the bittorrent code distributor as the ttorent li…

    …brary does not support Distrubted hash table for trackerless torrents.
    Parth-Brahmbhatt committed Dec 19, 2014
    Configuration menu
    Copy the full SHA
    dc24e44 View commit details
    Browse the repository at this point in the history
  14. Configuration menu
    Copy the full SHA
    eb242e0 View commit details
    Browse the repository at this point in the history

Commits on Dec 20, 2014

  1. Configuration menu
    Copy the full SHA
    a17167e View commit details
    Browse the repository at this point in the history

Commits on Jan 8, 2015

  1. Configuration menu
    Copy the full SHA
    7681686 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    5658bbb View commit details
    Browse the repository at this point in the history
  3. Merge remote-tracking branch 'upstream/master' into STORM-166

    Conflicts:
    	storm-core/src/jvm/backtype/storm/Config.java
    Parth-Brahmbhatt committed Jan 8, 2015
    Configuration menu
    Copy the full SHA
    58667be View commit details
    Browse the repository at this point in the history

Commits on Jan 9, 2015

  1. Configuration menu
    Copy the full SHA
    c1e8782 View commit details
    Browse the repository at this point in the history

Commits on Jan 23, 2015

  1. Configuration menu
    Copy the full SHA
    27a381a View commit details
    Browse the repository at this point in the history

Commits on Feb 3, 2015

  1. Configuration menu
    Copy the full SHA
    01eee7a View commit details
    Browse the repository at this point in the history

Commits on Feb 4, 2015

  1. Configuration menu
    Copy the full SHA
    1b6491f View commit details
    Browse the repository at this point in the history

Commits on Feb 12, 2015

  1. Configuration menu
    Copy the full SHA
    4502bff View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    aa24375 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    7cae523 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    0b454e8 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    27ad1fb View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    726ce80 View commit details
    Browse the repository at this point in the history

Commits on Feb 13, 2015

  1. Configuration menu
    Copy the full SHA
    63f29f3 View commit details
    Browse the repository at this point in the history
  2. Merge remote-tracking branch 'upstream/master' into STORM-166

    Conflicts:
    	storm-core/src/clj/backtype/storm/daemon/nimbus.clj
    	storm-core/src/clj/backtype/storm/ui/core.clj
    Parth-Brahmbhatt committed Feb 13, 2015
    Configuration menu
    Copy the full SHA
    23ed537 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    a8aacca View commit details
    Browse the repository at this point in the history

Commits on Feb 17, 2015

  1. Configuration menu
    Copy the full SHA
    8d4e561 View commit details
    Browse the repository at this point in the history

Commits on Mar 2, 2015

  1. Merge remote-tracking branch 'upstream/master' into STORM-166

    Conflicts:
    	STORM-UI-REST-API.md
    	pom.xml
    	storm-core/src/clj/backtype/storm/daemon/nimbus.clj
    	storm-core/src/clj/backtype/storm/daemon/supervisor.clj
    	storm-core/src/clj/backtype/storm/ui/core.clj
    	storm-core/src/jvm/backtype/storm/generated/ClusterSummary.java
    	storm-core/src/jvm/backtype/storm/generated/TopologyInfo.java
    	storm-core/src/jvm/backtype/storm/generated/TopologySummary.java
    	storm-core/src/py/storm/ttypes.py
    	storm-core/test/clj/backtype/storm/nimbus_test.clj
    Parth-Brahmbhatt committed Mar 2, 2015
    Configuration menu
    Copy the full SHA
    85e4d5e View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    f7205d2 View commit details
    Browse the repository at this point in the history

Commits on Mar 19, 2015

  1. Merge remote-tracking branch 'upstream/master' into STORM-166

    Conflicts:
    	conf/defaults.yaml
    	storm-core/src/clj/backtype/storm/cluster.clj
    	storm-core/src/clj/backtype/storm/daemon/nimbus.clj
    	storm-core/src/clj/backtype/storm/ui/core.clj
    	storm-core/test/clj/backtype/storm/cluster_test.clj
    	storm-core/test/clj/backtype/storm/security/auth/auth_test.clj
    Parth-Brahmbhatt committed Mar 19, 2015
    Configuration menu
    Copy the full SHA
    5758718 View commit details
    Browse the repository at this point in the history

Commits on Mar 20, 2015

  1. Configuration menu
    Copy the full SHA
    eb3a837 View commit details
    Browse the repository at this point in the history
  2. Merge remote-tracking branch 'upstream/master' into STORM-166

    Conflicts:
    	storm-core/src/ui/public/index.html
    	storm-core/src/ui/public/templates/index-page-template.html
    	storm-core/src/ui/public/templates/topology-page-template.html
    Parth-Brahmbhatt committed Mar 20, 2015
    Configuration menu
    Copy the full SHA
    a11fcc3 View commit details
    Browse the repository at this point in the history

Commits on Apr 1, 2015

  1. Merge remote-tracking branch 'apache/master' into nimbus-ha

    Conflicts:
    	storm-core/src/jvm/backtype/storm/utils/NimbusClient.java
    	storm-core/test/clj/backtype/storm/security/auth/nimbus_auth_test.clj
    Parth-Brahmbhatt committed Apr 1, 2015
    Configuration menu
    Copy the full SHA
    765e4c2 View commit details
    Browse the repository at this point in the history

Commits on Aug 12, 2015

  1. Merge remote-tracking branch 'apache/nimbus-ha-branch' into ha-merge

    Conflicts:
    	STORM-UI-REST-API.md
    	conf/defaults.yaml
    	storm-core/src/clj/backtype/storm/daemon/nimbus.clj
    	storm-core/src/clj/backtype/storm/ui/core.clj
    	storm-core/src/jvm/backtype/storm/Config.java
    	storm-core/src/jvm/backtype/storm/generated/TopologySummary.java
    Parth-Brahmbhatt committed Aug 12, 2015
    Configuration menu
    Copy the full SHA
    d1afefd View commit details
    Browse the repository at this point in the history
  2. STORM-726: Adding nimbus.host config for backward compatibility.

    Conflicts:
    	storm-core/src/jvm/backtype/storm/utils/NimbusClient.java
    Parth-Brahmbhatt committed Aug 12, 2015
    Configuration menu
    Copy the full SHA
    3e20823 View commit details
    Browse the repository at this point in the history
  3. Changed the code-distributor entries to ephemeral nodes. Added code t…

    …o ensure nimbus sets up the correct code-distributor entries on startup.
    Parth-Brahmbhatt committed Aug 12, 2015
    Configuration menu
    Copy the full SHA
    3f66ffd View commit details
    Browse the repository at this point in the history
  4. modifed ui to reflect dead nimbus hosts based on nimbus.seeds.

    Conflicts:
    	storm-core/src/jvm/backtype/storm/utils/NimbusClient.java
    Parth-Brahmbhatt committed Aug 12, 2015
    Configuration menu
    Copy the full SHA
    16293e4 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    95fb680 View commit details
    Browse the repository at this point in the history
  6. STORM-726: Adding nimbus.host config for backward compatibility.

    Conflicts:
    	storm-core/src/clj/backtype/storm/ui/core.clj
    	storm-core/src/jvm/backtype/storm/Config.java
    	storm-core/src/jvm/backtype/storm/utils/NimbusClient.java
    
    Conflicts:
    	storm-core/src/clj/backtype/storm/ui/core.clj
    	storm-core/src/jvm/backtype/storm/Config.java
    	storm-core/src/jvm/backtype/storm/utils/NimbusClient.java
    Parth-Brahmbhatt committed Aug 12, 2015
    Configuration menu
    Copy the full SHA
    0395fd6 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    21ba9c1 View commit details
    Browse the repository at this point in the history
  8. BUG-40864: Handing zookeeper failures that can result in nimbus ephem…

    …eral entries getting deleted. Adding a sleep before cody-sycn thread executes ls /code-distributor/topology-id to ensure it gets the correct id back so users dont have to wait for upto 5 minutes to submit topology.
    Parth-Brahmbhatt committed Aug 12, 2015
    Configuration menu
    Copy the full SHA
    ef3cee6 View commit details
    Browse the repository at this point in the history
  9. Addressing TODOs. Calling sync before calling getChildren on code-dis…

    …tributor path as zookeeper does not gurantee Simultaneously Consistent Cross-Client Views unless sync is called.
    Parth-Brahmbhatt committed Aug 12, 2015
    Configuration menu
    Copy the full SHA
    93dbcaf View commit details
    Browse the repository at this point in the history

Commits on Aug 19, 2015

  1. Configuration menu
    Copy the full SHA
    51fdc1a View commit details
    Browse the repository at this point in the history