0.7.1 changelist

Release 0.7.1

In this version, it will follow storm 0.7.1 interface, so the topology running in storm 0.7.1 can run in jstorm without any change.

Stability

• Assign workers in balance
• add setting "zmq.max.queue.msg" for zeromq
• communication between worker and tasks without zeromq
• Add catch exception operation
  • in supervisor SyncProcess/SyncSupervisor
  • add catch exception and report_error in spout's open and bolt's prepare
  • in all IO operation
  • in all serialize/deserialize
  • in all ZK operation
  • in topology upload/download function
  • during initialization zeromq
• do assignmen/reassignment operation in one thread to avoid competition
• redesign nimbus 's topology assign algorithm, make the logic simple much.
• redesign supervisor's sync assignment algorithm, make the logic simple much
• reduce zookeeper load
  • redesign nimbus monitor logic, it will just scan tasks' hearbeat, frequency is 10s
  • nimbus cancel watch on supervisor
  • supervisor heartbeat frequence change to 10s
  • supervisor syncSupervisor/syncProcess frequence change to 10s
  • supervisor scan /$(ZKROOT)/assignment only once in one monitor loop
  • task hearbeat change to 10s
• create task pid file before connection zk, this is very import when zk is unstable.

Performance tuning

• reduce once memory copy when deserialize tuple, improve performance huge.
• split executor thread as two thread, one handing receive tuples, one sending tuples, improve performance much
• redeisign sample code, it will sampling every 5 seconds, not every 20 tuple once, improve performance much
• simplify the ack's logic, make acker more effeciency
• Communication between worker and tasks won't use zeromq, just memory share in process
• in worker's Drainer/virtualportdispatch thread, spout/bolt recv/send thread, the thread will sleep 1 ms when there is not tuple in one loop
• communication between worker and tasks without zeromq
• sampling frequence change to 5s, not every 20 tuple once.

Enhancement:

• add IFailValueSpout interface
• Redesign sampling code, collection statics model become more common.
  • Add sending/recving tps statics, statics is more precise.
• Atomatically do deactivate action when kill/rebalance topology, and the wait time is 2 * MSG_TIMEOUT
• fix nongrouping bug, random.nextInt will generate value less than 0.
• Sleep one setting time(default is 1 minute) after finish spout open, which is used to wait other task finish initialization.
• Add check component name when submit topology, forbidding the component which name start with "__"
• change the zk's node /$(ZKROOT)/storm to /$(ZKROOT)/topology
• abstract topology check logic from generating real topology function
• when supervisor is down and topology do rebalance, the alive task under down supervisor is unavailable.
• add close connection operation after finish download topology binary
• automatically create all local dirtorie, such as /$(LOCALDIR)/supervisor/localstate
• when killing worker, add "kill and sleep " operation before "kill -9" operation
• when generate real topology binary,
  • configuration priority different.
    component configuration > topology configuration > system configuration
  • skip the output stream which target component doesn't exist.
  • skip the component whose parallism is 0.
  • component's parallism is less than 0, throw exception.
• skip ack/fail when inputstream setting is empty
• add topology name to the log
• fix ui select option error, default is 10 minutes
• supervisor can display all worker's status