Release 0.17.0

Highlights

• Bug Fixes/Code Cleanups
• Streamlet enhancements/examples

Release Notes

• Introduce Kryo Serializer to Streamlet (#2540)
• improve passing of executor ports (#2516)
• Update README.md with service url config (#2544)
• Update README to use service url config (#2543)
• update gitignore (#2539)
• Use builder pattern in Streamlet API (#2538)
• Change heron api dependency (#2534)
• Add a config translation function for components and fix a typo. (#2536)
• Used for example (#2537)
• Since Heron does not use kryo for anything, there is no need to shade it out (#2535)
• Explicity setting heron apiserver to run as cluster-admin (#2531)
• Allow users to register timer tasks in topologies (#2522)
• Generalize the interface of a Source by allowing it to emit a collection at once (#2526)
• Fix log.error exception in check_direct_mode_cluster_definition() (#2532)
• bump docs/website version (#2529)
• Apply config translation for Bolt and Spout (#2527)
• Fix the command line for heron executor for aurora. (#2525)
• Java streamlet examples (#2501)

Release 0.16.5

Highlights

• Stateful Windowing
• Simplified yet more powerful Streamlet API
• fix packaging bugs for kubernetes/Yarn/aurora schedulers

Release Notes

• Simplified Streamlet interface (#2519)
• fix broken docker build script (#2521)
• Remove unneeded dep when submitting a tar (#2520)
• Use heron home cluster defaults when running in a cluster. (#2517)
• Added in static files for HeronUI so that Offline Heron Deployment works (#2513)
• fixes bookeeper statefulset's deployment url (#2514)
• increasing memory for examples (#2512)
• fixing formating in examples (#2511)
• Stateful windowing (#2398)
• Added a generalized reduceByKeyAndWindow operator inside the Streamlet (#2502)
• • Streamlet typing changes (removal of superfluous super-typing) (#2505)
• adding metricscache-sink entry to configs (#2506)
• Issue 2500: Provide a yaml file to deploy bookkeeper in a StatefulSet for k8s installation (#2503)
• Currently Storm and Heron have different encoding styles for Integer in api.Confg. (#2504)
• add processingdataskew detector (#2498)
• include heron.aurora file in the distribution (#2496)
• bump version (#2497)
• fix map inconsistency (#2494)

Release 0.16.4

Highlights

• Expanded Streamlet API
• Bug Fixes

Release Notes

• Fix the correct python package for building heronpy (#2493)
• Add gflags to cpp instance to get rid of positional arguments (#2491)
• Make stmgr and tmaster use gflags. This removes the ugly positional arguments (#2490)
• Make Streamlets and KVStreamets independent of each other (#2487)
• Add flags to the heron executor. (#2463)
• Python Instance now takes key/value based arguments (#2489)
• validate topology before submitting (#2483)
• remove unnecessary import statement (#2488)
• Add dual license to api files (#2435)
• Updated the workspace after we upgraded to newer requests (#2486)
• Acking topologies for Java documentation (#2485)
• Kill both containers when zombie host happens (#2480)
• Removed dependency on psutil since its not manylinux compliant (#2479)
• Update k8s/GKE docs for multiple storage systems (#2478)
• Documentation for the Prometheus metrics sink (#2457)
• Update python requests version (#2456)
• changing commandline arguments to be passed in a key value fashion (#2471)
• fix bugs and simplifying join API (#2476)
• Make cli python3 compliant (#2475)
• Add gcp bookkeeper deploy file. (#2477)
• Heron on Kubernetes docs (#2444)
• fixing bugs in python streamlet join (#2474)
• Change logging interface to accept ostringstream to be outputted to log (#2469)
• Clear bookkeeper data, when redeploying minikube (#2466)
• Guard empty execute state (#2453)
• bump version (#2470)
• Added a CPP example topology (#2465)
• Make container ram padding configurable for round robin packing. (#2464)
• rename link in TOC (#2467)
• Documentation for the Scribe metrics sink (#2459)
• Added cli/scheduler support for submitting/launching cpp topologies (#2462)
• add prometheus-sinks configs to the remaining base templates (#2458)
• Added tests for cpp api (#2461)
• Enforce Memory limit for Python Instance (#2460)
• Heron C++ Instance (#2449)
• Rename "DSL" to "Streamlet API" (#2424)
• Heron API server documentation (#2405)
• add heron config section (#2417)

Release 0.16.3

Highlights

• Additional Storm API compatibility
• Several bug fixes

Release Notes

• Change EMPTY_STRING defn to be a plain string so that we are ok getting its references (#2446)
• Revert "Guard empty execution state. (#2447)" (#2448)
• Guard empty execution state. (#2447)
• Increase state manager reachability timeout. (#2395)
• Theoretical documentation for windowing (#2414)
• Updated architecture and design goals documentation (WIP) (#2387)
• Added Heron Instance related vars in cpp. This will be used by CPP Instance (#2442)
• Add helper methods in Config Helper Functions that will be used by CPP Instance (#2445)
• Do minor Metrics Cleanup (#2441)
• Log an error message when we dont recognize message type (#2440)
• C++ api bindings for Heron (#2429)
• bump version (#2439)
• Updated README.md (#2437)
• fixing join operator (#2436)
• adding BaseTickTupleAwareRichBolt for Storm compatibility (#2434)
• adding StormTopology compatibility (#2433)
• fix travis to use new targets and also some checkstyle (#2432)
• bump version (#2430)
• Add kubernetes deploy scripts for minikube. (#2426)
• Add apiserver.yaml and move bookkeeper.yaml for Kubernetes baremetal and VM deployments (#2418)
• fix heron submit examples (#2423)
• General Cleanup of DSL (#2420)
• Move examples to updated config system (#2425)
• allow users to set topology environment properties (#2416)
• update auroraheronshellcontroller (#2419)
• update restartcontainerresolver (#2415)
• fix spelling mistakes (#2413)
• Refactory kubernetes deploy yamls. (#2412)
• Add bookkeeper kubernetes yaml script. (#2390)
• update healthmgr (#2406)
• Add basic zookeeper yaml for kubernetes. (#2411)
• update aurora heron shell controller (#2410)
• Update Heron Topology API docs for Java (#2399)
• fixing SerializableGenerator (#2408)
• fixing flapMap signature (#2407)
• update metricscachemgr (#2401)
• Rename toSink to consume since thats more representative of what it ds (#2404)
• s/setDeliverySemantics/set_delivery_semantics (#2402)
• Fix code block and collapse behavior (website) (#2403)
• Heron Explorer documentation (#2227)
• Effectively-once semantics for the Heron API (#2367)
• Enhance Python DSL to bring it par with Java (#2391)
• Add fallback for proto type mismatch (#2397)
• Enable python checkstyle for heronpy files (#2393)
• Issue 2286: Implement bookkeeper based stateful storage (#2287)
• Log exception when scheduler fails to deploy config. (#2378)
• Modify local scheduler to work without a core uri. (#2376)
• Colorful dry-run output when we output to TTY device. (#2385)
• For topologies submitted with older jars, topologuy reliability mode (#2383)
• introduce sandbox config (#2373)
• remove remark about heron in beta (#2381)
• Be explicit about subprocess pipe's EOF. (#2380)
• Fix bazel 6.0 issues (#2379)
• Move zookeeper yaml to gcp. (#2377)
• Adding new metrics from Apache Storm (#2375)
• Make bazel updates for 6.0 (#2372)
• fix typo (#2374)
• fixing docker build script for heron (#2350)

Release 0.16.2

Release Notes

• changing watermark generator to be triggered by tick tuples for event time based windowing (#2370)
• Fix operator to support python 2 and 3. (#2369)
• Add component type label for prometheus metrics. (#2364)
• Create connection buffer and metrics map every time the connection comes up (#2366)
• Add helper method to specify ram (#2365)
• Seperate out the low level api examples and higher level dsl examples (#2361)
• Changing TimeTriggerPolicy to use tick tuples as a timer (#2357)
• remove note on production usage (#2363)
• Make it clear that the Functional API is in beta (#2362)
• re-address submodule issue (#2360)
• Heron DSL documentation (#2352)
• Fix website submodule issue (#2359)
• Small fixes for new Java DSL docs (#2358)
• Heron Java DSL docs (WIP) (#2342)

Release 0.16.1

Highlights

• Java functional API
• Heron windowing functionality
• Several bug fixes

Release Notes

• Fix src collection for java api (#2356)
• More Python3 Friendliness (#2355)
• Fix intellij setup script (#2354)
• Move java dsl to be part of heron api (#2353)
• Added a generic source that has open/cleanup. Also renamed a few functions (#2351)
• Fix mktemp issue and revert ci-docker (#2344)
• fix simulator for not initializing instance (#2347)
• Fix website asset pipeline (#2348)
• Delivery semantics documentation (WIP) (#2324)
• heronpy now needs protobuf 3.4.0 (#2345)
• Checkpoint manager does not support override (#2343)
• Fix missing artifact for docker build (#2341)
• Add heron cli config setting for service_url. (#2326)
• clean heron java instance for stateful processing (#2336)
• After building make sure that all the components have been built (#2340)
• Consider override yaml (#2339)
• Fix default acking (#2338)
• Make delivery semantics public (#2337)
• Add prometheus scrape annotations to pods in kubernetes scheduler. (#2331)
• Added a Config class to remove api.Config depedency (#2334)
• Python3 compatibility (#2335)
• pex_pytest migration (#2333)
• Revert "Take override.yaml in stmgr, metricsmgr, metricsmgr-cache, tmaster, instance (#2311)" (#2332)
• Take override.yaml in stmgr, metricsmgr, metricsmgr-cache, tmaster, instance (#2311)
• clean python state manager (#2327)
• Add prometheus sink to default kubernetes metrics config. (#2329)
• Java DSL for Heron (#2285)
• upgrade to protobuf3.4 (#2316)
• take into account feedback from Neng (#2322)
• clean tmater client (#2321)
• Issue 2219: Provide a bookkeeper based uploader and downloader (#2220)
• fix python state manager (#2319)
• Added a Stateful example topology. (#2318)
• Add prometheus metrics web sink. (#2243)
• Seperate Stmgr and Instance Serving in Stmgr (#2313)
• Upgrade pex (#2314)
• Add image pull policy param for kubernetes scheduler. (#2315)
• Always show links button (logs, job, exceptions, etc..) for instances. (#2312)
• Use bufferevent to do Network I/O (#2297)
• Fixed the path of the python examples (#2305)
• Bug fixes and integration test for Python DSL (#2304)
• fix bugs in stmgr unit test (#2306)
• Split executor args in aurora file so easy to read diff. (#2303)
• Refactoring storm api (#2300)
• fix tmaster client sends incomplete register request to tmaster (#2291)
• Add verbose logging to apiserver. (#2288)
• Add error logging for topology actions on the apiserver. (#2296)
• Fix Python documentation generation issues (#2302)
• remove startup scripts (#2298)
• Register requests are sent by instance as request/response pairs (#2294)
• Heron Storm Windowing Compatibility Layer (#2292)
• Correcting integration tests structure (#2293)
• Use Piper class instead of repeating functionality (#2289)
• adding additional unit and integration tests for heron windowing support (#2284)
• Reduce example topologies resources. (#2276)
• healthmgr: aurora heron shell controller (#2283)
• Add config mapping for gcs bucket in api server deploy. (#2281)
• cleanup docker scripts (#2277)
• Add kubernetes service for apiserver. (#2280)
• Notify both producer and consumer when init the Communicator (#2279)
• Add error log for physical plan check in tmaster (#2267)

Release 0.15.2

Highlights:

• Upgraded bazel to the latest version 0.5.4
• Upgraded zookeeper client version to 4.5.10
• Fix shading bugs in heron and storm jars

Release Notes

• Add kubernetes deploy files. (#2265)
• consistent behavior about when to shade. Shading is done when the final artifacts are generated (#2273)
• Normalize Shading and fix simulator build (#2271)
• Calculate the hostname during constructor rather than during register (#2270)
• Make shading uniform accross instance and api (#2266)
• Add topology name validation for kubernetes scheduler. (#2264)
• [ISSUE-1124] - Windows Bolt support (#2241)
• Read http response for all http response codes. (#2263)
• ensure empty function with unique names (#2262)
• Add zookeeper to the heron dist docker ubuntu14.04 (#2259)
• changes to make heron compile with bazel 0.5.4 (#2207)
• [ISSUE-2170] fix tracker crash when zk root topologies path does not … (#2260)
• fix the executor unit test (#2261)
• healthmgr check null return value (#2258)
• Support components name generated by SummingBird. (#2256)
• Renamed Exactly Once to Effectively Once (#2254)
• add restartcontainer resolver to healthmgr (#2230)
• Add src id in exception message. (#2252)
• Move system_config to from common to instance since only instance uses it (#2251)
• Move EventLooper and GatewayLooper to instance since no one else uses them (#2250)
• Moved system_constants from common to instance since only instance uses it (#2249)
• Deleted unused file (#2248)
• Include only Topology proto in heron java api jar (#2247)
• Remove unused proto (#2246)
• Upgraded zookeeper to the latest stable version (#2245)

Release 0.15.1

Highlights:

• Several bug fixes
• Heron Python packaging
• Heron downloader
• Upgrade libevent, gflags, and glog to latest versions

Release Notes:

• Make metrics cache client in healthMgr fetch location from statemgr (#2232)
• Fix heronpy packaging (#2244)
• Remove unnecessary sub module BUILD files (#2240)
• Removing legacy kakfa connector (#2242)
• Convert kubernetes scheduler to use heron downloader. (#2223)
• Make bazel configure python3 friendly (#2228)
• Limit size of a single data tuple. (#2224)
• Fix yaml health manager config parsing error (#2226)
• 0.15.0 documentation release (#2221)
• Fix Join Bolt implementation (#2215)
• Ability to launch health manager on container-0 (#2203)
• Add a heron downloader to download and extract topology packages. (#2216)
• upgrade gflags and glog to newer versions (#2211)
• remove inputs from streamlet since the core streamlet class does not use it (#2214)
• Removed operation since it is unused (#2213)
• All of dsl methods need to output the same stream. This pr (#2212)
• Make the exception style compatible with both python2 and python3 (#2210)
• Fixed libtool install command in ubuntu 16.04 (#2208)
• Upgrade libevent to a more recent stable version (#2209)
• Replace all localhost by 127.0.0.1 (#2204)
• Documentation changes for 0.15.0 release (#2201)
• replace hard coded stmgr port in stmgr_unittest (#1852)
• Fix requests version, fix default env. (#2202)
• cleanup the dependencies for generating heron storm jar (#2191)
• Fix the stateful component overloaded functions in window bolt (#2199)
• Remove unused third party packages (#2198)
• Expose all API related classes in Python Packaging (#2197)
• Update cpplint to the latest release. Minor fixes in the code along the way (#2196)
• Add health manager operation mode (#2181)
• Fix log level and parameters (#2195)
• Fix python examples now that spouts have moved to connectors (#2193)
• Upgrade to gtest 1.8 (#2192)
• Follow pythonic idiom of _ instead of camelCase in DSL (#2190)
• Follow Python idiom of using _ instead of camelCase (#2189)
• HeronTupleSet needs src task id (#2187)
• Certain information should be printed only in debug mode (#2186)
• Fix import name in python examples (#2185)
• Create a connectors directory for spouts and bolts (sources/sinks) (#2176)
• Verify cluster name matches when submitting through apiserver. (#2184)
• Add topology resource unit tests. (#2179)
• Remove spi dependency from simulator since TopologyUtils moved from spi (#2182)

Release 0.15.0

Highlights:

• Complete support for running Heron in Kubernetes
• Heron API server for easy deployment and topology management
• Support for Storm like python API with PEX support
• Python DSL with PEX support
• Exactly once and stateful processing in Java and Python
• Initial version of Dhalion for policy based self regulation
• Pulsar Spout
• Google cloud storage uploader
• Several bug fixes

Release Notes:

• Add base framework for api server. (#2094)
• Heron Python Code Cleanup (#2173)
• Update pulsar to apache client. (#2171)
• Added docstrings to python dsl (#2166)
• Add kubernetes stateful config. (#2164)
• added update certificates for java (#2163)
• Fix NPE when logging exceptions that have no message using SLF4J (#2161)
• Dhalion: Add implementation for resolvers and health policy (#2162)
• Added Pulsar Spout and Pulsar Streamlet (#2160)
• Added exactly once semantics for python dsl (#2159)
• Don't print topology context since it just prints a class pointer info (#2157)
• Misc Python fixes (#2158)
• Sort the task ids to ensure that identical fields grouping tuples (#2155)
• Make sure that metric registration and adding tasks are done once. (#2152)
• GetInstances will return only active instances. Instead we want to (#2148)
• Refactored common code by moving it to base instance (#2151)
• Correct the way we refer to the class member (#2150)
• Add MetricsCacheMetricsProvider (#2113)
• Add Dhalion detectors and diagnoser implementation for heron (#2132)
• Use a default value for checkpoint interval if user does not specify it (#2147)
• Fix the paths for local stateful storage (#2145)
• For debugging purporses count all words as 1 (#2144)
• Corrected stateful checkpoint location (#2143)
• Add a string method to print out the state (#2142)
• Use appropriate method while manipulating state (#2141)
• Spout should read input tuples even in non acking mode (#2140)
• Initialize instance state if it's null (#2138)
• Register for stateful messages and ensure that state is not empty before parsing it (#2139)
• Initialize the instance state before open/prepare instance (#2134)
• Add java doc gen for heron api, spi and storm (#2123)
• Added exactly once semantics for python (#2135)
• Update sendOutState (#2131)
• While unzipping pex files, don't overwrite existing files. (#2129)
• Use Java ver 8 as source and target version (#2130)
• Add Dhalion-sensor implementation for heron (#2101)
• Limit stmgr client reconnect attempts (#2125)
• Make instance send out state (#2120)
• Register stateful processing messages with StreamManagerClient (#2119)
• Added a new socket option to take the maximum packet size accepted (#2118)
• Add stateful support into Slave (#2106)
• Introduce StatefulController in TMaster and handle all stateful messages (#2105)
• Added ability in tmaster to accept stateful processing messages (#2102)
• For stateful topologies, TMaster should connect with checkpoint manager (#2100)
• Dhalion health manager (#1980)
• Add stateful controller into tmaster dir (#1973)
• Add clean stateful checkpoint handler into tcontroller (#2096)
• Introduce stateful restorer in stmgr. (#2091)
• New API for topology reliability modes (#2089)
• Correct heap memory usage of stream manager (#2086)
• Limit number of messages in mempool, instead of total byte sizes (#2073)
• Add Pulsar spout. (#2069)
• Instantiate ClientMgr in constructor (#2078)
• Instantiate checkpoint manager client in stmgr (#2076)
• Added Stateful processing related message handling in TMasterClient (#2075)
• Fix kafka spout HeapByteBuffer not serializable problem (#2063)
• Added stateful processing restore state machine to stmgr. (#2070)
• packaging python with all the modules (#2072)
• Fix the comparison to be string rather than numerical (#2071)
• Change ci-docker.sh to use sh instead of bash (#2068)
• Added stateful processing message handlers in stmgr-server (#2067)
• Remove API dep on c.t.h.common.basics.TypeUtils (#2061)
• Add hooks in stmgr client/clientmgr to send checkpoint markers. (#2060)
• Added a script to build/publish docker images (#2065)
• Mempool acquire should always return a Clean protobuf (#2057)
• Adding debugging to integration test spout, increasing timeouts for get condition (#2055)
• Added Checkpoint Gateway Layer while sending data to instances (#2047)
• Increase timeout for integration test result checking (#2053)
• Add countdown latch to runnable in testRegisterTimerEventInNanoSeconds (#2050)
• Added Spout for Apache Pulsar (#2048)
• Created heron/spouts to house all well-known spouts. (#2046)
• Added applatix scripts and tools (#2045)
• Package python dsl package (#2043)
• Python DSL (#2033)
• Extract HeronTuple from current_tuples while acking it (#2032)
• Applatix workflow with build followed by test (#2015)
• Made cloudpickle the default serializer (#2030)
• Window tuples should take care of handling acks (#2028)
• Changed the custom grouping to take in an actual instance of (#2027)
• Added WindowBolt interface in python (#2021)
• Added Checkpoint Gateway (#2024)
• Add security check for shell-kill-executor (#2013)
• flatten the directory structure for integration tests (#2018)
• Updated Python documentation (#2017)
• Added Neighbour Calculator, a utility class to calculate (#2016)
• Expose TopologyContext in heronpy (#2014)
• Moved Herons pypi to heronpy. Updated requirements as well (#2012)
• Remove pyheron since now python is first class api in Heron (#2006)
• Rename tools/travis-ci to tools/travis and update the references to it (#2007)
• Restart subprocess with shell_env. (#2004)
• Discover the platform in travis scripts (#2001)
• change the default gcc 4.8 to just gcc (#1999)
• remove the space in applatix array parameters (#1998)
• Made heron instance and examples depend on api and not on pyheron (#1992)
• Applatix CI integration (#1986)
• Add support for src task id in the storm api (#1938)
• Kubernetes Manual Scaling (#1970)
• Use TopologyContext as the base for TopologyContextImpl (#1989)
• Refactored constants (#1988)
• Add GCSUploader for uploading topologies to Google Cloud Storage. (#1976)
• Heron client unit test (#1983)
• Rephrase references to Storm (#1979)
• Native Python API for Heron. (#1982)
• Add checking server starting return (#1984)
• Change TMaster to watch the packing plan for changes (#1955)
• Added get topology name to context (#1978)
• Pex files need to be unzipped so that depdencies can be set/found (#1977)
• Import metrics from pyheron module (#1975)
• Added the python requirements to be >= (#1974)
• Allow the tracker to take command line args for a state manager. (#1967)
• Fix inaccurate execute-latency metric (#1969)
• Changed the python requirements for pyheron so that it doesn't force to downgrade (#1966)
• Topic Real Time Streaming does not exist as list of allowed Topic Names (#1965)
• Streamline preparation of docker images for distribution (#1961)
• Remove extra initialize method from ContainerFileDownloadHandler. (#1962)
• Allow any type of file to be downloaded from a container through the shell. (#1959)
• Increase delta in Python loopers test (#1958)
• Add support for watching packing plan to c++ state managers (#1953)
• Scripts to build docker artifacts (#1948)
• Add note on hostname in /etc/hosts (#1945)

Version 0.14.9

Highlights:

• Use of docker images for heron containers in DC/OS scheduler
• Docker images published as tar balls for private registries
• Initial support for running Heron in Kubernetes
• Docker images published in dockerhub for centos7, ubuntu14.04

Release Notes:

• Add a readme file for pyheron (#1944)
• Update documentation regarding topology_bin in the SubmitterMain (#1939)
• Kubernetes Scheduler Support (#1928)
• UpdateTopologyManager using stale topology (#1937)
• Moved the docker builds and docker images to use open-jdk-8 (#1941)
• Add simple script to compile heron and package it in a docker image. (#1934)
• Remove heron core fetching for MarathonScheduler. (#1894)
• When the python pex file is copied over to the container, only (#1926)
• Python packaging (#1425)
• UpdateTopologyMaster using stale Topology from initial submission (#1924)
• Adding sleep to TestSpout to throttle down cpu usage (#1925)
• Fixing logging bug in UpdateTopologyManager (#1931)
• Fix heron-shell dependency (#1930)
• Decode serialized value in heron-tracker. (#1927)
• Asynchronously handle subprocess stdout/stderr in client. (#1923)
• Removed per client/server specific acquire/release and consolidated them in one place (#1914)
• Add back missing arguments. (#1921)
• Remove unused heron.instance.slave.fetch.pplan.interval.sec (#1920)
• Clean up exception handling in common tests (#1919)
• Remove unused heron.instance.state.check.interval.sec config (#1916)