Skip to content

SpookyStuff-0.3.0 Release Candidate 1

Pre-release
Pre-release
Compare
Choose a tag to compare
@tribbloid tribbloid released this 23 Dec 14:11
· 1543 commits to master since this release
0.3.0-RC1
ef3ac2a

API FREEZE!

SpookyStuff is close to its first release as many of its API will hardly change in the future. Highlights of this release are:

  • better proxy support: restful client actions (Wget) can use both http & socks proxy
  • stripped-to-the-core DSL: the new DSL provide several rich and succinct API that drastically decreases step required to define a workflow, many new functions (e.g. flatSelect, explore etc.) shrink steps by magnitudes. In addition, a functional-oriented expression system is introduced to make query more streamlined and SQL-ish.
  • Ability to read from both RDD web cache and HDFS web cache: previously downloaded web resources can be persisted into distributed storage (fast) & RDD in memory (faster) and reused later.
  • General QA & bugfix: Integration test & more unit test are enabled which leads to much less bugs and failures.
Assets 4