Asakusa is a full stack framework for distributed/parallel computing, which provides with a development platform and runtime libraries supporting various distributed/parallel computing environments such as Hadoop, Spark, M3 for Batch Processing, and so on. Users can enjoy the best performance on distributed/parallel computing transparently changing execution engines among MapReduce, SparkRDD, and C++ native based on their data size.
Other than query-based languages, Asakusa helps to develop more complicated data flow programs more easily, efficiently, and comprehensively due to following components.
Data-flow oriented DSL
Data-flow based approach is suitable for DAG constructions which is appropriate for distributed/parallel computing. Asakusa offers Domain Specific Language based on Java with data-flow design, which is integrated with compilers.
A multi-tier compiler is supported. Java based source code is once compiled to inter-mediated representation and then optimized for each execution environments such that Hadoop(MapReduce), Spark(RDD), M3 for Batch Processing(C++ Native), respectively.
Data-Model language is supported, which is comprehensive for mapping with relational models, CSVs, or other data formats.
JUnit based unit testing and end-to-end testing are supported, which are portable among each execution environments. Source code, test code, and test data are fully compatible across Hadoop, Spark, M3 for Batch Processing and others.
Runtime execution driver
A transparent job execution driver is supported.
All these features have been well designed and developed with the expertise from experiences on enterprise-scale system developments over decades and promised to contribute to large scale systems on distributed/parallel environments to be more robust and stable.
How to build
./mvnw install -DskipTests
How to run tests
- Install Hadoop with local-mode settings
hadoopcommand into your PATH variable, or set it to
- And then run
How to import projects into Eclipse
./mvnw install eclipse:eclipse -DskipTests
- And then import projects from Eclipse
If you run tests in Eclipse, please activate
Preferences > Java > Debug > 'Only include exported classpath entries when launching'.
- Asakusa Framework SDK
- Asakusa Framework Language Toolset
- Asakusa Framework DAG Toolset
- Asakusa on Spark
- Asakusa on M3BP
Bug reports, Patch contribution
- Please report any issues to repository for issue tracking
- Please contribute with patches according to our contribution guide (Japanese only, English version to be added)