Java Groovy Other
Switch branches/tags
Clone or download
akirakw Merge pull request #813 from ashigeru/wip/fix-asakusa-run-src-libs
`asakusa run` on win cannot load libs in src/main/libs.
Latest commit ae58b9f Jul 17, 2018
Permalink
Failed to load latest commit information.
.github
.mvn/wrapper
build-tools
core-project
directio-project
dmdl-project
docs
dsl-project
gradle
hive-project
info
integration
iterative-project [maven-release-plugin] prepare for next development iteration Jun 27, 2018
operation-project
operator [maven-release-plugin] prepare for next development iteration Jun 27, 2018
sandbox-project
testing-project
utils-project
windgate-project
workflow
yaess-project
.gitattributes
.gitignore
LICENSE
NOTICE
README.md
mvnw
mvnw.cmd
pom.xml

README.md

Asakusa Framework

Asakusa is a full stack framework for distributed/parallel computing, which provides with a development platform and runtime libraries supporting various distributed/parallel computing environments such as Hadoop, Spark, M3 for Batch Processing, and so on. Users can enjoy the best performance on distributed/parallel computing transparently changing execution engines among MapReduce, SparkRDD, and C++ native based on their data size.

Other than query-based languages, Asakusa helps to develop more complicated data flow programs more easily, efficiently, and comprehensively due to following components.

  • Data-flow oriented DSL

    Data-flow based approach is suitable for DAG constructions which is appropriate for distributed/parallel computing. Asakusa offers Domain Specific Language based on Java with data-flow design, which is integrated with compilers.

  • Compilers

    A multi-tier compiler is supported. Java based source code is once compiled to inter-mediated representation and then optimized for each execution environments such that Hadoop(MapReduce), Spark(RDD), M3 for Batch Processing(C++ Native), respectively.

  • Data-Modeling language

    Data-Model language is supported, which is comprehensive for mapping with relational models, CSVs, or other data formats.

  • Test Environment

    JUnit based unit testing and end-to-end testing are supported, which are portable among each execution environments. Source code, test code, and test data are fully compatible across Hadoop, Spark, M3 for Batch Processing and others.

  • Runtime execution driver

    A transparent job execution driver is supported.

All these features have been well designed and developed with the expertise from experiences on enterprise-scale system developments over decades and promised to contribute to large scale systems on distributed/parallel environments to be more robust and stable.

How to build

Maven artifacts

./mvnw clean install -DskipTests

Gradle plug-ins

cd gradle
./gradlew clean [build] install

How to run tests

Maven artifacts

export HADOOP_CMD=/path/to/bin/hadoop
./mvnw test

Gradle plug-ins

cd gradle
./gradlew [clean] check

How to import projects into Eclipse

Maven artifacts

./mvnw eclipse:eclipse

And then import existing projects from Eclipse.

If you run tests in Eclipse, please activate Preferences > Java > Debug > 'Only include exported classpath entries when launching'.

Gradle plug-ins

cd gradle
./gradlew eclipse

And then import existing projects from Eclipse.

Sub Projects

Related Projects

Resources

Bug reports, Patch contribution

License