Skip to content
Asakusa Framework
Java Other
Latest commit 1aa7a1b May 31, 2016 @akirakw akirakw Merge pull request #625 from ashigeru/wip/hive-schema-gen
Add Hive schema info generator for on-MapReduce compiler.
Failed to load latest commit information.
.github Add GitHub templates Apr 15, 2016
.mvn/wrapper Add Maven Wrapper May 1, 2016
build-tools Add licene-maven pom definition to build-tools May 6, 2016
core-project Tracing facilities for MapReduce should call Report API directly. Apr 14, 2016
directio-project Add Hive schema info generator for on-MapReduce compiler. May 31, 2016
dmdl-project Bump up checkstyle version. Apr 8, 2016
docs/ja Fix task name Apr 28, 2016
dsl-project Bump up checkstyle version. Apr 8, 2016
hive-project Add Hive schema info generator for on-MapReduce compiler. May 31, 2016
iterative-project [maven-release-plugin] prepare for next development iteration Apr 8, 2016
operation-project Bump up checkstyle version. Apr 8, 2016
sandbox-project Bump up checkstyle version. Apr 8, 2016
sdk-project [maven-release-plugin] prepare for next development iteration Apr 8, 2016
testing-project Tracing facilities for MapReduce should call Report API directly. Apr 14, 2016
utils-project Bump up checkstyle version. Apr 8, 2016
windgate-project Bump up checkstyle version. Apr 8, 2016
yaess-project Bump up checkstyle version. Apr 8, 2016
.gitattributes fixed #8. some asakusa-runtime tests fail because of the Windows NewL… Apr 28, 2011
.gitignore Add IntelliJ-generated files to .gitignore Apr 28, 2016
LICENSE Modified license year. May 8, 2013
NOTICE License year updated to 2016 Jan 5, 2016
README.md Revise README for using Maven Wrapper May 10, 2016
mvnw Add Maven Wrapper May 1, 2016
mvnw.cmd Add Maven Wrapper May 1, 2016
pom.xml Add `asakusa-hive-info` artifact. May 26, 2016

README.md

Asakusa Framework

Asakusa is a full stack framework for distributed/parallel computing, which provides with a development platform and runtime libraries supporting various distributed/parallel computing environments such as Hadoop, Spark, M3 for Batch Processing, and so on. Users can enjoy the best performance on distributed/parallel computing transparently changing execution engines among MapReduce, SparkRDD, and C++ native based on their data size.

Other than query-based languages, Asakusa helps to develop more complicated data flow programs more easily, efficiently, and comprehensively due to following components.

  • Data-flow oriented DSL

    Data-flow based approach is suitable for DAG constructions which is appropriate for distributed/parallel computing. Asakusa offers Domain Specific Language based on Java with data-flow design, which is integrated with compilers.

  • Compilers

    A multi-tier compiler is supported. Java based source code is once compiled to inter-mediated representation and then optimized for each execution environments such that Hadoop(MapReduce), Spark(RDD), M3 for Batch Processing(C++ Native), respectively.

  • Data-Modeling language

    Data-Model language is supported, which is comprehensive for mapping with relational models, CSVs, or other data formats.

  • Test Environment

    JUnit based unit testing and end-to-end testing are supported, which are portable among each execution environments. Source code, test code, and test data are fully compatible across Hadoop, Spark, M3 for Batch Processing and others.

  • Runtime execution driver

    A transparent job execution driver is supported.

All these features have been well designed and developed with the expertise from experiences on enterprise-scale system developments over decades and promised to contribute to large scale systems on distributed/parallel environments to be more robust and stable.

How to build

./mvnw install -DskipTests

How to run tests

  • Install Hadoop with local-mode settings
  • Set hadoop command into your PATH variable, or set it to $HADOOP_CMD
  • And then run ./mvnw test

How to import projects into Eclipse

  • Run ./mvnw install eclipse:eclipse -DskipTests
  • And then import projects from Eclipse

If you run tests in Eclipse, please activate Preferences > Java > Debug > 'Only include exported classpath entries when launching'.

Sub Projects

Related Projects

Resources

Bug reports, Patch contribution

License

Something went wrong with that request. Please try again.