New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP][NO-JIRA] Mahout Cylons Demo Donation #347

Open
wants to merge 4 commits into
base: master
from

Conversation

Projects
None yet
2 participants
@rawkintrevo
Contributor

rawkintrevo commented Oct 18, 2017

Purpose of PR:

The so-called "Cylons demo" was well received at multiple talks in early Fall 2017, and there has been interest in continuing work on the project. The Cylon demo showcases Mahout in the following ways:

  • A more robust use of the "Eigenfaces Demo" (on Apache Spark)
  • How Mahout "precanned" algos can be chopped up and used in Flink Streaming Applications
  • How Mahout facilitates the so-called "Lambda style" machine learning paradigm by training an offline
    model in Apache Spark and utilizing it in Apache Flink with Apache Solr as the "model server" as well as a so called "Kappa" style by continuously training and applying a Canopy Function.

As this was originally a demo- it is somewhat 'dirty' to say the least. Much work is to be done to fully integrate this as a nice clean demo- especially with respect to documentation.

Note that a working drone is NOT required to run this demo as one can tie into any RSTP video feed and there are many public ones available (as well as creating one with many webcams).

Opening this as a WIP PR that others may contribute and help me get it ready for merging into the trunk.

Important ToDos

Please mark each with an "x"

  • A JIRA ticket exists (if not, please create this first)[https://issues.apache.org/jira/browse/ZEPPELIN/]
  • Title of PR is "MAHOUT-XXXX Brief Description of Changes" where XXXX is the JIRA number.
  • Created unit tests where appropriate
  • Added licenses correct on newly added files
  • Assigned JIRA to self
  • Added documentation in scala docs/java docs, and to website
  • Successfully built and ran all unit tests, verified that all tests pass locally.

If all of these things aren't complete, but you still feel it is
appropriate to open a PR, please add [WIP] after MAHOUT-XXXX before the
descriptions- e.g. "MAHOUT-XXXX [WIP] Description of Change"

Does this change break earlier versions?

Is this the beginning of a larger project for which a feature branch should be made?

@andrewpalumbo

This comment has been minimized.

Member

andrewpalumbo commented Oct 19, 2017

Awesome! this is going to be an amazing example of an Apache Flink/Mahout/Kafka/Solr + OpenCV stack. I'm not sure how we want to tackle the PR here. I can follow the README builds at some point soon. Thanks for this donation @rawkintrevo.

@andrewpalumbo

This comment has been minimized.

Member

andrewpalumbo commented Nov 15, 2017

per a quick discussion offline, It makes sense to ship this as an alpha project into /examples/bin/.

@rawkintrevo

This comment has been minimized.

Contributor

rawkintrevo commented Nov 15, 2017

its a fairlay large example. would like to keep it in its own sub module.

@andrewpalumbo

This comment has been minimized.

Member

andrewpalumbo commented Nov 15, 2017

Gotcha.. So are you thinking a Cylons module?

@andrewpalumbo

This comment has been minimized.

Member

andrewpalumbo commented Nov 15, 2017

Adding in an extensible full-stack real time image processing/classification with a kappa arch pipeline would be pretty sweet IMO.

@rawkintrevo

This comment has been minimized.

Contributor

rawkintrevo commented Nov 16, 2017

Yea if you look at how I re-org'd the examples, you'll see a lot of the examples were refactored to a legacy examples module, and Cylons is a module of its own.

@andrewpalumbo

This comment has been minimized.

Member

andrewpalumbo commented Dec 3, 2017

just looking at them now. I like the Cylons module name. I think we could go broader by having an mr-classics and an up-to-date-or-something. But need to test this out first!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment