Schemas from the event driven ml experiment
Event-Driven Machine Learning is an experiment ran by @DivLoic and @Giulbia. The idea is to bring together all the components to build a system serving prediction in realtime. From exploring the data with interactive tools like Ai Platform to deploying streaming application in the cloud with Confluent Cloud the EDML project discuss the important points leading to model serving automation.
In order to share the same schemas between the modules:
- edml-replay: app generating the validation and test data points
- edml-serving: app creating the prediction
- edml-scoring: app join the prediction to the actual result
This module gather all the avro schemas. After the schema validation, the module generates plain java objects, add thel into a maven package to be used by all the listed module.
<dependency>
<groupId>fr.xebia.gbildi</groupId>
<artifactId>edml-schema</artifactId>
<version>0.1.0-SNAPSHOT</version>
</dependency>The module uses the avrohugger maven plugin.
<plugin>
<groupId>at.makubi.maven.plugin</groupId>
<artifactId>avrohugger-maven-plugin</artifactId>
<version>${avrohugger-maven-plugin.version}</version>
</plugin>The Github Action trigger the goal generate:
./mvnw -B avrohugger:generate-scala-sourcesWhich creates the scala classes (under: target/generated-sources) that get shipped with the JAR. Example:
case class KeyClass(var uuid: String) extends org.apache.avro.specific.SpecificRecordBase {
def this() = this("")
def get(field$: Int): AnyRef = {
(field$: @switch) match {
case 0 => {
uuid
}.asInstanceOf[AnyRef]
case _ => new org.apache.avro.AvroRuntimeException("Bad index")
}
}
def put(field$: Int, value: Any): Unit = {
(field$: @switch) match {
case 0 => this.uuid = {
value.toString
}.asInstanceOf[String]
case _ => new org.apache.avro.AvroRuntimeException("Bad index")
}
()
}
def getSchema: org.apache.avro.Schema = KeyClass.SCHEMA$
}