This repository centralizes all the proactive objects (workflows, rules,..)
mbenguig Merge pull request #281 from mbenguig/master
Add PJID when submitting job from another
Latest commit ce8aa84 Oct 19, 2018
Permalink
Failed to load latest commit information.
AWS Changing kinds of worklfows (#188) Jun 5, 2018
AzureCognitiveServices Fix encoding and update PCA client side behavious (#239) Aug 29, 2018
BigData add description to some Big Data and PCA workflows (#231) Aug 20, 2018
CNES replace pca.action.icon with workflow.icon (#103) Mar 23, 2018
Calendars change kind name of calendar objects to "Calendar" (#230) Aug 17, 2018
Cassandra Fix encoding and update PCA client side behavious (#239) Aug 29, 2018
Clearwater Changing kinds of worklfows (#188) Jun 5, 2018
CloudAutomationClient Update cloud-automation-service-client-8.2.0-SNAPSHOT.jar May 31, 2018
CloudAutomationDockerTemplate Add Template for creating new docker-based PCA services (#256) Sep 6, 2018
CloudAutomationTemplate Changing kinds of worklfows (#188) Jun 5, 2018
Cron Merge pull request #262 from fviale/master Sep 12, 2018
DataConnectors Merge branch 'master' into master Oct 19, 2018
DataStreaming add description to some Big Data and PCA workflows (#231) Aug 20, 2018
DataVisualization add more machine learning generic tasks (#274) Oct 1, 2018
DatabaseServices add SQL statements task to the SQL project in data-connectors bucket (#… Oct 15, 2018
Databricks Changing kinds of worklfows (#188) Jun 5, 2018
DeepLearning Fixed FCN Oct 10, 2018
DeepLearningWorkflows Update the Yolo, Train_Object_Detection and Object_Detection tasks. (#… Sep 14, 2018
Docker Fix encoding and update PCA client side behavious (#239) Aug 29, 2018
DockerBasics Changing kinds of worklfows (#188) Jun 5, 2018
DockerSwarm Add PJID when submitting job from another Oct 16, 2018
ELK Improve elastic (#203) Jun 22, 2018
Elasticsearch Fix encoding and update PCA client side behavious (#239) Aug 29, 2018
Email Changing kinds of worklfows (#188) Jun 5, 2018
FileFolderManagement Add PJID when submitting job from another Oct 16, 2018
Finance Add documention in stress testing wkw Sep 6, 2018
FluidDynamicsCFDParaviewFlexnetPBS Fix job submission issue in LaBSsolver_multiple.xml Oct 16, 2018
GetStarted Merge branch 'master' into master Oct 19, 2018
H2O Elk stack h2o (#190) Jun 6, 2018
HDFS Changing kinds of worklfows (#188) Jun 5, 2018
HDFSOrchestration Add descriptions in spark_pi and write_read_hdfs workflows Jul 10, 2018
HadoopOrchestration Changing kinds of worklfows (#188) Jun 5, 2018
ImageAnalysis Changing kinds of worklfows (#188) Jun 5, 2018
JobAnalysis Changing kinds of worklfows (#188) Jun 5, 2018
Kafka add description to some Big Data and PCA workflows (#231) Aug 20, 2018
Kibana Elk stack h2o (#190) Jun 6, 2018
LogAnalysis add test scenarios to 5 packages (#252) Sep 6, 2018
Logstash Elk stack h2o (#190) Jun 6, 2018
MLBasics updated ml workflows and visdom visualize results (#266) Sep 14, 2018
MLNodeSource added data analytics workflows (#211) Jul 3, 2018
MachineLearning added two new scalers Oct 5, 2018
MachineLearningH2O Elk stack h2o (#190) Jun 6, 2018
Machining Change date format in machining workflows Sep 20, 2018
MongoDB Fix encoding and update PCA client side behavious (#239) Aug 29, 2018
MySQL Fix encoding and update PCA client side behavious (#239) Aug 29, 2018
NodeSources Modify Cloud Node Source examples (#269) Sep 21, 2018
OpenShift Add OpenShift examples (#283) Oct 17, 2018
OpenStack Changing kinds of worklfows (#188) Jun 5, 2018
PCWRules Update rules (#284) Oct 17, 2018
PostgreSQL fix postgresql password key format (#285) Oct 17, 2018
RemoteVisualization Changing kinds of worklfows (#188) Jun 5, 2018
ReserveNodes Add PJID when submitting job from another Oct 16, 2018
Scripts fix script URL reference in all SQL data connectors workflow (#282) Oct 16, 2018
SelectionScripts Add collection of selection scripts Sep 5, 2018
Spark Add descriptions in Spark, Swarm and Machining packages Sep 3, 2018
SparkOrchestration Add descriptions in spark_pi and write_read_hdfs workflows 2/2 Jul 10, 2018
Storm Update Storm Docker image (#250) Sep 4, 2018
Synchronization add test scenarios to 5 packages (#252) Sep 6, 2018
TextAnalysis add test scenarios to 5 packages (#252) Sep 6, 2018
TwitterApi Changing kinds of worklfows (#188) Jun 5, 2018
Visdom fixed visdom service initialization Sep 12, 2018
WebNotification Changing kinds of worklfows (#188) Jun 5, 2018
Zookeeper add description to some Big Data and PCA workflows (#231) Aug 20, 2018
gradle/wrapper Add Metadata for hub and reorganize folders Oct 5, 2017
src/test/java/org/ow2/proactive add SQL statements task to the SQL project in data-connectors bucket (#… Oct 15, 2018
.gitignore Add .gitignore file Feb 12, 2018
README.md update pcw rules with new format, fix readme (#276) Oct 1, 2018
build.gradle openshiftExamples (#280) Oct 15, 2018
gradle.properties Increased SNAPSHOT Version number Sep 18, 2018
gradlew Add Metadata for hub and reorganize folders Oct 5, 2017
gradlew.bat Add Metadata for hub and reorganize folders Oct 5, 2017
ordered_bucket_list openshiftExamples (#280) Oct 15, 2018

README.md

ProActive Examples

This repository includes a sub directory per bucket, in which a metadata file centralizes object-related-information: bucket name, object name, object kind,.. If the "studio_template" section is specified, the object/workflow is also considered as a studio template.

The aim of this project is to centralize all proactive workflows and other related objects (scripts, images, etc). The workflows from the ProActive Examples project are pushed to Catalog storage inside proactive.

How to build

Please run next command: gradlew clean zip or gradlew clean build This will generate the proactive-examples.zip file inside project's build folder.

How to test locally

Copy the genarated proactive-examples.zip file to your PA_SCHEDULER_HOME/samples directory. Start your proactive distribution. From this point everything should work ok. During scheduling startup: the proactive-examples.zip archive will be extracted to PA_SCHEDULER_HOME/samples/workflows/proactive-examples folder. On the next step the special groovy script will automatically push the workflows from proactive-examples folder to Catalog storage. If you need to retest the extracting and loading of proactive-examples, please remove workflows/proactive-examples folder. Also to test the filling of catalog storage don't forget to clean database.

The example of exact commands to test locally on linux:

1) PA_SCHEDULER_HOME is the path to your local Proactive distribution folder. You need to `cd` to this folder.
2) rm -fr samples/workflows/proactive-examples*
3) rm -fr data/*
4) you need to `cd` into your locally cloned proactive-examples project folder
5) ./gradlew clean zip
6) cp build/proactive-examples.zip PA_SCHEDULER_HOME/samples/
7) go back to PA_SCHEDULER_HOME and start proactive-server
8) ./PA_SCHEDULER_HOME/bin/proactive-server

How to add a new package

  1. Create a folder with the desired package name (e.g. TextAnalysis).

  2. Add a METADATA.json file into the package (e.g. TextAnalysis/METADATA.json).

  3. Insert the following JSON structure into the METADATA.json file:

{
	"metadata": {
		"slug": "textanalysis",
		"name": "Text Analysis",
		"short_description": "Text analysis with machine learning and deep learning on Docker",
		"author": "ActiveEon's Team",
		"tags": ["Samples", "Machine Learning", "Text Analysis", "Big Data", "Analytics", "Deep Learning"],
		"version": "1.0"
	},
	"catalog" : {
		"bucket" : "machine-learning",
		"objects" : [
			{
				"name" : "text_analysis",
				"metadata" : {
					"kind": "Workflow/standard",
					"commitMessage": "First commit",
					"contentType": "application/xml"
				},
				"file" : "resources/catalog/text_analysis.xml"
			}
		]
	}
}

3.1) Update the metadata fields:

    • metadata->slug - compact name of the package.
    • metadata->name - name of the package.
    • metadata->short_description - short description of the package.
    • metadata->tags - key works of the package.

3.2) Update the catalog fields:

    • catalog->bucket - Set the name of the bucket(s) for this package. A package can be installed in one or multiple buckets. Therefore, the bucket name can be a String value as in the example above (i.e "Machine_Learning") or a JSON array value as in ["Bucket_1","Bucket_2","Bucket_X"].
    • catalog->objects - Add an object of each workflow of the package.

An example of a catalog object that represents a workflow:

{
				"name" : "text_analysis",
				"metadata" : {
					"kind": "Workflow/standard",
					"commitMessage": "First commit",
					"contentType": "application/xml"
				},
				"file" : "resources/catalog/text_analysis.xml"
			}
    • object->name - Name of the workflow.
    • object->file - Relative path of the XML file of the workflow.
  1. Add the XML file(s) of the workflow(s) into resources/catalog/ inside your package folder (e.g. TextAnalysis/resources/catalog/text_analysis.xml).

  2. By default all new buckets will be added after all existing buckets inside catalog. So no need by default to add bucket name to ordered_bucket_list file.

But if you need to have strict order of buckets, then please update ordered_bucket_list by adding the package name (order by name). The whole list should be stored as 1 line without any spaces or end line character.

  1. Update build.gradle by finding task zip (type: Zip) function and adding an include for your package (e.g. include 'TextAnalysis/**'). Example:
task zip (type: Zip){
    archiveName="proactive-examples.zip"
    destinationDir = file('build/')
    from '.'
    include 'AWS/**'
    include 'Clearwater/**'
    include 'CloudAutomationTemplate/**'
    include 'Cron/**'
    include 'DockerBasics/**'
    include 'DockerSwarm/**'
    include 'Email/**'
    include 'FileFolderManagement/**'
    include 'FinanceMonteCarlo/**'
    include 'GetStarted/**'
    include 'HDFS/**'
    include 'HDFSOrchestration/**'
    include 'ImageAnalysis/**'
    include 'JobAnalysis/**'
    include 'LogAnalysis/**'
    include 'MLBasics/**'
    include 'MLNodeSource/**'
    include 'OpenStack/**'
    include 'RemoteVisualization/**'
    include 'Spark/**'
    include 'SparkOrchestration/**'
    include 'Storm/**'
    include 'TextAnalysis/**'
    include 'TriggerTemplate/**'
    include 'TwitterApi/**'
    include 'WebNotification/**'
    include 'ordered_bucket_list'
}

The rules for added workflows

All the workflows added in proactive-examples project have to follow the next rules:

. Every single workflow of packages distributed by Activeeon (as all the workflows from proactive-examples), MUST HAVE a Workflow Generic Information "workflow.icon" with a meaningful Icon . URL of this icon MUST reference a local file . If a workflow has a single task, this task MUST HAVE a Task Generic Information "task.icon" with the same icon as the Workflow

  • In case if workflow is not corresponding to specified rules: the test inside the proactive-examples project will fail.

That's all!