Merge pull request #109 from Sunbird-Saral/v1-develop

V1 develop

Jenkinsfile-backend

@@ -20,9 +20,9 @@ node('server_build_slave') {
                                           fi
                                           commit_id=${BUILD_ID}-$(git rev-parse --short HEAD)
                                           echo $commit_id> commit_id.txt
-                                          docker build -t anuvaadio/$image_name:1.0-latest .
+                                          docker build -t anuvaadio/$image_name:$image_tag .
                                           docker login -u $dockerhub_user -p $dockerhub_pass
-                                          docker push anuvaadio/$image_name:1.0-latest
+                                          docker push anuvaadio/$image_name:$image_tag
                                           '''
 
                            }

Jenkinsfile-deploy

@@ -0,0 +1,41 @@
+#!groovy
+
+node('saral-poc') {
+
+try {
+
+   stage('Checkout'){
+      checkout scm
+	}
+
+
+	stage('Docker-compose')
+withCredentials([usernamePassword(credentialsId: 'anuvaad-docker-hub-credentials', passwordVariable: 'PASSWORD', usernameVariable: 'USERNAME')]) {
+	       sh '''
+
+		environment=$(echo "$JOB_BASE_NAME" | rev | cut -d'-' -f 1 | rev)
+		compose_file=$(echo "$image_name")
+		tag=$(echo "$image_tag")
+		sudo docker service rm saralpoc_$compose_file
+		sudo env image="$compose_file" tag=$tag env="$environment" docker stack deploy --compose-file=$compose_file.yml --with-registry-auth --resolve-image=always saralpoc
+		sudo docker service update saralpoc_gateway --force
+		
+		echo "Reload Saral Reference Data ? : $reloadSaralReferenceData"
+		if [ $reloadSaralReferenceData = true ]
+		then 
+			echo "sleeping for 30s for containers to start"
+			sleep 30s
+			sudo docker exec $(sudo docker ps -aqf "name=^saralpoc_saral-backend") node ./data/import-data.js --delete
+			sudo docker exec $(sudo docker ps -aqf "name=^saralpoc_saral-backend") node ./data/import-data.js --import
+			echo 'Reloaded Saral Reference Data'
+		else 
+			echo 'skipping Reload Saral Reference Data'
+		fi
+		'''
+		}   
+ }
+catch (err) {
+    currentBuild.result = "FAILURE"
+    throw err
+ }
+}

models/v1.5/README.md

@@ -0,0 +1,89 @@
+# Handwritten digit recognition with MNIST and Keras - Playbook
+
+This repository is for handwritten digit recognition which is integrated to Saral project. Saral is an OCR-plus application that is capable of doing OCR and can also understand the structure of the physical input.
+##### total no. of classes- 11(0-10)
+- empty or noisy type images are included in class10
+
+
+#### Architectures
+- [Resnet164](https://arxiv.org/abs/1603.05027)
+
+
+
+#### Augmentation and Normalization
+- Channel-wise normalization of input images: substracted by mean and divided by std
+- Data augmentation: rotation, width shift, height shift, shearing, zooming
+
+
+##### To install packages run this command line
+```bash
+$ pip install -r requirement.txt 
+```
+
+## Training
+The training can be executed by the following command.
+In config file we can have to update training data path and model specific parameters. Once the training is done trained model will be saved in the given path.
+
+#### How to run
+```bash
+$ python training.py 
+```
+
+## Keras to tflite model converter
+This can be done using keras_to_tf_converter.py module. As a input it takes path of keras model and save path where we want to save the tflite model. Define them in ./config.py.
+
+#### How to run
+```bash
+$ python keras_to_tf_converter.py 
+```
+
+## TFLite model configuration for Saral App
+1. Place the .tflite model file in below project path.
+2. Use it in ``org.ekstep.saral.saralsdk.hwmodel.HWClassifier.java`` file
+``/Project-Saral/v1.0/frontend/SaralApp/android/saralsdk/src/main/assets``
+
+### Evaluation
+##### prediction.ipynb for benchmarking the test dataset for both keras and tflite model at digit and roll number level
+#### Config Parameters
+```bash
+
+
+optional arguments:
+  --epochs EPOCHS       How many epochs you need to run (default: 10)
+  --batch_size BATCH_SIZE
+                        The number of images in a batch (default: 64)
+  --path_for_weights PATH_FOR_WEIGHTS
+                        The path from where the weights will be saved or
+                        loaded (default: ./models/VGG16.h5)
+  --path_for_image PATH_FOR_IMAGE
+                        The path from where the model image will be saved
+                        (default: ./images/VGG16.png)
+  --path_for_plot PATH_FOR_PLOT
+                        The path from where the training progress will be
+                        plotted (default: ./images/VGG16_plot.png)
+  --data_augmentation DATA_AUGMENTATION
+                        0: No, 1: Yes (default: 1)
+  --save_model_and_weights SAVE_MODEL_AND_WEIGHTS
+                        0: No, 1: Yes (default: 1)
+  --load_weights LOAD_WEIGHTS
+                        0: No, 1: Yes (default: 0)
+  --plot_training_progress PLOT_TRAINING_PROGRESS
+                        0: No, 1: Yes (default: 1)
+  --save_model_to_image SAVE_MODEL_TO_IMAGE
+                        0: No, 1: Yes (default: 1)
+```
+
+## File descriptions
+```bash
+├── images/ # training data: few samples are given for each digit 
+├── models/ # model weights (included in this repo)
+├── README.md
+├── base_model.py # base model interface
+├── utils.py # helper functions
+├── resnet164.py
+├── training.py # training script
+
+
+```
+#### ResNet Implementation
+- [ResNet Author's Implementation](https://github.com/KaimingHe/resnet-1k-layers/blob/master/resnet-pre-act.lua)

models/v1.5/config.py

@@ -0,0 +1,29 @@
+
+SAVE_MODEL=True
+
+### path where you want to save the trained model
+SAVE_MODEL_PATH="./models/trained_resnet_model_v4.h5"
+
+### any keras(.h5) model path which you want to convert into tflite
+H5_MODEL_PATH="./models/trained_resnet_model_v4.h5"
+
+### flite  model save path
+TF_LITE_SAVE_PATH="./models/hw_recog_digit_tmp.tflite"
+### total number of epochs you want to train the model
+EPOCH=1
+BATCH_SIZE=8
+### Path of pretrained model to finetune it on new batch of dataset
+
+PRETRAINED_WEIGHT_PATH='./models/trained_resnet_model_v2_10.h5'
+
+#### datset path
+IMAGE_PATH="./images/*/*.jpg"
+
+####size of the dataset to use for validation the trained model
+TEST_DATA_SIZE=4
+
+##### if you want to use pre rained model as an initializer thwn FINE_TUNE should be True
+FINE_TUNE=True
+
+#### we can data augmentation True if dataset is small with less variation.
+DATA_AUGMENTATION=False

models/v1.5/keras_to_tf_converter.py

@@ -0,0 +1,9 @@
+import config
+import tensorflow as tf
+
+h5_model_path = config.H5_MODEL_PATH
+
+model = tf.keras.models.load_model(h5_model_path)
+converter = tf.lite.TFLiteConverter.from_keras_model(model)
+tflite_model = converter.convert()
+open(config.TF_LITE_SAVE_PATH, "wb").write(tflite_model)