Machine learning(ML) models are algorithms that are trained for a particular set of data. E.g. y = mx + c is an algorithm, whereas y = 2x + 3 is a model. You provide input to a model and it gives a response. Cognitive systems are not programmed and they perform as per the data on which they are trained on; to sense, predict, infer, and in some ways, think, using artificial intelligence and machine learning algorithms.
In machine learning world numerous models are being created for achieving a specific task. With so many models available, how can one decide which model to use? Which model is performing better? What are the various performance parameters for different models? This code pattern shows you a way to compare Watson Cognitive services models so as to decide which model performs better for a particular set of data. It provides user a platform to configure models, provide input data, execute and prepare performance evaluation statistics such as confusion matrix and ROC curve.
After going through this code pattern, you should be able to:
- Create and deploy Watson services models.
- Configure model evaluation workbench to compare specific models and provide and test data.
- Understand model evaluation parameters and know which model best suits your needs.
- User launches the application.
- Cloud authenticates the request and redirects it to the application.
- Parses input data provided for evaluating the models.
- Invokes adapter which calls cognitive services like Natural Language Classifier, Natural Language Understanding, etc.
- Parses the cognitive model services configuration.
- Connects to cognitive services.
- Gets response from cognitive services.
- Compares the expected result with actual result and does performance evaluation.
- Performance results are sent back to client devices
- Performance analysis is shown on UI
- Java Liberty Runtime - Develop, deploy, and scale Java web apps with ease. IBM WebSphere Liberty Profile is a highly composable, ultra-fast, ultra-light profile of IBM WebSphere Application Server designed for the cloud.
- Natural Language Classifier(NLC) - The Natural Language Classifier service applies cognitive computing techniques to return the best matching classes for a sentence or phrase.
- IBM Cloud Object Storage: An IBM Cloud service that provides an unstructured/structured cloud data store to build and deliver cost effective apps and services with high reliability and fast speed to market.
- Java: The Java programming language is a high-level, object-oriented language. When written, a Java program is compiled into bytecode. The bytecode is interpreted at run time by a platform-specific Java component
- Artificial Intelligence: Intelligence demonstrated by machines, in contrast to the natural intelligence displayed by humans.
Follow these steps to setup and run this code pattern. The steps are described in detail below.
- Pre-requisites
- Create the Cognitive models
- Deploy the application to the IBM Cloud
- Deploy the application to local machine
- Run the application
- Analyse the results
- IBM Cloud account: You must have IBM Cloud account to work with this code pattern. If you do not have an IBM Cloud account, you can create a one month free trail account here.
For this pattern to demonstrate, we would be creating and using Natural Language Classifier(NLC) models. This code pattern requires at least two NLC models to compare and evaluate. So, we will create two classifier models in one NLC service instance.
- Follow the instructions from the given link to create and train the NLC models.
- While you are training the first model, Use
NLC_TrainngDataset1.csv
file from../src/main/data/NLC_TrainngDataset1.csv
. - While you are training the second model, Use
NLC_TrainngDataset2.csv
file from../src/main/data/NLC_TrainngDataset2.csv
. - Copy API Key, Url and classifier_id for each of the classifiers and save it in a text file. These credentials are required in later steps.
Now, we have created and trained two models in one NLC service instance.
The application can be deployed on IBM Cloud or locally. Execute Step 3 for deploying on IBM Cloud or Step 4 to deploy locally.
-
Maven should be installed. If not installed, you can download maven from here. You can refer to the installation instructions here.
-
Install IBM Cloud CLI if not already done
-
Login to IBM Cloud using CLI using the below command. Ensure that the target points to the region and space where you are running and deploying this code pattern.
bx login
-
open command prompt. Change directory to location where you want to download project files. Go to that directory.
-
run one of the below commands to clone the repository.
git clone git@github.com:IBM/model-evaluation-workbench.git
or
git clone https://github.com/IBM/model-evaluation-workbench.git
-
Change directory to model-evaluation-workbench.
-
run the below command.
mvn clean install
-
This should create the file
workbenchModelEval.war
undertarget
folder. -
Open the
manifest.yml
file and update values as explained below.- For each classifier, update
NLC_API_KEY_CONFIG_x
,NLC_CLASSIFIER_ID_CONFIG_x
with API Key and Classifier Id respectively. For first classifier,x
is replaces with1
and for second classifier,x
is replaced with2
and so on. - Update value against
NLC_URL
. It'll be same for all classifiers.
- For each classifier, update
-
Before pushing the application, set the target organisation and space using the below command:
bx target --cf
-
Push the application to IBM Cloud using the command:
bx cf push <Application_Name>
-
The application should get deployed successfully. If not check the logs to determine the error and fix it.
- Open IBM Cloud Console, Under Dashboard you should see the above deployed application running. Click on the application.
- Click on
Visit App URL
link to launch the application.
The application can be deployed on IBM Cloud or locally. Execute Step 3 for deploying on IBM Cloud or Step 4 to deploy locally.
-
open command prompt. Change directory to location where you want to download project files. Go to that directory.
-
run the below command to clone the repository
git clone git@github.com:IBM/model-evaluation-workbench.git
-
Change directory to model-evaluation-workbench
- If eclipse is not installed on your local machine then install Eclipse following instructions provided in this link
- After the installation, launch Eclipse
- In the eclipse menu, click
file
and then clickimport
- Under import wizard, expand
General
folder and selectExisting Project into Workspace
. - Click
Next
and browse to the cloned project folder. - Click the
Finish
button. - Setup Liberty server in eclipse as per this Link
- Application needs to access models to send requests to and get responses from them. Access details of these models have to be provided in server config (server.env) file of liberty server. A sample server.env file snapshot is as in below image. You will need to provide the credentials of the models. You can access NLC credentials as described in section
Create NLC service instance
- Start the Liberty Server
- Right Click on the Liberty Server created above, Click on the
Add and Remove
- Under Add and Remove wizard, from the available section move
ModelEvaluationWorkbench
to configured section. - Click on Finish.
- Model Evaluation application should be started.
- In Eclipse, under Console tab, you will see default_host link similar to http://localhost:9091/ModelEvaluationWorkbench/.
Click on that link to launch the application.
We created two models using Natural Language Classifier. Now, we would evaluate which model is performing better when compared to each other.
- On the application home page click on
NLC
box - Under
Attribute
selectClasses
- Select
MODEL 1
andMODEL 2
checkboxes - Two browse buttons will be available since we have selected two models. Click on each browse button and select the file
src/main/data/NLC_TruthFile.csv
from your git repo. The same file is selected for both browse buttons because this file is the actual results file which needs to be compared with both the model results.
- Click on Evaluate Performance button.
The application invokes both the NLC services, gets responses and displays various evaluation parameters.
The result shows a lot of statistical data for model performance evaluation. It is suggested that you become familiar with these terminologies
Results of running the two NLC models using the truth file enables the application to check how the results from each of the models are faring.
The below screenshot shows various parameters such as F1Score Optimization
and Accuracy Optimization
and also provides Model Rating
The application also shows ROC chart, for both the models, for various parameters
In this case we can see that Model 1 fares better compared to model 2. Users can analyse various performance parameters and can take appropriate decision.
This code pattern is licensed under the Apache Software License, Version 2. Separate third party code objects invoked within this code pattern are licensed by their respective providers pursuant to their own separate licenses. Contributions are subject to the Developer Certificate of Origin, Version 1.1 (DCO) and the Apache Software License, Version 2.