-
Background:
- For most, if not all, reported QSAR-based DILI prediction models, evaluation is based on the statistical performances which in turn are based on random or stratified data splits accompanied by a strategy involving cross-validation or external validation
- This approach does not reflect the real-world scenario where the validity of a model based on past approved drugs has to be verified for its predictivity for future drugs
- Moreover, an explainable QSAR model is of great importance for biologists and medical officers to facilitate their decision making
- Thus, exploring the features driving the model performance and predefined applicability domains is key to increasing the model explainability and facilitating real-world implementation
-
Aim:
- Combining model-level representation generated by conventional machine learning algorithms with a deep learning framework based on Mold2 descriptors
- Could the DILI potential of newly approved drugs be predicted by accumulated knowledge of early approved ones?
- Developed the DeepDILI model using drugs approved before 1997 to predict the DILI potential of those approved thereafter
- Is model-level representation more informative than molecule-based representation for DILI prediction?
- Demonstrated that the DeepDILI model’s performance was significantly improved compared with deep neural networks based on molecule-based representation
- Could improved model explainability be established?
- They found 21 chemical descriptors that were enriched, suggesting a strong association with DILI outcome
-
Data type:
- Binary DILI classification data set and Mold2 descriptors
-
Data curation:
- Excluded biologics, mixtures, organometallics, and inorganics from the DILI
- Removed the Mold2 descriptors with constant values
- Kept only one descriptor if any two descriptors have a pairwise correlation coefficient more than 0.9
-
Approaches:
- Five conventional ML algorithms, including K-nearest neighbor (KNN), logistic regression (LR), SVM, RF, and extreme gradient boosting (XGBoost)
- Training set – drugs approved before year 1997
- Test set – drugs approved after year 1997 (independent test set for evaluation)
- The probability outputs from base classifiers (model-level representation) are the input for the second level of NN. This approach tries to use the meta-classifier to learn the complementary information provided by the base classifiers
- A series of parameters imployed for model optimization
- Base classifier development:
- Specifically, for each model and hyperparameter combination, 100 models were developed by randomly selecting 80% of drugs in training set 1 and then predicting the drugs in training set 2. For each algorithm, the 100 models with hyperparameter combination yielding the highest average Matthews correlation coefficient (MCC) were selected. Next, they ranked order the 500 base classifiers (100 models X 5 algorithms = 500 models) from high to low based on their MCC values. To avoid overfitting or underfitting problems of the developed base classifiers, they only kept the models with MCC values in the range of 5−95% percentile of the 500 models as the optimized base classifiers
- The 100 models with hyperparameter combination yielding the highest average MCC values are selected for further analysis
- Meta-classifier development:
- Developed a three-layer NN model as a meta-classifier for DILI prediction. Specifically, they extracted the probability output of the optimized base classifiers (i.e., model-level representation) from the training set 2 as the NN input layer. In other words, the model level representation was constructed by a vector consisting of the predicted probability values from the filtered based classifiers
- Hidden layer with 10 nodes with hyperbolic tangent activation (tanh) activation, and Adam optimizer, the output layer is used to project the hidden layer information into probabilistic values of DILI prediction using sigmoid function. The learning rate is set to 0.0001
- Logistic regression:
- Drugs predicted with probability ≥0.5 belong to DILI positives, and the rest belongs to DILI negatives. The inverse regularization parameter (C) in LR was optimized
- KNN:
- In this study, the Euclidean distance was applied to measure the distance between the object and its neighbor. The hyperparameter (i.e., the number of neighbors K) was optimized
- SVM:
- The radial basis function (RBF) was used as the kernel function in our study. The polynomial and RBF kernel functions with hyperparameters, i.e., γ for the nonlinear kernel and penalty parameter C, were optimized
- XGBoost
- Hyperparameters, including learning rate, estimators, max depth, and subsample, were optimized
- RF:
- We optimized the hyperparameters, including the number of trees, the max depth of the tree, the minimum sample split, and the minimum sample leaf
-
Model comparison:
- DeepDILI against base classifiers
- DeepDIli against conventional DNN
- DeepDILI against a published DNN
- DeepDILI against other ensemble approaches
- DeepDILI based on Mold2 against DeepDILI based on Mold2vec and MACCS
-
Model interpretation:
- Rank order the individual descriptor based on its model contribution
- A higher area under the curve (AUC) indicates less influence on the model performance by the removed feature
- Stepwise feature selection
- The higher the OFS is, the more preservability of the model performance is kept with features removed
-
Applicability domain:
- It was defined based on WHO-ATC
- Only the therapeutic classes with drugs number more than 20 were kept for the analysis
-
Model performance:
- ROC curve analysis (AUC), Accuracy, sensitivity, specificity, F1, MCC, balanced accuracy
- DeepDILI was also used to screen DrugBank database by excluding the compounds present in the training set. The predictions grouped into five classes
-
Key learnings:
- Complex algorithms (i.e., RF, XGBoost, SVM) may have better model generality than simple ML classifiers (i.e., LR and KNN)
- The prediction performance of DeepDILI models on the test set was ranked in the following order: Mold2-DeepDILI > Mol2vec-DeepDILI > MACCSDeepDILI across all the performance metrics except specificity
- Except for sensitivity, the DeepDILI model outperformed the two ensemble approaches across all of the other six performance metrics
- The DeepDILI model aims to provide an explainable model and address the critical questions in the DILI field, such as using accumulative DILI knowledge from approved drugs to infer DILI potential for newly approved drugs
- Replaced molecule-level representation with model-level representation from the individual ML classifier to maximally extract DILI-related information from the data and develop DNN DeepDILI model
- More investigation of molecular descriptors’ impact on model performance should be further investigated with diverse biological end points under different ML algorithms
- A sequential feature selection strategy was developed to generate a more explainable model with important feature subsets