(None, 13, 21, 2)| 0 |
-| QSeparableConv2D | (None, 11, 19, 5)| 33 |
-| QActivation | (None, 11, 19, 5)| 0 |
-| QConv2D | (None, 11, 19, 5)| 30 |
-| QActivation | (None, 11, 19, 5)| 0 |
-| AveragePooling2D | (None, 3, 6, 5) | 0 |
-| QActivation | (None, 3, 6, 5) | 0 |
-| Flatten | (None, 90) | 0 |
-| QDense | (None, 16) | 1456 |
-| QActivation | (None, 16) | 0 |
-| QDense | (None, 16) | 272 |
-| QActivation | (None, 16) | 0 |
-| QDense | (None, 14) | 238 |
+This generates a model with the following architecture:
+| Type | +Output Shape | +Parameters | +
|---|---|---|
| InputLayer | +(None, 13, 21, 2) | +0 | +
| QSeparableConv2D | +(None, 11, 19, 5) | +33 | +
| QActivation | +(None, 11, 19, 5) | +0 | +
| QConv2D | +(None, 11, 19, 5) | +30 | +
| QActivation | +(None, 11, 19, 5) | +0 | +
| AveragePooling2D | +(None, 3, 6, 5) | +0 | +
| QActivation | +(None, 3, 6, 5) | +0 | +
| Flatten | +(None, 90) | +0 | +
| QDense | +(None, 16) | +1456 | +
| QActivation | +(None, 16) | +0 | +
| QDense | +(None, 16) | +272 | +
| QActivation | +(None, 16) | +0 | +
| QDense | +(None, 14) | +238 | +
Refer to Evaluate for more details
diff --git a/docs/search/search_index.json b/docs/search/search_index.json index e384095..9e4519d 100644 --- a/docs/search/search_index.json +++ b/docs/search/search_index.json @@ -1 +1 @@ -{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"index.html","title":"Smart Pixels (ML)","text":"Smart Pixels is a project focused on implementing machine learning models directly on silicon pixels or future detectors to enhance the inference of charged particle track parameters. We here use an MDN (Mixture Density Network) with one Gaussian. This model predicts the mean value of the target variable and the associated uncertainty to that prediction, with NLL (Negative Log-likelihood Loss) as the loss function to minimize.
"},{"location":"index.html#table-of-contents","title":"Table of Contents","text":"git clone https://github.com/smart-pix/smart-pixels-ml.git\ncd smart-pixels-ml\npip install -r requirements.txt\npython test_run.py`\nThe output should look like this:
For the usage guide refer to Usage Guide
"},{"location":"index.html#dataset","title":"Dataset","text":"Download the simulated data from: zenodo and PixelAV
"},{"location":"index.html#structure","title":"Structure","text":"Input Data (dataX): Consists of 2D images representing the charge deposited by a particle, with each channel showing a different time slice of the particle\u2019s passage. You can choose to work with either 2 time slices (reduced) or the full 20 time slices. These correspond to input shapes of (13, 21, 2) and (13, 21, 20) respectively.
Labels (dataY): Four target variables are chosen as the labls viz. local $x$, local $y$, $\\alpha$ angle and $\\beta$ angle, associated with the particle trajectory.
For visualization of the how the input data looks like, we have to define the path towards the dataX and optionally to the labels as
import pandas as pd\nimport matplotlib.pyplot as plt\n\ndataX = pd.read_parquet(\"path/to/recon3D_data_file\")\nlabels_df = pd.read_parquet(\"path/to/labels_data_file\")\nreshaped_dataX = dataX.values.reshape((len(dataX), 20, 13, 21))\n\nprint(labels_df.iloc[0])\nplt.imshow(reshaped_dataX[0, 0, :, :], cmap='coolwarm') # first time-step\nplt.show()\nplt.imshow(reshaped_dataX[0, -1, :, :], cmap='coolwarm') # last time-step\nplt.show()\nDue to the large size of the dataset, the entire dataset can not be loaded into RAM. Hence, we use data generators to load the dataset on the fly during training with inbuilt quantization, standardization, shuffling, etc. Refer to data generator for more details.
"},{"location":"index.html#testing","title":"Testing","text":"To test that everything is working fine try running the simple test_run.py file as
python test_run.py\nThe core model architecture is defined in model.py, which provides the baseline MDN architecture with quantized neural network layers. We use the Negative Log-Likelihood (NLL) as the loss function implemented in loss.py. A good reading about it can be found here
Refer to Model and Loss for more details.
As an example, to implement the model with 2 time slices:
from model import *\n\nmodel=CreateModel((13,21,2),n_filters=5,pool_size=3)\nmodel.summary()\n(None, 13, 21, 2)| 0 | | QSeparableConv2D | (None, 11, 19, 5)| 33 | | QActivation | (None, 11, 19, 5)| 0 | | QConv2D | (None, 11, 19, 5)| 30 | | QActivation | (None, 11, 19, 5)| 0 | | AveragePooling2D | (None, 3, 6, 5) | 0 | | QActivation | (None, 3, 6, 5) | 0 | | Flatten | (None, 90) | 0 | | QDense | (None, 16) | 1456 | | QActivation | (None, 16) | 0 | | QDense | (None, 16) | 272 | | QActivation | (None, 16) | 0 | | QDense | (None, 14) | 238 |"},{"location":"index.html#model-evaluation","title":"Model Evaluation","text":"Refer to Evaluate for more details
"},{"location":"plot.html","title":"Plots","text":"Add things about the plots here
"},{"location":"testing.html","title":"Testing Module","text":"This module contains unit tests for the Smart Pixels ML project.
"},{"location":"testing.html#scripts","title":"Scripts","text":""},{"location":"testing.html#test_runpy","title":"test_run().py","text":"Description:
Usage:
python test_run()\nExample:
check_directories()
TEST_ROOT, DATA_DIR, LABELS_DIR, TFRECORDS_DIR, etc.) exist. Logs an error if any directory is missing.True if all required directories exist; otherwise, logs an error and returns False.generate_dummy_data(num_files=NUM_DUMMY_FILES)
num_files (int): The number of dummy data files to generate.DATA_DIR and LABELS_DIR.generate_tfrecords()
TFRECORDS_DIR_TRAIN and TFRECORDS_DIR_VALIDATION.load_tfrecords():
training_generator and validation_generator.test_model_generation()
test_train_model()
run_smoke_test():
some_test.py","text":""},{"location":"testing.html#functions_1","title":"Functions","text":"...
"},{"location":"testing.html#todo","title":"TODO \ud83d\udcdd","text":"This is a guide for using the Smart Pixels ML project to train and evaluate the models on the simulated data.
"},{"location":"usage.html#1-installation","title":"1. Installation","text":"Refer to Readme for installation instructions.
"},{"location":"usage.html#2-data-collection","title":"2. Data Collection","text":"Download the simulated data from zenodo and PixelAV Add other links here
Ensure the two directories Data and Labels are present.
"},{"location":"usage.html#3-data-preparation","title":"3. Data Preparation","text":"If everything is set up correctly, the training should start and run seamlessly. For example:
model.fit(\n x=training_generator, \n validation_data=validation_generator, \n epochs=200, \n verbose=1)\nAfter training, check the loss and accuracy of the model. And save the model weights.
"},{"location":"usage.html#6-model-evaluation","title":"6. Model Evaluation","text":"Initiate the model and Load the weights as
model=CreateModel((13,21,2),n_filters=5,pool_size=3)\nmodel.load_weights(\"model_weights.h5\")\nLook at predict
"},{"location":"usage.html#8-add-additional-instructions","title":"8. Add Additional Instructions","text":"Here are some additional instructions
"},{"location":"api/data_generator.html","title":"datagenerator module","text":"This module contains the OptimizedDataGenerator class, which generates batches of data for training and validation during model training. This datagenerator handles the loading and processing of the data, including shuffling, standardization, and quantization of the data. It does by pre-processing the data and saving it as TFRecord files and then loading the batches on the fly during training.
__init__(...)","text":"Initialize the OptimizedDataGenerator class with the specified parameters to configure the data generator for preprocessing and batching.
Described in the comments of the __init__ method of the OptimizedDataGenerator.py file.
training_generator = OptimizedDataGenerator(\n data_directory_path = \"path/to/data/\",\n labels_directory_path = \"path/to/labels/\",\n is_directory_recursive = False,\n file_type = \"parquet\",\n data_format = \"3D\",\n batch_size = val_batch_size,\n file_count = val_file_size,\n to_standardize= True,\n include_y_local= False,\n labels_list = ['x-midplane','y-midplane','cotAlpha','cotBeta'],\n input_shape = (2,13,21), # (20,13,21),\n transpose = (0,2,3,1),\n shuffle = False, \n files_from_end=True,\n\n tfrecords_dir = \"path/to/tfrecords/\",\n use_time_stamps = [0, 19], #-1\n max_workers = 1, # Don't make this too large (will use up all RAM)\n seed = 10, \n quantize = True # Quantization ON\n)\nAlready generated TFRecords can be reused by setting load_from_tfrecords_dir as
training_generator = OptimizedDataGenerator(\n load_from_tfrecords_dir = \"path/to/tfrecords/\",\n shuffle = True,\n seed = 13,\n quantize = True\n)\nThe same goes for the validation generator.
The data generators can be directly passed to the fit method of a Keras model.
history = model.fit(\n x=training_generator,\n validation_data=validation_generator,\n #callbacks=[es, mcp, csv_logger],\n epochs=1000,\n shuffle=False,\n verbose=1\n )\nThe evaluate.py file defines the evaluation function for the model.
evaluate(config)","text":"config (dict): Configuration dictionary containing the parameters for evaluation.config = {\n \"weightsPath\": \"path/to/weights.hdf5\",\n \"outFileName\": \"path/to/evaluation_results.csv\",\n \"data_directory_path\": \"path/to/data/\",\n \"labels_directory_path\": \"path/to/labels/\",\n \"n_filters\": 5,\n \"pool_size\": 3,\n \"val_batch_size\": 500,\n \"val_file_size\": 10\n}\nevaluate(config)\nThis module uses a Mixture Density Network(MDN) and hence a negative log-likelihood loss function. It uses TensorFlow and TensorFlow Probability for the loss computation.
"},{"location":"api/loss.html#loss-function","title":"Loss Function","text":""},{"location":"api/loss.html#custom_lossy-p_base-minval1e-9-maxval1e9-scale512","title":"custom_loss(y, p_base, minval=1e-9, maxval=1e9, scale=512)","text":"Calculates the Negative Log-Likelihood (NLL) for a batch of data using the model's predicted parameters.
"},{"location":"api/loss.html#brief-description","title":"Brief Description","text":"The model parameters are the mean and the lower triangular part of the covariance matrix.
loss function is vectorized with batches.
"},{"location":"api/loss.html#arguments","title":"Arguments","text":"y (tf.Tensor): The target data, shape (batch_size, 4).p_base (tf.Tensor): The predicted parameters, shape (batch_size, 16).minval (float): The minimum value for the likelihood, default 1e-9.maxval (float): The maximum value for the likelihood, default 1e9.scale (float): .tf.Tensor: Negative Log-Likelihood (NLL) for the given batch., shape (batch_size,).import tensorflow as tf\nfrom tensorflow.keras.optimizers import Adam\nfrom loss import custom_loss\nfrom models import CreateModel\n\nmodel = CreateModel((13, 21, 2), n_filters=5, pool_size=3)\nmodel.compile(optimizer=Adam(learning_rate=0.001), loss=custom_loss)\nThe models.py file defines Mixture Density Network (MDN) network with a 4D Multivariate Normal Distribution neural network architecture using quantized layers. The implementation uses QKeras to quantize the weights and activations of the network.
CreateModel(shape, n_filters, pool_size)","text":"Creates a quantized neural network model for regression task with quantized layers and activations as in Model. The model has 14 output nodes with 4 being the target variables and the rest 10 being the co-variances.
shape (tuple): Input shape (e.g., (13, 21, 2)/ (13, 21, 20)).n_filters (int): Number of filters for the convolutional layers.pool_size (int): Size of the pool for the pooling layer.keras.Model: A compiled Keras model instance.model = CreateModel((13, 21, 2), n_filters=5, pool_size=3) model.summary()
"},{"location":"api/models.html#additional-helper-functions","title":"Additional Helper Functions","text":""},{"location":"api/models.html#conv_networkvar-n_filters5-kernel_size3","title":"conv_network(var, n_filters=5, kernel_size=3)","text":"Defines the convolutional network block, with quantized layers and activations.
var (InputLayer: tf.Tensor): Input tensor.n_filters (int): Number of filters.kernel_size (int): Kernel size.
Returns:
tf.Tensor: Output tensor.var_network(var, hidden=10, output=2)","text":"Defines the dense network block, with quantized layers and activations.
var (InputLayer: tf.Tensor): Input tensor.hidden (int): Number of hidden units.output (int): Number of output units.
Returns:
tf.Tensor: Output tensor.Add Plot results here
"},{"location":"api/utils.html","title":"Utils Module","text":"This module contains utility functions to manage file operations and GPU configurations.
"},{"location":"api/utils.html#functions","title":"Functions","text":""},{"location":"api/utils.html#safe_remove_directorydirectory_path","title":"safe_remove_directory(directory_path)","text":"Safely removes a directory if it exists.
directory_path (str): Path to the directory to be removed.check_GPU()","text":"Checks for available GPUs and sets memory growth to prevent allocation issues.
Smart Pixels is a project focused on implementing machine learning models directly on silicon pixels or future detectors to enhance the inference of charged particle track parameters. We here use an MDN (Mixture Density Network) with one Gaussian. This model predicts the mean value of the target variable and the associated uncertainty to that prediction, with NLL (Negative Log-likelihood Loss) as the loss function to minimize.
"},{"location":"index.html#table-of-contents","title":"Table of Contents","text":"git clone https://github.com/smart-pix/smart-pixels-ml.git\ncd smart-pixels-ml\npip install -r requirements.txt\npython test_run.py`\nThe output should look like this:
For the usage guide refer to Usage Guide
"},{"location":"index.html#dataset","title":"Dataset","text":"Download the simulated data from: zenodo and PixelAV
"},{"location":"index.html#structure","title":"Structure","text":"Input Data (dataX): Consists of 2D images representing the charge deposited by a particle, with each channel showing a different time slice of the particle\u2019s passage. You can choose to work with either 2 time slices (reduced) or the full 20 time slices. These correspond to input shapes of (13, 21, 2) and (13, 21, 20) respectively.
Labels (dataY): Four target variables are chosen as the labls viz. local $x$, local $y$, $\\alpha$ angle and $\\beta$ angle, associated with the particle trajectory.
For visualization of the how the input data looks like, we have to define the path towards the dataX and optionally to the labels as
import pandas as pd\nimport matplotlib.pyplot as plt\n\ndataX = pd.read_parquet(\"path/to/recon3D_data_file\")\nlabels_df = pd.read_parquet(\"path/to/labels_data_file\")\nreshaped_dataX = dataX.values.reshape((len(dataX), 20, 13, 21))\n\nprint(labels_df.iloc[0])\nplt.imshow(reshaped_dataX[0, 0, :, :], cmap='coolwarm') # first time-step\nplt.show()\nplt.imshow(reshaped_dataX[0, -1, :, :], cmap='coolwarm') # last time-step\nplt.show()\nDue to the large size of the dataset, the entire dataset can not be loaded into RAM. Hence, we use data generators to load the dataset on the fly during training with inbuilt quantization, standardization, shuffling, etc. Refer to data generator for more details.
"},{"location":"index.html#testing","title":"Testing","text":"To test that everything is working fine try running the simple test_run.py file as
python test_run.py\nThe core model architecture is defined in model.py, which provides the baseline MDN architecture with quantized neural network layers. We use the Negative Log-Likelihood (NLL) as the loss function implemented in loss.py. A good reading about it can be found here
Refer to Model and Loss for more details.
As an example, to implement the model with 2 time slices:
from model import *\n\nmodel=CreateModel((13,21,2),n_filters=5,pool_size=3)\nmodel.summary()\nRefer to Evaluate for more details
"},{"location":"plot.html","title":"Plots","text":"Add things about the plots here
"},{"location":"testing.html","title":"Testing Module","text":"This module contains unit tests for the Smart Pixels ML project.
"},{"location":"testing.html#scripts","title":"Scripts","text":""},{"location":"testing.html#test_runpy","title":"test_run().py","text":"Description:
Usage:
python test_run()\nExample:
check_directories()
TEST_ROOT, DATA_DIR, LABELS_DIR, TFRECORDS_DIR, etc.) exist. Logs an error if any directory is missing.True if all required directories exist; otherwise, logs an error and returns False.generate_dummy_data(num_files=NUM_DUMMY_FILES)
num_files (int): The number of dummy data files to generate.DATA_DIR and LABELS_DIR.generate_tfrecords()
TFRECORDS_DIR_TRAIN and TFRECORDS_DIR_VALIDATION.load_tfrecords():
training_generator and validation_generator.test_model_generation()
test_train_model()
run_smoke_test():
some_test.py","text":""},{"location":"testing.html#functions_1","title":"Functions","text":"...
"},{"location":"testing.html#todo","title":"TODO \ud83d\udcdd","text":"This is a guide for using the Smart Pixels ML project to train and evaluate the models on the simulated data.
"},{"location":"usage.html#1-installation","title":"1. Installation","text":"Refer to Readme for installation instructions.
"},{"location":"usage.html#2-data-collection","title":"2. Data Collection","text":"Download the simulated data from zenodo and PixelAV Add other links here
Ensure the two directories Data and Labels are present.
"},{"location":"usage.html#3-data-preparation","title":"3. Data Preparation","text":"If everything is set up correctly, the training should start and run seamlessly. For example:
model.fit(\n x=training_generator, \n validation_data=validation_generator, \n epochs=200, \n verbose=1)\nAfter training, check the loss and accuracy of the model. And save the model weights.
"},{"location":"usage.html#6-model-evaluation","title":"6. Model Evaluation","text":"Initiate the model and Load the weights as
model=CreateModel((13,21,2),n_filters=5,pool_size=3)\nmodel.load_weights(\"model_weights.h5\")\nLook at predict
"},{"location":"usage.html#8-add-additional-instructions","title":"8. Add Additional Instructions","text":"Here are some additional instructions
"},{"location":"api/data_generator.html","title":"datagenerator module","text":"This module contains the OptimizedDataGenerator class, which generates batches of data for training and validation during model training. This datagenerator handles the loading and processing of the data, including shuffling, standardization, and quantization of the data. It does by pre-processing the data and saving it as TFRecord files and then loading the batches on the fly during training.
__init__(...)","text":"Initialize the OptimizedDataGenerator class with the specified parameters to configure the data generator for preprocessing and batching.
Described in the comments of the __init__ method of the OptimizedDataGenerator.py file.
training_generator = OptimizedDataGenerator(\n data_directory_path = \"path/to/data/\",\n labels_directory_path = \"path/to/labels/\",\n is_directory_recursive = False,\n file_type = \"parquet\",\n data_format = \"3D\",\n batch_size = val_batch_size,\n file_count = val_file_size,\n to_standardize= True,\n include_y_local= False,\n labels_list = ['x-midplane','y-midplane','cotAlpha','cotBeta'],\n input_shape = (2,13,21), # (20,13,21),\n transpose = (0,2,3,1),\n shuffle = False, \n files_from_end=True,\n\n tfrecords_dir = \"path/to/tfrecords/\",\n use_time_stamps = [0, 19], #-1\n max_workers = 1, # Don't make this too large (will use up all RAM)\n seed = 10, \n quantize = True # Quantization ON\n)\nAlready generated TFRecords can be reused by setting load_from_tfrecords_dir as
training_generator = OptimizedDataGenerator(\n load_from_tfrecords_dir = \"path/to/tfrecords/\",\n shuffle = True,\n seed = 13,\n quantize = True\n)\nThe same goes for the validation generator.
The data generators can be directly passed to the fit method of a Keras model.
history = model.fit(\n x=training_generator,\n validation_data=validation_generator,\n #callbacks=[es, mcp, csv_logger],\n epochs=1000,\n shuffle=False,\n verbose=1\n )\nThe evaluate.py file defines the evaluation function for the model.
evaluate(config)","text":"config (dict): Configuration dictionary containing the parameters for evaluation.config = {\n \"weightsPath\": \"path/to/weights.hdf5\",\n \"outFileName\": \"path/to/evaluation_results.csv\",\n \"data_directory_path\": \"path/to/data/\",\n \"labels_directory_path\": \"path/to/labels/\",\n \"n_filters\": 5,\n \"pool_size\": 3,\n \"val_batch_size\": 500,\n \"val_file_size\": 10\n}\nevaluate(config)\nThis module uses a Mixture Density Network(MDN) and hence a negative log-likelihood loss function. It uses TensorFlow and TensorFlow Probability for the loss computation.
"},{"location":"api/loss.html#loss-function","title":"Loss Function","text":""},{"location":"api/loss.html#custom_lossy-p_base-minval1e-9-maxval1e9-scale512","title":"custom_loss(y, p_base, minval=1e-9, maxval=1e9, scale=512)","text":"Calculates the Negative Log-Likelihood (NLL) for a batch of data using the model's predicted parameters.
"},{"location":"api/loss.html#brief-description","title":"Brief Description","text":"The model parameters are the mean and the lower triangular part of the covariance matrix.
loss function is vectorized with batches.
"},{"location":"api/loss.html#arguments","title":"Arguments","text":"y (tf.Tensor): The target data, shape (batch_size, 4).p_base (tf.Tensor): The predicted parameters, shape (batch_size, 16).minval (float): The minimum value for the likelihood, default 1e-9.maxval (float): The maximum value for the likelihood, default 1e9.scale (float): .tf.Tensor: Negative Log-Likelihood (NLL) for the given batch., shape (batch_size,).import tensorflow as tf\nfrom tensorflow.keras.optimizers import Adam\nfrom loss import custom_loss\nfrom models import CreateModel\n\nmodel = CreateModel((13, 21, 2), n_filters=5, pool_size=3)\nmodel.compile(optimizer=Adam(learning_rate=0.001), loss=custom_loss)\nThe models.py file defines Mixture Density Network (MDN) network with a 4D Multivariate Normal Distribution neural network architecture using quantized layers. The implementation uses QKeras to quantize the weights and activations of the network.
CreateModel(shape, n_filters, pool_size)","text":"Creates a quantized neural network model for regression task with quantized layers and activations as in Model. The model has 14 output nodes with 4 being the target variables and the rest 10 being the co-variances.
shape (tuple): Input shape (e.g., (13, 21, 2)/ (13, 21, 20)).n_filters (int): Number of filters for the convolutional layers.pool_size (int): Size of the pool for the pooling layer.keras.Model: A compiled Keras model instance.model = CreateModel((13, 21, 2), n_filters=5, pool_size=3) model.summary()
"},{"location":"api/models.html#additional-helper-functions","title":"Additional Helper Functions","text":""},{"location":"api/models.html#conv_networkvar-n_filters5-kernel_size3","title":"conv_network(var, n_filters=5, kernel_size=3)","text":"Defines the convolutional network block, with quantized layers and activations.
var (InputLayer: tf.Tensor): Input tensor.n_filters (int): Number of filters.kernel_size (int): Kernel size.
Returns:
tf.Tensor: Output tensor.var_network(var, hidden=10, output=2)","text":"Defines the dense network block, with quantized layers and activations.
var (InputLayer: tf.Tensor): Input tensor.hidden (int): Number of hidden units.output (int): Number of output units.
Returns:
tf.Tensor: Output tensor.Add Plot results here
"},{"location":"api/utils.html","title":"Utils Module","text":"This module contains utility functions to manage file operations and GPU configurations.
"},{"location":"api/utils.html#functions","title":"Functions","text":""},{"location":"api/utils.html#safe_remove_directorydirectory_path","title":"safe_remove_directory(directory_path)","text":"Safely removes a directory if it exists.
directory_path (str): Path to the directory to be removed.check_GPU()","text":"Checks for available GPUs and sets memory growth to prevent allocation issues.
| Type | +Output Shape | +Parameters | +
|---|---|---|
| InputLayer | +(None, 13, 21, 2) | +0 | +
| QSeparableConv2D | +(None, 11, 19, 5) | +33 | +
| QActivation | +(None, 11, 19, 5) | +0 | +
| QConv2D | +(None, 11, 19, 5) | +30 | +
| QActivation | +(None, 11, 19, 5) | +0 | +
| AveragePooling2D | +(None, 3, 6, 5) | +0 | +
| QActivation | +(None, 3, 6, 5) | +0 | +
| Flatten | +(None, 90) | +0 | +
| QDense | +(None, 16) | +1456 | +
| QActivation | +(None, 16) | +0 | +
| QDense | +(None, 16) | +272 | +
| QActivation | +(None, 16) | +0 | +
| QDense | +(None, 14) | +238 | +