diff --git a/README.md b/README.md index 732fa6a..d1a7666 100644 --- a/README.md +++ b/README.md @@ -15,15 +15,15 @@ PyTorchPipe (PTP) is a component-oriented framework that facilitates development of computational _multi-modal pipelines_ and comparison of diverse neural network-based models. PTP frames training and testing procedures as _pipelines_ consisting of many components communicating through data streams. -Each such a stream can consist of several components, including one problem instance (providing batches of data), any number of trainable components (models) and additional components providing required transformations and computations. +Each such a stream can consist of several components, including one task instance (providing batches of data), any number of trainable components (models) and additional components providing required transformations and computations. -As a result, the training & testing procedures are no longer pinned to a specific problem or model, and built-in mechanisms for compatibility checking (handshaking), configuration and global variables management & statistics collection facilitate rapid development of complex pipelines and running diverse experiments. +As a result, the training & testing procedures are no longer pinned to a specific task or model, and built-in mechanisms for compatibility checking (handshaking), configuration and global variables management & statistics collection facilitate rapid development of complex pipelines and running diverse experiments. In its core, to _accelerate the computations_ on their own, PTP relies on PyTorch and extensively uses its mechanisms for distribution of computations on CPUs/GPUs, including multi-process data loaders and multi-GPU data parallelism. The models are _agnostic_ to those operations and one indicates whether to use them in configuration files (data loaders) or by passing adequate run-time arguments (--gpu). **Datasets:** -PTP focuses on multi-modal reasoning combining vision and language. Currently it offers the following _Problems_ from the following problem domains: +PTP focuses on multi-modal reasoning combining vision and language. Currently it offers the following _Tasks_ from the following task domains: * CLEVR, GQA, ImageCLEF VQA-Med 2019 (Visual Question Answering) * MNIST, CIFAR-100 (Image Classification) @@ -31,8 +31,8 @@ PTP focuses on multi-modal reasoning combining vision and language. Currently it * WikiText-2 / WikiText-103 (Language Modelling) * ANKI (Machine Translation) -Aside of providing batches of samples, the Problem class will automatically download the files associated with a given dataset (as long as the dataset is publicly available). -The diversity of those problems (and associated models) proves the flexibility of the framework, we are working on incorporation of new ones into PTP. +Aside of providing batches of samples, the Task class will automatically download the files associated with a given dataset (as long as the dataset is publicly available). +The diversity of those tasks (and associated models) proves the flexibility of the framework, we are working on incorporation of new ones into PTP. **Pipelines:** What people typically define as a _model_ in PTP is framed as a _pipeline_, consisting of many inter-connected components, with one or more _Models_ containing trainable elements. @@ -72,14 +72,14 @@ The framework also offers several components useful when working with text: and several general-purpose components, from tensor transformations (List to Tensor, Reshape Tensor, Reduce Tensor, Concatenate Tensor), to components calculating losses (NLL Loss) and statistics (Accuracy Statistics, Precision/Recall Statistics, BLEU Statistics etc.) to viewers (Stream Viewer, Stream File Exporter etc.). **Workers:** -PTP workers are python scripts that are _agnostic_ to the problems/models/pipelines that they are supposed to work with. +PTP workers are python scripts that are _agnostic_ to the tasks/models/pipelines that they are supposed to work with. Currently framework offers three workers: - * ptp-offline-trainer (a trainer relying on classical methodology interlacing training and validation at the end of every epoch, creates separate instances of training and validation problems and trains the models by feeding the created pipeline with batches of data, relying on the notion of an _epoch_) + * ptp-offline-trainer (a trainer relying on classical methodology interlacing training and validation at the end of every epoch, creates separate instances of training and validation tasks and trains the models by feeding the created pipeline with batches of data, relying on the notion of an _epoch_) - * ptp-online-trainer (a flexible trainer creating separate instances of training and validation problems and training the models by feeding the created pipeline with batches of data, relying on the notion of an _episode_) + * ptp-online-trainer (a flexible trainer creating separate instances of training and validation tasks and training the models by feeding the created pipeline with batches of data, relying on the notion of an _episode_) - * ptp-processor (performing one pass over the all samples returned by a given problem instance, useful for collecting scores on test set, answers for submissions to competitions etc.) + * ptp-processor (performing one pass over the all samples returned by a given task instance, useful for collecting scores on test set, answers for submissions to competitions etc.) ## Installation diff --git a/configs/cifar100/cifar100_classification_convnet_softmax.yml b/configs/cifar100/cifar100_classification_convnet_softmax.yml index c7a0442..bcdae1f 100644 --- a/configs/cifar100/cifar100_classification_convnet_softmax.yml +++ b/configs/cifar100/cifar100_classification_convnet_softmax.yml @@ -1,4 +1,4 @@ -# Load config defining CIFAR100 problems for training, validation and testing. +# Load config defining CIFAR100 tasks for training, validation and testing. default_configs: cifar100/default_cifar100.yml # Definition of the pipeline. diff --git a/configs/cifar100/default_cifar100.yml b/configs/cifar100/default_cifar100.yml index 718ce88..0ffd41f 100644 --- a/configs/cifar100/default_cifar100.yml +++ b/configs/cifar100/default_cifar100.yml @@ -1,6 +1,6 @@ # Training parameters: training: - problem: + task: type: CIFAR100 batch_size: &b 1024 use_train_data: True @@ -25,7 +25,7 @@ training: # Validation parameters: validation: #partial_validation_interval: 100 - problem: + task: type: CIFAR100 batch_size: *b use_train_data: True # True because we are splitting the training set to: validation and training @@ -37,7 +37,7 @@ validation: # Testing parameters: test: - problem: + task: type: MNIST batch_size: *b use_train_data: False @@ -94,7 +94,7 @@ pipeline: image_viewer: priority: 100.5 - type: ImageToClassViewer + type: ImageViewer streams: images: inputs labels: fine_labels diff --git a/configs/clevr/clevr_all_vgg_glove_lstm_concat_ffn.yml b/configs/clevr/clevr_all_vgg_glove_lstm_concat_ffn.yml index da57752..13a53c4 100644 --- a/configs/clevr/clevr_all_vgg_glove_lstm_concat_ffn.yml +++ b/configs/clevr/clevr_all_vgg_glove_lstm_concat_ffn.yml @@ -1,19 +1,19 @@ -# Load config defining CLEVR problems for training, validation and testing. +# Load config defining CLEVR tasks for training, validation and testing. default_configs: clevr/default_clevr.yml # Resize and normalize images - in all sets. training: - problem: + task: resize_image: [224, 224] image_preprocessing: normalize validation: - problem: + task: resize_image: [224, 224] image_preprocessing: normalize test: - problem: + task: resize_image: [224, 224] image_preprocessing: normalize @@ -76,7 +76,7 @@ pipeline: # Image encoder. image_encoder: priority: 2.1 - type: TorchVisionWrapper + type: GenericImageEncoder model_type: vgg16 streams: inputs: images @@ -87,7 +87,7 @@ pipeline: ################################################################## # 3rd subpipeline: concatenation + FF. concat: - type: Concatenation + type: ConcatenateTensor priority: 3.1 input_streams: [question_activations,image_activations] dim: 1 # default diff --git a/configs/clevr/clevr_image_convnet_ffn.yml b/configs/clevr/clevr_image_convnet_ffn.yml index 469008d..6b665ad 100644 --- a/configs/clevr/clevr_image_convnet_ffn.yml +++ b/configs/clevr/clevr_image_convnet_ffn.yml @@ -1,4 +1,4 @@ -# Load config defining CLEVR problems for training, validation and testing. +# Load config defining CLEVR tasks for training, validation and testing. default_configs: clevr/default_clevr.yml # Definition of the pipeline. diff --git a/configs/clevr/clevr_question_glove_lstm.yml b/configs/clevr/clevr_question_glove_lstm.yml index 02a5399..352dc49 100644 --- a/configs/clevr/clevr_question_glove_lstm.yml +++ b/configs/clevr/clevr_question_glove_lstm.yml @@ -1,17 +1,17 @@ -# Load config defining CLEVR problems for training, validation and testing. +# Load config defining CLEVR tasks for training, validation and testing. default_configs: clevr/default_clevr.yml # This is unimodal (questino-based) baseline, thus stop streaming images - in all sets. training: - problem: + task: stream_images: False validation: - problem: + task: stream_images: False test: - problem: + task: stream_images: False # Definition of the pipeline. diff --git a/configs/clevr/default_clevr.yml b/configs/clevr/default_clevr.yml index 16d517d..e4ebec9 100644 --- a/configs/clevr/default_clevr.yml +++ b/configs/clevr/default_clevr.yml @@ -1,6 +1,6 @@ # Training parameters: training: - problem: + task: type: CLEVR batch_size: &b 64 split: training @@ -18,7 +18,7 @@ training: # Validation parameters: validation: - problem: + task: type: CLEVR batch_size: *b split: validation @@ -26,7 +26,7 @@ validation: # Testing parameters: test: - problem: + task: type: CLEVR batch_size: *b split: test @@ -96,7 +96,7 @@ pipeline: #image_viewer: # priority: 100.5 - # type: ImageToClassViewer + # type: ImageViewer # streams: # images: inputs # labels: labels diff --git a/configs/default/components/text/bow_encoder.yml b/configs/default/components/language/bow_encoder.yml similarity index 100% rename from configs/default/components/text/bow_encoder.yml rename to configs/default/components/language/bow_encoder.yml diff --git a/configs/default/components/text/label_indexer.yml b/configs/default/components/language/label_indexer.yml similarity index 98% rename from configs/default/components/text/label_indexer.yml rename to configs/default/components/language/label_indexer.yml index f2c0a48..925dbdc 100644 --- a/configs/default/components/text/label_indexer.yml +++ b/configs/default/components/language/label_indexer.yml @@ -4,7 +4,7 @@ # 1. CONFIGURATION PARAMETERS that will be LOADED by the component. #################################################################### -# Folder where problem will store data (LOADED) +# Folder where task will store data (LOADED) data_folder: '~/data/' # Source files that will be used to create the vocabulary (LOADED) diff --git a/configs/default/components/text/sentence_indexer.yml b/configs/default/components/language/sentence_indexer.yml similarity index 98% rename from configs/default/components/text/sentence_indexer.yml rename to configs/default/components/language/sentence_indexer.yml index c3684e3..a8d0e46 100644 --- a/configs/default/components/text/sentence_indexer.yml +++ b/configs/default/components/language/sentence_indexer.yml @@ -4,7 +4,7 @@ # 1. CONFIGURATION PARAMETERS that will be LOADED by the component. #################################################################### -# Folder where problem will store data (LOADED) +# Folder where task will store data (LOADED) data_folder: '~/data/' # Source files that will be used to create the vocabulary (LOADED) diff --git a/configs/default/components/text/sentence_one_hot_encoder.yml b/configs/default/components/language/sentence_one_hot_encoder.yml similarity index 97% rename from configs/default/components/text/sentence_one_hot_encoder.yml rename to configs/default/components/language/sentence_one_hot_encoder.yml index 17f2be5..b86b303 100644 --- a/configs/default/components/text/sentence_one_hot_encoder.yml +++ b/configs/default/components/language/sentence_one_hot_encoder.yml @@ -4,7 +4,7 @@ # 1. CONFIGURATION PARAMETERS that will be LOADED by the component. #################################################################### -# Folder where problem will store data (LOADED) +# Folder where task will store data (LOADED) data_folder: '~/data/' # Source files that will be used to create the vocabulary (LOADED) diff --git a/configs/default/components/text/sentence_tokenizer.yml b/configs/default/components/language/sentence_tokenizer.yml similarity index 100% rename from configs/default/components/text/sentence_tokenizer.yml rename to configs/default/components/language/sentence_tokenizer.yml diff --git a/configs/default/components/text/word_decoder.yml b/configs/default/components/language/word_decoder.yml similarity index 97% rename from configs/default/components/text/word_decoder.yml rename to configs/default/components/language/word_decoder.yml index 96064ec..8d07760 100644 --- a/configs/default/components/text/word_decoder.yml +++ b/configs/default/components/language/word_decoder.yml @@ -4,7 +4,7 @@ # 1. CONFIGURATION PARAMETERS that will be LOADED by the component. #################################################################### -# Folder where problem will store data (LOADED) +# Folder where task will store data (LOADED) data_folder: '~/data/' # Source files that will be used to create the vocabulary (LOADED) diff --git a/configs/default/components/models/attn_decoder_rnn.yml b/configs/default/components/models/general_usage/attention_decoder.yml similarity index 100% rename from configs/default/components/models/attn_decoder_rnn.yml rename to configs/default/components/models/general_usage/attention_decoder.yml diff --git a/configs/default/components/models/feed_forward_network.yml b/configs/default/components/models/general_usage/feed_forward_network.yml similarity index 100% rename from configs/default/components/models/feed_forward_network.yml rename to configs/default/components/models/general_usage/feed_forward_network.yml diff --git a/configs/default/components/models/recurrent_neural_network.yml b/configs/default/components/models/general_usage/recurrent_neural_network.yml similarity index 100% rename from configs/default/components/models/recurrent_neural_network.yml rename to configs/default/components/models/general_usage/recurrent_neural_network.yml diff --git a/configs/default/components/models/seq2seq_rnn.yml b/configs/default/components/models/general_usage/seq2seq.yml similarity index 100% rename from configs/default/components/models/seq2seq_rnn.yml rename to configs/default/components/models/general_usage/seq2seq.yml diff --git a/configs/default/components/models/index_embeddings.yml b/configs/default/components/models/language/index_embeddings.yml similarity index 100% rename from configs/default/components/models/index_embeddings.yml rename to configs/default/components/models/language/index_embeddings.yml diff --git a/configs/default/components/models/sentence_embeddings.yml b/configs/default/components/models/language/sentence_embeddings.yml similarity index 98% rename from configs/default/components/models/sentence_embeddings.yml rename to configs/default/components/models/language/sentence_embeddings.yml index dceb725..2b95f02 100644 --- a/configs/default/components/models/sentence_embeddings.yml +++ b/configs/default/components/models/language/sentence_embeddings.yml @@ -4,7 +4,7 @@ # 1. CONFIGURATION PARAMETERS that will be LOADED by the component. #################################################################### -# Folder where problem will store data (LOADED) +# Folder where task will store data (LOADED) data_folder: '~/data/' # Source files that will be used to create the vocabulary (LOADED) diff --git a/configs/default/components/models/vqa/multimodal_compact_bilinear_pooling.yml b/configs/default/components/models/multi_modal_reasoning/compact_bilinear_pooling.yml similarity index 100% rename from configs/default/components/models/vqa/multimodal_compact_bilinear_pooling.yml rename to configs/default/components/models/multi_modal_reasoning/compact_bilinear_pooling.yml diff --git a/configs/default/components/models/vqa/multimodal_factorized_bilinear_pooling.yml b/configs/default/components/models/multi_modal_reasoning/factorized_bilinear_pooling.yml similarity index 95% rename from configs/default/components/models/vqa/multimodal_factorized_bilinear_pooling.yml rename to configs/default/components/models/multi_modal_reasoning/factorized_bilinear_pooling.yml index 1c55fe9..9cf5874 100644 --- a/configs/default/components/models/vqa/multimodal_factorized_bilinear_pooling.yml +++ b/configs/default/components/models/multi_modal_reasoning/factorized_bilinear_pooling.yml @@ -1,4 +1,4 @@ -# This file defines the default values for the MultimodalFactorizedBilinearPooling model. +# This file defines the default values for the FactorizedBilinearPooling model. #################################################################### # 1. CONFIGURATION PARAMETERS that will be LOADED by the component. diff --git a/configs/default/components/models/vqa/element_wise_multiplication.yml b/configs/default/components/models/multi_modal_reasoning/low_rank_bilinear_pooling.yml similarity index 95% rename from configs/default/components/models/vqa/element_wise_multiplication.yml rename to configs/default/components/models/multi_modal_reasoning/low_rank_bilinear_pooling.yml index f0f02d8..2c38456 100644 --- a/configs/default/components/models/vqa/element_wise_multiplication.yml +++ b/configs/default/components/models/multi_modal_reasoning/low_rank_bilinear_pooling.yml @@ -1,4 +1,4 @@ -# This file defines the default values for the ElementWiseMultiplication model. +# This file defines the default values for the LowRankBilinearPooling model. #################################################################### # 1. CONFIGURATION PARAMETERS that will be LOADED by the component. diff --git a/configs/default/components/models/vqa/attention.yml b/configs/default/components/models/multi_modal_reasoning/question_driven_attention.yml similarity index 96% rename from configs/default/components/models/vqa/attention.yml rename to configs/default/components/models/multi_modal_reasoning/question_driven_attention.yml index 4a65a76..ce883b8 100644 --- a/configs/default/components/models/vqa/attention.yml +++ b/configs/default/components/models/multi_modal_reasoning/question_driven_attention.yml @@ -1,4 +1,4 @@ -# This file defines the default values for the VQA_Attention model. +# This file defines the default values for the QuestionDrivenAttention model. #################################################################### # 1. CONFIGURATION PARAMETERS that will be LOADED by the component. diff --git a/configs/default/components/models/vqa/relational_network.yml b/configs/default/components/models/multi_modal_reasoning/relational_network.yml similarity index 96% rename from configs/default/components/models/vqa/relational_network.yml rename to configs/default/components/models/multi_modal_reasoning/relational_network.yml index 1d19ec2..2b5604c 100644 --- a/configs/default/components/models/vqa/relational_network.yml +++ b/configs/default/components/models/multi_modal_reasoning/relational_network.yml @@ -1,4 +1,4 @@ -# This file defines the default values for the ElementWiseMultiplication model. +# This file defines the default values for the LowRankBilinearPooling model. #################################################################### # 1. CONFIGURATION PARAMETERS that will be LOADED by the component. diff --git a/configs/default/components/models/vqa/self_attention.yml b/configs/default/components/models/multi_modal_reasoning/self_attention.yml similarity index 100% rename from configs/default/components/models/vqa/self_attention.yml rename to configs/default/components/models/multi_modal_reasoning/self_attention.yml diff --git a/configs/default/components/models/convnet_encoder.yml b/configs/default/components/models/vision/convnet_encoder.yml similarity index 100% rename from configs/default/components/models/convnet_encoder.yml rename to configs/default/components/models/vision/convnet_encoder.yml diff --git a/configs/default/components/models/torch_vision_wrapper.yml b/configs/default/components/models/vision/generic_image_encoder.yml similarity index 100% rename from configs/default/components/models/torch_vision_wrapper.yml rename to configs/default/components/models/vision/generic_image_encoder.yml diff --git a/configs/default/components/models/lenet5.yml b/configs/default/components/models/vision/lenet5.yml similarity index 100% rename from configs/default/components/models/lenet5.yml rename to configs/default/components/models/vision/lenet5.yml diff --git a/configs/default/components/publishers/accuracy_statistics.yml b/configs/default/components/statistics/accuracy_statistics.yml similarity index 100% rename from configs/default/components/publishers/accuracy_statistics.yml rename to configs/default/components/statistics/accuracy_statistics.yml diff --git a/configs/default/components/publishers/batch_size_statistics.yml b/configs/default/components/statistics/batch_size_statistics.yml similarity index 100% rename from configs/default/components/publishers/batch_size_statistics.yml rename to configs/default/components/statistics/batch_size_statistics.yml diff --git a/configs/default/components/publishers/bleu_statistics.yml b/configs/default/components/statistics/bleu_statistics.yml similarity index 100% rename from configs/default/components/publishers/bleu_statistics.yml rename to configs/default/components/statistics/bleu_statistics.yml diff --git a/configs/default/components/publishers/precision_recall_statistics.yml b/configs/default/components/statistics/precision_recall_statistics.yml similarity index 100% rename from configs/default/components/publishers/precision_recall_statistics.yml rename to configs/default/components/statistics/precision_recall_statistics.yml diff --git a/configs/default/components/problems/image_text_to_class/clevr.yml b/configs/default/components/tasks/image_text_to_class/clevr.yml similarity index 91% rename from configs/default/components/problems/image_text_to_class/clevr.yml rename to configs/default/components/tasks/image_text_to_class/clevr.yml index 0d5b571..c1fe428 100644 --- a/configs/default/components/problems/image_text_to_class/clevr.yml +++ b/configs/default/components/tasks/image_text_to_class/clevr.yml @@ -1,17 +1,17 @@ -# This file defines the default values for the CLEVR problem. +# This file defines the default values for the CLEVR task. #################################################################### # 1. CONFIGURATION PARAMETERS that will be LOADED by the component. #################################################################### -# Folder where problem will store data (LOADED) +# Folder where task will store data (LOADED) data_folder: '~/data/CLEVR_v1.0' # Defines the set (split) that will be used (LOADED) # Options: training | validation | test | cogent_a_training | cogent_a_validation | cogent_b_validation split: training -# Flag indicating whether the problem will load and return images (LOADED) +# Flag indicating whether the task will load and return images (LOADED) stream_images: True # Resize parameter (LOADED) @@ -31,7 +31,7 @@ streams: #################################################################### # Stream containing batch of indices (OUTPUT) - # Every problem MUST return that stream. + # Every task MUST return that stream. indices: indices # Stream containing batch of images (OUTPUT) diff --git a/configs/default/components/problems/image_text_to_class/gqa.yml b/configs/default/components/tasks/image_text_to_class/gqa.yml similarity index 91% rename from configs/default/components/problems/image_text_to_class/gqa.yml rename to configs/default/components/tasks/image_text_to_class/gqa.yml index e1c662f..ed9929e 100644 --- a/configs/default/components/problems/image_text_to_class/gqa.yml +++ b/configs/default/components/tasks/image_text_to_class/gqa.yml @@ -1,10 +1,10 @@ -# This file defines the default values for the GQA problem. +# This file defines the default values for the GQA task. #################################################################### # 1. CONFIGURATION PARAMETERS that will be LOADED by the component. #################################################################### -# Folder where problem will store data (LOADED) +# Folder where task will store data (LOADED) data_folder: '~/data/gqa' # Defines the set (split) that will be used (LOADED) @@ -12,7 +12,7 @@ data_folder: '~/data/gqa' # Note: test_dev should be used for validation. split: training_0 -# Flag indicating whether the problem will load and return images (LOADED) +# Flag indicating whether the task will load and return images (LOADED) stream_images: True # Resize parameter (LOADED) @@ -32,7 +32,7 @@ streams: #################################################################### # Stream containing batch of indices (OUTPUT) - # Every problem MUST return that stream. + # Every task MUST return that stream. indices: indices # Stream containing batch of sample (original) identifiers (OUTPUT) diff --git a/configs/default/components/problems/image_text_to_class/vqa_med_2019.yml b/configs/default/components/tasks/image_text_to_class/vqa_med_2019.yml similarity index 91% rename from configs/default/components/problems/image_text_to_class/vqa_med_2019.yml rename to configs/default/components/tasks/image_text_to_class/vqa_med_2019.yml index bc66b0c..a354e9e 100644 --- a/configs/default/components/problems/image_text_to_class/vqa_med_2019.yml +++ b/configs/default/components/tasks/image_text_to_class/vqa_med_2019.yml @@ -1,10 +1,10 @@ -# This file defines the default values for the VQAMED2019 problem. +# This file defines the default values for the VQAMED2019 task. #################################################################### # 1. CONFIGURATION PARAMETERS that will be LOADED by the component. #################################################################### -# Folder where problem will store data (LOADED) +# Folder where task will store data (LOADED) data_folder: '~/data/vqa-med' # Defines the set (split) that will be used (LOADED) @@ -15,7 +15,7 @@ split: training # Options: all | c1 | c2 | c3 | c4 (or any combination of the latter 4) categories: all -# Flag indicating whether the problem will load and return images (LOADED) +# Flag indicating whether the task will load and return images (LOADED) stream_images: True # Flag indicating whether images will be preloaded (i.e. loaded once at start) (LOADED) @@ -29,7 +29,7 @@ preload_images: False #resize_image: [height, width] # Scale parameter [height, width] (LOADED) -# Problem will use those values to rescale the image_sizes to range (0, 1). +# Task will use those values to rescale the image_sizes to range (0, 1). scale_image_size: [2414, 2323] # Select applied image preprocessing/augmentations (LOADED) @@ -50,7 +50,7 @@ question_preprocessing: lowercase, remove_punctuation # Accepted formats: a,b,c or [a,b,c] answer_preprocessing: none -# When filename is not empty, problem will calculate weights associated with all samples +# When filename is not empty, task will calculate weights associated with all samples # by looking at the distribution of all answers from all loaded samples (LOADED) # Those weights can be next used by weighted samplers (e.g. kFoldWeightedSampler) export_sample_weights: '' @@ -78,7 +78,7 @@ streams: #################################################################### # Stream containing batch of indices (OUTPUT) - # Every problem MUST return that stream. + # Every task MUST return that stream. indices: indices # Stream containing batch of images (OUTPUT) diff --git a/configs/default/components/problems/image_to_class/cifar_100.yml b/configs/default/components/tasks/image_to_class/cifar_100.yml similarity index 93% rename from configs/default/components/problems/image_to_class/cifar_100.yml rename to configs/default/components/tasks/image_to_class/cifar_100.yml index 957b422..c0722aa 100644 --- a/configs/default/components/problems/image_to_class/cifar_100.yml +++ b/configs/default/components/tasks/image_to_class/cifar_100.yml @@ -1,10 +1,10 @@ -# This file defines the default values for the CIFAR-100 problem. +# This file defines the default values for the CIFAR-100 task. #################################################################### # 1. CONFIGURATION PARAMETERS that will be LOADED by the component. #################################################################### -# Folder where problem will store data (LOADED) +# Folder where task will store data (LOADED) data_folder: '~/data/cifar-100' # Defines the set that will be used used (LOADED) @@ -21,7 +21,7 @@ streams: #################################################################### # Stream containing batch of indices (OUTPUT) - # Every problem MUST return that stream. + # Every task MUST return that stream. indices: indices # Stream containing batch of images (OUTPUT) diff --git a/configs/default/components/problems/image_to_class/mnist.yml b/configs/default/components/tasks/image_to_class/mnist.yml similarity index 92% rename from configs/default/components/problems/image_to_class/mnist.yml rename to configs/default/components/tasks/image_to_class/mnist.yml index 3dacab6..2e4bd8c 100644 --- a/configs/default/components/problems/image_to_class/mnist.yml +++ b/configs/default/components/tasks/image_to_class/mnist.yml @@ -1,10 +1,10 @@ -# This file defines the default values for the MNIST problem. +# This file defines the default values for the MNIST task. #################################################################### # 1. CONFIGURATION PARAMETERS that will be LOADED by the component. #################################################################### -# Folder where problem will store data (LOADED) +# Folder where task will store data (LOADED) data_folder: '~/data/mnist' # Defines the set that will be used used (LOADED) @@ -21,7 +21,7 @@ streams: #################################################################### # Stream containing batch of indices (OUTPUT) - # Every problem MUST return that stream. + # Every task MUST return that stream. indices: indices # Stream containing batch of images (OUTPUT) diff --git a/configs/default/components/problems/text_to_class/dummy_language_identification.yml b/configs/default/components/tasks/text_to_class/dummy_language_identification.yml similarity index 89% rename from configs/default/components/problems/text_to_class/dummy_language_identification.yml rename to configs/default/components/tasks/text_to_class/dummy_language_identification.yml index 1f9cd3e..9af05c7 100644 --- a/configs/default/components/problems/text_to_class/dummy_language_identification.yml +++ b/configs/default/components/tasks/text_to_class/dummy_language_identification.yml @@ -1,17 +1,17 @@ -# This file defines the default values for the dummy language identification problem. +# This file defines the default values for the dummy language identification task. #################################################################### # 1. CONFIGURATION PARAMETERS that will be LOADED by the component. #################################################################### -# Folder where problem will store data (LOADED) +# Folder where task will store data (LOADED) data_folder: '~/data/language_identification/dummy' # Defines the set that will be used used (LOADED) # True: training set | False: test set. use_train_data: True -# If set, problem will always (re)generate data (LOADED) +# If set, task will always (re)generate data (LOADED) regenerate: False streams: @@ -20,7 +20,7 @@ streams: #################################################################### # Stream containing batch of indices (OUTPUT) - # Every problem MUST return that stream. + # Every task MUST return that stream. indices: indices # Stream containing batch of images (OUTPUT) diff --git a/configs/default/components/problems/text_to_class/wily_language_identification.yml b/configs/default/components/tasks/text_to_class/wily_language_identification.yml similarity index 89% rename from configs/default/components/problems/text_to_class/wily_language_identification.yml rename to configs/default/components/tasks/text_to_class/wily_language_identification.yml index 95c9306..b2f03ad 100644 --- a/configs/default/components/problems/text_to_class/wily_language_identification.yml +++ b/configs/default/components/tasks/text_to_class/wily_language_identification.yml @@ -1,17 +1,17 @@ -# This file defines the default values for the WiLY language identification problem. +# This file defines the default values for the WiLY language identification task. #################################################################### # 1. CONFIGURATION PARAMETERS that will be LOADED by the component. #################################################################### -# Folder where problem will store data (LOADED) +# Folder where task will store data (LOADED) data_folder: '~/data/language_identification/wily' # Defines the set that will be used used (LOADED) # True: training set | False: test set. use_train_data: True -# If set, problem will always (re)generate data (LOADED) +# If set, task will always (re)generate data (LOADED) regenerate: False streams: @@ -20,7 +20,7 @@ streams: #################################################################### # Stream containing batch of indices (OUTPUT) - # Every problem MUST return that stream. + # Every task MUST return that stream. indices: indices # Stream containing batch of images (OUTPUT) diff --git a/configs/default/components/problems/text_to_class/wily_ngram_language_modeling.yml b/configs/default/components/tasks/text_to_class/wily_ngram_language_modeling.yml similarity index 94% rename from configs/default/components/problems/text_to_class/wily_ngram_language_modeling.yml rename to configs/default/components/tasks/text_to_class/wily_ngram_language_modeling.yml index 5d7dd6f..715ab96 100644 --- a/configs/default/components/problems/text_to_class/wily_ngram_language_modeling.yml +++ b/configs/default/components/tasks/text_to_class/wily_ngram_language_modeling.yml @@ -5,7 +5,7 @@ # 1. CONFIGURATION PARAMETERS that will be LOADED by the component. #################################################################### -# Folder where problem will store data (LOADED) +# Folder where task will store data (LOADED) data_folder: '~/data/language_identification/wily' # Defines the set that will be used used (LOADED) @@ -21,7 +21,7 @@ streams: #################################################################### # Stream containing batch of indices (OUTPUT) - # Every problem MUST return that stream. + # Every task MUST return that stream. indices: indices # Stream containing batch of images (OUTPUT) diff --git a/configs/default/components/problems/text_to_text/translation_pairs.yml b/configs/default/components/tasks/text_to_text/translation_pairs.yml similarity index 94% rename from configs/default/components/problems/text_to_text/translation_pairs.yml rename to configs/default/components/tasks/text_to_text/translation_pairs.yml index f48f650..e118a8d 100644 --- a/configs/default/components/problems/text_to_text/translation_pairs.yml +++ b/configs/default/components/tasks/text_to_text/translation_pairs.yml @@ -4,7 +4,7 @@ # 1. CONFIGURATION PARAMETERS that will be LOADED by the component. #################################################################### -# Folder where problem will store data (LOADED) +# Folder where task will store data (LOADED) data_folder: ~/data/language_modeling/translation_pairs # Defines the dataset that will be used used (LOADED) @@ -25,7 +25,7 @@ streams: #################################################################### # Stream containing batch of indices (OUTPUT) - # Every problem MUST return that stream. + # Every task MUST return that stream. indices: indices # Stream containing batch of tokenized source sentences (OUTPUT) diff --git a/configs/default/components/problems/text_to_text/wikitext_language_modeling.yml b/configs/default/components/tasks/text_to_text/wikitext_language_modeling.yml similarity index 94% rename from configs/default/components/problems/text_to_text/wikitext_language_modeling.yml rename to configs/default/components/tasks/text_to_text/wikitext_language_modeling.yml index 763db2a..bf607ed 100644 --- a/configs/default/components/problems/text_to_text/wikitext_language_modeling.yml +++ b/configs/default/components/tasks/text_to_text/wikitext_language_modeling.yml @@ -4,7 +4,7 @@ # 1. CONFIGURATION PARAMETERS that will be LOADED by the component. #################################################################### -# Folder where problem will store data (LOADED) +# Folder where task will store data (LOADED) data_folder: ~/data/language_modeling/wikitext # Defines the dataset that will be used used (LOADED) @@ -24,7 +24,7 @@ streams: #################################################################### # Stream containing batch of indices (OUTPUT) - # Every problem MUST return that stream. + # Every task MUST return that stream. indices: indices # Stream containing batch of tokenized source sentences (OUTPUT) diff --git a/configs/default/components/transforms/concatenation.yml b/configs/default/components/transforms/concatenate_tensor.yml similarity index 96% rename from configs/default/components/transforms/concatenation.yml rename to configs/default/components/transforms/concatenate_tensor.yml index d91427f..03c5db1 100644 --- a/configs/default/components/transforms/concatenation.yml +++ b/configs/default/components/transforms/concatenate_tensor.yml @@ -1,4 +1,4 @@ -# This file defines the default values for the Tensor Reshaper. +# This file defines the default values for the ConcatenateTensor. #################################################################### # 1. CONFIGURATION PARAMETERS that will be LOADED by the component. diff --git a/configs/default/components/transforms/non_linearity.yml b/configs/default/components/transforms/non_linearity.yml index 18d77dc..904dd25 100644 --- a/configs/default/components/transforms/non_linearity.yml +++ b/configs/default/components/transforms/non_linearity.yml @@ -1,4 +1,4 @@ -# This file defines the default values for the Tensor Reshaper. +# This file defines the default values for the NonLinearity. #################################################################### # 1. CONFIGURATION PARAMETERS that will be LOADED by the component. diff --git a/configs/default/components/viewers/image_to_class_viewer.yml b/configs/default/components/viewers/image_viewer.yml similarity index 95% rename from configs/default/components/viewers/image_to_class_viewer.yml rename to configs/default/components/viewers/image_viewer.yml index b43b7bd..4b2a8b5 100644 --- a/configs/default/components/viewers/image_to_class_viewer.yml +++ b/configs/default/components/viewers/image_viewer.yml @@ -1,4 +1,4 @@ -# This file defines the default values for the ImageToClassViewer. +# This file defines the default values for the ImageViewer. #################################################################### # 1. CONFIGURATION PARAMETERS that will be LOADED by the component. diff --git a/configs/default/workers/offline_trainer.yml b/configs/default/workers/offline_trainer.yml index 110231c..e5616b2 100644 --- a/configs/default/workers/offline_trainer.yml +++ b/configs/default/workers/offline_trainer.yml @@ -11,17 +11,17 @@ default_training: # Default batch size. batch_size: 64 - # Definition of the problem (Mandatory!) - #problem: + # Definition of the task (Mandatory!) + #task: # One must define its type (Mandatory!) # type: ? - # The rest of the content of that section is problem-specific... + # The rest of the content of that section is task-specific... # Section describing curriculum learning (Optional) #curriculum_learning: # # Flag indicating whether curriculum learning has to finish before (eventual) termination of the training. # must_finish: True - # The rest of the content of that section is problem-specific... + # The rest of the content of that section is task-specific... # Definition of optimizer (Mandatory!) #optimizer: @@ -74,11 +74,11 @@ default_validation: # In this trainer Partial Validation is optional (negative value means it is disabled) partial_validation_interval: -1 - # Definition of the problem (mandatory!) - #problem: + # Definition of the task (mandatory!) + #task: # One must define its type (Mandatory!) # type: ? - # The rest of the content of that section is problem-specific... + # The rest of the content of that section is task-specific... # Set a default configuration section for data loader. dataloader: diff --git a/configs/default/workers/online_trainer.yml b/configs/default/workers/online_trainer.yml index 07532d6..924ff05 100644 --- a/configs/default/workers/online_trainer.yml +++ b/configs/default/workers/online_trainer.yml @@ -11,17 +11,17 @@ default_training: # Default batch size. batch_size: 64 - # Definition of the problem (Mandatory!) - #problem: + # Definition of the task (Mandatory!) + #task: # One must define its type (Mandatory!) # type: ? - # The rest of the content of that section is problem-specific... + # The rest of the content of that section is task-specific... # Section describing curriculum learning (Optional) #curriculum_learning: # # Flag indicating whether curriculum learning has to finish before (eventual) termination of the training. # must_finish: True - # The rest of the content of that section is problem-specific... + # The rest of the content of that section is task-specific... # Definition of optimizer (Mandatory!) #optimizer: @@ -74,11 +74,11 @@ default_validation: # In this trainer Partial Validation is mandatory, hence interval must be > 0. partial_validation_interval: 100 - # Definition of the problem (mandatory!) - #problem: + # Definition of the task (mandatory!) + #task: # One must define its type (Mandatory!) # type: ? - # The rest of the content of that section is problem-specific... + # The rest of the content of that section is task-specific... # Set a default configuration section for data loader. dataloader: diff --git a/configs/default/workers/processor.yml b/configs/default/workers/processor.yml index ed1ed28..b4a12f8 100644 --- a/configs/default/workers/processor.yml +++ b/configs/default/workers/processor.yml @@ -10,11 +10,11 @@ default_test: # Default batch size. batch_size: 64 - # Definition of the problem (Mandatory!) - #problem: + # Definition of the task (Mandatory!) + #task: # One must define its type (Mandatory!) # type: ? - # The rest of the content of that section is problem-specific... + # The rest of the content of that section is task-specific... # Set a default configuration section for data loader. dataloader: diff --git a/configs/mnist/default_mnist.yml b/configs/mnist/default_mnist.yml index 6cf6c94..4d56066 100644 --- a/configs/mnist/default_mnist.yml +++ b/configs/mnist/default_mnist.yml @@ -1,6 +1,6 @@ # Training parameters: training: - problem: + task: type: MNIST batch_size: &b 64 use_train_data: True @@ -23,7 +23,7 @@ training: # Validation parameters: validation: #partial_validation_interval: 100 - problem: + task: type: MNIST batch_size: *b use_train_data: True # True because we are splitting the training set to: validation and training @@ -35,7 +35,7 @@ validation: # Testing parameters: test: - problem: + task: type: MNIST batch_size: *b use_train_data: False @@ -84,7 +84,7 @@ pipeline: image_viewer: priority: 100.5 - type: ImageToClassViewer + type: ImageViewer streams: images: inputs labels: labels diff --git a/configs/mnist/mnist_classification_convnet_softmax.yml b/configs/mnist/mnist_classification_convnet_softmax.yml index 8b2580f..ab2ceef 100644 --- a/configs/mnist/mnist_classification_convnet_softmax.yml +++ b/configs/mnist/mnist_classification_convnet_softmax.yml @@ -1,4 +1,4 @@ -# Load config defining MNIST problems for training, validation and testing. +# Load config defining MNIST tasks for training, validation and testing. default_configs: mnist/default_mnist.yml pipeline: diff --git a/configs/mnist/mnist_classification_kfold_softmax.yml b/configs/mnist/mnist_classification_kfold_softmax.yml index 9685e82..5107aed 100644 --- a/configs/mnist/mnist_classification_kfold_softmax.yml +++ b/configs/mnist/mnist_classification_kfold_softmax.yml @@ -1,9 +1,9 @@ -# Load config defining MNIST problems for training, validation and testing. +# Load config defining MNIST tasks for training, validation and testing. default_configs: mnist/default_mnist.yml # Training parameters: training: - problem: + task: type: MNIST batch_size: &b 64 use_train_data: True @@ -24,7 +24,7 @@ training: # Validation parameters: validation: partial_validation_interval: 500 - problem: + task: type: MNIST batch_size: *b use_train_data: True # True because we are splitting the training set to: validation and training diff --git a/configs/mnist/mnist_classification_lenet5.yml b/configs/mnist/mnist_classification_lenet5.yml index 1dded53..98b7561 100644 --- a/configs/mnist/mnist_classification_lenet5.yml +++ b/configs/mnist/mnist_classification_lenet5.yml @@ -1,19 +1,19 @@ -# Load config defining MNIST problems for training, validation and testing. +# Load config defining MNIST tasks for training, validation and testing. default_configs: mnist/default_mnist.yml # Training parameters - overwrite defaults: training: - problem: + task: resize_image: [32, 32] # Validation parameters - overwrite defaults: validation: - problem: + task: resize_image: [32, 32] # Testing parameters - overwrite defaults: test: - problem: + task: resize_image: [32, 32] # Definition of the pipeline. diff --git a/configs/mnist/mnist_classification_softmax.yml b/configs/mnist/mnist_classification_softmax.yml index 1875f42..f225470 100644 --- a/configs/mnist/mnist_classification_softmax.yml +++ b/configs/mnist/mnist_classification_softmax.yml @@ -1,4 +1,4 @@ -# Load config defining MNIST problems for training, validation and testing. +# Load config defining MNIST tasks for training, validation and testing. default_configs: mnist/default_mnist.yml pipeline: diff --git a/configs/mnist/mnist_classification_vf_2lenet5_2losses.yml b/configs/mnist/mnist_classification_vf_2lenet5_2losses.yml index 783283b..21c49d3 100644 --- a/configs/mnist/mnist_classification_vf_2lenet5_2losses.yml +++ b/configs/mnist/mnist_classification_vf_2lenet5_2losses.yml @@ -1,19 +1,19 @@ -# Load config defining MNIST problems for training, validation and testing. +# Load config defining MNIST tasks for training, validation and testing. default_configs: mnist/default_mnist.yml # Training parameters - overwrite defaults: training: - problem: + task: resize_image: [32, 32] # Validation parameters - overwrite defaults: validation: - problem: + task: resize_image: [32, 32] # Testing parameters - overwrite defaults: test: - problem: + task: resize_image: [32, 32] # Definition of the pipeline. @@ -182,7 +182,7 @@ pipeline: input_mask_streams: [flow1_masks, flow2_masks] input_word_mappings: [word_to_ix1, word_to_ix2] globals: - output_word_mappings: label_word_mappings # from MNIST problem. + output_word_mappings: label_word_mappings # from MNIST task. streams: output_strings: merged_predictions output_indices: merged_indices diff --git a/configs/mnist/mnist_classification_vf_shared_convnet_2softmaxes_2losses.yml b/configs/mnist/mnist_classification_vf_shared_convnet_2softmaxes_2losses.yml index efea8a4..7192e81 100644 --- a/configs/mnist/mnist_classification_vf_shared_convnet_2softmaxes_2losses.yml +++ b/configs/mnist/mnist_classification_vf_shared_convnet_2softmaxes_2losses.yml @@ -1,9 +1,9 @@ -# Load config defining MNIST problems for training, validation and testing. +# Load config defining MNIST tasks for training, validation and testing. default_configs: mnist/default_mnist.yml # Training parameters - overwrite defaults: training: - problem: + task: #resize_image: [32, 32] batch_size: 64 #optimizer: @@ -15,12 +15,12 @@ training: # Validation parameters - overwrite defaults: #validation: # partial_validation_interval: 10 -# problem: +# task: # resize_image: [32, 32] # Testing parameters - overwrite defaults: #test: -# problem: +# task: # resize_image: [32, 32] # Definition of the pipeline. @@ -213,7 +213,7 @@ pipeline: input_mask_streams: [flow1_masks, flow2_masks] input_word_mappings: [word_to_ix1, word_to_ix2] globals: - output_word_mappings: label_word_mappings # from MNIST problem. + output_word_mappings: label_word_mappings # from MNIST task. streams: output_strings: merged_predictions output_indices: merged_indices diff --git a/configs/translation/eng_fra_translation_enc_attndec.yml b/configs/translation/eng_fra_translation_enc_attndec.yml index a299011..7c80e2c 100644 --- a/configs/translation/eng_fra_translation_enc_attndec.yml +++ b/configs/translation/eng_fra_translation_enc_attndec.yml @@ -4,7 +4,7 @@ # Training parameters: training: - problem: + task: type: &p_type TranslationPairs data_folder: &data_folder ~/data/language_modeling/translation_pairs dataset: &dataset eng-fra @@ -26,7 +26,7 @@ training: # Validation parameters: validation: partial_validation_interval: 100 - problem: + task: type: *p_type data_folder: *data_folder dataset: *dataset @@ -36,7 +36,7 @@ validation: # Testing parameters: test: - problem: + task: type: *p_type data_folder: *data_folder dataset: *dataset @@ -104,7 +104,7 @@ pipeline: # Single layer GRU Decoder with attention decoder: - type: Attn_Decoder_RNN + type: AttentionDecoder priority: 4 hidden_size: 50 use_logsoftmax: False diff --git a/configs/vqa_med_2019/c1_classification/c1_classification_all_bow_vgg16_concat.yml b/configs/vqa_med_2019/c1_classification/c1_classification_all_bow_vgg16_concat.yml index 17d7b75..66c65c8 100644 --- a/configs/vqa_med_2019/c1_classification/c1_classification_all_bow_vgg16_concat.yml +++ b/configs/vqa_med_2019/c1_classification/c1_classification_all_bow_vgg16_concat.yml @@ -1,4 +1,4 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/c1_classification/default_c1_classification.yml pipeline: @@ -56,7 +56,7 @@ pipeline: # 3rd subpipeline: image. # Image encoder. image_encoder: - type: TorchVisionWrapper + type: GenericImageEncoder priority: 3.1 streams: inputs: images @@ -66,10 +66,10 @@ pipeline: # 4th subpipeline: concatenation + FF. concat: - type: Concatenation + type: ConcatenateTensor priority: 4.1 input_streams: [question_activations,image_size_activations,image_activations] - # Concatenation + # ConcatenateTensor dim: 1 # default input_dims: [[-1,99],[-1,10],[-1,100]] output_dims: [-1,209] diff --git a/configs/vqa_med_2019/c1_classification/c1_classification_all_rnn_vgg16_concat.yml b/configs/vqa_med_2019/c1_classification/c1_classification_all_rnn_vgg16_concat.yml index b8d9dfc..ff3f5e1 100644 --- a/configs/vqa_med_2019/c1_classification/c1_classification_all_rnn_vgg16_concat.yml +++ b/configs/vqa_med_2019/c1_classification/c1_classification_all_rnn_vgg16_concat.yml @@ -1,4 +1,4 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/c1_classification/default_c1_classification.yml pipeline: @@ -63,7 +63,7 @@ pipeline: # 3rd subpipeline: image. # Image encoder. image_encoder: - type: TorchVisionWrapper + type: GenericImageEncoder priority: 3.1 streams: inputs: images @@ -73,10 +73,10 @@ pipeline: # 4th subpipeline: concatenation + FF. concat: - type: Concatenation + type: ConcatenateTensor priority: 4.1 input_streams: [question_activations,image_size_activations,image_activations] - # Concatenation + # ConcatenateTensor dim: 1 # default input_dims: [[-1,100],[-1,10],[-1,100]] output_dims: [-1,210] diff --git a/configs/vqa_med_2019/c1_classification/c1_classification_image_cnn_softmax.yml b/configs/vqa_med_2019/c1_classification/c1_classification_image_cnn_softmax.yml index 623a5e4..afc250f 100644 --- a/configs/vqa_med_2019/c1_classification/c1_classification_image_cnn_softmax.yml +++ b/configs/vqa_med_2019/c1_classification/c1_classification_image_cnn_softmax.yml @@ -1,4 +1,4 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/c1_classification/default_c1_classification.yml pipeline: diff --git a/configs/vqa_med_2019/c1_classification/c1_classification_image_size_softmax.yml b/configs/vqa_med_2019/c1_classification/c1_classification_image_size_softmax.yml index 0201388..c119df6 100644 --- a/configs/vqa_med_2019/c1_classification/c1_classification_image_size_softmax.yml +++ b/configs/vqa_med_2019/c1_classification/c1_classification_image_size_softmax.yml @@ -1,4 +1,4 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/c1_classification/default_c1_classification.yml pipeline: diff --git a/configs/vqa_med_2019/c1_classification/c1_classification_question_mimic_rnn.yml b/configs/vqa_med_2019/c1_classification/c1_classification_question_mimic_rnn.yml index 4f137e1..0cf7ae3 100644 --- a/configs/vqa_med_2019/c1_classification/c1_classification_question_mimic_rnn.yml +++ b/configs/vqa_med_2019/c1_classification/c1_classification_question_mimic_rnn.yml @@ -1,4 +1,4 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/c1_classification/default_c1_classification.yml pipeline: diff --git a/configs/vqa_med_2019/c1_classification/c1_classification_question_onehot_bow.yml b/configs/vqa_med_2019/c1_classification/c1_classification_question_onehot_bow.yml index d879a86..78d7672 100644 --- a/configs/vqa_med_2019/c1_classification/c1_classification_question_onehot_bow.yml +++ b/configs/vqa_med_2019/c1_classification/c1_classification_question_onehot_bow.yml @@ -1,4 +1,4 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/c1_classification/default_c1_classification.yml pipeline: diff --git a/configs/vqa_med_2019/c1_classification/c1_classification_question_rnn.yml b/configs/vqa_med_2019/c1_classification/c1_classification_question_rnn.yml index 0beba15..47f8dbd 100644 --- a/configs/vqa_med_2019/c1_classification/c1_classification_question_rnn.yml +++ b/configs/vqa_med_2019/c1_classification/c1_classification_question_rnn.yml @@ -1,4 +1,4 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/c1_classification/default_c1_classification.yml pipeline: diff --git a/configs/vqa_med_2019/c1_classification/c1_classification_vf_question_rnn_separate_q_categorization.yml b/configs/vqa_med_2019/c1_classification/c1_classification_vf_question_rnn_separate_q_categorization.yml index cce673d..12d9398 100644 --- a/configs/vqa_med_2019/c1_classification/c1_classification_vf_question_rnn_separate_q_categorization.yml +++ b/configs/vqa_med_2019/c1_classification/c1_classification_vf_question_rnn_separate_q_categorization.yml @@ -1,4 +1,4 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/c1_classification/default_c1_classification.yml pipeline: diff --git a/configs/vqa_med_2019/c1_classification/default_c1_classification.yml b/configs/vqa_med_2019/c1_classification/default_c1_classification.yml index fa26a80..f5246ea 100644 --- a/configs/vqa_med_2019/c1_classification/default_c1_classification.yml +++ b/configs/vqa_med_2019/c1_classification/default_c1_classification.yml @@ -1,9 +1,9 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/default_vqa_med_2019.yml # Training parameters: training: - problem: + task: categories: C1 export_sample_weights: ~/data/vqa-med/answers.c1.weights.csv sampler: @@ -11,7 +11,7 @@ training: # Validation parameters: validation: - problem: + task: categories: C1 diff --git a/configs/vqa_med_2019/c2_classification/c2_class_lstm_resnet152_ewm_cat_is.yml b/configs/vqa_med_2019/c2_classification/c2_class_lstm_resnet152_ewm_cat_is.yml index b27aea1..78624b3 100644 --- a/configs/vqa_med_2019/c2_classification/c2_class_lstm_resnet152_ewm_cat_is.yml +++ b/configs/vqa_med_2019/c2_classification/c2_class_lstm_resnet152_ewm_cat_is.yml @@ -1,8 +1,8 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/c2_classification/default_c2_classification.yml training: - problem: + task: batch_size: 48 # Appy all preprocessing/data augmentations. image_preprocessing: normalize @@ -10,16 +10,16 @@ training: question_preprocessing: lowercase,remove_punctuation,tokenize # none | lowercase | remove_punctuation | tokenize | random_remove_stop_words | random_shuffle_words | all streams: - # Problem is returning tokenized questions. + # Task is returning tokenized questions. questions: tokenized_questions validation: - problem: + task: batch_size: 48 question_preprocessing: lowercase,remove_punctuation,tokenize # none | lowercase | remove_punctuation | tokenize | random_remove_stop_words | random_shuffle_words | all streams: - # Problem is returning tokenized questions. + # Task is returning tokenized questions. questions: tokenized_questions @@ -67,7 +67,7 @@ pipeline: # Image encoder. image_encoder: priority: 3.1 - type: TorchVisionWrapper + type: GenericImageEncoder model_type: resnet152 streams: inputs: images @@ -79,7 +79,7 @@ pipeline: # Element wise multiplication + FF. question_image_fusion: priority: 4.1 - type: ElementWiseMultiplication + type: LowRankBilinearPooling dropout_rate: 0.5 streams: image_encodings: image_activations @@ -118,9 +118,9 @@ pipeline: # 4th subpipeline: concatenation + FF. concat: priority: 5.2 - type: Concatenation + type: ConcatenateTensor input_streams: [question_image_activations,image_size_activations] - # Concatenation + # ConcatenateTensor dim: 1 # default input_dims: [[-1,100],[-1,10]] output_dims: [-1,110] diff --git a/configs/vqa_med_2019/c2_classification/c2_class_lstm_resnet152_rn_cat_is.yml b/configs/vqa_med_2019/c2_classification/c2_class_lstm_resnet152_rn_cat_is.yml index 1a1a05c..f026cd3 100644 --- a/configs/vqa_med_2019/c2_classification/c2_class_lstm_resnet152_rn_cat_is.yml +++ b/configs/vqa_med_2019/c2_classification/c2_class_lstm_resnet152_rn_cat_is.yml @@ -1,8 +1,8 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/c2_classification/default_c2_classification.yml training: - problem: + task: batch_size: 32 # Appy all preprocessing/data augmentations. image_preprocessing: normalize @@ -10,16 +10,16 @@ training: question_preprocessing: lowercase,remove_punctuation,tokenize # none | lowercase | remove_punctuation | tokenize | random_remove_stop_words | random_shuffle_words | all streams: - # Problem is returning tokenized questions. + # Task is returning tokenized questions. questions: tokenized_questions validation: - problem: + task: batch_size: 32 question_preprocessing: lowercase,remove_punctuation,tokenize # none | lowercase | remove_punctuation | tokenize | random_remove_stop_words | random_shuffle_words | all streams: - # Problem is returning tokenized questions. + # Task is returning tokenized questions. questions: tokenized_questions @@ -67,7 +67,7 @@ pipeline: # Image encoder. image_encoder: priority: 3.1 - type: TorchVisionWrapper + type: GenericImageEncoder model_type: resnet152 return_feature_maps: True streams: @@ -116,9 +116,9 @@ pipeline: # 6th subpipeline: concatenation + FF. concat: priority: 5.2 - type: Concatenation + type: ConcatenateTensor input_streams: [rn_activation,image_size_activations] - # Concatenation + # ConcatenateTensor dim: 1 # default input_dims: [[-1,100],[-1,10]] output_dims: [-1,110] diff --git a/configs/vqa_med_2019/c2_classification/c2_class_lstm_resnet50_attn_cat_is.yml b/configs/vqa_med_2019/c2_classification/c2_class_lstm_resnet50_attn_cat_is.yml index 08b043e..f9226f3 100644 --- a/configs/vqa_med_2019/c2_classification/c2_class_lstm_resnet50_attn_cat_is.yml +++ b/configs/vqa_med_2019/c2_classification/c2_class_lstm_resnet50_attn_cat_is.yml @@ -1,22 +1,22 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/c2_classification/default_c2_classification.yml training: - problem: + task: batch_size: 48 # Appy all preprocessing/data augmentations. question_preprocessing: lowercase,remove_punctuation,tokenize streams: - # Problem is returning tokenized questions. + # Task is returning tokenized questions. questions: tokenized_questions validation: - problem: + task: batch_size: 48 # Appy all preprocessing/data augmentations. question_preprocessing: lowercase,remove_punctuation,tokenize streams: - # Problem is returning tokenized questions. + # Task is returning tokenized questions. questions: tokenized_questions @@ -64,7 +64,7 @@ pipeline: # Image encoder. image_encoder: priority: 3.1 - type: TorchVisionWrapper + type: GenericImageEncoder model_type: resnet50 return_feature_maps: True streams: @@ -75,7 +75,7 @@ pipeline: # Attention + FF. question_image_fusion: priority: 4.1 - type: VQA_Attention + type: QuestionDrivenAttention dropout_rate: 0.5 latent_size: 100 num_attention_heads: 2 diff --git a/configs/vqa_med_2019/c2_classification/c2_class_lstm_resnet50_coattn_mfb_cat_is.yml b/configs/vqa_med_2019/c2_classification/c2_class_lstm_resnet50_coattn_mfb_cat_is.yml index cf8bd70..9eddf8f 100644 --- a/configs/vqa_med_2019/c2_classification/c2_class_lstm_resnet50_coattn_mfb_cat_is.yml +++ b/configs/vqa_med_2019/c2_classification/c2_class_lstm_resnet50_coattn_mfb_cat_is.yml @@ -1,22 +1,22 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/c2_classification/default_c2_classification.yml training: - problem: + task: batch_size: 64 # Appy all preprocessing/data augmentations. question_preprocessing: lowercase,remove_punctuation,tokenize streams: - # Problem is returning tokenized questions. + # Task is returning tokenized questions. questions: tokenized_questions validation: - problem: + task: batch_size: 64 # Appy all preprocessing/data augmentations. question_preprocessing: lowercase,remove_punctuation,tokenize streams: - # Problem is returning tokenized questions. + # Task is returning tokenized questions. questions: tokenized_questions @@ -78,7 +78,7 @@ pipeline: # Image encoder. image_encoder: priority: 2.1 - type: TorchVisionWrapper + type: GenericImageEncoder model_type: resnet50 return_feature_maps: True streams: @@ -88,7 +88,7 @@ pipeline: image_attention: priority: 2.2 - type: VQA_Attention + type: QuestionDrivenAttention dropout_rate: 0.3 latent_size: 1024 output_mode: 'Image' @@ -105,7 +105,7 @@ pipeline: # MFB question_image_fusion: priority: 3.1 - type: MultimodalFactorizedBilinearPooling + type: FactorizedBilinearPooling dropout_rate: 0.3 latent_size: 512 pool_factor: 2 diff --git a/configs/vqa_med_2019/c2_classification/c2_class_lstm_resnet50_ewm_cat_is.yml b/configs/vqa_med_2019/c2_classification/c2_class_lstm_resnet50_ewm_cat_is.yml index 2db4248..bd8e0c0 100644 --- a/configs/vqa_med_2019/c2_classification/c2_class_lstm_resnet50_ewm_cat_is.yml +++ b/configs/vqa_med_2019/c2_classification/c2_class_lstm_resnet50_ewm_cat_is.yml @@ -1,22 +1,22 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/c2_classification/default_c2_classification.yml training: - problem: + task: batch_size: 48 # Appy all preprocessing/data augmentations. question_preprocessing: lowercase,remove_punctuation,tokenize streams: - # Problem is returning tokenized questions. + # Task is returning tokenized questions. questions: tokenized_questions validation: - problem: + task: batch_size: 48 # Appy all preprocessing/data augmentations. question_preprocessing: lowercase,remove_punctuation,tokenize streams: - # Problem is returning tokenized questions. + # Task is returning tokenized questions. questions: tokenized_questions @@ -64,7 +64,7 @@ pipeline: # Image encoder. image_encoder: priority: 3.1 - type: TorchVisionWrapper + type: GenericImageEncoder model_type: resnet50 streams: inputs: images @@ -76,7 +76,7 @@ pipeline: # Element wise multiplication + FF. question_image_fusion: priority: 4.1 - type: ElementWiseMultiplication + type: LowRankBilinearPooling dropout_rate: 0.5 streams: image_encodings: image_activations @@ -115,9 +115,9 @@ pipeline: # 4th subpipeline: concatenation + FF. concat: priority: 5.2 - type: Concatenation + type: ConcatenateTensor input_streams: [question_image_activations,image_size_activations] - # Concatenation + # ConcatenateTensor dim: 1 # default input_dims: [[-1,100],[-1,10]] output_dims: [-1,110] diff --git a/configs/vqa_med_2019/c2_classification/c2_class_lstm_resnet50_mfb_cat_is.yml b/configs/vqa_med_2019/c2_classification/c2_class_lstm_resnet50_mfb_cat_is.yml index af55490..944348d 100644 --- a/configs/vqa_med_2019/c2_classification/c2_class_lstm_resnet50_mfb_cat_is.yml +++ b/configs/vqa_med_2019/c2_classification/c2_class_lstm_resnet50_mfb_cat_is.yml @@ -1,22 +1,22 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/c2_classification/default_c2_classification.yml training: - problem: + task: batch_size: 48 # Appy all preprocessing/data augmentations. question_preprocessing: lowercase,remove_punctuation,tokenize streams: - # Problem is returning tokenized questions. + # Task is returning tokenized questions. questions: tokenized_questions validation: - problem: + task: batch_size: 48 # Appy all preprocessing/data augmentations. question_preprocessing: lowercase,remove_punctuation,tokenize streams: - # Problem is returning tokenized questions. + # Task is returning tokenized questions. questions: tokenized_questions @@ -64,7 +64,7 @@ pipeline: # Image encoder. image_encoder: priority: 3.1 - type: TorchVisionWrapper + type: GenericImageEncoder model_type: resnet50 streams: inputs: images @@ -76,7 +76,7 @@ pipeline: # Attention + FF. question_image_fusion: priority: 4.1 - type: MultimodalFactorizedBilinearPooling + type: FactorizedBilinearPooling dropout_rate: 0.5 latent_size: 256 pool_factor: 2 diff --git a/configs/vqa_med_2019/c2_classification/c2_class_lstm_resnet50_rn_cat_is.yml b/configs/vqa_med_2019/c2_classification/c2_class_lstm_resnet50_rn_cat_is.yml index 51dd275..2d9508e 100644 --- a/configs/vqa_med_2019/c2_classification/c2_class_lstm_resnet50_rn_cat_is.yml +++ b/configs/vqa_med_2019/c2_classification/c2_class_lstm_resnet50_rn_cat_is.yml @@ -1,8 +1,8 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/c2_classification/default_c2_classification.yml training: - problem: + task: batch_size: 32 # Appy all preprocessing/data augmentations. image_preprocessing: normalize @@ -10,16 +10,16 @@ training: question_preprocessing: lowercase,remove_punctuation,tokenize # none | lowercase | remove_punctuation | tokenize | random_remove_stop_words | random_shuffle_words | all streams: - # Problem is returning tokenized questions. + # Task is returning tokenized questions. questions: tokenized_questions validation: - problem: + task: batch_size: 32 question_preprocessing: lowercase,remove_punctuation,tokenize # none | lowercase | remove_punctuation | tokenize | random_remove_stop_words | random_shuffle_words | all streams: - # Problem is returning tokenized questions. + # Task is returning tokenized questions. questions: tokenized_questions @@ -67,7 +67,7 @@ pipeline: # Image encoder. image_encoder: priority: 3.1 - type: TorchVisionWrapper + type: GenericImageEncoder model_type: resnet50 return_feature_maps: True streams: @@ -116,9 +116,9 @@ pipeline: # 6th subpipeline: concatenation + FF. concat: priority: 5.2 - type: Concatenation + type: ConcatenateTensor input_streams: [rn_activation,image_size_activations] - # Concatenation + # ConcatenateTensor dim: 1 # default input_dims: [[-1,100],[-1,10]] output_dims: [-1,110] diff --git a/configs/vqa_med_2019/c2_classification/c2_class_lstm_selfattn.yml b/configs/vqa_med_2019/c2_classification/c2_class_lstm_selfattn.yml index 3066fee..6e30601 100644 --- a/configs/vqa_med_2019/c2_classification/c2_class_lstm_selfattn.yml +++ b/configs/vqa_med_2019/c2_classification/c2_class_lstm_selfattn.yml @@ -1,22 +1,22 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/c2_classification/default_c2_classification.yml training: - problem: + task: batch_size: 48 # Appy all preprocessing/data augmentations. question_preprocessing: lowercase,remove_punctuation,tokenize streams: - # Problem is returning tokenized questions. + # Task is returning tokenized questions. questions: tokenized_questions validation: - problem: + task: batch_size: 48 # Appy all preprocessing/data augmentations. question_preprocessing: lowercase,remove_punctuation,tokenize streams: - # Problem is returning tokenized questions. + # Task is returning tokenized questions. questions: tokenized_questions diff --git a/configs/vqa_med_2019/c2_classification/c2_class_lstm_vgg16_rn.yml b/configs/vqa_med_2019/c2_classification/c2_class_lstm_vgg16_rn.yml index 14e4de2..807f597 100644 --- a/configs/vqa_med_2019/c2_classification/c2_class_lstm_vgg16_rn.yml +++ b/configs/vqa_med_2019/c2_classification/c2_class_lstm_vgg16_rn.yml @@ -1,23 +1,23 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/c2_classification/default_c2_classification.yml training: - problem: + task: # Appy all preprocessing/data augmentations. image_preprocessing: normalize # none | random_affine | random_horizontal_flip | normalize | all question_preprocessing: lowercase,remove_punctuation,tokenize # none | lowercase | remove_punctuation | tokenize | random_remove_stop_words | random_shuffle_words | all streams: - # Problem is returning tokenized questions. + # Task is returning tokenized questions. questions: tokenized_questions validation: - problem: + task: question_preprocessing: lowercase,remove_punctuation,tokenize # none | lowercase | remove_punctuation | tokenize | random_remove_stop_words | random_shuffle_words | all streams: - # Problem is returning tokenized questions. + # Task is returning tokenized questions. questions: tokenized_questions @@ -65,7 +65,7 @@ pipeline: # Image encoder. image_encoder: priority: 3.1 - type: TorchVisionWrapper + type: GenericImageEncoder return_feature_maps: True streams: inputs: images diff --git a/configs/vqa_med_2019/c2_classification/c2_class_lstm_vgg16_rn_cat_is.yml b/configs/vqa_med_2019/c2_classification/c2_class_lstm_vgg16_rn_cat_is.yml index 22c25e4..10c710f 100644 --- a/configs/vqa_med_2019/c2_classification/c2_class_lstm_vgg16_rn_cat_is.yml +++ b/configs/vqa_med_2019/c2_classification/c2_class_lstm_vgg16_rn_cat_is.yml @@ -1,23 +1,23 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/c2_classification/default_c2_classification.yml training: - problem: + task: # Appy all preprocessing/data augmentations. image_preprocessing: normalize # none | random_affine | random_horizontal_flip | normalize | all question_preprocessing: lowercase,remove_punctuation,tokenize # none | lowercase | remove_punctuation | tokenize | random_remove_stop_words | random_shuffle_words | all streams: - # Problem is returning tokenized questions. + # Task is returning tokenized questions. questions: tokenized_questions validation: - problem: + task: question_preprocessing: lowercase,remove_punctuation,tokenize # none | lowercase | remove_punctuation | tokenize | random_remove_stop_words | random_shuffle_words | all streams: - # Problem is returning tokenized questions. + # Task is returning tokenized questions. questions: tokenized_questions @@ -65,7 +65,7 @@ pipeline: # Image encoder. image_encoder: priority: 3.1 - type: TorchVisionWrapper + type: GenericImageEncoder return_feature_maps: True streams: inputs: images @@ -113,9 +113,9 @@ pipeline: # 6th subpipeline: concatenation + FF. concat: priority: 5.2 - type: Concatenation + type: ConcatenateTensor input_streams: [rn_activation,image_size_activations] - # Concatenation + # ConcatenateTensor dim: 1 # default input_dims: [[-1,100],[-1,10]] output_dims: [-1,110] diff --git a/configs/vqa_med_2019/c2_classification/c2_classification_all_rnn_vgg16_concat.yml b/configs/vqa_med_2019/c2_classification/c2_classification_all_rnn_vgg16_concat.yml index 51fba8d..700c4b1 100644 --- a/configs/vqa_med_2019/c2_classification/c2_classification_all_rnn_vgg16_concat.yml +++ b/configs/vqa_med_2019/c2_classification/c2_classification_all_rnn_vgg16_concat.yml @@ -1,4 +1,4 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/c2_classification/default_c2_classification.yml pipeline: @@ -64,7 +64,7 @@ pipeline: # Image encoder. image_encoder: priority: 3.1 - type: TorchVisionWrapper + type: GenericImageEncoder streams: inputs: images outputs: image_activations @@ -74,9 +74,9 @@ pipeline: # 4th subpipeline: concatenation + FF. concat: priority: 4.1 - type: Concatenation + type: ConcatenateTensor input_streams: [question_activations,image_size_activations,image_activations] - # Concatenation + # ConcatenateTensor dim: 1 # default input_dims: [[-1,100],[-1,10],[-1,100]] output_dims: [-1,210] diff --git a/configs/vqa_med_2019/c2_classification/c2_classification_all_rnn_vgg16_ewm.yml b/configs/vqa_med_2019/c2_classification/c2_classification_all_rnn_vgg16_ewm.yml index 5447526..eae957f 100644 --- a/configs/vqa_med_2019/c2_classification/c2_classification_all_rnn_vgg16_ewm.yml +++ b/configs/vqa_med_2019/c2_classification/c2_classification_all_rnn_vgg16_ewm.yml @@ -1,4 +1,4 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/c2_classification/default_c2_classification.yml pipeline: @@ -51,7 +51,7 @@ pipeline: # Image encoder. image_encoder: priority: 3.1 - type: TorchVisionWrapper + type: GenericImageEncoder streams: inputs: images outputs: image_activations @@ -62,7 +62,7 @@ pipeline: # Element wise multiplication + FF. question_image_fusion: priority: 4.1 - type: ElementWiseMultiplication + type: LowRankBilinearPooling dropout_rate: 0.5 streams: image_encodings: image_activations diff --git a/configs/vqa_med_2019/c2_classification/c2_classification_all_rnn_vgg16_ewm_size.yml b/configs/vqa_med_2019/c2_classification/c2_classification_all_rnn_vgg16_ewm_size.yml index 1a1f774..d5c8a02 100644 --- a/configs/vqa_med_2019/c2_classification/c2_classification_all_rnn_vgg16_ewm_size.yml +++ b/configs/vqa_med_2019/c2_classification/c2_classification_all_rnn_vgg16_ewm_size.yml @@ -1,4 +1,4 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/c2_classification/default_c2_classification.yml pipeline: @@ -52,7 +52,7 @@ pipeline: # Image encoder. image_encoder: priority: 3.1 - type: TorchVisionWrapper + type: GenericImageEncoder streams: inputs: images outputs: image_activations @@ -63,7 +63,7 @@ pipeline: # Element wise multiplication + FF. question_image_fusion: priority: 4.1 - type: ElementWiseMultiplication + type: LowRankBilinearPooling dropout_rate: 0.5 streams: image_encodings: image_activations @@ -102,9 +102,9 @@ pipeline: # 4th subpipeline: concatenation + FF. concat: priority: 5.2 - type: Concatenation + type: ConcatenateTensor input_streams: [question_image_activations,image_size_activations] - # Concatenation + # ConcatenateTensor dim: 1 # default input_dims: [[-1,100],[-1,10]] output_dims: [-1,110] diff --git a/configs/vqa_med_2019/c2_classification/c2_classification_all_rnn_vgg16_mcb.yml b/configs/vqa_med_2019/c2_classification/c2_classification_all_rnn_vgg16_mcb.yml index 075c958..2e70145 100644 --- a/configs/vqa_med_2019/c2_classification/c2_classification_all_rnn_vgg16_mcb.yml +++ b/configs/vqa_med_2019/c2_classification/c2_classification_all_rnn_vgg16_mcb.yml @@ -1,4 +1,4 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/c2_classification/default_c2_classification.yml pipeline: @@ -51,7 +51,7 @@ pipeline: # Image encoder. image_encoder: priority: 3.1 - type: TorchVisionWrapper + type: GenericImageEncoder streams: inputs: images outputs: image_activations @@ -62,7 +62,7 @@ pipeline: # Element wise multiplication + FF. question_image_fusion: priority: 4.1 - type: MultimodalCompactBilinearPooling + type: CompactBilinearPooling streams: image_encodings: image_activations question_encodings: question_activations diff --git a/configs/vqa_med_2019/c2_classification/c2_word_answer_onehot_bow.yml b/configs/vqa_med_2019/c2_classification/c2_word_answer_onehot_bow.yml index 2d28708..2b8fb3c 100644 --- a/configs/vqa_med_2019/c2_classification/c2_word_answer_onehot_bow.yml +++ b/configs/vqa_med_2019/c2_classification/c2_word_answer_onehot_bow.yml @@ -1,16 +1,16 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/c2_classification/default_c2_classification.yml # Training parameters: training: - problem: + task: batch_size: 128 terminal_conditions: episode_limit: 1000 # Validation parameters: validation: - problem: + task: batch_size: 128 pipeline: diff --git a/configs/vqa_med_2019/c2_classification/default_c2_classification.yml b/configs/vqa_med_2019/c2_classification/default_c2_classification.yml index 9511a28..39dcd1c 100644 --- a/configs/vqa_med_2019/c2_classification/default_c2_classification.yml +++ b/configs/vqa_med_2019/c2_classification/default_c2_classification.yml @@ -1,9 +1,9 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/default_vqa_med_2019.yml # Training parameters: training: - problem: + task: categories: C2 export_sample_weights: ~/data/vqa-med/answers.c2.weights.csv sampler: @@ -11,7 +11,7 @@ training: # Validation parameters: validation: - problem: + task: categories: C2 diff --git a/configs/vqa_med_2019/c3_classification/c3_classification_all_bow_vgg16_concat.yml b/configs/vqa_med_2019/c3_classification/c3_classification_all_bow_vgg16_concat.yml index 8d88cd9..a28a0d5 100644 --- a/configs/vqa_med_2019/c3_classification/c3_classification_all_bow_vgg16_concat.yml +++ b/configs/vqa_med_2019/c3_classification/c3_classification_all_bow_vgg16_concat.yml @@ -1,4 +1,4 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/c3_classification/default_c3_classification.yml pipeline: @@ -56,7 +56,7 @@ pipeline: # 3rd subpipeline: image. # Image encoder. image_encoder: - type: TorchVisionWrapper + type: GenericImageEncoder priority: 3.1 streams: inputs: images @@ -66,10 +66,10 @@ pipeline: # 4th subpipeline: concatenation + FF. concat: - type: Concatenation + type: ConcatenateTensor priority: 4.1 input_streams: [question_activations,image_size_activations,image_activations] - # Concatenation + # ConcatenateTensor dim: 1 # default input_dims: [[-1,99],[-1,10],[-1,100]] output_dims: [-1,209] diff --git a/configs/vqa_med_2019/c3_classification/c3_classification_all_concat.yml b/configs/vqa_med_2019/c3_classification/c3_classification_all_concat.yml index 668949c..ab93130 100644 --- a/configs/vqa_med_2019/c3_classification/c3_classification_all_concat.yml +++ b/configs/vqa_med_2019/c3_classification/c3_classification_all_concat.yml @@ -1,4 +1,4 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/c3_classification/default_c3_classification.yml pipeline: @@ -95,10 +95,10 @@ pipeline: # 4th subpipeline: concatenation + FF. concat: - type: Concatenation + type: ConcatenateTensor priority: 4.1 input_streams: [question_activations,image_size_activations,image_activations] - # Concatenation + # ConcatenateTensor dim: 1 # default input_dims: [[-1,100],[-1,100],[-1,100]] output_dims: [-1,300] diff --git a/configs/vqa_med_2019/c3_classification/c3_classification_all_rnn_vgg16_concat.yml b/configs/vqa_med_2019/c3_classification/c3_classification_all_rnn_vgg16_concat.yml index 15a94e9..070f6c0 100644 --- a/configs/vqa_med_2019/c3_classification/c3_classification_all_rnn_vgg16_concat.yml +++ b/configs/vqa_med_2019/c3_classification/c3_classification_all_rnn_vgg16_concat.yml @@ -1,4 +1,4 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/c3_classification/default_c3_classification.yml pipeline: @@ -63,7 +63,7 @@ pipeline: # 3rd subpipeline: image. # Image encoder. image_encoder: - type: TorchVisionWrapper + type: GenericImageEncoder priority: 3.1 streams: inputs: images @@ -73,10 +73,10 @@ pipeline: # 4th subpipeline: concatenation + FF. concat: - type: Concatenation + type: ConcatenateTensor priority: 4.1 input_streams: [question_activations,image_size_activations,image_activations] - # Concatenation + # ConcatenateTensor dim: 1 # default input_dims: [[-1,100],[-1,10],[-1,100]] output_dims: [-1,210] diff --git a/configs/vqa_med_2019/c3_classification/c3_classification_image_cnn_softmax.yml b/configs/vqa_med_2019/c3_classification/c3_classification_image_cnn_softmax.yml index e93a77d..1744f27 100644 --- a/configs/vqa_med_2019/c3_classification/c3_classification_image_cnn_softmax.yml +++ b/configs/vqa_med_2019/c3_classification/c3_classification_image_cnn_softmax.yml @@ -1,4 +1,4 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/c3_classification/default_c3_classification.yml pipeline: diff --git a/configs/vqa_med_2019/c3_classification/c3_classification_image_plus_size_concat.yml b/configs/vqa_med_2019/c3_classification/c3_classification_image_plus_size_concat.yml index 586fbe1..3264cfe 100644 --- a/configs/vqa_med_2019/c3_classification/c3_classification_image_plus_size_concat.yml +++ b/configs/vqa_med_2019/c3_classification/c3_classification_image_plus_size_concat.yml @@ -1,4 +1,4 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/c3_classification/default_c3_classification.yml pipeline: @@ -57,10 +57,10 @@ pipeline: # 4th subpipeline: concatenation + FF. concat: - type: Concatenation + type: ConcatenateTensor priority: 4.1 input_streams: [image_size_activations,image_activations] - # Concatenation + # ConcatenateTensor dim: 1 # default input_dims: [[-1,100],[-1,10]] output_dims: [-1,110] diff --git a/configs/vqa_med_2019/c3_classification/c3_classification_image_size_softmax.yml b/configs/vqa_med_2019/c3_classification/c3_classification_image_size_softmax.yml index 68491a2..5c98f11 100644 --- a/configs/vqa_med_2019/c3_classification/c3_classification_image_size_softmax.yml +++ b/configs/vqa_med_2019/c3_classification/c3_classification_image_size_softmax.yml @@ -1,4 +1,4 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/c3_classification/default_c3_classification.yml pipeline: diff --git a/configs/vqa_med_2019/c3_classification/c3_classification_image_softmax.yml b/configs/vqa_med_2019/c3_classification/c3_classification_image_softmax.yml index 0b0405b..7c27016 100644 --- a/configs/vqa_med_2019/c3_classification/c3_classification_image_softmax.yml +++ b/configs/vqa_med_2019/c3_classification/c3_classification_image_softmax.yml @@ -1,12 +1,12 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/c3_classification/default_c3_classification.yml training: - problem: + task: resize_image: &resize_image [28, 28] validation: - problem: + task: resize_image: *resize_image diff --git a/configs/vqa_med_2019/c3_classification/c3_classification_image_vgg16_softmax.yml b/configs/vqa_med_2019/c3_classification/c3_classification_image_vgg16_softmax.yml index 03ceadd..a716371 100644 --- a/configs/vqa_med_2019/c3_classification/c3_classification_image_vgg16_softmax.yml +++ b/configs/vqa_med_2019/c3_classification/c3_classification_image_vgg16_softmax.yml @@ -1,11 +1,11 @@ - # Load config defining problems for training, validation and testing. + # Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/c3_classification/default_c3_classification.yml pipeline: # Image encoder. image_encoder: - type: TorchVisionWrapper + type: GenericImageEncoder priority: 1.1 streams: inputs: images diff --git a/configs/vqa_med_2019/c3_classification/c3_classification_question_onehot_bow.yml b/configs/vqa_med_2019/c3_classification/c3_classification_question_onehot_bow.yml index f6a0e3e..a7432e0 100644 --- a/configs/vqa_med_2019/c3_classification/c3_classification_question_onehot_bow.yml +++ b/configs/vqa_med_2019/c3_classification/c3_classification_question_onehot_bow.yml @@ -1,4 +1,4 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/c3_classification/default_c3_classification.yml pipeline: diff --git a/configs/vqa_med_2019/c3_classification/c3_classification_question_rnn.yml b/configs/vqa_med_2019/c3_classification/c3_classification_question_rnn.yml index a79da98..39e6163 100644 --- a/configs/vqa_med_2019/c3_classification/c3_classification_question_rnn.yml +++ b/configs/vqa_med_2019/c3_classification/c3_classification_question_rnn.yml @@ -1,4 +1,4 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/c3_classification/default_c3_classification.yml pipeline: diff --git a/configs/vqa_med_2019/c3_classification/default_c3_classification.yml b/configs/vqa_med_2019/c3_classification/default_c3_classification.yml index 5da853b..fa91f68 100644 --- a/configs/vqa_med_2019/c3_classification/default_c3_classification.yml +++ b/configs/vqa_med_2019/c3_classification/default_c3_classification.yml @@ -1,9 +1,9 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/default_vqa_med_2019.yml # Training parameters: training: - problem: + task: categories: C3 export_sample_weights: ~/data/vqa-med/answers.c3.weights.csv sampler: @@ -11,7 +11,7 @@ training: # Validation parameters: validation: - problem: + task: categories: C3 diff --git a/configs/vqa_med_2019/c4_classification/c4_classification_all_rnn_vgg16_ewm_size.yml b/configs/vqa_med_2019/c4_classification/c4_classification_all_rnn_vgg16_ewm_size.yml index e6831f9..1c9578d 100644 --- a/configs/vqa_med_2019/c4_classification/c4_classification_all_rnn_vgg16_ewm_size.yml +++ b/configs/vqa_med_2019/c4_classification/c4_classification_all_rnn_vgg16_ewm_size.yml @@ -1,4 +1,4 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/c4_classification/default_c4_classification.yml pipeline: @@ -52,7 +52,7 @@ pipeline: # Image encoder. image_encoder: priority: 3.1 - type: TorchVisionWrapper + type: GenericImageEncoder streams: inputs: images outputs: image_activations @@ -63,7 +63,7 @@ pipeline: # Element wise multiplication + FF. question_image_fusion: priority: 4.1 - type: ElementWiseMultiplication + type: LowRankBilinearPooling dropout_rate: 0.5 streams: image_encodings: image_activations @@ -102,9 +102,9 @@ pipeline: # 4th subpipeline: concatenation + FF. concat: priority: 5.2 - type: Concatenation + type: ConcatenateTensor input_streams: [question_image_activations,image_size_activations] - # Concatenation + # ConcatenateTensor dim: 1 # default input_dims: [[-1,100],[-1,10]] output_dims: [-1,110] diff --git a/configs/vqa_med_2019/c4_classification/c4_enc_attndec.yml b/configs/vqa_med_2019/c4_classification/c4_enc_attndec.yml index 022a16a..bdd77c8 100644 --- a/configs/vqa_med_2019/c4_classification/c4_enc_attndec.yml +++ b/configs/vqa_med_2019/c4_classification/c4_enc_attndec.yml @@ -1,9 +1,9 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/default_vqa_med_2019.yml # Training parameters: training: - problem: + task: batch_size: 64 categories: C4 question_preprocessing: lowercase, remove_punctuation, tokenize, random_remove_stop_words #,random_shuffle_words @@ -21,7 +21,7 @@ training: # Validation parameters: validation: - problem: + task: batch_size: 64 categories: C4 question_preprocessing: lowercase, remove_punctuation, tokenize @@ -87,7 +87,7 @@ pipeline: # Single layer GRU Decoder with attention decoder: - type: Attn_Decoder_RNN + type: AttentionDecoder priority: 4 hidden_size: 100 use_logsoftmax: False diff --git a/configs/vqa_med_2019/c4_classification/c4_enc_attndec_resnet152_ewm_cat_is.yml b/configs/vqa_med_2019/c4_classification/c4_enc_attndec_resnet152_ewm_cat_is.yml index 92c722c..7ab71cf 100644 --- a/configs/vqa_med_2019/c4_classification/c4_enc_attndec_resnet152_ewm_cat_is.yml +++ b/configs/vqa_med_2019/c4_classification/c4_enc_attndec_resnet152_ewm_cat_is.yml @@ -1,9 +1,9 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/default_vqa_med_2019.yml # Training parameters: training: - problem: + task: batch_size: 200 # requires to use 4 GPUs! categories: C4 question_preprocessing: lowercase, remove_punctuation, tokenize #, random_remove_stop_words #,random_shuffle_words @@ -21,7 +21,7 @@ training: # Validation parameters: validation: - problem: + task: batch_size: 200 categories: C4 question_preprocessing: lowercase, remove_punctuation, tokenize @@ -75,7 +75,7 @@ pipeline: # Image encoder. image_encoder: priority: 2.0 - type: TorchVisionWrapper + type: GenericImageEncoder model_type: resnet152 streams: inputs: images @@ -87,8 +87,6 @@ pipeline: encoder: priority: 3 type: RecurrentNeuralNetwork - # Do not wrap that model with DataDictParallel! - parallelize: False cell_type: GRU initial_state: Trainable hidden_size: 100 @@ -120,7 +118,7 @@ pipeline: # Element wise multiplication + FF. question_image_fusion: priority: 3.1 - type: ElementWiseMultiplication + type: LowRankBilinearPooling dropout_rate: 0.5 streams: image_encodings: image_activations @@ -156,7 +154,7 @@ pipeline: # Single layer GRU Decoder with attention decoder: - type: Attn_Decoder_RNN + type: AttentionDecoder priority: 4 hidden_size: 100 use_logsoftmax: False diff --git a/configs/vqa_med_2019/c4_classification/c4_frozen_if_gru_dec.yml b/configs/vqa_med_2019/c4_classification/c4_frozen_if_gru_dec.yml index e9ff65a..82fa093 100644 --- a/configs/vqa_med_2019/c4_classification/c4_frozen_if_gru_dec.yml +++ b/configs/vqa_med_2019/c4_classification/c4_frozen_if_gru_dec.yml @@ -1,4 +1,4 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/default_vqa_med_2019.yml, vqa_med_2019/frozen_pipelines/frozen_input_fusion_glove_lstm_vgg_att_is_cat.yml, @@ -21,7 +21,7 @@ c4_hyperparameters: # Training parameters: training: - problem: + task: batch_size: *batch_size categories: C4 export_sample_weights: ~/data/vqa-med/answers.c4.weights.csv @@ -51,7 +51,7 @@ training: # Validation parameters: validation: - problem: + task: batch_size: *batch_size categories: C4 # Appy all preprocessing/data augmentations. diff --git a/configs/vqa_med_2019/c4_classification/c4_word_answer_glove_sum.yml b/configs/vqa_med_2019/c4_classification/c4_word_answer_glove_sum.yml index 2c12145..af80903 100644 --- a/configs/vqa_med_2019/c4_classification/c4_word_answer_glove_sum.yml +++ b/configs/vqa_med_2019/c4_classification/c4_word_answer_glove_sum.yml @@ -1,9 +1,9 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/c4_classification/default_c4_classification.yml # Training parameters: training: - problem: + task: categories: C4 batch_size: 512 # In here we won't use images at all. @@ -13,7 +13,7 @@ training: # Validation parameters: validation: - problem: + task: categories: C4 batch_size: 512 # In here we won't use images at all. diff --git a/configs/vqa_med_2019/c4_classification/c4_word_answer_mimic_sum.yml b/configs/vqa_med_2019/c4_classification/c4_word_answer_mimic_sum.yml index db28cc4..bc32444 100644 --- a/configs/vqa_med_2019/c4_classification/c4_word_answer_mimic_sum.yml +++ b/configs/vqa_med_2019/c4_classification/c4_word_answer_mimic_sum.yml @@ -1,9 +1,9 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/c4_classification/default_c4_classification.yml # Training parameters: training: - problem: + task: categories: C4 batch_size: 512 # In here we won't use images at all. @@ -13,7 +13,7 @@ training: # Validation parameters: validation: - problem: + task: categories: C4 batch_size: 512 # In here we won't use images at all. diff --git a/configs/vqa_med_2019/c4_classification/c4_word_answer_onehot_bow.yml b/configs/vqa_med_2019/c4_classification/c4_word_answer_onehot_bow.yml index 17168c2..4255481 100644 --- a/configs/vqa_med_2019/c4_classification/c4_word_answer_onehot_bow.yml +++ b/configs/vqa_med_2019/c4_classification/c4_word_answer_onehot_bow.yml @@ -1,9 +1,9 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/c4_classification/default_c4_classification.yml # Training parameters: training: - problem: + task: categories: C4 batch_size: 512 # In here we won't use images at all. @@ -13,7 +13,7 @@ training: # Validation parameters: validation: - problem: + task: categories: C4 batch_size: 512 # In here we won't use images at all. diff --git a/configs/vqa_med_2019/c4_classification/c4_word_answer_onehot_sum.yml b/configs/vqa_med_2019/c4_classification/c4_word_answer_onehot_sum.yml index 640f202..424d3fc 100644 --- a/configs/vqa_med_2019/c4_classification/c4_word_answer_onehot_sum.yml +++ b/configs/vqa_med_2019/c4_classification/c4_word_answer_onehot_sum.yml @@ -1,9 +1,9 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/c4_classification/default_c4_classification.yml # Training parameters: training: - problem: + task: categories: C4 batch_size: 512 # In here we won't use images at all. @@ -13,7 +13,7 @@ training: # Validation parameters: validation: - problem: + task: categories: C4 batch_size: 512 # In here we won't use images at all. diff --git a/configs/vqa_med_2019/c4_classification/default_c4_classification.yml b/configs/vqa_med_2019/c4_classification/default_c4_classification.yml index dcfcd52..833c97f 100644 --- a/configs/vqa_med_2019/c4_classification/default_c4_classification.yml +++ b/configs/vqa_med_2019/c4_classification/default_c4_classification.yml @@ -1,9 +1,9 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/default_vqa_med_2019.yml # Training parameters: training: - problem: + task: categories: C4 export_sample_weights: ~/data/vqa-med/answers.c4.weights.csv sampler: @@ -11,7 +11,7 @@ training: # Validation parameters: validation: - problem: + task: categories: C4 diff --git a/configs/vqa_med_2019/default_vqa_med_2019.yml b/configs/vqa_med_2019/default_vqa_med_2019.yml index aa04ba4..c237739 100644 --- a/configs/vqa_med_2019/default_vqa_med_2019.yml +++ b/configs/vqa_med_2019/default_vqa_med_2019.yml @@ -1,6 +1,6 @@ # Training parameters: training: - problem: + task: type: &p_type VQAMED2019 data_folder: &data_folder ~/data/vqa-med split: training_validation @@ -33,7 +33,7 @@ training: # Validation parameters: validation: partial_validation_interval: 10 - problem: + task: type: *p_type data_folder: *data_folder split: training_validation diff --git a/configs/vqa_med_2019/evaluation/deepta/glove_gru_resnet50_coattn_mfb_is_cat_ffn_c123_loss.yml b/configs/vqa_med_2019/evaluation/deepta/glove_gru_resnet50_coattn_mfb_is_cat_ffn_c123_loss.yml index ad78835..94f16c5 100644 --- a/configs/vqa_med_2019/evaluation/deepta/glove_gru_resnet50_coattn_mfb_is_cat_ffn_c123_loss.yml +++ b/configs/vqa_med_2019/evaluation/deepta/glove_gru_resnet50_coattn_mfb_is_cat_ffn_c123_loss.yml @@ -1,4 +1,4 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/default_vqa_med_2019.yml hyperparameters: @@ -29,8 +29,8 @@ hyperparameters: question_attention_output_size_val: &question_attention_output_size_val 800 # Fusion I: image + question - question_image_fusion_type_val: &question_image_fusion_type MultimodalFactorizedBilinearPooling - # Options: ElementWiseMultiplication | MultimodalFactorizedBilinearPooling (component: question_image_fusion) + question_image_fusion_type_val: &question_image_fusion_type FactorizedBilinearPooling + # Options: LowRankBilinearPooling | FactorizedBilinearPooling (component: question_image_fusion) question_image_fusion_size_val: &question_image_fusion_size_val 512 # Image size encoder. @@ -46,7 +46,7 @@ hyperparameters: # Training parameters: training: - problem: + task: batch_size: *batch_size categories: C1,C2,C3 export_sample_weights: ~/data/vqa-med/answers.c1_c2_c3_binary_yn.weights.csv @@ -71,7 +71,7 @@ training: # Validation parameters: validation: - problem: + task: batch_size: *batch_size categories: C1,C2,C3 # Appy all preprocessing/data augmentations. @@ -165,7 +165,7 @@ pipeline: # Image encoder. image_encoder: priority: 2.1 - type: TorchVisionWrapper + type: GenericImageEncoder model_type: *image_encoder_model return_feature_maps: True streams: @@ -175,7 +175,7 @@ pipeline: image_attention: priority: 2.2 - type: VQA_Attention + type: QuestionDrivenAttention dropout_rate: 0.3 latent_size: 256 output_mode: 'Image' @@ -224,9 +224,9 @@ pipeline: # 5th subpipeline: concatenation concat: priority: 5.1 - type: Concatenation + type: ConcatenateTensor input_streams: [fused_activations,image_size_activations] - # Concatenation + # ConcatenateTensor dim: 1 # default input_dims: [[-1,*question_image_fusion_size_val],[-1,*image_size_encoder_output_size_val]] output_dims: [-1,*question_image_size_fusion_size_val] diff --git a/configs/vqa_med_2019/evaluation/deepta/glove_gru_vgg16_coattn_mfb_is_cat_ffn_c1234_loss.yml b/configs/vqa_med_2019/evaluation/deepta/glove_gru_vgg16_coattn_mfb_is_cat_ffn_c1234_loss.yml index da816bd..d8a9fef 100644 --- a/configs/vqa_med_2019/evaluation/deepta/glove_gru_vgg16_coattn_mfb_is_cat_ffn_c1234_loss.yml +++ b/configs/vqa_med_2019/evaluation/deepta/glove_gru_vgg16_coattn_mfb_is_cat_ffn_c1234_loss.yml @@ -1,4 +1,4 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/default_vqa_med_2019.yml hyperparameters: @@ -29,8 +29,8 @@ hyperparameters: question_attention_output_size_val: &question_attention_output_size_val 800 # Fusion I: image + question - question_image_fusion_type_val: &question_image_fusion_type MultimodalFactorizedBilinearPooling - # Options: ElementWiseMultiplication | MultimodalFactorizedBilinearPooling (component: question_image_fusion) + question_image_fusion_type_val: &question_image_fusion_type FactorizedBilinearPooling + # Options: LowRankBilinearPooling | FactorizedBilinearPooling (component: question_image_fusion) question_image_fusion_size_val: &question_image_fusion_size_val 512 # Image size encoder. @@ -46,7 +46,7 @@ hyperparameters: # Training parameters: training: - problem: + task: batch_size: *batch_size categories: all #C1,C2,C3 # TODO: all export_sample_weights: ~/data/vqa-med/answers.c1_c2_c3_c4_binary_yn.weights.csv @@ -71,7 +71,7 @@ training: # Validation parameters: validation: - problem: + task: batch_size: *batch_size categories: all #C1,C2,C3 # TODO: all # Appy all preprocessing/data augmentations. @@ -165,7 +165,7 @@ pipeline: # Image encoder. image_encoder: priority: 2.1 - type: TorchVisionWrapper + type: GenericImageEncoder model_type: *image_encoder_model return_feature_maps: True streams: @@ -175,7 +175,7 @@ pipeline: image_attention: priority: 2.2 - type: VQA_Attention + type: QuestionDrivenAttention dropout_rate: 0.3 latent_size: 256 output_mode: 'Image' @@ -224,9 +224,9 @@ pipeline: # 5th subpipeline: concatenation concat: priority: 5.1 - type: Concatenation + type: ConcatenateTensor input_streams: [fused_activations,image_size_activations] - # Concatenation + # ConcatenateTensor dim: 1 # default input_dims: [[-1,*question_image_fusion_size_val],[-1,*image_size_encoder_output_size_val]] output_dims: [-1,*question_image_size_fusion_size_val] diff --git a/configs/vqa_med_2019/evaluation/example_mimic_lstm_vgg16_ewm_is_cat_ffn_c123_loss.yml b/configs/vqa_med_2019/evaluation/example_mimic_lstm_vgg16_ewm_is_cat_ffn_c123_loss.yml index 03857bd..81efdd7 100644 --- a/configs/vqa_med_2019/evaluation/example_mimic_lstm_vgg16_ewm_is_cat_ffn_c123_loss.yml +++ b/configs/vqa_med_2019/evaluation/example_mimic_lstm_vgg16_ewm_is_cat_ffn_c123_loss.yml @@ -1,4 +1,4 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/default_vqa_med_2019.yml #,vqa_med_2019/frozen_pipelines/frozen_question_categorization_glove_rnn_ffn.yml @@ -26,8 +26,8 @@ hyperparameters: question_encoder_output_size_val: &question_encoder_output_size_val 103 # Fusion I: image + question - question_image_fusion_type_val: &question_image_fusion_type ElementWiseMultiplication - # Options: ElementWiseMultiplication | ? (component: question_image_fusion) + question_image_fusion_type_val: &question_image_fusion_type LowRankBilinearPooling + # Options: LowRankBilinearPooling | ? (component: question_image_fusion) question_image_fusion_size_val: &question_image_fusion_size_val 115 # Image size encoder. @@ -46,7 +46,7 @@ hyperparameters: # Training parameters: training: - problem: + task: batch_size: *batch_size categories: C1,C2,C3 export_sample_weights: ~/data/vqa-med/answers.c1_c2_c3_binary_yn.weights.csv @@ -76,7 +76,7 @@ training: # Validation parameters: validation: - problem: + task: batch_size: *batch_size categories: C1,C2,C3 # Appy all preprocessing/data augmentations. @@ -159,7 +159,7 @@ pipeline: # Image encoder. image_encoder: priority: 2.1 - type: TorchVisionWrapper + type: GenericImageEncoder model: *image_encoder_model streams: inputs: images @@ -214,9 +214,9 @@ pipeline: # 5th subpipeline: concatenation concat: priority: 5.1 - type: Concatenation + type: ConcatenateTensor input_streams: [question_image_activations,image_size_activations] - # Concatenation + # ConcatenateTensor dim: 1 # default input_dims: [[-1,*question_image_fusion_size_val],[-1,*image_size_encoder_output_size_val]] output_dims: [-1,*question_image_size_fusion_size_val] diff --git a/configs/vqa_med_2019/evaluation/frozen_if_ffn_c1234_loss.yml b/configs/vqa_med_2019/evaluation/frozen_if_ffn_c1234_loss.yml index f5c9328..7391ab9 100644 --- a/configs/vqa_med_2019/evaluation/frozen_if_ffn_c1234_loss.yml +++ b/configs/vqa_med_2019/evaluation/frozen_if_ffn_c1234_loss.yml @@ -1,4 +1,4 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/default_vqa_med_2019.yml, vqa_med_2019/frozen_pipelines/frozen_input_fusion_glove_lstm_vgg_att_is_cat.yml @@ -24,7 +24,7 @@ c123_hyperparameters: # Training parameters: training: - problem: + task: batch_size: *batch_size categories: all export_sample_weights: ~/data/vqa-med/answers.c1_c2_c3_c4_binary_yn.weights.csv @@ -54,7 +54,7 @@ training: # Validation parameters: validation: - problem: + task: batch_size: *batch_size categories: all # Appy all preprocessing/data augmentations. diff --git a/configs/vqa_med_2019/evaluation/frozen_if_ffn_c123_loss.yml b/configs/vqa_med_2019/evaluation/frozen_if_ffn_c123_loss.yml index 6340fe3..6d4b77d 100644 --- a/configs/vqa_med_2019/evaluation/frozen_if_ffn_c123_loss.yml +++ b/configs/vqa_med_2019/evaluation/frozen_if_ffn_c123_loss.yml @@ -1,4 +1,4 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/default_vqa_med_2019.yml, vqa_med_2019/frozen_pipelines/frozen_input_fusion_glove_lstm_vgg_att_is_cat.yml, @@ -24,7 +24,7 @@ c123_hyperparameters: # Training parameters: training: - problem: + task: batch_size: *batch_size categories: C1,C2,C3 export_sample_weights: ~/data/vqa-med/answers.c1_c2_c3_binary_yn.weights.csv @@ -54,7 +54,7 @@ training: # Validation parameters: validation: - problem: + task: batch_size: *batch_size categories: C1,C2,C3 # Appy all preprocessing/data augmentations. diff --git a/configs/vqa_med_2019/evaluation/frozen_if_vf_5ffn_c1234yn_5losses.yml b/configs/vqa_med_2019/evaluation/frozen_if_vf_5ffn_c1234yn_5losses.yml index ea732e9..f6bea20 100644 --- a/configs/vqa_med_2019/evaluation/frozen_if_vf_5ffn_c1234yn_5losses.yml +++ b/configs/vqa_med_2019/evaluation/frozen_if_vf_5ffn_c1234yn_5losses.yml @@ -1,4 +1,4 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/default_vqa_med_2019.yml, vqa_med_2019/frozen_pipelines/frozen_input_fusion_glove_lstm_vgg_att_is_cat.yml, @@ -24,7 +24,7 @@ c123_hyperparameters: # Training parameters: training: - problem: + task: batch_size: *batch_size categories: all export_sample_weights: ~/data/vqa-med/answers.c1_c2_c3_c4_binary_yn.weights.csv @@ -54,7 +54,7 @@ training: # Validation parameters: validation: - problem: + task: batch_size: *batch_size categories: all # Appy all preprocessing/data augmentations. diff --git a/configs/vqa_med_2019/evaluation/frozen_if_vf_5ffn_support_c1234yn_5losses.yml b/configs/vqa_med_2019/evaluation/frozen_if_vf_5ffn_support_c1234yn_5losses.yml index 073b78e..8c49970 100644 --- a/configs/vqa_med_2019/evaluation/frozen_if_vf_5ffn_support_c1234yn_5losses.yml +++ b/configs/vqa_med_2019/evaluation/frozen_if_vf_5ffn_support_c1234yn_5losses.yml @@ -1,4 +1,4 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/default_vqa_med_2019.yml, vqa_med_2019/frozen_pipelines/frozen_input_fusion_glove_lstm_vgg_att_is_cat.yml, @@ -39,7 +39,7 @@ c1234_hyperparameters: # Training parameters: training: - problem: + task: batch_size: *batch_size categories: all export_sample_weights: ~/data/vqa-med/answers.c1_c2_c3_c4_binary_yn.weights.csv @@ -69,7 +69,7 @@ training: # Validation parameters: validation: - problem: + task: batch_size: *batch_size categories: all # Appy all preprocessing/data augmentations. @@ -442,9 +442,9 @@ pipeline: # Concatenate input fused activatons with three supporting streams. activation_support_concat: priority: 11.1 - type: Concatenation + type: ConcatenateTensor input_streams: [concatenated_activations,c1_support,c2_support,c3_support] - # Concatenation + # ConcatenateTensor dim: 1 # default input_dims: [[-1,*activation_size_val],[-1,*c123_support_output_size_val],[-1,*c123_support_output_size_val],[-1,*c123_support_output_size_val]] output_dims: [-1,*activation_support_size_val] diff --git a/configs/vqa_med_2019/evaluation/tom/glove_lstm_resnet152_att_is_cat_ffn_c123_loss.yml b/configs/vqa_med_2019/evaluation/tom/glove_lstm_resnet152_att_is_cat_ffn_c123_loss.yml index bf81cb7..6e3c4a9 100644 --- a/configs/vqa_med_2019/evaluation/tom/glove_lstm_resnet152_att_is_cat_ffn_c123_loss.yml +++ b/configs/vqa_med_2019/evaluation/tom/glove_lstm_resnet152_att_is_cat_ffn_c123_loss.yml @@ -1,4 +1,4 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/default_vqa_med_2019.yml #,vqa_med_2019/frozen_pipelines/frozen_question_categorization_glove_rnn_ffn.yml @@ -27,10 +27,10 @@ hyperparameters: question_encoder_output_size_val: &question_encoder_output_size_val 100 # Fusion I: image + question - question_image_fusion_type_val: &question_image_fusion_type VQA_Attention - # Options: ElementWiseMultiplication | VQA_Attention + question_image_fusion_type_val: &question_image_fusion_type QuestionDrivenAttention + # Options: LowRankBilinearPooling | QuestionDrivenAttention #question_image_fusion_size_val: &question_image_fusion_size_val 1124 - # INFO: this variable is set by VQA_Attention component! + # INFO: this variable is set by QuestionDrivenAttention component! # Image size encoder. image_size_encoder_output_size_val: &image_size_encoder_output_size_val 10 @@ -47,7 +47,7 @@ hyperparameters: # Training parameters: training: - problem: + task: batch_size: *batch_size categories: C1,C2,C3 export_sample_weights: ~/data/vqa-med/answers.c1_c2_c3_binary_yn.weights.csv @@ -77,7 +77,7 @@ training: # Validation parameters: validation: - problem: + task: batch_size: *batch_size categories: C1,C2,C3 # Appy all preprocessing/data augmentations. @@ -161,7 +161,7 @@ pipeline: # Image encoder. image_encoder: priority: 2.1 - type: TorchVisionWrapper + type: GenericImageEncoder model: *image_encoder_model return_feature_maps: True streams: @@ -218,9 +218,9 @@ pipeline: # 5th subpipeline: concatenation concat: priority: 5.1 - type: Concatenation + type: ConcatenateTensor input_streams: [question_image_activations,image_size_activations] - # Concatenation + # ConcatenateTensor dim: 1 # default input_dims: [[-1,1124],[-1,*image_size_encoder_output_size_val]] output_dims: [-1,*question_image_size_fusion_size_val] diff --git a/configs/vqa_med_2019/evaluation/tom/glove_lstm_resnet152_mcb_is_cat_ffn_c123_loss.yml b/configs/vqa_med_2019/evaluation/tom/glove_lstm_resnet152_mcb_is_cat_ffn_c123_loss.yml index f765357..bc2d2bb 100644 --- a/configs/vqa_med_2019/evaluation/tom/glove_lstm_resnet152_mcb_is_cat_ffn_c123_loss.yml +++ b/configs/vqa_med_2019/evaluation/tom/glove_lstm_resnet152_mcb_is_cat_ffn_c123_loss.yml @@ -1,4 +1,4 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/default_vqa_med_2019.yml #,vqa_med_2019/frozen_pipelines/frozen_question_categorization_glove_rnn_ffn.yml @@ -26,8 +26,8 @@ hyperparameters: question_encoder_output_size_val: &question_encoder_output_size_val 100 # Fusion I: image + question - question_image_fusion_type_val: &question_image_fusion_type MultimodalCompactBilinearPooling - # Options: ElementWiseMultiplication | MultimodalCompactBilinearPooling | VQA_Attention + question_image_fusion_type_val: &question_image_fusion_type CompactBilinearPooling + # Options: LowRankBilinearPooling | CompactBilinearPooling | QuestionDrivenAttention question_image_fusion_size_val: &question_image_fusion_size_val 100 # Image size encoder. @@ -45,7 +45,7 @@ hyperparameters: # Training parameters: training: - problem: + task: batch_size: *batch_size categories: C1,C2,C3 export_sample_weights: ~/data/vqa-med/answers.c1_c2_c3_binary_yn.weights.csv @@ -75,7 +75,7 @@ training: # Validation parameters: validation: - problem: + task: batch_size: *batch_size categories: C1,C2,C3 # Appy all preprocessing/data augmentations. @@ -159,7 +159,7 @@ pipeline: # Image encoder. image_encoder: priority: 2.1 - type: TorchVisionWrapper + type: GenericImageEncoder model: *image_encoder_model streams: inputs: images @@ -214,9 +214,9 @@ pipeline: # 5th subpipeline: concatenation concat: priority: 5.1 - type: Concatenation + type: ConcatenateTensor input_streams: [question_image_activations,image_size_activations] - # Concatenation + # ConcatenateTensor dim: 1 # default input_dims: [[-1,*question_image_fusion_size_val],[-1,*image_size_encoder_output_size_val]] output_dims: [-1,*question_image_size_fusion_size_val] diff --git a/configs/vqa_med_2019/evaluation/tom/glove_lstm_vgg16_att_is_cat_ffn_c123_loss.yml b/configs/vqa_med_2019/evaluation/tom/glove_lstm_vgg16_att_is_cat_ffn_c123_loss.yml index 9bfb01f..f4a5e1b 100644 --- a/configs/vqa_med_2019/evaluation/tom/glove_lstm_vgg16_att_is_cat_ffn_c123_loss.yml +++ b/configs/vqa_med_2019/evaluation/tom/glove_lstm_vgg16_att_is_cat_ffn_c123_loss.yml @@ -1,4 +1,4 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/default_vqa_med_2019.yml #,vqa_med_2019/frozen_pipelines/frozen_question_categorization_glove_rnn_ffn.yml @@ -27,10 +27,10 @@ hyperparameters: question_encoder_output_size_val: &question_encoder_output_size_val 100 # Fusion I: image + question - question_image_fusion_type_val: &question_image_fusion_type VQA_Attention - # Options: ElementWiseMultiplication | VQA_Attention + question_image_fusion_type_val: &question_image_fusion_type QuestionDrivenAttention + # Options: LowRankBilinearPooling | QuestionDrivenAttention #question_image_fusion_size_val: &question_image_fusion_size_val 1124 - # INFO: this variable is set by VQA_Attention component! + # INFO: this variable is set by QuestionDrivenAttention component! # Image size encoder. image_size_encoder_output_size_val: &image_size_encoder_output_size_val 10 @@ -47,7 +47,7 @@ hyperparameters: # Training parameters: training: - problem: + task: batch_size: *batch_size categories: C1,C2,C3 export_sample_weights: ~/data/vqa-med/answers.c1_c2_c3_binary_yn.weights.csv @@ -77,7 +77,7 @@ training: # Validation parameters: validation: - problem: + task: batch_size: *batch_size categories: C1,C2,C3 # Appy all preprocessing/data augmentations. @@ -161,7 +161,7 @@ pipeline: # Image encoder. image_encoder: priority: 2.1 - type: TorchVisionWrapper + type: GenericImageEncoder model: *image_encoder_model return_feature_maps: True streams: @@ -218,9 +218,9 @@ pipeline: # 5th subpipeline: concatenation concat: priority: 5.1 - type: Concatenation + type: ConcatenateTensor input_streams: [question_image_activations,image_size_activations] - # Concatenation + # ConcatenateTensor dim: 1 # default input_dims: [[-1,1124],[-1,*image_size_encoder_output_size_val]] output_dims: [-1,*question_image_size_fusion_size_val] diff --git a/configs/vqa_med_2019/evaluation/tom/glove_lstm_vgg16_ewm_is_cat_ffn_c123_loss.yml b/configs/vqa_med_2019/evaluation/tom/glove_lstm_vgg16_ewm_is_cat_ffn_c123_loss.yml index d72c1e6..765c77a 100644 --- a/configs/vqa_med_2019/evaluation/tom/glove_lstm_vgg16_ewm_is_cat_ffn_c123_loss.yml +++ b/configs/vqa_med_2019/evaluation/tom/glove_lstm_vgg16_ewm_is_cat_ffn_c123_loss.yml @@ -1,4 +1,4 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/default_vqa_med_2019.yml #,vqa_med_2019/frozen_pipelines/frozen_question_categorization_glove_rnn_ffn.yml @@ -26,8 +26,8 @@ hyperparameters: question_encoder_output_size_val: &question_encoder_output_size_val 100 # Fusion I: image + question - question_image_fusion_type_val: &question_image_fusion_type ElementWiseMultiplication - # Options: ElementWiseMultiplication | ? (component: question_image_fusion) + question_image_fusion_type_val: &question_image_fusion_type LowRankBilinearPooling + # Options: LowRankBilinearPooling | ? (component: question_image_fusion) question_image_fusion_size_val: &question_image_fusion_size_val 100 # Image size encoder. @@ -45,7 +45,7 @@ hyperparameters: # Training parameters: training: - problem: + task: batch_size: *batch_size categories: C1,C2,C3 export_sample_weights: ~/data/vqa-med/answers.c1_c2_c3_binary_yn.weights.csv @@ -75,7 +75,7 @@ training: # Validation parameters: validation: - problem: + task: batch_size: *batch_size categories: C1,C2,C3 # Appy all preprocessing/data augmentations. @@ -159,7 +159,7 @@ pipeline: # Image encoder. image_encoder: priority: 2.1 - type: TorchVisionWrapper + type: GenericImageEncoder model: *image_encoder_model streams: inputs: images @@ -214,9 +214,9 @@ pipeline: # 5th subpipeline: concatenation concat: priority: 5.1 - type: Concatenation + type: ConcatenateTensor input_streams: [question_image_activations,image_size_activations] - # Concatenation + # ConcatenateTensor dim: 1 # default input_dims: [[-1,*question_image_fusion_size_val],[-1,*image_size_encoder_output_size_val]] output_dims: [-1,*question_image_size_fusion_size_val] diff --git a/configs/vqa_med_2019/evaluation/tom/glove_lstm_vgg16_mcb_is_cat_ffn_c123_loss.yml b/configs/vqa_med_2019/evaluation/tom/glove_lstm_vgg16_mcb_is_cat_ffn_c123_loss.yml index 5a8bf30..3a5c78c 100644 --- a/configs/vqa_med_2019/evaluation/tom/glove_lstm_vgg16_mcb_is_cat_ffn_c123_loss.yml +++ b/configs/vqa_med_2019/evaluation/tom/glove_lstm_vgg16_mcb_is_cat_ffn_c123_loss.yml @@ -1,4 +1,4 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/default_vqa_med_2019.yml #,vqa_med_2019/frozen_pipelines/frozen_question_categorization_glove_rnn_ffn.yml @@ -26,8 +26,8 @@ hyperparameters: question_encoder_output_size_val: &question_encoder_output_size_val 100 # Fusion I: image + question - question_image_fusion_type_val: &question_image_fusion_type MultimodalCompactBilinearPooling - # Options: ElementWiseMultiplication | MultimodalCompactBilinearPooling | + question_image_fusion_type_val: &question_image_fusion_type CompactBilinearPooling + # Options: LowRankBilinearPooling | CompactBilinearPooling | question_image_fusion_size_val: &question_image_fusion_size_val 200 # Image size encoder. @@ -45,7 +45,7 @@ hyperparameters: # Training parameters: training: - problem: + task: batch_size: *batch_size categories: C1,C2,C3 export_sample_weights: ~/data/vqa-med/answers.c1_c2_c3_binary_yn.weights.csv @@ -75,7 +75,7 @@ training: # Validation parameters: validation: - problem: + task: batch_size: *batch_size categories: C1,C2,C3 # Appy all preprocessing/data augmentations. @@ -159,7 +159,7 @@ pipeline: # Image encoder. image_encoder: priority: 2.1 - type: TorchVisionWrapper + type: GenericImageEncoder model: *image_encoder_model streams: inputs: images @@ -214,9 +214,9 @@ pipeline: # 5th subpipeline: concatenation concat: priority: 5.1 - type: Concatenation + type: ConcatenateTensor input_streams: [question_image_activations,image_size_activations] - # Concatenation + # ConcatenateTensor dim: 1 # default input_dims: [[-1,*question_image_fusion_size_val],[-1,*image_size_encoder_output_size_val]] output_dims: [-1,*question_image_size_fusion_size_val] diff --git a/configs/vqa_med_2019/extend_answers.yml b/configs/vqa_med_2019/extend_answers.yml index eed7f39..59636c6 100644 --- a/configs/vqa_med_2019/extend_answers.yml +++ b/configs/vqa_med_2019/extend_answers.yml @@ -2,7 +2,7 @@ # It adds new sections (sets) without samplers and components for saving answers that we can use for getting final answers. training_answers: - problem: + task: type: &p_type VQAMED2019 data_folder: &data_folder ~/data/vqa-med split: training @@ -20,7 +20,7 @@ training_answers: num_workers: 1 validation_answers: - problem: + task: type: *p_type data_folder: *data_folder split: validation @@ -40,7 +40,7 @@ validation_answers: # Testing parameters: test_answers: - problem: + task: type: *p_type data_folder: *data_folder split: test_answers diff --git a/configs/vqa_med_2019/extend_answers_c4.yml b/configs/vqa_med_2019/extend_answers_c4.yml index 40a1381..a4bf26d 100644 --- a/configs/vqa_med_2019/extend_answers_c4.yml +++ b/configs/vqa_med_2019/extend_answers_c4.yml @@ -2,7 +2,7 @@ # It adds new sections (sets) without samplers and components for saving answers that we can use for getting final answers. training_answers: - problem: + task: type: &p_type VQAMED2019 data_folder: &data_folder ~/data/vqa-med split: training @@ -20,7 +20,7 @@ training_answers: num_workers: 1 validation_answers: - problem: + task: type: *p_type data_folder: *data_folder split: validation diff --git a/configs/vqa_med_2019/frozen_pipelines/frozen_input_fusion_glove_lstm_vgg_att_is_cat.yml b/configs/vqa_med_2019/frozen_pipelines/frozen_input_fusion_glove_lstm_vgg_att_is_cat.yml index 60a489b..1c9bd60 100644 --- a/configs/vqa_med_2019/frozen_pipelines/frozen_input_fusion_glove_lstm_vgg_att_is_cat.yml +++ b/configs/vqa_med_2019/frozen_pipelines/frozen_input_fusion_glove_lstm_vgg_att_is_cat.yml @@ -21,9 +21,9 @@ checkpoint: &checkpoint ~/image-clef-2019/experiments/c4_encoders/20190504_20244 # ~/image-clef-2019/experiments/c4_encoders/20190504_202441/checkpoints/glove_lstm_vgg16_att_is_cat_ffn_c123_loss_best.pt # + Model 'pipe1_question_embeddings' [SentenceEmbeddings] params saved # + Model 'pipe1_lstm' [RecurrentNeuralNetwork] params saved -# + Model 'image_encoder' [TorchVisionWrapper] params saved +# + Model 'image_encoder' [GenericImageEncoder] params saved # + Model 'image_size_encoder' [FeedForwardNetwork] params saved -# + Model 'question_image_fusion' [VQA_Attention] params saved +# + Model 'question_image_fusion' [QuestionDrivenAttention] params saved # + Model 'question_image_ffn' [FeedForwardNetwork] params saved # This one will be skipped, as this is C123 classifier! @@ -43,7 +43,7 @@ pipe_if0_hyperparameters: question_encoder_output_size_val: &question_encoder_output_size_val 100 # Fusion I: image + question - question_image_fusion_type_val: &question_image_fusion_type VQA_Attention + question_image_fusion_type_val: &question_image_fusion_type QuestionDrivenAttention # Image size encoder. image_size_encoder_output_size_val: &image_size_encoder_output_size_val 10 @@ -118,7 +118,7 @@ pipeline: # Image encoder. pipe_if2_image_encoder: priority: 2.1 - type: TorchVisionWrapper + type: GenericImageEncoder model: *image_encoder_model return_feature_maps: True # LOAD AND FREEZE # @@ -198,9 +198,9 @@ pipeline: # 5th subpipeline: concatenation pipe_if5_concat: priority: 5.1 - type: Concatenation + type: ConcatenateTensor input_streams: [question_image_activations,image_size_activations] - # Concatenation + # ConcatenateTensor dim: 1 # default input_dims: [[-1,1124],[-1,*image_size_encoder_output_size_val]] output_dims: [-1,*question_image_size_fusion_size_val] diff --git a/configs/vqa_med_2019/frozen_pipelines/input_fusion_processor_io.yml b/configs/vqa_med_2019/frozen_pipelines/input_fusion_processor_io.yml index 542461e..4e69bd3 100644 --- a/configs/vqa_med_2019/frozen_pipelines/input_fusion_processor_io.yml +++ b/configs/vqa_med_2019/frozen_pipelines/input_fusion_processor_io.yml @@ -1,11 +1,11 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/frozen_pipelines/frozen_input_fusion_glove_lstm_vgg_att_is_cat.yml, vqa_med_2019/frozen_pipelines/frozen_question_categorization_glove_rnn_ffn.yml training_validation: - problem: + task: type: &p_type VQAMED2019 data_folder: &data_folder ~/data/vqa-med split: training_validation diff --git a/configs/vqa_med_2019/question_categorization/default_question_categorization.yml b/configs/vqa_med_2019/question_categorization/default_question_categorization.yml index ee3873f..793fea1 100644 --- a/configs/vqa_med_2019/question_categorization/default_question_categorization.yml +++ b/configs/vqa_med_2019/question_categorization/default_question_categorization.yml @@ -1,8 +1,8 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/default_vqa_med_2019.yml training: - problem: + task: categories: all export_sample_weights: ~/data/vqa-med/answers.all.weights.csv # Do not load and stream images! @@ -14,7 +14,7 @@ training: loss_stop_threshold: 1.0e-3 validation: - problem: + task: categories: all # Do not load and stream images! stream_images: False diff --git a/configs/vqa_med_2019/question_categorization/question_categorization_onehot_bow.yml b/configs/vqa_med_2019/question_categorization/question_categorization_onehot_bow.yml index 29baea5..cfd5010 100644 --- a/configs/vqa_med_2019/question_categorization/question_categorization_onehot_bow.yml +++ b/configs/vqa_med_2019/question_categorization/question_categorization_onehot_bow.yml @@ -1,4 +1,4 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/question_categorization/default_question_categorization.yml pipeline: diff --git a/configs/vqa_med_2019/question_categorization/question_categorization_onehot_rnn.yml b/configs/vqa_med_2019/question_categorization/question_categorization_onehot_rnn.yml index bb7b77d..2bca202 100644 --- a/configs/vqa_med_2019/question_categorization/question_categorization_onehot_rnn.yml +++ b/configs/vqa_med_2019/question_categorization/question_categorization_onehot_rnn.yml @@ -1,4 +1,4 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/question_categorization/default_question_categorization.yml pipeline: diff --git a/configs/vqa_med_2019/question_categorization/question_categorization_rnn.yml b/configs/vqa_med_2019/question_categorization/question_categorization_rnn.yml index 8b53f64..f5324fa 100644 --- a/configs/vqa_med_2019/question_categorization/question_categorization_rnn.yml +++ b/configs/vqa_med_2019/question_categorization/question_categorization_rnn.yml @@ -1,4 +1,4 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/question_categorization/default_question_categorization.yml pipeline: diff --git a/configs/vqa_med_2019/question_categorization/question_categorization_rnn_ffn.yml b/configs/vqa_med_2019/question_categorization/question_categorization_rnn_ffn.yml index eb770d0..4ab1925 100644 --- a/configs/vqa_med_2019/question_categorization/question_categorization_rnn_ffn.yml +++ b/configs/vqa_med_2019/question_categorization/question_categorization_rnn_ffn.yml @@ -1,4 +1,4 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/question_categorization/default_question_categorization.yml pipeline: diff --git a/configs/vqa_med_2019/vf/c1_binary_vf_cat_hard_shared_question_rnn_two_ffns_losses.yml b/configs/vqa_med_2019/vf/c1_binary_vf_cat_hard_shared_question_rnn_two_ffns_losses.yml index adbc985..25d48b9 100644 --- a/configs/vqa_med_2019/vf/c1_binary_vf_cat_hard_shared_question_rnn_two_ffns_losses.yml +++ b/configs/vqa_med_2019/vf/c1_binary_vf_cat_hard_shared_question_rnn_two_ffns_losses.yml @@ -1,9 +1,9 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/default_vqa_med_2019.yml # Training parameters: training: - problem: + task: categories: C1 export_sample_weights: ~/data/vqa-med/answers.c1.weights.csv sampler: @@ -15,7 +15,7 @@ training: # Validation parameters: validation: - problem: + task: categories: C1 sampler: type: kFoldRandomSampler diff --git a/configs/vqa_med_2019/vf/c1_binary_vf_cat_rnn_shared_all_encoders_two_ffns_losses.yml b/configs/vqa_med_2019/vf/c1_binary_vf_cat_rnn_shared_all_encoders_two_ffns_losses.yml index 1450d46..4e81cce 100644 --- a/configs/vqa_med_2019/vf/c1_binary_vf_cat_rnn_shared_all_encoders_two_ffns_losses.yml +++ b/configs/vqa_med_2019/vf/c1_binary_vf_cat_rnn_shared_all_encoders_two_ffns_losses.yml @@ -1,9 +1,9 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/default_vqa_med_2019.yml # Training parameters: training: - problem: + task: categories: C1 export_sample_weights: ~/data/vqa-med/answers.c1.weights.csv sampler: @@ -15,7 +15,7 @@ training: # Validation parameters: validation: - problem: + task: categories: C1 sampler: type: kFoldRandomSampler @@ -182,7 +182,7 @@ pipeline: # Image encoder. image_encoder: - type: TorchVisionWrapper + type: GenericImageEncoder priority: 2.1 streams: inputs: images @@ -206,10 +206,10 @@ pipeline: ################# PIPE 4: SHARED CONCAT ################# concat: - type: Concatenation + type: ConcatenateTensor priority: 4.1 input_streams: [questions_activations,image_activations,image_size_activations] - # Concatenation + # ConcatenateTensor dim: 1 # default input_dims: [[-1,100],[-1,100],[-1,10]] output_dims: [-1,210] diff --git a/configs/vqa_med_2019/vf/c1_binary_vf_cat_rnn_shared_question_rnn_two_ffns_losses.yml b/configs/vqa_med_2019/vf/c1_binary_vf_cat_rnn_shared_question_rnn_two_ffns_losses.yml index 5b882cc..a2641ac 100644 --- a/configs/vqa_med_2019/vf/c1_binary_vf_cat_rnn_shared_question_rnn_two_ffns_losses.yml +++ b/configs/vqa_med_2019/vf/c1_binary_vf_cat_rnn_shared_question_rnn_two_ffns_losses.yml @@ -1,9 +1,9 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/default_vqa_med_2019.yml # Training parameters: training: - problem: + task: categories: C1 export_sample_weights: ~/data/vqa-med/answers.c1.weights.csv sampler: @@ -11,7 +11,7 @@ training: # Validation parameters: validation: - problem: + task: categories: C1 diff --git a/configs/vqa_med_2019/vf/c1_c2_c3_binary_vf_cat_rnn_shared_all_encoders_four_ffns_losses.yml b/configs/vqa_med_2019/vf/c1_c2_c3_binary_vf_cat_rnn_shared_all_encoders_four_ffns_losses.yml index 909a872..784461f 100644 --- a/configs/vqa_med_2019/vf/c1_c2_c3_binary_vf_cat_rnn_shared_all_encoders_four_ffns_losses.yml +++ b/configs/vqa_med_2019/vf/c1_c2_c3_binary_vf_cat_rnn_shared_all_encoders_four_ffns_losses.yml @@ -1,9 +1,9 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/default_vqa_med_2019.yml # Training parameters: training: - problem: + task: categories: C1,C2,C3 export_sample_weights: ~/data/vqa-med/answers.c1_c2_c3_binary_yn.weights.csv sampler: @@ -11,7 +11,7 @@ training: # Validation parameters: validation: - problem: + task: categories: C1,C2,C3 @@ -173,7 +173,7 @@ pipeline: # Image encoder. image_encoder: - type: TorchVisionWrapper + type: GenericImageEncoder priority: 2.1 streams: inputs: images @@ -197,10 +197,10 @@ pipeline: ################# PIPE 4: SHARED CONCAT ################# concat: - type: Concatenation + type: ConcatenateTensor priority: 4.1 input_streams: [questions_activations,image_activations,image_size_activations] - # Concatenation + # ConcatenateTensor dim: 1 # default input_dims: [[-1,100],[-1,100],[-1,10]] output_dims: [-1,210] diff --git a/configs/vqa_med_2019/vf/c1_c3_binary_vf_cat_rnn_shared_all_encoders_three_ffns_losses.yml b/configs/vqa_med_2019/vf/c1_c3_binary_vf_cat_rnn_shared_all_encoders_three_ffns_losses.yml index 46d967a..c406759 100644 --- a/configs/vqa_med_2019/vf/c1_c3_binary_vf_cat_rnn_shared_all_encoders_three_ffns_losses.yml +++ b/configs/vqa_med_2019/vf/c1_c3_binary_vf_cat_rnn_shared_all_encoders_three_ffns_losses.yml @@ -1,9 +1,9 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/default_vqa_med_2019.yml # Training parameters: training: - problem: + task: categories: C1,C3 export_sample_weights: ~/data/vqa-med/answers.c1_c3_binary_yn.weights.csv sampler: @@ -11,7 +11,7 @@ training: # Validation parameters: validation: - problem: + task: categories: C1,C3 @@ -173,7 +173,7 @@ pipeline: # Image encoder. image_encoder: - type: TorchVisionWrapper + type: GenericImageEncoder priority: 2.1 streams: inputs: images @@ -197,10 +197,10 @@ pipeline: ################# PIPE 4: SHARED CONCAT ################# concat: - type: Concatenation + type: ConcatenateTensor priority: 4.1 input_streams: [questions_activations,image_activations,image_size_activations] - # Concatenation + # ConcatenateTensor dim: 1 # default input_dims: [[-1,100],[-1,100],[-1,10]] output_dims: [-1,210] diff --git a/configs/vqa_med_2019/vf/lstm_resnet152_is_cat_ffn_c123_no_binary_loss.yml b/configs/vqa_med_2019/vf/lstm_resnet152_is_cat_ffn_c123_no_binary_loss.yml index 5a541f2..d4ad23e 100644 --- a/configs/vqa_med_2019/vf/lstm_resnet152_is_cat_ffn_c123_no_binary_loss.yml +++ b/configs/vqa_med_2019/vf/lstm_resnet152_is_cat_ffn_c123_no_binary_loss.yml @@ -1,9 +1,9 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/default_vqa_med_2019.yml # Training parameters: training: - problem: + task: categories: C1,C2,C3 export_sample_weights: ~/data/vqa-med/answers.c1_c2_c3_binary_yn.weights.csv # Appy all preprocessing/data augmentations. @@ -15,7 +15,7 @@ training: # Validation parameters: validation: - problem: + task: categories: C1,C2,C3 # Appy all preprocessing/data augmentations. question_preprocessing: lowercase,remove_punctuation,tokenize @@ -169,7 +169,7 @@ pipeline: # Image encoder. image_encoder: - type: TorchVisionWrapper + type: GenericImageEncoder model: resnet152 priority: 2.1 streams: @@ -195,10 +195,10 @@ pipeline: ################# PIPE 4: SHARED CONCAT ################# concat: - type: Concatenation + type: ConcatenateTensor priority: 4.1 input_streams: [questions_activations,image_activations,image_size_activations] - # Concatenation + # ConcatenateTensor dim: 1 # default input_dims: [[-1,100],[-1,100],[-1,10]] output_dims: [-1,210] diff --git a/configs/vqa_med_2019/vf/lstm_resnet50_ewm_is_cat_ffn_c123_loss_ffn_yn_loss.yml b/configs/vqa_med_2019/vf/lstm_resnet50_ewm_is_cat_ffn_c123_loss_ffn_yn_loss.yml index 1b3f29d..261a0a3 100644 --- a/configs/vqa_med_2019/vf/lstm_resnet50_ewm_is_cat_ffn_c123_loss_ffn_yn_loss.yml +++ b/configs/vqa_med_2019/vf/lstm_resnet50_ewm_is_cat_ffn_c123_loss_ffn_yn_loss.yml @@ -1,9 +1,9 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/default_vqa_med_2019.yml # Training parameters: training: - problem: + task: categories: C1,C2,C3 export_sample_weights: ~/data/vqa-med/answers.c1_c2_c3_binary_yn.weights.csv # Appy all preprocessing/data augmentations. @@ -15,7 +15,7 @@ training: # Validation parameters: validation: - problem: + task: categories: C1,C2,C3 # Appy all preprocessing/data augmentations. question_preprocessing: lowercase,remove_punctuation,tokenize @@ -170,7 +170,7 @@ pipeline: # Image encoder. image_encoder: priority: 2.1 - type: TorchVisionWrapper + type: GenericImageEncoder model: resnet50 streams: inputs: images @@ -196,7 +196,7 @@ pipeline: # Element wise multiplication + FF. question_image_fusion: priority: 4.1 - type: ElementWiseMultiplication + type: LowRankBilinearPooling dropout_rate: 0.5 streams: image_encodings: image_activations @@ -225,9 +225,9 @@ pipeline: # 5th subpipeline: concatenation concat: priority: 5.1 - type: Concatenation + type: ConcatenateTensor input_streams: [question_image_activations,image_size_activations] - # Concatenation + # ConcatenateTensor dim: 1 # default input_dims: [[-1,100],[-1,10]] output_dims: [-1,110] diff --git a/configs/vqa_med_2019/vf/lstm_resnet50_ewm_is_cat_ffn_c123_no_binary_loss.yml b/configs/vqa_med_2019/vf/lstm_resnet50_ewm_is_cat_ffn_c123_no_binary_loss.yml index 2a46463..cf01ba1 100644 --- a/configs/vqa_med_2019/vf/lstm_resnet50_ewm_is_cat_ffn_c123_no_binary_loss.yml +++ b/configs/vqa_med_2019/vf/lstm_resnet50_ewm_is_cat_ffn_c123_no_binary_loss.yml @@ -1,9 +1,9 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/default_vqa_med_2019.yml # Training parameters: training: - problem: + task: categories: C1,C2,C3 export_sample_weights: ~/data/vqa-med/answers.c1_c2_c3_binary_yn.weights.csv # Appy all preprocessing/data augmentations. @@ -15,7 +15,7 @@ training: # Validation parameters: validation: - problem: + task: categories: C1,C2,C3 # Appy all preprocessing/data augmentations. question_preprocessing: lowercase,remove_punctuation,tokenize @@ -170,7 +170,7 @@ pipeline: # Image encoder. image_encoder: priority: 2.1 - type: TorchVisionWrapper + type: GenericImageEncoder model: resnet50 streams: inputs: images @@ -196,7 +196,7 @@ pipeline: # Element wise multiplication + FF. question_image_fusion: priority: 4.1 - type: ElementWiseMultiplication + type: LowRankBilinearPooling dropout_rate: 0.5 streams: image_encodings: image_activations @@ -225,9 +225,9 @@ pipeline: # 5th subpipeline: concatenation concat: priority: 5.1 - type: Concatenation + type: ConcatenateTensor input_streams: [question_image_activations,image_size_activations] - # Concatenation + # ConcatenateTensor dim: 1 # default input_dims: [[-1,100],[-1,10]] output_dims: [-1,110] diff --git a/configs/vqa_med_2019/vf/lstm_resnet50_is_cat_ffn_c123_no_binary_loss.yml b/configs/vqa_med_2019/vf/lstm_resnet50_is_cat_ffn_c123_no_binary_loss.yml index 2364e06..cc0f555 100644 --- a/configs/vqa_med_2019/vf/lstm_resnet50_is_cat_ffn_c123_no_binary_loss.yml +++ b/configs/vqa_med_2019/vf/lstm_resnet50_is_cat_ffn_c123_no_binary_loss.yml @@ -1,9 +1,9 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/default_vqa_med_2019.yml # Training parameters: training: - problem: + task: categories: C1,C2,C3 export_sample_weights: ~/data/vqa-med/answers.c1_c2_c3_binary_yn.weights.csv # Appy all preprocessing/data augmentations. @@ -15,7 +15,7 @@ training: # Validation parameters: validation: - problem: + task: categories: C1,C2,C3 # Appy all preprocessing/data augmentations. question_preprocessing: lowercase,remove_punctuation,tokenize @@ -169,7 +169,7 @@ pipeline: # Image encoder. image_encoder: - type: TorchVisionWrapper + type: GenericImageEncoder model: resnet50 priority: 2.1 streams: @@ -195,10 +195,10 @@ pipeline: ################# PIPE 4: SHARED CONCAT ################# concat: - type: Concatenation + type: ConcatenateTensor priority: 4.1 input_streams: [questions_activations,image_activations,image_size_activations] - # Concatenation + # ConcatenateTensor dim: 1 # default input_dims: [[-1,100],[-1,100],[-1,10]] output_dims: [-1,210] diff --git a/configs/vqa_med_2019/vf/lstm_vgg16_is_cat_ffn_c123_binary_yn_loss.yml b/configs/vqa_med_2019/vf/lstm_vgg16_is_cat_ffn_c123_binary_yn_loss.yml index 71c3946..32c74b2 100644 --- a/configs/vqa_med_2019/vf/lstm_vgg16_is_cat_ffn_c123_binary_yn_loss.yml +++ b/configs/vqa_med_2019/vf/lstm_vgg16_is_cat_ffn_c123_binary_yn_loss.yml @@ -1,9 +1,9 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/default_vqa_med_2019.yml # Training parameters: training: - problem: + task: categories: C1,C2,C3 export_sample_weights: ~/data/vqa-med/answers.c1_c2_c3_binary_yn.weights.csv # Appy all preprocessing/data augmentations. @@ -15,7 +15,7 @@ training: # Validation parameters: validation: - problem: + task: categories: C1,C2,C3 # Appy all preprocessing/data augmentations. question_preprocessing: lowercase,remove_punctuation,tokenize @@ -168,7 +168,7 @@ pipeline: # Image encoder. image_encoder: - type: TorchVisionWrapper + type: GenericImageEncoder model: vgg16 priority: 2.1 streams: @@ -193,10 +193,10 @@ pipeline: ################# PIPE 4: SHARED CONCAT ################# concat: - type: Concatenation + type: ConcatenateTensor priority: 4.1 input_streams: [questions_activations,image_activations,image_size_activations] - # Concatenation + # ConcatenateTensor dim: 1 # default input_dims: [[-1,100],[-1,100],[-1,10]] output_dims: [-1,210] diff --git a/configs/vqa_med_2019/vf/lstm_vgg16_is_cat_ffn_c123_no_yn_loss.yml b/configs/vqa_med_2019/vf/lstm_vgg16_is_cat_ffn_c123_no_yn_loss.yml index 1bf7bdc..ce52710 100644 --- a/configs/vqa_med_2019/vf/lstm_vgg16_is_cat_ffn_c123_no_yn_loss.yml +++ b/configs/vqa_med_2019/vf/lstm_vgg16_is_cat_ffn_c123_no_yn_loss.yml @@ -1,9 +1,9 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/default_vqa_med_2019.yml # Training parameters: training: - problem: + task: categories: C1,C2,C3 export_sample_weights: ~/data/vqa-med/answers.c1_c2_c3_binary_yn.weights.csv # Appy all preprocessing/data augmentations. @@ -15,7 +15,7 @@ training: # Validation parameters: validation: - problem: + task: categories: C1,C2,C3 # Appy all preprocessing/data augmentations. question_preprocessing: lowercase,remove_punctuation,tokenize @@ -168,7 +168,7 @@ pipeline: # Image encoder. image_encoder: - type: TorchVisionWrapper + type: GenericImageEncoder model: vgg16 priority: 2.1 streams: @@ -193,10 +193,10 @@ pipeline: ################# PIPE 4: SHARED CONCAT ################# concat: - type: Concatenation + type: ConcatenateTensor priority: 4.1 input_streams: [questions_activations,image_activations,image_size_activations] - # Concatenation + # ConcatenateTensor dim: 1 # default input_dims: [[-1,100],[-1,100],[-1,10]] output_dims: [-1,210] diff --git a/configs/vqa_med_2019/vf/lstm_vgg16_is_cat_ffn_only_yn_loss.yml b/configs/vqa_med_2019/vf/lstm_vgg16_is_cat_ffn_only_yn_loss.yml index 7cbe09e..b39acbc 100644 --- a/configs/vqa_med_2019/vf/lstm_vgg16_is_cat_ffn_only_yn_loss.yml +++ b/configs/vqa_med_2019/vf/lstm_vgg16_is_cat_ffn_only_yn_loss.yml @@ -1,9 +1,9 @@ -# Load config defining problems for training, validation and testing. +# Load config defining tasks for training, validation and testing. default_configs: vqa_med_2019/default_vqa_med_2019.yml # Training parameters: training: - problem: + task: categories: C1,C2,C3 export_sample_weights: ~/data/vqa-med/answers.binary_yn.weights.csv # Appy all preprocessing/data augmentations. @@ -15,7 +15,7 @@ training: # Validation parameters: validation: - problem: + task: categories: C1,C2,C3 # Appy all preprocessing/data augmentations. question_preprocessing: lowercase,remove_punctuation,tokenize @@ -168,7 +168,7 @@ pipeline: # Image encoder. image_encoder: - type: TorchVisionWrapper + type: GenericImageEncoder model: vgg16 priority: 2.1 streams: @@ -193,10 +193,10 @@ pipeline: ################# PIPE 4: SHARED CONCAT ################# concat: - type: Concatenation + type: ConcatenateTensor priority: 4.1 input_streams: [questions_activations,image_activations,image_size_activations] - # Concatenation + # ConcatenateTensor dim: 1 # default input_dims: [[-1,100],[-1,100],[-1,10]] output_dims: [-1,210] diff --git a/configs/wikitext/wikitext_language_modeling_encoder_attndecoder.yml b/configs/wikitext/wikitext_language_modeling_encoder_attndecoder.yml index dbfe301..6c5d3e4 100644 --- a/configs/wikitext/wikitext_language_modeling_encoder_attndecoder.yml +++ b/configs/wikitext/wikitext_language_modeling_encoder_attndecoder.yml @@ -6,7 +6,7 @@ # Training parameters: training: - problem: + task: type: &p_type WikiTextLanguageModeling data_folder: &data_folder ~/data/language_modeling/wikitext-2 dataset: &dataset wikitext-2 @@ -28,7 +28,7 @@ training: # Validation parameters: validation: partial_validation_interval: 100 - problem: + task: type: *p_type data_folder: *data_folder dataset: *dataset @@ -38,7 +38,7 @@ validation: # Testing parameters: test: - problem: + task: type: *p_type data_folder: *data_folder dataset: *dataset @@ -95,7 +95,7 @@ pipeline: # LSTM Decoder decoder: - type: Attn_Decoder_RNN + type: AttentionDecoder priority: 4 hidden_size: 50 num_layers: 1 diff --git a/configs/wikitext/wikitext_language_modeling_rnn.yml b/configs/wikitext/wikitext_language_modeling_rnn.yml index d92e13d..c7d4313 100644 --- a/configs/wikitext/wikitext_language_modeling_rnn.yml +++ b/configs/wikitext/wikitext_language_modeling_rnn.yml @@ -1,6 +1,6 @@ # Training parameters: training: - problem: + task: type: &p_type WikiTextLanguageModeling data_folder: &data_folder ~/data/language_modeling/wikitext-2 dataset: &dataset wikitext-2 @@ -22,7 +22,7 @@ training: # Validation parameters: validation: partial_validation_interval: 10 - problem: + task: type: *p_type data_folder: *data_folder dataset: *dataset @@ -32,7 +32,7 @@ validation: # Testing parameters: test: - problem: + task: type: *p_type data_folder: *data_folder dataset: *dataset diff --git a/configs/wikitext/wikitext_language_modeling_seq2seq.yml b/configs/wikitext/wikitext_language_modeling_seq2seq.yml index e9b2388..eeff49d 100644 --- a/configs/wikitext/wikitext_language_modeling_seq2seq.yml +++ b/configs/wikitext/wikitext_language_modeling_seq2seq.yml @@ -6,7 +6,7 @@ # Training parameters: training: - problem: + task: type: &p_type WikiTextLanguageModeling data_folder: &data_folder ~/data/language_modeling/wikitext-2 dataset: &dataset wikitext-2 @@ -28,7 +28,7 @@ training: # Validation parameters: validation: partial_validation_interval: 100 - problem: + task: type: *p_type data_folder: *data_folder dataset: *dataset @@ -38,7 +38,7 @@ validation: # Testing parameters: test: - problem: + task: type: *p_type data_folder: *data_folder dataset: *dataset diff --git a/configs/wikitext/wikitext_language_modeling_seq2seq_simple.yml b/configs/wikitext/wikitext_language_modeling_seq2seq_simple.yml index 606e506..29b95a4 100644 --- a/configs/wikitext/wikitext_language_modeling_seq2seq_simple.yml +++ b/configs/wikitext/wikitext_language_modeling_seq2seq_simple.yml @@ -2,11 +2,11 @@ # It's been made for test purposes only, as it is doing: # [word 0 , ... , word 49] -> [word 1 , ... , word 50] (basically copying most of the input) # -# The seq2seq here is implemented throught the use of a simplified seq2seq component `Seq2Seq_RNN` +# The seq2seq here is implemented throught the use of a simplified seq2seq component `Seq2Seq` # Training parameters: training: - problem: + task: type: &p_type WikiTextLanguageModeling data_folder: &data_folder ~/data/language_modeling/wikitext-2 dataset: &dataset wikitext-2 @@ -28,7 +28,7 @@ training: # Validation parameters: validation: partial_validation_interval: 100 - problem: + task: type: *p_type data_folder: *data_folder dataset: *dataset @@ -38,7 +38,7 @@ validation: # Testing parameters: test: - problem: + task: type: *p_type data_folder: *data_folder dataset: *dataset @@ -75,7 +75,7 @@ pipeline: # LSTM seq2seq lstm_encoder: - type: Seq2Seq_RNN + type: Seq2Seq priority: 3 initial_state: Trainable hidden_size: 300 diff --git a/configs/wily/dummy_language_identification_bow.yml b/configs/wily/dummy_language_identification_bow.yml index 6a31ac7..b3b2fff 100644 --- a/configs/wily/dummy_language_identification_bow.yml +++ b/configs/wily/dummy_language_identification_bow.yml @@ -1,6 +1,6 @@ # Training parameters: training: - problem: + task: type: &p_type DummyLanguageIdentification batch_size: 2 use_train_data: True @@ -22,7 +22,7 @@ training: # Validation parameters: validation: partial_validation_interval: 10 - problem: + task: type: *p_type batch_size: 2 use_train_data: True @@ -30,7 +30,7 @@ validation: # Testing parameters: test: - problem: + task: type: *p_type batch_size: 2 use_train_data: False diff --git a/configs/wily/wily_language_identification_bow.yml b/configs/wily/wily_language_identification_bow.yml index b5f4c29..2c9e4cd 100644 --- a/configs/wily/wily_language_identification_bow.yml +++ b/configs/wily/wily_language_identification_bow.yml @@ -1,6 +1,6 @@ # Training parameters: training: - problem: + task: type: &p_type WiLYLanguageIdentification data_folder: &data_folder '~/data/language_identification/wily' batch_size: 64 @@ -27,7 +27,7 @@ training: # Validation parameters: validation: partial_validation_interval: 10 - problem: + task: type: *p_type data_folder: *data_folder batch_size: 64 @@ -40,7 +40,7 @@ validation: # Testing parameters: test: - problem: + task: type: *p_type data_folder: *data_folder batch_size: 64 diff --git a/configs/wily/wily_ngram_language_modeling.yml b/configs/wily/wily_ngram_language_modeling.yml index 65b2650..ec5d56f 100644 --- a/configs/wily/wily_ngram_language_modeling.yml +++ b/configs/wily/wily_ngram_language_modeling.yml @@ -1,6 +1,6 @@ # Training parameters: training: - problem: + task: type: &p_type WiLYNGramLanguageModeling data_folder: &data_folder '~/data/language_identification/wily' context: &context 2 @@ -29,7 +29,7 @@ training: # Validation parameters: validation: partial_validation_interval: 10 - problem: + task: type: *p_type data_folder: *data_folder context: *context @@ -44,7 +44,7 @@ validation: # Testing parameters: test: - problem: + task: type: *p_type data_folder: *data_folder context: *context diff --git a/ptp/__init__.py b/ptp/__init__.py index a9c69b2..f6a2601 100644 --- a/ptp/__init__.py +++ b/ptp/__init__.py @@ -5,24 +5,26 @@ # Components. from .components.component import Component +from .components.language import * + from .components.losses import * from .components.masking import * from .components.models import * -from .components.problems.problem import Problem -from .components.problems.image_text_to_class import * -from .components.problems.image_to_class import * -from .components.problems.text_to_class import * -from .components.problems.text_to_text import * +from .components.tasks.task import Task +from .components.tasks.image_text_to_class import * +from .components.tasks.image_to_class import * +from .components.tasks.text_to_class import * +from .components.tasks.text_to_text import * from .components.publishers import * -from .components.text import * - from .components.transforms import * +from .components.statistics import * + from .components.viewers import * # Rest. diff --git a/ptp/application/__init__.py b/ptp/application/__init__.py index ad55e93..568c266 100644 --- a/ptp/application/__init__.py +++ b/ptp/application/__init__.py @@ -1,11 +1,11 @@ from .component_factory import ComponentFactory from .pipeline_manager import PipelineManager -from .problem_manager import ProblemManager +from .task_manager import TaskManager from .sampler_factory import SamplerFactory __all__ = [ 'ComponentFactory', 'PipelineManager', - 'ProblemManager', + 'TaskManager', 'SamplerFactory', ] diff --git a/ptp/application/pipeline_manager.py b/ptp/application/pipeline_manager.py index 3f48cc2..492f20a 100644 --- a/ptp/application/pipeline_manager.py +++ b/ptp/application/pipeline_manager.py @@ -28,7 +28,7 @@ from ptp.utils.app_state import AppState from ptp.configuration.configuration_error import ConfigurationError from ptp.application.component_factory import ComponentFactory -from ptp.utils.data_dict_parallel import DataDictParallel +from ptp.utils.data_streams_parallel import DataStreamsParallel components_to_skip_in_data_parallel = ["SentenceEmbeddings", "IndexEmbeddings"] @@ -74,7 +74,7 @@ def build(self, use_logger=True): """ Method creating the pipeline, consisting of: - a list components ordered by the priority (dictionary). - - problem (as a separate "link" to object in the list of components, instance of a class derrived from Problem class) + - task (as a separate "link" to object in the list of components, instance of a class derrived from Task class) - models (separate list with link to objects in components dict) - losses (selarate list with links to objects in components dict) @@ -162,10 +162,10 @@ def build(self, use_logger=True): # Create component. component, class_obj = ComponentFactory.build(c_key, c_config) - # Check if class is derived (even indirectly) from Problem. - if ComponentFactory.check_inheritance(class_obj, ptp.Problem.__name__): + # Check if class is derived (even indirectly) from Task. + if ComponentFactory.check_inheritance(class_obj, ptp.Task.__name__): raise ConfigurationError("Object '{}' cannot be instantiated as part of pipeline, \ - as its class type '{}' is derived from Problem class!".format(c_key, class_obj.__name__)) + as its class type '{}' is derived from Task class!".format(c_key, class_obj.__name__)) # Add it to dict. self.__components[c_priority] = component @@ -222,7 +222,7 @@ def save(self, chkpt_dir, training_status, loss): # Save state dicts of all models. for model in self.models: # Check if model is wrapped in dataparallel. - if (type(model).__name__ == "DataDictParallel"): + if (type(model).__name__ == "DataStreamsParallel"): model.module.save_to_checkpoint(chkpt) model_str += " + Model '{}' [{}] params saved \n".format(model.module.name, type(model.module).__name__) else: @@ -414,7 +414,7 @@ def __getitem__(self, number): def __len__(self): """ - Returns the number of objects in the pipeline (excluding problems) + Returns the number of objects in the pipeline (excluding tasks) :return: Length of the :py:class:`Pipeline`. """ @@ -442,7 +442,7 @@ def summarize_all_components_header(self): def summarize_all_components(self): """ - Summarizes the pipeline by showing all its components (excluding problem). + Summarizes the pipeline by showing all its components (excluding task). :return: Summary as a str. """ @@ -475,7 +475,7 @@ def summarize_models_header(self): def summarize_models(self): """ - Summarizes the pipeline by showing all its components (excluding problem). + Summarizes the pipeline by showing all its components (excluding task). :return: Summary as a str. """ @@ -485,11 +485,11 @@ def summarize_models(self): return summary_str - def handshake(self, data_dict, log=True): + def handshake(self, data_streams, log=True): """ Performs handshaking of inputs and outputs definitions of all components in the pipeline. - :param data_dict: Initial datadict returned by the problem. + :param data_streams: Initial datadict returned by the task. :param log: Logs the detected errors and info (DEFAULT: True) @@ -501,15 +501,15 @@ def handshake(self, data_dict, log=True): # Get component comp = self.__components[prio] # Handshake inputs and outputs. - errors += comp.handshake_input_definitions(data_dict, log) - errors += comp.export_output_definitions(data_dict, log) + errors += comp.handshake_input_definitions(data_streams, log) + errors += comp.export_output_definitions(data_streams, log) # Log final definition. if errors == 0 and log: self.logger.info("Handshake successfull") - def_str = "Final definition of DataDict used in pipeline:\n" + def_str = "Final definition of DataStreams used in pipeline:\n" def_str += '='*80 + '\n' - for item in data_dict.items(): + for item in data_streams.items(): def_str += ' {}\n'.format(item) def_str += '='*80 + '\n' self.logger.info(def_str) @@ -517,30 +517,30 @@ def handshake(self, data_dict, log=True): return errors - def forward(self, data_dict): + def forward(self, data_streams): """ Method responsible for processing the data dict, using all components in the components queue. - :param data_dict: :py:class:`ptp.utils.DataDict` object containing both input data to be processed and that will be extended by the results. + :param data_streams: :py:class:`ptp.utils.DataStreams` object containing both input data to be processed and that will be extended by the results. """ if self.app_state.args.use_gpu: - data_dict.to(device = self.app_state.device) + data_streams.to(device = self.app_state.device) for prio in self.__priorities: # Get component comp = self.__components[prio] - if (type(comp).__name__ == "DataDictParallel"): - # Forward of wrapper returns outputs in separate DataDict. - outputs = comp(data_dict) + if (type(comp).__name__ == "DataStreamsParallel"): + # Forward of wrapper returns outputs in separate DataStreams. + outputs = comp(data_streams) # Postprocessing: copy only the outputs of the wrapped model. for key in comp.module.output_data_definitions().keys(): - data_dict.extend({key: outputs[key]}) + data_streams.publish({key: outputs[key]}) else: # "Normal" forward step. - comp(data_dict) + comp(data_streams) # Move data to device. - data_dict.to(device = self.app_state.device) + data_streams.to(device = self.app_state.device) def eval(self): @@ -574,10 +574,10 @@ def cuda(self): # Check if class is derived (even indirectly) from Model. if ComponentFactory.check_inheritance(type(component), ptp.Model.__name__): model = component - # Wrap model with DataDictParallel when required. + # Wrap model with DataStreamsParallel when required. if self.app_state.use_dataparallel and type(model).__name__ not in components_to_skip_in_data_parallel: print("Moving to GPU", model.name) - model = DataDictParallel(model) + model = DataStreamsParallel(model) # Mode to cuda. model.to(self.app_state.device) @@ -594,12 +594,12 @@ def zero_grad(self): model.zero_grad() - def backward(self, data_dict): + def backward(self, data_streams): """ Propagates gradients backwards, starting from losses returned by every loss component in the pipeline. If using many losses the components derived from loss must overwrite the ''loss_keys()'' method. - :param data_dict: :py:class:`ptp.utils.DataDict` object containing both input data to be processed and that will be extended by the results. + :param data_streams: :py:class:`ptp.utils.DataStreams` object containing both input data to be processed and that will be extended by the results. """ if (len(self.losses) == 0): @@ -614,17 +614,17 @@ def backward(self, data_dict): pass_counter += 1 if pass_counter == total_passes: # Last pass. - data_dict[key].backward() + data_streams[key].backward() else: # "Other pass." - data_dict[key].backward(retain_graph=True) + data_streams[key].backward(retain_graph=True) def return_loss_on_batch(self, stat_col): """ Sums all losses and returns a single value that can be used e.g. in terminal condition or model(s) saving. - :param data_dict: :py:class:`ptp.utils.DataDict` object containing both input data to be processed and that will be extended by the results. + :param data_streams: :py:class:`ptp.utils.DataStreams` object containing both input data to be processed and that will be extended by the results. :return: Loss (scalar value). """ @@ -635,7 +635,7 @@ def return_loss_on_set(self, stat_agg): """ Sums all losses and returns a single value that can be used e.g. in terminal condition or model(s) saving. - :param data_dict: :py:class:`ptp.utils.DataDict` object containing both input data to be processed and that will be extended by the results. + :param data_streams: :py:class:`ptp.utils.DataStreams` object containing both input data to be processed and that will be extended by the results. :return: Loss (scalar value). """ @@ -700,27 +700,27 @@ def add_statistics(self, stat_col): stat_col.add_statistics("total_loss_support", None) - def collect_statistics(self, stat_col, data_dict): + def collect_statistics(self, stat_col, data_streams): """ Collects statistics for every component in the pipeline. :param stat_col: :py:class:`ptp.utils.StatisticsCollector`. - :param data_dict: ``DataDict`` containing inputs, targets etc. - :type data_dict: :py:class:`ptp.core_types.DataDict` + :param data_streams: ``DataStreams`` containing inputs, targets etc. + :type data_streams: :py:class:`ptp.data_types.DataStreams` """ for prio in self.__priorities: comp = self.__components[prio] - comp.collect_statistics(stat_col, data_dict) + comp.collect_statistics(stat_col, data_streams) # Additional "total loss" (for single- and multi-loss pipelines). loss_sum = 0 for loss in self.losses: for key in loss.loss_keys(): - loss_sum += data_dict[key].cpu().item() + loss_sum += data_streams[key].cpu().item() stat_col["total_loss"] = loss_sum - stat_col["total_loss_support"] = len(data_dict["indices"]) # batch size + stat_col["total_loss_support"] = len(data_streams["indices"]) # batch size def add_aggregators(self, stat_agg): diff --git a/ptp/application/sampler_factory.py b/ptp/application/sampler_factory.py index d3bbbae..8684e10 100644 --- a/ptp/application/sampler_factory.py +++ b/ptp/application/sampler_factory.py @@ -34,18 +34,18 @@ class SamplerFactory(object): """ @staticmethod - def build(problem, config, problem_subset_name): + def build(task, config, task_subset_name): """ Static method returning particular sampler, depending on the name \ - provided in the list of parameters & the specified problem class. + provided in the list of parameters & the specified task class. - :param problem: Instance of an object derived from the Problem class. - :type problem: ``problems.Problem`` + :param task: Instance of an object derived from the Task class. + :type task: ``tasks.Task`` :param config: Parameters used to instantiate the sampler. :type config: :py:class:`ptp.configuration.ConfigInterface` - :param problem_subset_name: Name of problem subset (and associated ProblemManager object) + :param task_subset_name: Name of task subset (and associated TaskManager object) ..note:: @@ -135,10 +135,10 @@ def build(problem, config, problem_subset_name): # Else: use them as they are, including single index. # Check if indices are within range. - if max(indices) >= len(problem): + if max(indices) >= len(task): raise ConfigurationError("SubsetRandomSampler cannot work properly when indices are out of range ({}) " - "considering that there are {} samples in the problem".format( - max(indices), len(problem))) + "considering that there are {} samples in the task".format( + max(indices), len(task))) # Create the sampler object. sampler = pt_samplers.SubsetRandomSampler(indices) @@ -156,7 +156,7 @@ def build(problem, config, problem_subset_name): weights = np.fromfile(os.path.expanduser(config['weights']), dtype=float, count=-1, sep=',') # Create sampler class. - sampler = pt_samplers.WeightedRandomSampler(weights, len(problem), replacement=True) + sampler = pt_samplers.WeightedRandomSampler(weights, len(task), replacement=True) ########################################################################### # Handle third special case: kFoldRandomSampler. @@ -175,7 +175,7 @@ def build(problem, config, problem_subset_name): epochs_per_fold = config.get("epochs_per_fold", 1) # Create the sampler object. - sampler = ptp_samplers.kFoldRandomSampler(len(problem), folds, epochs_per_fold, problem_subset_name == 'training') + sampler = ptp_samplers.kFoldRandomSampler(len(task), folds, epochs_per_fold, task_subset_name == 'training') ########################################################################### # Handle fourd special case: kFoldWeightedRandomSampler. @@ -202,7 +202,7 @@ def build(problem, config, problem_subset_name): epochs_per_fold = config.get("epochs_per_fold", 1) # Create the sampler object. - sampler = ptp_samplers.kFoldWeightedRandomSampler(weights, len(problem), folds, epochs_per_fold, problem_subset_name == 'training') + sampler = ptp_samplers.kFoldWeightedRandomSampler(weights, len(task), folds, epochs_per_fold, task_subset_name == 'training') elif typename in ['BatchSampler', 'DistributedSampler']: # Sorry, don't support those. Yet;) @@ -216,7 +216,7 @@ def build(problem, config, problem_subset_name): # Get the sampler class. sampler_class = getattr(pt_samplers, typename) # Create "regular" sampler. - sampler = sampler_class(problem) + sampler = sampler_class(task) # Return sampler. return sampler diff --git a/ptp/application/problem_manager.py b/ptp/application/task_manager.py similarity index 77% rename from ptp/application/problem_manager.py rename to ptp/application/task_manager.py index bb48460..b940c28 100644 --- a/ptp/application/problem_manager.py +++ b/ptp/application/task_manager.py @@ -31,18 +31,18 @@ from ptp.application.sampler_factory import SamplerFactory -class ProblemManager(object): +class TaskManager(object): """ - Class that instantiates and manages problem and associated entities (dataloader, sampler etc.). + Class that instantiates and manages task and associated entities (dataloader, sampler etc.). """ def __init__(self, name, config): """ Initializes the manager. - :param name: Name of the manager (e.g. 'training', 'validation'). + :param name: Name of the manager (associated with a given config section e.g. 'training', 'validation'). - :param config: 'ConfigInterface' object, referring to one of main sections (training/validation/testing/...). + :param config: 'ConfigInterface' object, referring to one of main sections (training/validation/test/...). :type config: :py:class:`ptp.configuration.ConfigInterface` """ @@ -55,7 +55,7 @@ def __init__(self, name, config): # Initialize logger. self.logger = logging.initialize_logger(self.name) - # Single batch that will be used for validation (for validation problem manager). + # Single batch that will be used for validation (for validation task manager). self.batch = None @@ -84,7 +84,7 @@ def worker_init_fn(self, worker_id): def build(self, log=True): """ - Method creates a problem on the basis of configuration section. + Method creates a task on the basis of configuration section. :param log: Logs information and the detected errors (DEFAULT: TRUE) @@ -92,14 +92,14 @@ def build(self, log=True): """ try: # Create component. - component, class_obj = ComponentFactory.build("problem", self.config["problem"]) + component, class_obj = ComponentFactory.build("task", self.config["task"]) - # Check if class is derived (even indirectly) from Problem. - if not ComponentFactory.check_inheritance(class_obj, ptp.Problem.__name__): - raise ConfigurationError("Class '{}' is not derived from the Problem class!".format(class_obj.__name__)) + # Check if class is derived (even indirectly) from Task. + if not ComponentFactory.check_inheritance(class_obj, ptp.Task.__name__): + raise ConfigurationError("Class '{}' is not derived from the Task class!".format(class_obj.__name__)) - # Set problem. - self.problem = component + # Set task. + self.task = component # Try to build the sampler. # Check if sampler is required, i.e. 'sampler' section is empty. @@ -108,18 +108,18 @@ def build(self, log=True): # Set sampler to none. self.sampler = None else: - self.sampler = SamplerFactory.build(self.problem, self.config["sampler"], self.name) + self.sampler = SamplerFactory.build(self.task, self.config["sampler"], self.name) # Set shuffle to False - REQUIRED as those two are exclusive. self.config['dataloader'].add_config_params({'shuffle': False}) - # build the DataLoader on top of the validation problem - self.dataloader = DataLoader(dataset=self.problem, - batch_size=self.config['problem']['batch_size'], + # build the DataLoader on top of the validation task + self.dataloader = DataLoader(dataset=self.task, + batch_size=self.config['task']['batch_size'], shuffle=self.config['dataloader']['shuffle'], sampler=self.sampler, batch_sampler= None, num_workers=self.config['dataloader']['num_workers'], - collate_fn=self.problem.collate_fn, + collate_fn=self.task.collate_fn, pin_memory=self.config['dataloader']['pin_memory'], drop_last=self.config['dataloader']['drop_last'], timeout=self.config['dataloader']['timeout'], @@ -127,7 +127,7 @@ def build(self, log=True): # Display sizes. if log: - self.logger.info("Problem for '{}' loaded (size: {})".format(self.name, len(self.problem))) + self.logger.info("Task for '{}' loaded (size: {})".format(self.name, len(self.task))) if (self.sampler is not None): self.logger.info("Sampler for '{}' created (size: {})".format(self.name, len(self.sampler))) @@ -136,12 +136,12 @@ def build(self, log=True): except ConfigurationError as e: if log: - self.logger.error("Detected configuration error while creating the problem instance:\n {}".format(e)) + self.logger.error("Detected configuration error while creating the task instance:\n {}".format(e)) # Return error. return 1 except KeyError as e: if log: - self.logger.error("Detected key error while creating the problem instance: required key {} is missing".format(e)) + self.logger.error("Detected key error while creating the task instance: required key {} is missing".format(e)) # Return error. return 1 @@ -156,7 +156,7 @@ def __len__(self): elif self.sampler is not None: total_num_samples = len(self.sampler) else: - total_num_samples = len(self.problem) + total_num_samples = len(self.task) return total_num_samples @@ -181,23 +181,23 @@ def get_epoch_size(self): """ # "Estimate" dataset size. if (self.sampler is not None): - problem_size = len(self.sampler) + task_size = len(self.sampler) else: - problem_size = len(self.problem) + task_size = len(self.task) - # If problem_size is a multiciplity of batch_size OR drop last is set. - if (problem_size % self.dataloader.batch_size) == 0 or self.dataloader.drop_last: - return problem_size // self.dataloader.batch_size + # If task_size is a multiciplity of batch_size OR drop last is set. + if (task_size % self.dataloader.batch_size) == 0 or self.dataloader.drop_last: + return task_size // self.dataloader.batch_size else: - return (problem_size // self.dataloader.batch_size) + 1 + return (task_size // self.dataloader.batch_size) + 1 def initialize_epoch(self): """ Function called to initialize a new epoch. """ epoch = self.app_state.epoch - # Update problem settings depending on the epoch. - self.problem.initialize_epoch(epoch) + # Update task settings depending on the epoch. + self.task.initialize_epoch(epoch) # Generate a single batch used for partial validation. if self.name == 'validation': @@ -213,4 +213,4 @@ def finalize_epoch(self): Function called at the end of an epoch to finalize it. """ epoch = self.app_state.epoch - self.problem.initialize_epoch(epoch) + self.task.initialize_epoch(epoch) diff --git a/ptp/components/component.py b/ptp/components/component.py index f0e0f5c..901c35e 100644 --- a/ptp/components/component.py +++ b/ptp/components/component.py @@ -142,14 +142,14 @@ def handshake_input_definitions(self, all_definitions, log_errors=True): # Check presence of key. if key not in all_definitions.keys(): if log_errors: - self.logger.error("Input definition: expected field '{}' not found in DataDict keys ({})".format(key, all_definitions.keys())) + self.logger.error("Input definition: expected field '{}' not found in DataStreams keys ({})".format(key, all_definitions.keys())) errors += 1 continue # Check number of dimensions. dd = all_definitions[key] if len(id.dimensions) != len (dd.dimensions): if log_errors: - self.logger.error("Input definition: field '{}' in DataDict has different dimensions from expected (expected {} while received {})".format(key, id.dimensions, dd.dimensions)) + self.logger.error("Input definition: field '{}' in DataStreams has different dimensions from expected (expected {} while received {})".format(key, id.dimensions, dd.dimensions)) errors += 1 else: # Check dimensions one by one. @@ -157,12 +157,12 @@ def handshake_input_definitions(self, all_definitions, log_errors=True): # -1 means that it can handle different values here. if did != -1 and did != ddd: if log_errors: - self.logger.error("Input definition: field '{}' in DataDict has dimension {} different from expected (expected {} while received {})".format(key,index, id.dimensions, dd.dimensions)) + self.logger.error("Input definition: field '{}' in DataStreams has dimension {} different from expected (expected {} while received {})".format(key,index, id.dimensions, dd.dimensions)) errors += 1 # Check number of types. if len(id.types) != len (dd.types): if log_errors: - self.logger.error("Input definition: field '{}' in DataDict has number of types different from expected (expected {} while received {})".format(key, id.types, dd.types)) + self.logger.error("Input definition: field '{}' in DataStreams has number of types different from expected (expected {} while received {})".format(key, id.types, dd.types)) errors += 1 else: # Check types one by one. @@ -170,7 +170,7 @@ def handshake_input_definitions(self, all_definitions, log_errors=True): # -1 means that it can handle different values here. if tid != tdd: if log_errors: - self.logger.error("Input definition: field '{}' in DataDict has type {} different from expected (expected {} while received {})".format(key,index, id.types, dd.types)) + self.logger.error("Input definition: field '{}' in DataStreams has type {} different from expected (expected {} while received {})".format(key,index, id.types, dd.types)) errors += 1 return errors @@ -190,7 +190,7 @@ def export_output_definitions(self, all_definitions, log_errors=True): # Check presence of key. if key in all_definitions.keys(): if log_errors: - self.logger.error("Output definition error: field '{}' cannot be added to DataDict, as it is already present in its keys ({})".format(key, all_definitions.keys())) + self.logger.error("Output definition error: field '{}' cannot be added to DataStreams, as it is already present in its keys ({})".format(key, all_definitions.keys())) errors += 1 else: # Add field to definitions. @@ -200,12 +200,12 @@ def export_output_definitions(self, all_definitions, log_errors=True): @abc.abstractmethod - def __call__(self, data_dict): + def __call__(self, data_streams): """ Method responsible for processing the data dict. Abstract, must be implemented by all derived classes. - :param data_dict: :py:class:`ptp.core_types.DataDict` object containing both input data to be processed and that will be extended by the results. + :param data_streams: :py:class:`ptp.data_types.DataStreams` object containing both input data to be processed and that will be extended by the results. """ pass @@ -224,7 +224,7 @@ def add_statistics(self, stat_col): pass - def collect_statistics(self, stat_col, data_dict): + def collect_statistics(self, stat_col, data_streams): """ Base statistics collection. @@ -236,8 +236,8 @@ def collect_statistics(self, stat_col, data_dict): :param stat_col: :py:class:`ptp.configuration.StatisticsCollector`. - :param data_dict: ``DataDict`` containing inputs, targets etc. - :type data_dict: :py:class:`ptp.core_types.DataDict` + :param data_streams: ``DataStreams`` containing inputs, targets etc. + :type data_streams: :py:class:`ptp.data_types.DataStreams` """ pass diff --git a/ptp/components/text/__init__.py b/ptp/components/language/__init__.py similarity index 100% rename from ptp/components/text/__init__.py rename to ptp/components/language/__init__.py diff --git a/ptp/components/text/bow_encoder.py b/ptp/components/language/bow_encoder.py similarity index 93% rename from ptp/components/text/bow_encoder.py rename to ptp/components/language/bow_encoder.py index 0313232..4bac20f 100644 --- a/ptp/components/text/bow_encoder.py +++ b/ptp/components/language/bow_encoder.py @@ -68,18 +68,18 @@ def output_data_definitions(self): self.key_outputs: DataDefinition([-1, self.bow_size], [torch.Tensor], "Batch of sentences, each represented as a single vector [BATCH_SIZE x ITEM_SIZE] (agnostic to item size)") } - def __call__(self, data_dict): + def __call__(self, data_streams): """ Encodes batch, or, in fact, only one field of batch ("inputs"). - Stores result in "outputs" field of data_dict. + Stores result in "outputs" field of data_streams. - :param data_dict: :py:class:`ptp.utils.DataDict` object containing (among others): + :param data_streams: :py:class:`ptp.utils.DataStreams` object containing (among others): - "inputs": expected input containing list of (list of tokens) [BATCH SIZE] x [SEQ_LEN] x [ITEM_SIZE] - "outputs": added output tensor with encoded words [BATCH_SIZE x ITEM_SIZE] """ # Get inputs to be encoded. - inputs = data_dict[self.key_inputs] + inputs = data_streams[self.key_inputs] outputs_list = [] # Process samples 1 by one. for sample in inputs: @@ -90,7 +90,7 @@ def __call__(self, data_dict): # Concatenate output tensors. outputs = torch.cat(outputs_list, dim=0) # Add result to the data dict. - data_dict.extend({self.key_outputs: outputs}) + data_streams.publish({self.key_outputs: outputs}) def encode_sample(self, list_of_tokens): """ diff --git a/ptp/components/text/label_indexer.py b/ptp/components/language/label_indexer.py similarity index 92% rename from ptp/components/text/label_indexer.py rename to ptp/components/language/label_indexer.py index c3090cd..667fd02 100644 --- a/ptp/components/text/label_indexer.py +++ b/ptp/components/language/label_indexer.py @@ -68,19 +68,19 @@ def output_data_definitions(self): self.key_outputs: DataDefinition([-1], [torch.Tensor], "Batch of labels, each represented as a single index [BATCH_SIZE]") } - def __call__(self, data_dict): + def __call__(self, data_streams): """ Encodes "inputs" in the format of a single word. - Stores result in "outputs" field of in data_dict. + Stores result in "outputs" field of in data_streams. - :param data_dict: :py:class:`ptp.utils.DataDict` object containing (among others): + :param data_streams: :py:class:`ptp.utils.DataStreams` object containing (among others): - "inputs": expected input field containing list of words [BATCH_SIZE] x x [string] - "outputs": added output field containing list of indices [BATCH_SIZE] """ # Get inputs to be encoded. - inputs = data_dict[self.key_inputs] + inputs = data_streams[self.key_inputs] outputs_list = [] # Process samples 1 by 1. for sample in inputs: @@ -95,4 +95,4 @@ def __call__(self, data_dict): # Transform to tensor. output_tensor = torch.tensor(outputs_list) # Create the returned dict. - data_dict.extend({self.key_outputs: output_tensor}) + data_streams.publish({self.key_outputs: output_tensor}) diff --git a/ptp/components/text/sentence_indexer.py b/ptp/components/language/sentence_indexer.py similarity index 86% rename from ptp/components/text/sentence_indexer.py rename to ptp/components/language/sentence_indexer.py index b13c83b..68dab9b 100644 --- a/ptp/components/text/sentence_indexer.py +++ b/ptp/components/language/sentence_indexer.py @@ -17,9 +17,8 @@ import torch from ptp.components.component import Component -from ptp.components.mixins.word_mappings import WordMappings from ptp.data_types.data_definition import DataDefinition -from ptp.components.utils.word_mappings import pad_trunc_list +from ptp.components.mixins.word_mappings import WordMappings, pad_trunc_list class SentenceIndexer(Component, WordMappings): @@ -102,37 +101,37 @@ def output_data_definitions(self): } - def __call__(self, data_dict): + def __call__(self, data_streams): """ Encodes inputs into outputs. Depending on the mode (set by 'reverse' config param) calls sentences_to_tensor() (when False) or tensor_to_sentences() (when set to True). - :param data_dict: :py:class:`ptp.datatypes.DataDict` object. + :param data_streams: :py:class:`ptp.datatypes.DataStreams` object. """ if self.mode_reverse: if self.use_input_distributions: # Produce list of words. - self.tensor_distributions_to_sentences(data_dict) + self.tensor_distributions_to_sentences(data_streams) else: # Produce list of words. - self.tensor_indices_to_sentences(data_dict) + self.tensor_indices_to_sentences(data_streams) else: # Produce indices. - self.sentences_to_tensor(data_dict) + self.sentences_to_tensor(data_streams) - def sentences_to_tensor(self, data_dict): + def sentences_to_tensor(self, data_streams): """ Encodes "inputs" in the format of batch of list of words into a single tensor with corresponding indices. - :param data_dict: :py:class:`ptp.datatypes.DataDict` object containing (among others): + :param data_streams: :py:class:`ptp.datatypes.DataStreams` object containing (among others): - "inputs": expected input field containing list of lists of words [BATCH_SIZE] x [SEQ_SIZE] x [string] - "outputs": added output field containing tensor with indices [BATCH_SIZE x SEQ_SIZE] """ # Get inputs to be encoded. - inputs = data_dict[self.key_inputs] + inputs = data_streams[self.key_inputs] # Get index of padding. pad_index = self.word_to_ix[''] @@ -162,13 +161,13 @@ def sentences_to_tensor(self, data_dict): # output = self.app_state.LongTensor(outputs_list) output = torch.nn.utils.rnn.pad_sequence(outputs_list, batch_first=True, padding_value=pad_index) # Create the returned dict. - data_dict.extend({self.key_outputs: output}) + data_streams.publish({self.key_outputs: output}) - def tensor_indices_to_sentences(self, data_dict): + def tensor_indices_to_sentences(self, data_streams): """ Encodes "inputs" in the format of tensor with indices into a batch of list of words. - :param data_dict: :py:class:`ptp.datatypes.DataDict` object containing (among others): + :param data_streams: :py:class:`ptp.datatypes.DataStreams` object containing (among others): - "inputs": added output field containing tensor with indices [BATCH_SIZE x SEQ_SIZE] @@ -176,7 +175,7 @@ def tensor_indices_to_sentences(self, data_dict): """ # Get inputs to be changed to words. - inputs = data_dict[self.key_inputs].data.cpu().numpy().tolist() + inputs = data_streams[self.key_inputs].data.cpu().numpy().tolist() outputs_list = [] # Process samples 1 by 1. @@ -194,13 +193,13 @@ def tensor_indices_to_sentences(self, data_dict): outputs_list.append(output_sample) # Create the returned dict. - data_dict.extend({self.key_outputs: outputs_list}) + data_streams.publish({self.key_outputs: outputs_list}) - def tensor_distributions_to_sentences(self, data_dict): + def tensor_distributions_to_sentences(self, data_streams): """ Encodes "inputs" in the format of tensor with probability distributions into a batch of list of words. - :param data_dict: :py:class:`ptp.datatypes.DataDict` object containing (among others): + :param data_streams: :py:class:`ptp.datatypes.DataStreams` object containing (among others): - "inputs": added output field containing tensor with indices [BATCH_SIZE x SEQ_SIZE x ITEM_SIZE] @@ -208,7 +207,7 @@ def tensor_distributions_to_sentences(self, data_dict): """ # Get inputs to be changed to words. - inputs = data_dict[self.key_inputs].max(2)[1].data.cpu().numpy().tolist() + inputs = data_streams[self.key_inputs].max(2)[1].data.cpu().numpy().tolist() outputs_list = [] # Process samples 1 by 1. @@ -227,4 +226,4 @@ def tensor_distributions_to_sentences(self, data_dict): outputs_list.append(output_sample) # Create the returned dict. - data_dict.extend({self.key_outputs: outputs_list}) + data_streams.publish({self.key_outputs: outputs_list}) diff --git a/ptp/components/text/sentence_one_hot_encoder.py b/ptp/components/language/sentence_one_hot_encoder.py similarity index 92% rename from ptp/components/text/sentence_one_hot_encoder.py rename to ptp/components/language/sentence_one_hot_encoder.py index b25a3e8..3996236 100644 --- a/ptp/components/text/sentence_one_hot_encoder.py +++ b/ptp/components/language/sentence_one_hot_encoder.py @@ -65,19 +65,19 @@ def output_data_definitions(self): self.key_outputs: DataDefinition([-1, -1, len(self.word_to_ix)], [list, list, torch.Tensor], "Batch of sentences, each represented as a list of vectors [BATCH_SIZE] x [SEQ_LENGTH] x [VOCABULARY_SIZE]"), } - def __call__(self, data_dict): + def __call__(self, data_streams): """ Encodes "inputs" in the format of list of tokens (for a single sample) - Stores result in "encoded_inputs" field of in data_dict. + Stores result in "encoded_inputs" field of in data_streams. - :param data_dict: :py:class:`ptp.utils.DataDict` object containing (among others): + :param data_streams: :py:class:`ptp.utils.DataStreams` object containing (among others): - "inputs": expected input field containing list of words [BATCH_SIZE] x [SEQ_SIZE] x [string] - "encoded_targets": added output field containing list of indices [BATCH_SIZE] x [SEQ_SIZE] x [VOCABULARY_SIZE1] """ # Get inputs to be encoded. - inputs = data_dict[self.key_inputs] + inputs = data_streams[self.key_inputs] outputs_list = [] # Process samples 1 by one. for sample in inputs: @@ -95,4 +95,4 @@ def __call__(self, data_dict): outputs_list.append(output_sample) # Create the returned dict. - data_dict.extend({self.key_outputs: outputs_list}) + data_streams.publish({self.key_outputs: outputs_list}) diff --git a/ptp/components/text/sentence_tokenizer.py b/ptp/components/language/sentence_tokenizer.py similarity index 94% rename from ptp/components/text/sentence_tokenizer.py rename to ptp/components/language/sentence_tokenizer.py index 1f8f63c..3941077 100644 --- a/ptp/components/text/sentence_tokenizer.py +++ b/ptp/components/language/sentence_tokenizer.py @@ -137,19 +137,19 @@ def detokenize_sample(self, sample): """ return ' '.join([str(x) for x in sample]) - def __call__(self, data_dict): + def __call__(self, data_streams): """ Encodes batch, or, in fact, only one field of bach ("inputs"). - Stores result in "encoded_inputs" field of in data_dict. + Stores result in "encoded_inputs" field of in data_streams. - :param data_dict: :py:class:`ptp.utils.DataDict` object containing (among others): + :param data_streams: :py:class:`ptp.utils.DataStreams` object containing (among others): - "inputs": expected input field containing list of words - "encoded_targets": added field containing output, tensor with encoded samples [BATCH_SIZE x 1] """ # Get inputs to be encoded. - inputs = data_dict[self.key_inputs] + inputs = data_streams[self.key_inputs] outputs_list = [] # Process samples 1 by one. for sample in inputs: @@ -157,4 +157,4 @@ def __call__(self, data_dict): # Add to outputs. outputs_list.append( output ) # Create the returned dict. - data_dict.extend({self.key_outputs: outputs_list}) + data_streams.publish({self.key_outputs: outputs_list}) diff --git a/ptp/components/text/word_decoder.py b/ptp/components/language/word_decoder.py similarity index 91% rename from ptp/components/text/word_decoder.py rename to ptp/components/language/word_decoder.py index e75dd15..ef9e4dc 100644 --- a/ptp/components/text/word_decoder.py +++ b/ptp/components/language/word_decoder.py @@ -68,19 +68,19 @@ def output_data_definitions(self): self.key_outputs: DataDefinition([-1, 1], [list, str], "Batch of words, each represented as a single string [BATCH_SIZE] x [string]") } - def __call__(self, data_dict): + def __call__(self, data_streams): """ Encodes "inputs" in the format of a single word. - Stores result in "outputs" field of in data_dict. + Stores result in "outputs" field of in data_streams. - :param data_dict: :py:class:`ptp.utils.DataDict` object containing (among others): + :param data_streams: :py:class:`ptp.utils.DataStreams` object containing (among others): - "inputs": expected input field containing tensor [BATCH_SIZE x ITEM_SIZE] - "outputs": added output field containing list of words [BATCH_SIZE] x [string] """ # Get inputs to be encoded. - inputs = data_dict[self.key_inputs] + inputs = data_streams[self.key_inputs] outputs_list = [] # Process samples 1 by 1. for sample in inputs.chunk(inputs.size(0), 0): @@ -89,5 +89,5 @@ def __call__(self, data_dict): output_sample = self.ix_to_word[max_index] outputs_list.append(output_sample) # Create the returned dict. - data_dict.extend({self.key_outputs: outputs_list}) + data_streams.publish({self.key_outputs: outputs_list}) diff --git a/ptp/components/losses/loss.py b/ptp/components/losses/loss.py index b30dc32..9173611 100644 --- a/ptp/components/losses/loss.py +++ b/ptp/components/losses/loss.py @@ -52,7 +52,7 @@ def __init__(self, name, class_type, config): def loss_keys(self): """ - Function returns a list containing keys used to return losses in DataDict. + Function returns a list containing keys used to return losses in DataStreams. Those keys will be used to find objects that will be roots for backpropagation of gradients. :return: list of keys associated with losses returned by the component. @@ -70,14 +70,14 @@ def add_statistics(self, stat_col): # Add loss statistics with formatting. stat_col.add_statistics(self.key_loss, '{:12.10f}') - def collect_statistics(self, stat_col, data_dict): + def collect_statistics(self, stat_col, data_streams): """ Collects statistics (loss) for given episode. :param stat_col: ``StatisticsCollector``. """ - stat_col[self.key_loss] = data_dict[self.key_loss].item() + stat_col[self.key_loss] = data_streams[self.key_loss].item() def add_aggregators(self, stat_agg): """ diff --git a/ptp/components/losses/nll_loss.py b/ptp/components/losses/nll_loss.py index ee75212..835cc7e 100644 --- a/ptp/components/losses/nll_loss.py +++ b/ptp/components/losses/nll_loss.py @@ -81,11 +81,11 @@ def output_data_definitions(self): } - def __call__(self, data_dict): + def __call__(self, data_streams): """ Calculates loss (negative log-likelihood) and adds it to data dict. - :param data_dict: DataDict containing: + :param data_streams: DataStreams containing: - "targets": input batch of targets (class indices) [BATCH_SIZE x NUM_CLASSES] - "predictions": input batch of predictions (log_probs) being outputs of the model [BATCH_SIZE x 1] @@ -94,8 +94,8 @@ def __call__(self, data_dict): """ # Load inputs. - targets = data_dict[self.key_targets] - predictions = data_dict[self.key_predictions] + targets = data_streams[self.key_targets] + predictions = data_streams[self.key_predictions] #print("targets = ",targets) #print("predictions = ",predictions) @@ -107,7 +107,7 @@ def __call__(self, data_dict): # Mask predictions if option set. if self.use_masking: - masks = data_dict[self.key_masks] + masks = data_streams[self.key_masks] targets = targets * masks.type(self.app_state.LongTensor) #print("unsqueezed masks = ", masks.unsqueeze(1)) predictions = predictions * masks.unsqueeze(1).type(self.app_state.FloatTensor) @@ -123,4 +123,4 @@ def __call__(self, data_dict): # Calculate loss. loss = self.loss_function(predictions.view(-1, last_dim), targets.view(-1)) # Add it to datadict. - data_dict.extend({self.key_loss: loss}) + data_streams.publish({self.key_loss: loss}) diff --git a/ptp/components/masking/join_masked_predictions.py b/ptp/components/masking/join_masked_predictions.py index 07496b1..6a7dc9c 100644 --- a/ptp/components/masking/join_masked_predictions.py +++ b/ptp/components/masking/join_masked_predictions.py @@ -112,12 +112,12 @@ def output_data_definitions(self): } - def __call__(self, data_dict): + def __call__(self, data_streams): """ Encodes "inputs" in the format of a single tensor. - Stores reshaped tensor in "outputs" field of in data_dict. + Stores reshaped tensor in "outputs" field of in data_streams. - :param data_dict: :py:class:`ptp.utils.DataDict` object containing (among others): + :param data_streams: :py:class:`ptp.utils.DataStreams` object containing (among others): - "inputs": expected input field containing tensor [BATCH_SIZE x ...] @@ -127,7 +127,7 @@ def __call__(self, data_dict): masks = [] for imsk in self.input_mask_stream_keys: # Get mask from stream. - mask = data_dict[imsk] + mask = data_streams[imsk] masks.append(mask.data.cpu().numpy()) # Sum all masks and make sure that they are complementary. @@ -155,7 +155,7 @@ def __call__(self, data_dict): #print(ix_to_word) # Get the right sample from the right prediction stream. - sample_prediction = data_dict[self.input_prediction_stream_keys[mapping[sample]]][sample] + sample_prediction = data_streams[self.input_prediction_stream_keys[mapping[sample]]][sample] #print(sample_prediction) output_predictions_lst.append(sample_prediction) @@ -171,7 +171,7 @@ def __call__(self, data_dict): output_indices.append(self.output_word_to_ix[word]) #print(output_predictions_lst) - #targets = data_dict["targets"].data.cpu().numpy() + #targets = data_streams["targets"].data.cpu().numpy() #print("targets = \n",targets.tolist()) #print("joined answers = \n",output_indices) @@ -179,7 +179,7 @@ def __call__(self, data_dict): output_indices_tensor = torch.tensor(output_indices) # Extend the dict by returned output streams. - data_dict.extend({ + data_streams.publish({ self.key_output_indices: output_indices_tensor, self.key_output_strings: output_answers }) diff --git a/ptp/components/masking/string_to_mask.py b/ptp/components/masking/string_to_mask.py index 175521d..a86b6da 100644 --- a/ptp/components/masking/string_to_mask.py +++ b/ptp/components/masking/string_to_mask.py @@ -69,19 +69,19 @@ def output_data_definitions(self): } - def __call__(self, data_dict): + def __call__(self, data_streams): """ Encodes "inputs" in the format of a single tensor. - Stores reshaped tensor in "outputs" field of in data_dict. + Stores reshaped tensor in "outputs" field of in data_streams. - :param data_dict: :py:class:`ptp.utils.DataDict` object containing (among others): + :param data_streams: :py:class:`ptp.utils.DataStreams` object containing (among others): - "inputs": expected input field containing tensor [BATCH_SIZE x ...] - "outputs": added output field containing tensor [BATCH_SIZE x ...] """ # Get inputs strings. - strings = data_dict[self.key_strings] + strings = data_streams[self.key_strings] masks = torch.zeros(len(strings), requires_grad=False).type(self.app_state.ByteTensor) @@ -93,7 +93,7 @@ def __call__(self, data_dict): masks[i] = 1 # Extend the dict by returned stream with masks. - data_dict.extend({ + data_streams.publish({ self.key_masks: masks }) diff --git a/ptp/components/mixins/__init__.py b/ptp/components/mixins/__init__.py deleted file mode 100644 index f748621..0000000 --- a/ptp/components/mixins/__init__.py +++ /dev/null @@ -1,5 +0,0 @@ -from .word_mappings import WordMappings - -__all__ = [ - 'WordMappings', - ] diff --git a/ptp/components/utils/embeddings.py b/ptp/components/mixins/embeddings.py similarity index 99% rename from ptp/components/utils/embeddings.py rename to ptp/components/mixins/embeddings.py index 099684d..1b5e5ef 100644 --- a/ptp/components/utils/embeddings.py +++ b/ptp/components/mixins/embeddings.py @@ -23,7 +23,7 @@ import torch -import ptp.components.utils.io as io +import ptp.components.mixins.io as io def load_pretrained_embeddings(logger, folder, embeddings_name, word_to_ix, embeddings_size): diff --git a/ptp/components/utils/io.py b/ptp/components/mixins/io.py similarity index 100% rename from ptp/components/utils/io.py rename to ptp/components/mixins/io.py diff --git a/ptp/components/mixins/word_mappings.py b/ptp/components/mixins/word_mappings.py index a6df45f..110f2c0 100644 --- a/ptp/components/mixins/word_mappings.py +++ b/ptp/components/mixins/word_mappings.py @@ -15,9 +15,7 @@ __author__ = "Tomasz Kornuta" import os - -import ptp.components.utils.word_mappings as wm - +import csv class WordMappings(object): """ @@ -60,15 +58,15 @@ def __init__(self): #, name, class_type, config): exit(-1) # Try to load the preprocessed word mappings. - self.word_to_ix = wm.load_word_mappings_from_csv_file(self.logger, self.data_folder, self.word_mappings_file) + self.word_to_ix = load_word_mappings_from_csv_file(self.logger, self.data_folder, self.word_mappings_file) assert (len(self.word_to_ix) > 0), "The word mappings loaded from file are empty!" else: # Try to generate new word mappings from source files. - self.word_to_ix = wm.generate_word_mappings_from_source_files(self.logger, self.data_folder, self.source_vocabulary_files) + self.word_to_ix = generate_word_mappings_from_source_files(self.logger, self.data_folder, self.source_vocabulary_files) assert (len(self.word_to_ix) > 0), "The word mappings generated from sources are empty!" # Ok, save mappings, so next time we will simply load them. - wm.save_word_mappings_to_csv_file(self.logger, self.data_folder, self.word_mappings_file, self.word_to_ix) + save_word_mappings_to_csv_file(self.logger, self.data_folder, self.word_mappings_file, self.word_to_ix) # Check if additional tokens are present. self.additional_tokens = self.config["additional_tokens"].split(',') @@ -90,3 +88,99 @@ def __init__(self): #, name, class_type, config): +def load_word_mappings_from_csv_file(logger, folder, filename): + """ + Loads (word:index) mappings from csv file. + + .. warning:: + There is an assumption that file will contain key:value pairs (no content checking for now!) + + :param logger: Logger object. + + :param folder: Relative path to to the folder. + :type folder: str + + :param filename: File with encodings (absolute path + filename). + + :return: dictionary with word:index keys + """ + file_path = os.path.join(os.path.expanduser(folder), filename) + + if not os.path.exists(file_path): + logger.warning("Cannot load word mappings from '{}' because the file does not exist".format(file_path)) + + with open(file_path, mode='rt') as csvfile: + # Check the presence of the header. + sniffer = csv.Sniffer() + first_bytes = str(csvfile.read(256)) + has_header = sniffer.has_header(first_bytes) + # Rewind. + csvfile.seek(0) + reader = csv.reader(csvfile) + # Skip the header row. + if has_header: + next(reader) + # Read the remaining rows. + word_to_ix = {rows[0]:int(rows[1]) for rows in reader} + + logger.info("Loaded mappings of size {}".format(len(word_to_ix))) + return word_to_ix + + +def save_word_mappings_to_csv_file(logger, folder, filename, word_to_ix, fieldnames = ["word","index"]): + """ + Saves (word:index) mappings dictionary to a file. + + :param logger: Logger object. + + :param folder: Relative path to to the folder. + :type folder: str + + :param filename: Name of file with encodings. + + :param word_to_ix: Dictionary with word:index mappings to be saved. + + """ + # Expand path. + folder = os.path.expanduser(folder) + # Make sure directory exists. + os.makedirs(os.path.dirname(folder +'/'), exist_ok=True) + + file_path = os.path.join(folder, filename) + + with open(file_path, mode='w+') as csvfile: + writer = csv.DictWriter(csvfile, fieldnames=fieldnames) + # Create header. + writer.writeheader() + + # Write word-index pairs. + for (k,v) in word_to_ix.items(): + #print("{} : {}".format(k,v)) + writer.writerow({fieldnames[0]:k, fieldnames[1]: v}) + + logger.info("Saved mappings of size {} to file '{}'".format(len(word_to_ix), file_path)) + +def pad_trunc_list(l: list, length: int, padding_value = 0, eos_value = None): + """ + Will apply padding / clipping to list to meet requested length. + Works on the list in-place. + + :param l: List to manipulate + + :param length: Target length + + :param padding_value: Value to fill when padding. Default is int(0). + + :return: None + """ + if len(l) < length: + if eos_value is not None: + l.append(eos_value) + l.extend([padding_value]*(length-len(l))) + + elif len(l) > length: + #print("pad_trunc_list to cat!: {}".format(len(l))) + #exit(1) + del l[length:] + if eos_value is not None: + l[length-1] = eos_value diff --git a/ptp/components/models/__init__.py b/ptp/components/models/__init__.py index ce5f509..f08a60e 100644 --- a/ptp/components/models/__init__.py +++ b/ptp/components/models/__init__.py @@ -1,36 +1,51 @@ -from .convnet_encoder import ConvNetEncoder -from .feed_forward_network import FeedForwardNetwork -from .index_embeddings import IndexEmbeddings -from .torch_vision_wrapper import TorchVisionWrapper -from .lenet5 import LeNet5 from .model import Model -from .recurrent_neural_network import RecurrentNeuralNetwork -from .sentence_embeddings import SentenceEmbeddings -from .seq2seq_rnn import Seq2Seq_RNN -from .attn_decoder_rnn import Attn_Decoder_RNN - -from .vqa.element_wise_multiplication import ElementWiseMultiplication -from .vqa.multimodal_compact_bilinear_pooling import MultimodalCompactBilinearPooling -from .vqa.relational_network import RelationalNetwork -from .vqa.attention import VQA_Attention -from .vqa.multimodal_factorized_bilinear_pooling import MultimodalFactorizedBilinearPooling -from .vqa.self_attention import SelfAttention + +# General usage +from .general_usage.feed_forward_network import FeedForwardNetwork +from .general_usage.recurrent_neural_network import RecurrentNeuralNetwork +from .general_usage.seq2seq import Seq2Seq +from .general_usage.attention_decoder import AttentionDecoder + +# Language +from .language.index_embeddings import IndexEmbeddings +from .language.sentence_embeddings import SentenceEmbeddings + +# Vision +from .vision.convnet_encoder import ConvNetEncoder +from .vision.generic_image_encoder import GenericImageEncoder +from .vision.lenet5 import LeNet5 + +# Multi-modal reasoning +from .multi_modal_reasoning.compact_bilinear_pooling import CompactBilinearPooling +from .multi_modal_reasoning.factorized_bilinear_pooling import FactorizedBilinearPooling +from .multi_modal_reasoning.low_rank_bilinear_pooling import LowRankBilinearPooling +from .multi_modal_reasoning.question_driven_attention import QuestionDrivenAttention +from .multi_modal_reasoning.relational_network import RelationalNetwork +from .multi_modal_reasoning.self_attention import SelfAttention __all__ = [ - 'ConvNetEncoder', - 'FeedForwardNetwork', - 'IndexEmbeddings', - 'TorchVisionWrapper', - 'LeNet5', 'Model', + + # General usage + 'FeedForwardNetwork', 'RecurrentNeuralNetwork', + 'Seq2Seq', + 'AttentionDecoder', + + # Language + 'IndexEmbeddings', 'SentenceEmbeddings', - 'Seq2Seq_RNN', - 'ElementWiseMultiplication', - 'MultimodalCompactBilinearPooling', + + # Vision + 'ConvNetEncoder', + 'GenericImageEncoder', + 'LeNet5', + + # Multi-modal reasoning + 'CompactBilinearPooling', + 'FactorizedBilinearPooling', + 'LowRankBilinearPooling', + 'QuestionDrivenAttention', 'RelationalNetwork', - 'Attn_Decoder_RNN', - 'VQA_Attention', - 'MultimodalFactorizedBilinearPooling', 'SelfAttention' ] diff --git a/ptp/components/models/attn_decoder_rnn.py b/ptp/components/models/general_usage/attention_decoder.py similarity index 95% rename from ptp/components/models/attn_decoder_rnn.py rename to ptp/components/models/general_usage/attention_decoder.py index 32a2c14..3cd7891 100644 --- a/ptp/components/models/attn_decoder_rnn.py +++ b/ptp/components/models/general_usage/attention_decoder.py @@ -21,7 +21,7 @@ from ptp.data_types.data_definition import DataDefinition -class Attn_Decoder_RNN(Model): +class AttentionDecoder(Model): """ Single layer GRU decoder with attention: Bahdanau, D., Cho, K., & Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473. @@ -38,7 +38,7 @@ def __init__(self, name, config): :type config: ``ptp.configuration.ConfigInterface`` """ # Call constructors of parent classes. - Model.__init__(self, name, Attn_Decoder_RNN, config) + Model.__init__(self, name, AttentionDecoder, config) # Get input/output mode self.output_last_state = self.config["output_last_state"] @@ -173,22 +173,22 @@ def output_data_definitions(self): return d - def forward(self, data_dict): + def forward(self, data_streams): """ Forward pass of the model. - :param data_dict: DataDict({'inputs', 'predictions ...}), where: + :param data_streams: DataStreams({'inputs', 'predictions ...}), where: - inputs: expected inputs [BATCH_SIZE x SEQ_LEN x INPUT_SIZE], - predictions: returned output with predictions (log_probs) [BATCH_SIZE x SEQ_LEN x PREDICTION_SIZE] """ - inputs = data_dict[self.key_inputs] + inputs = data_streams[self.key_inputs] batch_size = inputs.shape[0] #print("{}: input shape: {}, device: {}\n".format(self.name, inputs.shape, inputs.device)) # Initialize hidden state from inputs - as last hidden state from external component. - hidden = data_dict[self.key_input_state] + hidden = data_streams[self.key_input_state] # For RNNs (aside of LSTM): [BATCH_SIZE x NUM_LAYERS x HIDDEN_SIZE] -> [NUM_LAYERS x BATCH_SIZE x HIDDEN_SIZE] hidden = hidden.transpose(0,1) #print("{}: hidden shape: {}, device: {}\n".format(self.name, hidden.shape, hidden.device)) @@ -227,16 +227,16 @@ def forward(self, data_dict): if self.use_logsoftmax: outputs = self.log_softmax(outputs) # Add predictions to datadict. - data_dict.extend({self.key_predictions: outputs}) + data_streams.publish({self.key_predictions: outputs}) elif self.prediction_mode == "Last": if self.use_logsoftmax: outputs = self.log_softmax(activations_partial.squeeze(1)) # Add predictions to datadict. - data_dict.extend({self.key_predictions: outputs}) + data_streams.publish({self.key_predictions: outputs}) # Output last hidden state, if requested if self.output_last_state: # For others: [NUM_LAYERS x BATCH_SIZE x HIDDEN_SIZE] -> [BATCH_SIZE x NUM_LAYERS x HIDDEN_SIZE] hidden = hidden.transpose(0,1) # Export last hidden state. - data_dict.extend({self.key_output_state: hidden}) + data_streams.publish({self.key_output_state: hidden}) diff --git a/ptp/components/models/feed_forward_network.py b/ptp/components/models/general_usage/feed_forward_network.py similarity index 96% rename from ptp/components/models/feed_forward_network.py rename to ptp/components/models/general_usage/feed_forward_network.py index e766ba9..75ec2f2 100644 --- a/ptp/components/models/feed_forward_network.py +++ b/ptp/components/models/general_usage/feed_forward_network.py @@ -124,17 +124,17 @@ def output_data_definitions(self): self.key_predictions: DataDefinition(([-1] * (self.dimensions -1)) + [self.prediction_size], [torch.Tensor], "Batch of predictions, each represented as probability distribution over classes [BATCH_SIZE x ... x PREDICTION_SIZE]") } - def forward(self, data_dict): + def forward(self, data_streams): """ Forward pass of the model. - :param data_dict: DataDict({'inputs', 'predictions ...}), where: + :param data_streams: DataStreams({'inputs', 'predictions ...}), where: - inputs: expected inputs [BATCH_SIZE x ... x INPUT_SIZE], - predictions: returned output with predictions (log_probs) [BATCH_SIZE x ... x NUM_CLASSES] """ # Get inputs. - x = data_dict[self.key_inputs] + x = data_streams[self.key_inputs] #print("{}: input shape: {}, device: {}\n".format(self.name, x.shape, x.device)) @@ -154,4 +154,4 @@ def forward(self, data_dict): x = x.view(*origin_shape[0:self.dimensions-1], -1) # Add predictions to datadict. - data_dict.extend({self.key_predictions: x}) + data_streams.publish({self.key_predictions: x}) diff --git a/ptp/components/models/recurrent_neural_network.py b/ptp/components/models/general_usage/recurrent_neural_network.py similarity index 96% rename from ptp/components/models/recurrent_neural_network.py rename to ptp/components/models/general_usage/recurrent_neural_network.py index 26105b6..84f07d9 100644 --- a/ptp/components/models/recurrent_neural_network.py +++ b/ptp/components/models/general_usage/recurrent_neural_network.py @@ -259,11 +259,11 @@ def output_data_definitions(self): return d - def forward(self, data_dict): + def forward(self, data_streams): """ Forward pass of the model. - :param data_dict: DataDict({'inputs', 'predictions ...}), where: + :param data_streams: DataStreams({'inputs', 'predictions ...}), where: - inputs: expected inputs [BATCH_SIZE x SEQ_LEN x INPUT_SIZE], - predictions: returned output with predictions (log_probs) [BATCH_SIZE x SEQ_LEN x PREDICTION_SIZE] @@ -275,13 +275,13 @@ def forward(self, data_dict): # Get inputs if "None" in self.input_mode: # Must be in autoregressive mode - retrieve batch_size from initial hidden state from encoder. - batch_size = data_dict[self.key_input_state][0].shape[1] + batch_size = data_streams[self.key_input_state][0].shape[1] # Set zero inputs [BATCH_SIZE x SEQ_LEN x INPUT_SIZE]. inputs = torch.zeros(batch_size, self.hidden_size, requires_grad=False).type(self.app_state.FloatTensor) else: # Get inputs [BATCH_SIZE x SEQ_LEN x INPUT_SIZE] - inputs = data_dict[self.key_inputs] + inputs = data_streams[self.key_inputs] if inputs.dim() == 2: inputs = inputs.unsqueeze(1) batch_size = inputs.shape[0] @@ -290,7 +290,7 @@ def forward(self, data_dict): # Get initial state, depending on the settings. if self.initial_state == "Input": # Initialize hidden state from inputs - as last hidden state from external component. - hidden = data_dict[self.key_input_state] + hidden = data_streams[self.key_input_state] # Flip batch and num_layer dims so batch will be third/second! if self.cell_type == 'LSTM': # For LSTM: [BATCH_SIZE x NUM_LAYERS x 2 x HIDDEN_SIZE] -> [2 x NUM_LAYERS x BATCH_SIZE x HIDDEN_SIZE] @@ -325,14 +325,14 @@ def forward(self, data_dict): if self.use_logsoftmax: outputs = self.log_softmax(outputs) # Add predictions to datadict. - data_dict.extend({self.key_predictions: outputs}) + data_streams.publish({self.key_predictions: outputs}) elif self.prediction_mode == "Last": # Take only the last activations. outputs = activations_partial.squeeze(1) if self.use_logsoftmax: outputs = self.log_softmax(outputs) # Add predictions to datadict. - data_dict.extend({self.key_predictions: outputs}) + data_streams.publish({self.key_predictions: outputs}) else: # Normal mode - feed the entire input sequence at once @@ -347,7 +347,7 @@ def forward(self, data_dict): outputs = self.log_softmax(outputs) # Add predictions to datadict. - data_dict.extend({self.key_predictions: outputs}) + data_streams.publish({self.key_predictions: outputs}) elif self.prediction_mode == "Last": outputs = self.activation_to_output_pass(activations.contiguous()[:, -1, :].unsqueeze(1)) outputs = outputs.squeeze(1) @@ -357,7 +357,7 @@ def forward(self, data_dict): outputs = self.log_softmax(outputs) # Add predictions to datadict. - data_dict.extend({self.key_predictions: outputs}) + data_streams.publish({self.key_predictions: outputs}) elif self.prediction_mode == "None": # Nothing, since we don't want to keep the RNN's outputs pass @@ -371,4 +371,4 @@ def forward(self, data_dict): # For others: [NUM_LAYERS x BATCH_SIZE x HIDDEN_SIZE] -> [BATCH_SIZE x NUM_LAYERS x HIDDEN_SIZE] hidden = hidden.transpose(0,1) # Export last hidden state. - data_dict.extend({self.key_output_state: hidden}) + data_streams.publish({self.key_output_state: hidden}) diff --git a/ptp/components/models/seq2seq_rnn.py b/ptp/components/models/general_usage/seq2seq.py similarity index 96% rename from ptp/components/models/seq2seq_rnn.py rename to ptp/components/models/general_usage/seq2seq.py index 16380a8..a131e20 100644 --- a/ptp/components/models/seq2seq_rnn.py +++ b/ptp/components/models/general_usage/seq2seq.py @@ -21,7 +21,7 @@ from ptp.data_types.data_definition import DataDefinition -class Seq2Seq_RNN(Model): +class Seq2Seq(Model): """ Simple Classifier consisting of fully connected layer with log softmax non-linearity. """ @@ -33,7 +33,7 @@ def __init__(self, name, config): :type config: ``ptp.configuration.ConfigInterface`` """ # Call constructors of parent classes. - Model.__init__(self, name, Seq2Seq_RNN, config) + Model.__init__(self, name, Seq2Seq, config) # Get input/output mode self.input_mode = self.config["input_mode"] @@ -162,18 +162,18 @@ def output_data_definitions(self): return d - def forward(self, data_dict): + def forward(self, data_streams): """ Forward pass of the model. - :param data_dict: DataDict({'inputs', 'predictions ...}), where: + :param data_streams: DataStreams({'inputs', 'predictions ...}), where: - inputs: expected inputs [BATCH_SIZE x SEQ_LEN x INPUT_SIZE], - predictions: returned output with predictions (log_probs) [BATCH_SIZE x SEQ_LEN x PREDICTION_SIZE] """ # Get inputs [BATCH_SIZE x SEQ_LEN x INPUT_SIZE] - inputs = data_dict[self.key_inputs] + inputs = data_streams[self.key_inputs] if inputs.dim() == 2: inputs = inputs.unsqueeze(1) batch_size = inputs.shape[0] @@ -204,5 +204,5 @@ def forward(self, data_dict): outputs = self.log_softmax(outputs) # Add predictions to datadict. - data_dict.extend({self.key_predictions: outputs}) + data_streams.publish({self.key_predictions: outputs}) diff --git a/ptp/components/models/index_embeddings.py b/ptp/components/models/language/index_embeddings.py similarity index 91% rename from ptp/components/models/index_embeddings.py rename to ptp/components/models/language/index_embeddings.py index 49bb40f..7d5ce1d 100644 --- a/ptp/components/models/index_embeddings.py +++ b/ptp/components/models/language/index_embeddings.py @@ -81,23 +81,23 @@ def output_data_definitions(self): } - def forward(self, data_dict): + def forward(self, data_streams): """ Forward pass - performs embedding. - :param data_dict: DataDict({'images',**}), where: + :param data_streams: DataStreams({'images',**}), where: - inputs: expected indexed sentences [BATCH_SIZE x SENTENCE_LENGTH] - outputs: added embedded sentences [BATCH_SIZE x SENTENCE_LENGTH x EMBEDDING_SIZE] - :type data_dict: ``miprometheus.utils.DataDict`` + :type data_streams: ``miprometheus.utils.DataStreams`` """ - # Unpack DataDict. - inputs = data_dict[self.key_inputs] + # Unpack DataStreams. + inputs = data_streams[self.key_inputs] # Embedd inputs. embeds = self.embeddings(inputs) # Add embeddings to datadict. - data_dict.extend({self.key_outputs: embeds}) + data_streams.publish({self.key_outputs: embeds}) diff --git a/ptp/components/models/sentence_embeddings.py b/ptp/components/models/language/sentence_embeddings.py similarity index 92% rename from ptp/components/models/sentence_embeddings.py rename to ptp/components/models/language/sentence_embeddings.py index 29671c6..2671d6d 100644 --- a/ptp/components/models/sentence_embeddings.py +++ b/ptp/components/models/language/sentence_embeddings.py @@ -21,11 +21,10 @@ import torch from ptp.components.models.model import Model -from ptp.components.mixins.word_mappings import WordMappings +from ptp.components.mixins.word_mappings import WordMappings, pad_trunc_list from ptp.data_types.data_definition import DataDefinition -import ptp.components.utils.embeddings as emb -from ptp.components.utils.word_mappings import pad_trunc_list +import ptp.components.mixins.embeddings as emb class SentenceEmbeddings(Model, WordMappings): @@ -100,20 +99,20 @@ def output_data_definitions(self): } - def forward(self, data_dict): + def forward(self, data_streams): """ Forward pass - performs embedding. - :param data_dict: DataDict({'images',**}), where: + :param data_streams: DataStreams({'images',**}), where: - inputs: expected tokenized sentences [BATCH_SIZE x SENTENCE_LENGTH] x [string] - outputs: added embedded sentences [BATCH_SIZE x SENTENCE_LENGTH x EMBEDDING_SIZE] - :type data_dict: ``miprometheus.utils.DataDict`` + :type data_streams: ``miprometheus.utils.DataStreams`` """ - # Unpack DataDict. - inputs = data_dict[self.key_inputs] + # Unpack DataStreams. + inputs = data_streams[self.key_inputs] #print("{}: input len: {}, device: {}\n".format(self.name, len(inputs), "-")) indices_list = [] @@ -145,4 +144,4 @@ def forward(self, data_dict): #print("{}: embedds shape: {}, device: {}\n".format(self.name, embedds.shape, embedds.device)) # Add embeddings to datadict. - data_dict.extend({self.key_outputs: embedds}) + data_streams.publish({self.key_outputs: embedds}) diff --git a/ptp/components/models/vqa/multimodal_compact_bilinear_pooling.py b/ptp/components/models/multi_modal_reasoning/compact_bilinear_pooling.py similarity index 93% rename from ptp/components/models/vqa/multimodal_compact_bilinear_pooling.py rename to ptp/components/models/multi_modal_reasoning/compact_bilinear_pooling.py index 0a3953c..a8d0e6e 100644 --- a/ptp/components/models/vqa/multimodal_compact_bilinear_pooling.py +++ b/ptp/components/models/multi_modal_reasoning/compact_bilinear_pooling.py @@ -1,7 +1,7 @@ #!/usr/bin/env python3 # -*- coding: utf-8 -*- # -# Copyright (C) IBM Corporation 2018 +# Copyright (C) IBM Corporation 2019 # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -25,7 +25,7 @@ from ptp.data_types.data_definition import DataDefinition -class MultimodalCompactBilinearPooling(Model): +class CompactBilinearPooling(Model): """ Element of one of classical baselines for Visual Question Answering. @@ -48,7 +48,7 @@ def __init__(self, name, config): :type config: ``ptp.configuration.ConfigInterface`` """ - super(MultimodalCompactBilinearPooling, self).__init__(name, MultimodalCompactBilinearPooling, config) + super(CompactBilinearPooling, self).__init__(name, CompactBilinearPooling, config) # Get key mappings. self.key_image_encodings = self.stream_keys["image_encodings"] @@ -124,16 +124,16 @@ def output_data_definitions(self): self.key_outputs: DataDefinition([-1, self.output_size], [torch.Tensor], "Batch of outputs [BATCH_SIZE x OUTPUT_SIZE]") } - def forward(self, data_dict): + def forward(self, data_streams): """ Main forward pass of the model. - :param data_dict: DataDict({'images',**}) - :type data_dict: ``ptp.dadatypes.DataDict`` + :param data_streams: DataStreams({'images',**}) + :type data_streams: ``ptp.dadatypes.DataStreams`` """ - # Unpack DataDict. - enc_img = data_dict[self.key_image_encodings] - enc_q = data_dict[self.key_question_encodings] + # Unpack DataStreams. + enc_img = data_streams[self.key_image_encodings] + enc_q = data_streams[self.key_question_encodings] sketch_pm_img = self.image_sketch_projection_matrix sketch_pm_q = self.question_sketch_projection_matrix @@ -169,4 +169,4 @@ def forward(self, data_dict): #print("cbp=",cbp) # Add predictions to datadict. - data_dict.extend({self.key_outputs: cbp}) + data_streams.publish({self.key_outputs: cbp}) diff --git a/ptp/components/models/vqa/multimodal_factorized_bilinear_pooling.py b/ptp/components/models/multi_modal_reasoning/factorized_bilinear_pooling.py similarity index 90% rename from ptp/components/models/vqa/multimodal_factorized_bilinear_pooling.py rename to ptp/components/models/multi_modal_reasoning/factorized_bilinear_pooling.py index e9fcc33..cd6506f 100644 --- a/ptp/components/models/vqa/multimodal_factorized_bilinear_pooling.py +++ b/ptp/components/models/multi_modal_reasoning/factorized_bilinear_pooling.py @@ -1,7 +1,7 @@ #!/usr/bin/env python3 # -*- coding: utf-8 -*- # -# Copyright (C) IBM Corporation 2018 +# Copyright (C) IBM Corporation 2019 # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -24,7 +24,7 @@ from ptp.data_types.data_definition import DataDefinition import torch.nn.functional as F -class MultimodalFactorizedBilinearPooling(Model): +class FactorizedBilinearPooling(Model): """ Element of one of the classical baselines for Visual Question Answering. The multi-modal data is fused via sum-pooling of the element-wise multiplied high-dimensional representations and returned (for subsequent classification, done in a separate component e.g. ffn). @@ -42,7 +42,7 @@ def __init__(self, name, config): :type config: ``ptp.configuration.ConfigInterface`` """ - super(MultimodalFactorizedBilinearPooling, self).__init__(name, MultimodalFactorizedBilinearPooling, config) + super(FactorizedBilinearPooling, self).__init__(name, FactorizedBilinearPooling, config) # Get key mappings. self.key_image_encodings = self.stream_keys["image_encodings"] @@ -98,17 +98,17 @@ def output_data_definitions(self): self.key_outputs: DataDefinition([-1, self.output_size], [torch.Tensor], "Batch of outputs [BATCH_SIZE x OUTPUT_SIZE]") } - def forward(self, data_dict): + def forward(self, data_streams): """ Main forward pass of the model. - :param data_dict: DataDict({'images',**}) - :type data_dict: ``ptp.dadatypes.DataDict`` + :param data_streams: DataStreams({'images',**}) + :type data_streams: ``ptp.dadatypes.DataStreams`` """ - # Unpack DataDict. - enc_img = data_dict[self.key_image_encodings] #[48, 2048] - enc_q = data_dict[self.key_question_encodings] #[48, 100] + # Unpack DataStreams. + enc_img = data_streams[self.key_image_encodings] #[48, 2048] + enc_q = data_streams[self.key_question_encodings] #[48, 100] # Map image and question encodings to high-dimensional space using FF latent_img = self.dropout(self.image_encodings_ff(enc_img)) # [48, 512] @@ -127,4 +127,4 @@ def forward(self, data_dict): outputs = F.normalize(enc_z, p=2, dim=1) # [48, 256] # Add predictions to datadict. - data_dict.extend({self.key_outputs: outputs}) + data_streams.publish({self.key_outputs: outputs}) diff --git a/ptp/components/models/vqa/element_wise_multiplication.py b/ptp/components/models/multi_modal_reasoning/low_rank_bilinear_pooling.py similarity index 89% rename from ptp/components/models/vqa/element_wise_multiplication.py rename to ptp/components/models/multi_modal_reasoning/low_rank_bilinear_pooling.py index a5c88c8..8879752 100644 --- a/ptp/components/models/vqa/element_wise_multiplication.py +++ b/ptp/components/models/multi_modal_reasoning/low_rank_bilinear_pooling.py @@ -1,7 +1,7 @@ #!/usr/bin/env python3 # -*- coding: utf-8 -*- # -# Copyright (C) IBM Corporation 2018 +# Copyright (C) IBM Corporation 2019 # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -24,7 +24,7 @@ from ptp.data_types.data_definition import DataDefinition -class ElementWiseMultiplication(Model): +class LowRankBilinearPooling(Model): """ Element of one of classical baselines for Visual Question Answering. The model inputs (question and image encodings) are fused via element-wise multiplication and returned (for subsequent classification, done in a separate component e.g. ffn). @@ -41,7 +41,7 @@ def __init__(self, name, config): :type config: ``ptp.configuration.ConfigInterface`` """ - super(ElementWiseMultiplication, self).__init__(name, ElementWiseMultiplication, config) + super(LowRankBilinearPooling, self).__init__(name, LowRankBilinearPooling, config) # Get key mappings. self.key_image_encodings = self.stream_keys["image_encodings"] @@ -91,17 +91,17 @@ def output_data_definitions(self): self.key_outputs: DataDefinition([-1, self.output_size], [torch.Tensor], "Batch of outputs [BATCH_SIZE x OUTPUT_SIZE]") } - def forward(self, data_dict): + def forward(self, data_streams): """ Main forward pass of the model. - :param data_dict: DataDict({'images',**}) - :type data_dict: ``ptp.dadatypes.DataDict`` + :param data_streams: DataStreams({'images',**}) + :type data_streams: ``ptp.dadatypes.DataStreams`` """ - # Unpack DataDict. - enc_img = data_dict[self.key_image_encodings] - enc_q = data_dict[self.key_question_encodings] + # Unpack DataStreams. + enc_img = data_streams[self.key_image_encodings] + enc_q = data_streams[self.key_question_encodings] # Apply nonlinearities and dropout on images. enc_img = self.activation(enc_img) @@ -119,4 +119,4 @@ def forward(self, data_dict): outputs = latent_img * latent_q # Add predictions to datadict. - data_dict.extend({self.key_outputs: outputs}) + data_streams.publish({self.key_outputs: outputs}) diff --git a/ptp/components/models/vqa/attention.py b/ptp/components/models/multi_modal_reasoning/question_driven_attention.py similarity index 93% rename from ptp/components/models/vqa/attention.py rename to ptp/components/models/multi_modal_reasoning/question_driven_attention.py index e9d3af7..2606426 100644 --- a/ptp/components/models/vqa/attention.py +++ b/ptp/components/models/multi_modal_reasoning/question_driven_attention.py @@ -1,7 +1,7 @@ #!/usr/bin/env python3 # -*- coding: utf-8 -*- # -# Copyright (C) IBM Corporation 2018 +# Copyright (C) IBM Corporation 2019 # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -24,7 +24,7 @@ from ptp.data_types.data_definition import DataDefinition -class VQA_Attention(Model): +class QuestionDrivenAttention(Model): """ Element of one of the classical baselines for Visual Question Answering. Attention-weighted image maps are computed based on the question. @@ -43,7 +43,7 @@ def __init__(self, name, config): :type config: ``ptp.configuration.ConfigInterface`` """ - super(VQA_Attention, self).__init__(name, VQA_Attention, config) + super(QuestionDrivenAttention, self).__init__(name, QuestionDrivenAttention, config) # Get key mappings. self.key_feature_maps = self.stream_keys["feature_maps"] @@ -113,17 +113,17 @@ def output_data_definitions(self): self.key_outputs: DataDefinition([-1, self.output_size], [torch.Tensor], "Batch of outputs [BATCH_SIZE x OUTPUT_SIZE]") } - def forward(self, data_dict): + def forward(self, data_streams): """ Main forward pass of the model. - :param data_dict: DataDict({'images',**}) - :type data_dict: ``ptp.dadatypes.DataDict`` + :param data_streams: DataStreams({'images',**}) + :type data_streams: ``ptp.dadatypes.DataStreams`` """ - # Unpack DataDict. - enc_img = data_dict[self.key_feature_maps] #[48, 2048, 7, 7] - enc_q = data_dict[self.key_question_encodings] #[48, 100] + # Unpack DataStreams. + enc_img = data_streams[self.key_feature_maps] #[48, 2048, 7, 7] + enc_q = data_streams[self.key_question_encodings] #[48, 100] # print("im_enc", enc_img.shape) # print("enc_q", enc_q.shape) @@ -154,7 +154,7 @@ def forward(self, data_dict): outputs = torch.cat([attention_enc_img, enc_q], dim=1) # print("outputs", outputs.shape) # Add predictions to datadict. - data_dict.extend({self.key_outputs: outputs}) + data_streams.publish({self.key_outputs: outputs}) def tile_2d_over_nd(feature_vector, feature_map): diff --git a/ptp/components/models/vqa/relational_network.py b/ptp/components/models/multi_modal_reasoning/relational_network.py similarity index 94% rename from ptp/components/models/vqa/relational_network.py rename to ptp/components/models/multi_modal_reasoning/relational_network.py index 7ea2afa..c20e615 100644 --- a/ptp/components/models/vqa/relational_network.py +++ b/ptp/components/models/multi_modal_reasoning/relational_network.py @@ -1,7 +1,7 @@ #!/usr/bin/env python3 # -*- coding: utf-8 -*- # -# Copyright (C) IBM Corporation 2018 +# Copyright (C) IBM Corporation 2019 # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -125,16 +125,16 @@ def output_data_definitions(self): self.key_outputs: DataDefinition([-1, self.output_size], [torch.Tensor], "Batch of outputs [BATCH_SIZE x OUTPUT_SIZE]") } - def forward(self, data_dict): + def forward(self, data_streams): """ Main forward pass of the model. - :param data_dict: DataDict({'images',**}) - :type data_dict: ``ptp.dadatypes.DataDict`` + :param data_streams: DataStreams({'images',**}) + :type data_streams: ``ptp.dadatypes.DataStreams`` """ - # Unpack DataDict. - feat_m = data_dict[self.key_feature_maps] - enc_q = data_dict[self.key_question_encodings] + # Unpack DataStreams. + feat_m = data_streams[self.key_feature_maps] + enc_q = data_streams[self.key_question_encodings] # List [FEAT_WIDTH x FEAT_HEIGHT] of tensors [BATCH SIZE x (2 * FEAT_DEPTH + QUESTION_SIZE)] relational_inputs = [] @@ -169,4 +169,4 @@ def forward(self, data_dict): summed_relations = torch.sum(stacked_relations, dim=1) # Add outputs to datadict. - data_dict.extend({self.key_outputs: summed_relations}) + data_streams.publish({self.key_outputs: summed_relations}) diff --git a/ptp/components/models/vqa/self_attention.py b/ptp/components/models/multi_modal_reasoning/self_attention.py similarity index 91% rename from ptp/components/models/vqa/self_attention.py rename to ptp/components/models/multi_modal_reasoning/self_attention.py index 080f9a9..43bc6b4 100644 --- a/ptp/components/models/vqa/self_attention.py +++ b/ptp/components/models/multi_modal_reasoning/self_attention.py @@ -1,7 +1,7 @@ #!/usr/bin/env python3 # -*- coding: utf-8 -*- # -# Copyright (C) IBM Corporation 2018 +# Copyright (C) IBM Corporation 2019 # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -88,16 +88,16 @@ def output_data_definitions(self): self.key_outputs: DataDefinition([-1, self.output_size], [torch.Tensor], "Batch of outputs [BATCH_SIZE x OUTPUT_SIZE]") } - def forward(self, data_dict): + def forward(self, data_streams): """ Main forward pass of the model. - :param data_dict: DataDict({'images',**}) - :type data_dict: ``ptp.dadatypes.DataDict`` + :param data_streams: DataStreams({'images',**}) + :type data_streams: ``ptp.dadatypes.DataStreams`` """ - # Unpack DataDict. - input_enc = data_dict[self.key_question_encodings] # [batch, num_words, embed_dim] # Dense prediction from RNN + # Unpack DataStreams. + input_enc = data_streams[self.key_question_encodings] # [batch, num_words, embed_dim] # Dense prediction from RNN batch_size = input_enc.size()[0] # [48, 8, 100] # Attention computed as two FF layers with ReLU activation and softmax for probabilities ==> softmax(FF(ReLU(FF(input)))) @@ -112,4 +112,4 @@ def forward(self, data_dict): # outputs = torch.sum(input_enc_weighted,1)/self.num_attention_heads # Add predictions to datadict. - data_dict.extend({self.key_outputs: outputs}) + data_streams.publish({self.key_outputs: outputs}) diff --git a/ptp/components/models/convnet_encoder.py b/ptp/components/models/vision/convnet_encoder.py similarity index 97% rename from ptp/components/models/convnet_encoder.py rename to ptp/components/models/vision/convnet_encoder.py index 788ef5f..88f485f 100644 --- a/ptp/components/models/convnet_encoder.py +++ b/ptp/components/models/vision/convnet_encoder.py @@ -1,7 +1,7 @@ #!/usr/bin/env python3 # -*- coding: utf-8 -*- # -# Copyright (C) IBM Corporation 2018 +# Copyright (C) IBM Corporation 2019 # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -227,18 +227,18 @@ def output_data_definitions(self): self.key_feature_maps: DataDefinition([-1, self.out_channels_conv3, self.height_features_maxpool3, self.width_features_maxpool3], [torch.Tensor], "Batch of filter maps [BATCH_SIZE x FEAT_DEPTH x FEAT_HEIGHT x FEAT_WIDTH]") } - def forward(self, data_dict): + def forward(self, data_streams): """ forward pass of the ``SimpleConvNet`` model. - :param data_dict: DataDict({'inputs','outputs'}), where: + :param data_streams: DataStreams({'inputs','outputs'}), where: - inputs: [batch_size, in_depth, in_height, in_width], - feature_maps: batch of feature maps [batch_size, out_depth, out_height, out_width] """ # get images - images = data_dict[self.key_inputs] + images = data_streams[self.key_inputs] # apply Convolutional layer 1 out_conv1 = self.conv1(images) @@ -259,4 +259,4 @@ def forward(self, data_dict): out_maxpool3 = torch.nn.functional.relu(self.maxpool3(out_conv3)) # Add output to datadict. - data_dict.extend({self.key_feature_maps: out_maxpool3}) + data_streams.publish({self.key_feature_maps: out_maxpool3}) diff --git a/ptp/components/models/torch_vision_wrapper.py b/ptp/components/models/vision/generic_image_encoder.py similarity index 94% rename from ptp/components/models/torch_vision_wrapper.py rename to ptp/components/models/vision/generic_image_encoder.py index d7416c8..f50b701 100644 --- a/ptp/components/models/torch_vision_wrapper.py +++ b/ptp/components/models/vision/generic_image_encoder.py @@ -1,7 +1,7 @@ #!/usr/bin/env python3 # -*- coding: utf-8 -*- # -# Copyright (C) IBM Corporation 2018 +# Copyright (C) IBM Corporation 2019 # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -27,7 +27,7 @@ from ptp.data_types.data_definition import DataDefinition -class TorchVisionWrapper(Model): +class GenericImageEncoder(Model): """ Class """ @@ -41,7 +41,7 @@ def __init__(self, name, config): :type config: ``ptp.configuration.ConfigInterface`` """ - super(TorchVisionWrapper, self).__init__(name, TorchVisionWrapper, config) + super(GenericImageEncoder, self).__init__(name, GenericImageEncoder, config) # Get key mappings. self.key_inputs = self.stream_keys["inputs"] @@ -165,24 +165,24 @@ def output_data_definitions(self): self.key_outputs: DataDefinition([-1, self.output_size], [torch.Tensor], "Batch of outputs, each represented as probability distribution over classes [BATCH_SIZE x PREDICTION_SIZE]") } - def forward(self, data_dict): + def forward(self, data_streams): """ Main forward pass of the model. - :param data_dict: DataDict({'inputs', ....}), where: + :param data_streams: DataStreams({'inputs', ....}), where: - inputs: expected stream containing images [BATCH_SIZE x IMAGE_DEPTH x IMAGE_HEIGHT x IMAGE WIDTH] - outpus: added stream containing outputs [BATCH_SIZE x PREDICTION_SIZE] - :type data_dict: ``ptp.data_types.DataDict`` + :type data_streams: ``ptp.data_types.DataStreams`` """ - # Unpack DataDict. - img = data_dict[self.key_inputs] + # Unpack DataStreams. + img = data_streams[self.key_inputs] #print("{}: input shape: {}, device: {}\n".format(self.name, img.shape, img.device)) outputs = self.model(img) # Add outputs to datadict. - data_dict.extend({self.key_outputs: outputs}) + data_streams.publish({self.key_outputs: outputs}) diff --git a/ptp/components/models/lenet5.py b/ptp/components/models/vision/lenet5.py similarity index 92% rename from ptp/components/models/lenet5.py rename to ptp/components/models/vision/lenet5.py index f12d511..f7abaa9 100644 --- a/ptp/components/models/lenet5.py +++ b/ptp/components/models/vision/lenet5.py @@ -1,7 +1,7 @@ #!/usr/bin/env python3 # -*- coding: utf-8 -*- # -# Copyright (C) IBM Corporation 2018 +# Copyright (C) IBM Corporation 2019 # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -77,15 +77,15 @@ def output_data_definitions(self): self.key_predictions: DataDefinition([-1, self.prediction_size], [torch.Tensor], "Batch of predictions, each represented as probability distribution over classes [BATCH_SIZE x PREDICTION_SIZE]") } - def forward(self, data_dict): + def forward(self, data_streams): """ Main forward pass of the ``LeNet5`` model. - :param data_dict: DataDict({'images',**}), where: + :param data_streams: DataStreams({'images',**}), where: - images: [batch_size, num_channels, width, height] - :type data_dict: ``miprometheus.utils.DataDict`` + :type data_streams: ``miprometheus.utils.DataStreams`` :return: Predictions [batch_size, num_classes] @@ -100,8 +100,8 @@ def forward(self, data_dict): # #print (name, param.data) - # Unpack DataDict. - img = data_dict[self.key_inputs] + # Unpack DataStreams. + img = data_streams[self.key_inputs] # Pass inputs through layers. x = self.conv1(img) @@ -120,4 +120,4 @@ def forward(self, data_dict): # Log softmax. predictions = torch.nn.functional.log_softmax(x, dim=1) # Add predictions to datadict. - data_dict.extend({self.key_predictions: predictions}) + data_streams.publish({self.key_predictions: predictions}) diff --git a/ptp/components/publishers/__init__.py b/ptp/components/publishers/__init__.py index 2c8e764..8114e03 100644 --- a/ptp/components/publishers/__init__.py +++ b/ptp/components/publishers/__init__.py @@ -1,15 +1,7 @@ -from .accuracy_statistics import AccuracyStatistics -from .batch_size_statistics import BatchSizeStatistics -from .bleu_statistics import BLEUStatistics from .global_variable_publisher import GlobalVariablePublisher -from .precision_recall_statistics import PrecisionRecallStatistics from .stream_file_exporter import StreamFileExporter __all__ = [ - 'AccuracyStatistics', - 'BatchSizeStatistics', - 'BLEUStatistics', 'GlobalVariablePublisher', - 'PrecisionRecallStatistics', 'StreamFileExporter', ] diff --git a/ptp/components/publishers/global_variable_publisher.py b/ptp/components/publishers/global_variable_publisher.py index 9f8b2b4..53b40fe 100644 --- a/ptp/components/publishers/global_variable_publisher.py +++ b/ptp/components/publishers/global_variable_publisher.py @@ -79,11 +79,11 @@ def output_data_definitions(self): return { } - def __call__(self, data_dict): + def __call__(self, data_streams): """ Empty method. - :param data_dict: :py:class:`ptp.utils.DataDict` object. + :param data_streams: :py:class:`ptp.utils.DataStreams` object. """ pass \ No newline at end of file diff --git a/ptp/components/publishers/stream_file_exporter.py b/ptp/components/publishers/stream_file_exporter.py index f656374..77c83d4 100644 --- a/ptp/components/publishers/stream_file_exporter.py +++ b/ptp/components/publishers/stream_file_exporter.py @@ -86,20 +86,20 @@ def output_data_definitions(self): return { } - def __call__(self, data_dict): + def __call__(self, data_streams): """ Exports values from the indicated streams to file. - :param data_dict: :py:class:`ptp.utils.DataDict` object containing "indices" and other streams that will be exported to file. + :param data_streams: :py:class:`ptp.utils.DataStreams` object containing "indices" and other streams that will be exported to file. """ # Get batch size. - indices = data_dict[self.key_indices] + indices = data_streams[self.key_indices] batch_size = len(indices) # Check present streams. absent_streams = [] present_streams = [] for stream_key in self.input_stream_keys: - if stream_key in data_dict.keys(): + if stream_key in data_streams.keys(): present_streams.append(stream_key) else: absent_streams.append(stream_key) @@ -122,7 +122,7 @@ def __call__(self, data_dict): val_str = '' for stream_key in self.input_stream_keys: if stream_key in present_streams: - value = data_dict[stream_key][i] + value = data_streams[stream_key][i] # Add value changed to string along with separator. val_str = val_str + '{}'.format(value) + self.separator # Remove the last separator. diff --git a/ptp/components/statistics/__init__.py b/ptp/components/statistics/__init__.py new file mode 100644 index 0000000..40f75e7 --- /dev/null +++ b/ptp/components/statistics/__init__.py @@ -0,0 +1,11 @@ +from .accuracy_statistics import AccuracyStatistics +from .batch_size_statistics import BatchSizeStatistics +from .bleu_statistics import BLEUStatistics +from .precision_recall_statistics import PrecisionRecallStatistics + +__all__ = [ + 'AccuracyStatistics', + 'BatchSizeStatistics', + 'BLEUStatistics', + 'PrecisionRecallStatistics', + ] diff --git a/ptp/components/publishers/accuracy_statistics.py b/ptp/components/statistics/accuracy_statistics.py similarity index 91% rename from ptp/components/publishers/accuracy_statistics.py rename to ptp/components/statistics/accuracy_statistics.py index 4ff9d9b..76effea 100644 --- a/ptp/components/publishers/accuracy_statistics.py +++ b/ptp/components/statistics/accuracy_statistics.py @@ -88,31 +88,31 @@ def output_data_definitions(self): return {} - def __call__(self, data_dict): + def __call__(self, data_streams): """ Call method - empty for all statistics. """ pass - def calculate_accuracy(self, data_dict): + def calculate_accuracy(self, data_streams): """ Calculates accuracy equal to mean number of correct classification in a given batch. - :param data_dict: DataDict containing the targets. - :type data_dict: DataDict + :param data_streams: DataStreams containing the targets. + :type data_streams: DataStreams :return: Accuracy. """ # Get targets. - targets = data_dict[self.key_targets].data.cpu().numpy() + targets = data_streams[self.key_targets].data.cpu().numpy() if self.use_prediction_distributions: # Get indices of the max log-probability. - preds = data_dict[self.key_predictions].max(1)[1].data.cpu().numpy() + preds = data_streams[self.key_predictions].max(1)[1].data.cpu().numpy() else: - preds = data_dict[self.key_predictions].data.cpu().numpy() + preds = data_streams[self.key_predictions].data.cpu().numpy() # Calculate the correct predictinos. correct = np.equal(preds, targets) @@ -121,7 +121,7 @@ def calculate_accuracy(self, data_dict): if self.use_masking: # Get masks from inputs. - masks = data_dict[self.key_masks].data.cpu().numpy() + masks = data_streams[self.key_masks].data.cpu().numpy() correct = correct * masks batch_size = masks.sum() else: @@ -153,14 +153,14 @@ def add_statistics(self, stat_col): stat_col.add_statistics(self.key_accuracy, '{:6.4f}') stat_col.add_statistics(self.key_accuracy+'_support', None) - def collect_statistics(self, stat_col, data_dict): + def collect_statistics(self, stat_col, data_streams): """ Collects statistics (accuracy and support set size) for given episode. :param stat_col: ``StatisticsCollector``. """ - acc, batch_size = self.calculate_accuracy(data_dict) + acc, batch_size = self.calculate_accuracy(data_streams) stat_col[self.key_accuracy] = acc stat_col[self.key_accuracy+'_support'] = batch_size diff --git a/ptp/components/publishers/batch_size_statistics.py b/ptp/components/statistics/batch_size_statistics.py similarity index 94% rename from ptp/components/publishers/batch_size_statistics.py rename to ptp/components/statistics/batch_size_statistics.py index 4e1088f..01535f3 100644 --- a/ptp/components/publishers/batch_size_statistics.py +++ b/ptp/components/statistics/batch_size_statistics.py @@ -62,7 +62,7 @@ def output_data_definitions(self): return {} - def __call__(self, data_dict): + def __call__(self, data_streams): """ Call method - empty for all statistics. """ @@ -77,14 +77,14 @@ def add_statistics(self, stat_col): """ stat_col.add_statistics('batch_size', '{:06d}') - def collect_statistics(self, stat_col, data_dict): + def collect_statistics(self, stat_col, data_streams): """ Collects statistics (batch_size) for given episode. :param stat_col: ``StatisticsCollector``. """ - stat_col['batch_size'] = len(data_dict[self.key_indices]) + stat_col['batch_size'] = len(data_streams[self.key_indices]) def add_aggregators(self, stat_agg): """ diff --git a/ptp/components/publishers/bleu_statistics.py b/ptp/components/statistics/bleu_statistics.py similarity index 92% rename from ptp/components/publishers/bleu_statistics.py rename to ptp/components/statistics/bleu_statistics.py index 6432c06..8b9a465 100644 --- a/ptp/components/publishers/bleu_statistics.py +++ b/ptp/components/statistics/bleu_statistics.py @@ -104,35 +104,35 @@ def output_data_definitions(self): return {} - def __call__(self, data_dict): + def __call__(self, data_streams): """ Call method - empty for all statistics. """ pass - def calculate_BLEU(self, data_dict): + def calculate_BLEU(self, data_streams): """ Calculates BLEU for predictions of a given batch. - :param data_dict: DataDict containing the targets and predictions (and optionally masks). - :type data_dict: DataDict + :param data_streams: DataStreams containing the targets and predictions (and optionally masks). + :type data_streams: DataStreams :return: Accuracy. """ # Get targets. - targets = data_dict[self.key_targets].data.cpu().numpy().tolist() + targets = data_streams[self.key_targets].data.cpu().numpy().tolist() if self.use_prediction_distributions: # Get indices of the max log-probability. - preds = data_dict[self.key_predictions].max(-1)[1].data.cpu().numpy().tolist() + preds = data_streams[self.key_predictions].max(-1)[1].data.cpu().numpy().tolist() else: - preds = data_dict[self.key_predictions].data.cpu().numpy().tolist() + preds = data_streams[self.key_predictions].data.cpu().numpy().tolist() #if self.use_masking: # # Get masks from inputs. - # masks = data_dict[self.key_masks].data.cpu().numpy().tolist() + # masks = data_streams[self.key_masks].data.cpu().numpy().tolist() #else: # batch_size = preds.shape[0] @@ -185,14 +185,14 @@ def add_statistics(self, stat_col): """ stat_col.add_statistics(self.key_bleu, '{:6.4f}') - def collect_statistics(self, stat_col, data_dict): + def collect_statistics(self, stat_col, data_streams): """ Collects statistics (batch_size) for given episode. :param stat_col: ``StatisticsCollector``. """ - stat_col[self.key_bleu] = self.calculate_BLEU(data_dict) + stat_col[self.key_bleu] = self.calculate_BLEU(data_streams) def add_aggregators(self, stat_agg): """ diff --git a/ptp/components/publishers/precision_recall_statistics.py b/ptp/components/statistics/precision_recall_statistics.py similarity index 94% rename from ptp/components/publishers/precision_recall_statistics.py rename to ptp/components/statistics/precision_recall_statistics.py index 36b83ea..43fd08d 100644 --- a/ptp/components/publishers/precision_recall_statistics.py +++ b/ptp/components/statistics/precision_recall_statistics.py @@ -109,19 +109,19 @@ def output_data_definitions(self): """ return {} - def __call__(self, data_dict): + def __call__(self, data_streams): """ Calculates precission recall statistics. - :param data_dict: DataDict containing the targets. - :type data_dict: DataDict + :param data_streams: DataStreams containing the targets. + :type data_streams: DataStreams """ # Use worker interval. if self.app_state.episode % self.app_state.args.logging_interval == 0: # Calculate all four statistics. - confusion_matrix, precisions, recalls, f1scores, supports = self.calculate_statistics(data_dict) + confusion_matrix, precisions, recalls, f1scores, supports = self.calculate_statistics(data_streams) if self.show_confusion_matrix: self.logger.info("Confusion matrix:\n{}".format(confusion_matrix)) @@ -150,28 +150,28 @@ def __call__(self, data_dict): self.logger.info(log_str) - def calculate_statistics(self, data_dict): + def calculate_statistics(self, data_streams): """ Calculates confusion_matrix, precission, recall, f1score and support statistics. - :param data_dict: DataDict containing the targets. - :type data_dict: DataDict + :param data_streams: DataStreams containing the targets. + :type data_streams: DataStreams :return: Calculated statistics. """ - targets = data_dict[self.key_targets].data.cpu().numpy() + targets = data_streams[self.key_targets].data.cpu().numpy() #print("Targets :", targets) if self.use_prediction_distributions: # Get indices of the max log-probability. - preds = data_dict[self.key_predictions].max(1)[1].data.cpu().numpy() + preds = data_streams[self.key_predictions].max(1)[1].data.cpu().numpy() else: - preds = data_dict[self.key_predictions].data.cpu().numpy() + preds = data_streams[self.key_predictions].data.cpu().numpy() #print("Predictions :", preds) if self.use_masking: # Get masks from inputs. - masks = data_dict[self.key_masks].data.cpu().numpy() + masks = data_streams[self.key_masks].data.cpu().numpy() else: # Create vector full of ones. masks = np.ones(targets.shape[0]) @@ -245,7 +245,7 @@ def add_statistics(self, stat_col): stat_col.add_statistics(self.key_f1score+'_support', None) - def collect_statistics(self, stat_col, data_dict): + def collect_statistics(self, stat_col, data_streams): """ Collects statistics (batch_size) for given episode. @@ -253,7 +253,7 @@ def collect_statistics(self, stat_col, data_dict): """ # Calculate all four statistics. - _, precisions, recalls, f1scores, supports = self.calculate_statistics(data_dict) + _, precisions, recalls, f1scores, supports = self.calculate_statistics(data_streams) # Calculate weighted averages. precision_sum = sum([pi*si for (pi,si) in zip(precisions,supports)]) diff --git a/ptp/components/problems/image_text_to_class/__init__.py b/ptp/components/tasks/image_text_to_class/__init__.py similarity index 77% rename from ptp/components/problems/image_text_to_class/__init__.py rename to ptp/components/tasks/image_text_to_class/__init__.py index eb52c35..2f50ed1 100644 --- a/ptp/components/problems/image_text_to_class/__init__.py +++ b/ptp/components/tasks/image_text_to_class/__init__.py @@ -1,7 +1,9 @@ from .clevr import CLEVR +from .gqa import GQA from .vqa_med_2019 import VQAMED2019 __all__ = [ 'CLEVR', + 'GQA', 'VQAMED2019', ] diff --git a/ptp/components/problems/image_text_to_class/clevr.py b/ptp/components/tasks/image_text_to_class/clevr.py similarity index 86% rename from ptp/components/problems/image_text_to_class/clevr.py rename to ptp/components/tasks/image_text_to_class/clevr.py index ff18441..5a0f1a4 100644 --- a/ptp/components/problems/image_text_to_class/clevr.py +++ b/ptp/components/tasks/image_text_to_class/clevr.py @@ -24,16 +24,16 @@ import torch from torchvision import transforms -from ptp.components.problems.problem import Problem +from ptp.components.tasks.task import Task from ptp.data_types.data_definition import DataDefinition -#from ptp.components.utils.io import save_nparray_to_csv_file +#from ptp.components.mixins.io import save_nparray_to_csv_file from ptp.configuration.config_parsing import get_value_from_dictionary, get_value_list_from_dictionary from ptp.configuration.configuration_error import ConfigurationError -class CLEVR(Problem): +class CLEVR(Task): """ - Problem providing data associated with CLEVR (Compositional Language andElementary Visual Reasoning) diagnostics dataset + Task providing data associated with CLEVR (Compositional Language andElementary Visual Reasoning) diagnostics dataset The dataset consists of three splits: - A training set of 70,000 images and 699,989 questions @@ -60,7 +60,7 @@ class CLEVR(Problem): """ def __init__(self, name, config): """ - Initializes problem object. Calls base constructor. Downloads the dataset if not present and loads the adequate files depending on the mode. + Initializes task object. Calls base constructor. Downloads the dataset if not present and loads the adequate files depending on the mode. :param name: Name of the component. @@ -69,7 +69,7 @@ def __init__(self, name, config): :param config: Dictionary of parameters (read from configuration ``.yaml`` file). """ # Call constructors of parent classes. - Problem.__init__(self, name, CLEVR, config) + Task.__init__(self, name, CLEVR, config) # Get key mappings of all output streams. self.key_images = self.stream_keys["images"] @@ -249,9 +249,9 @@ def output_data_definitions(self): def __len__(self): """ - Returns the "size" of the "problem" (total number of samples). + Returns the "size" of the "task" (total number of samples). - :return: The size of the problem. + :return: The size of the task. """ return len(self.dataset) @@ -317,70 +317,70 @@ def __getitem__(self, index): :param index: index of the sample to return. :type index: int - :return: DataDict({'indices', 'images', 'images_ids','questions', 'answers', 'question_type_ids', 'question_type_names'}) + :return: DataStreams({'indices', 'images', 'images_ids','questions', 'answers', 'question_type_ids', 'question_type_names'}) """ # Get item. item = self.dataset[index] # Create the resulting sample (data dict). - data_dict = self.create_data_dict(index) + data_streams = self.create_data_streams(index) # Load and stream the image ids. img_id = item["image_filename"] - data_dict[self.key_image_ids] = img_id + data_streams[self.key_image_ids] = img_id # Load the adequate image - only when required. if self.stream_images: img = self.get_image(img_id) # Image related variables. - data_dict[self.key_images] = img + data_streams[self.key_images] = img # Return question. - data_dict[self.key_questions] = item["question"] + data_streams[self.key_questions] = item["question"] # Return answer. if "answer" in item.keys(): - data_dict[self.key_answers] = item["answer"] + data_streams[self.key_answers] = item["answer"] else: - data_dict[self.key_answers] = "" + data_streams[self.key_answers] = "" # Question type related variables. if "question_family_index" in item.keys(): - data_dict[self.key_question_type_ids] = self.question_family_id_to_subtype_id_mapping[item["question_family_index"]] - data_dict[self.key_question_type_names] = self.question_family_id_to_subtype_mapping[item["question_family_index"]] + data_streams[self.key_question_type_ids] = self.question_family_id_to_subtype_id_mapping[item["question_family_index"]] + data_streams[self.key_question_type_names] = self.question_family_id_to_subtype_mapping[item["question_family_index"]] else: - data_dict[self.key_question_type_ids] = -1 - data_dict[self.key_question_type_names] = "" + data_streams[self.key_question_type_ids] = -1 + data_streams[self.key_question_type_names] = "" # Return sample. - return data_dict + return data_streams def collate_fn(self, batch): """ - Combines a list of DataDict (retrieved with :py:func:`__getitem__`) into a batch. + Combines a list of DataStreams (retrieved with :py:func:`__getitem__`) into a batch. :param batch: list of individual samples to combine :type batch: list - :return: DataDict({'indices', 'images', 'images_ids','questions', 'answers', 'category_ids', 'image_sizes'}) + :return: DataStreams({'indices', 'images', 'images_ids','questions', 'answers', 'category_ids', 'image_sizes'}) """ # Collate indices. - data_dict = self.create_data_dict([sample[self.key_indices] for sample in batch]) + data_streams = self.create_data_streams([sample[self.key_indices] for sample in batch]) # Stack images. - data_dict[self.key_image_ids] = [item[self.key_image_ids] for item in batch] + data_streams[self.key_image_ids] = [item[self.key_image_ids] for item in batch] if self.stream_images: - data_dict[self.key_images] = torch.stack([item[self.key_images] for item in batch]).type(torch.FloatTensor) + data_streams[self.key_images] = torch.stack([item[self.key_images] for item in batch]).type(torch.FloatTensor) # Collate lists/lists of lists. - data_dict[self.key_questions] = [item[self.key_questions] for item in batch] - data_dict[self.key_answers] = [item[self.key_answers] for item in batch] + data_streams[self.key_questions] = [item[self.key_questions] for item in batch] + data_streams[self.key_answers] = [item[self.key_answers] for item in batch] # Stack categories. - data_dict[self.key_question_type_ids] = torch.tensor([item[self.key_question_type_ids] for item in batch]) - data_dict[self.key_question_type_names] = [item[self.key_question_type_names] for item in batch] + data_streams[self.key_question_type_ids] = torch.tensor([item[self.key_question_type_ids] for item in batch]) + data_streams[self.key_question_type_names] = [item[self.key_question_type_names] for item in batch] # Return collated dict. - return data_dict + return data_streams diff --git a/ptp/components/problems/image_text_to_class/gqa.py b/ptp/components/tasks/image_text_to_class/gqa.py similarity index 85% rename from ptp/components/problems/image_text_to_class/gqa.py rename to ptp/components/tasks/image_text_to_class/gqa.py index 0f2da3c..0ab3148 100644 --- a/ptp/components/problems/image_text_to_class/gqa.py +++ b/ptp/components/tasks/image_text_to_class/gqa.py @@ -25,15 +25,15 @@ import torch from torchvision import transforms -from ptp.components.problems.problem import Problem +from ptp.components.tasks.task import Task from ptp.data_types.data_definition import DataDefinition from ptp.configuration.config_parsing import get_value_from_dictionary, get_value_list_from_dictionary from ptp.configuration.configuration_error import ConfigurationError -class GQA(Problem): +class GQA(Task): """ - Problem providing data associated with the GQA dataset (Question Answering on Image Scene Graphs). + Task providing data associated with the GQA dataset (Question Answering on Image Scene Graphs). The dataset consists of 22M questions about various day-to-day images. Each image is associated with a scene graph of the image's objects, attributes and relations. @@ -50,7 +50,7 @@ class GQA(Problem): """ def __init__(self, name, config): """ - Initializes problem object. Calls base constructor. Downloads the dataset if not present and loads the adequate files depending on the mode. + Initializes task object. Calls base constructor. Downloads the dataset if not present and loads the adequate files depending on the mode. :param name: Name of the component. @@ -59,7 +59,7 @@ def __init__(self, name, config): :param config: Dictionary of parameters (read from configuration ``.yaml`` file). """ # Call constructors of parent classes. - Problem.__init__(self, name, GQA, config) + Task.__init__(self, name, GQA, config) # Get key mappings of all output streams. self.key_sample_ids = self.stream_keys["sample_ids"] @@ -183,9 +183,9 @@ def output_data_definitions(self): def __len__(self): """ - Returns the "size" of the "problem" (total number of samples). + Returns the "size" of the "task" (total number of samples). - :return: The size of the problem. + :return: The size of the task. """ return len(self.dataset) @@ -278,61 +278,61 @@ def __getitem__(self, index): :param index: index of the sample to return. :type index: int - :return: DataDict({'indices', 'sample_ids', images', 'images_ids','questions', 'answers', 'full_answers'}) + :return: DataStreams({'indices', 'sample_ids', images', 'images_ids','questions', 'answers', 'full_answers'}) """ # Get item. item = self.dataset[index] # Create the resulting sample (data dict). - data_dict = self.create_data_dict(index) + data_streams = self.create_data_streams(index) # Return sample id. - data_dict[self.key_sample_ids] = item[self.key_sample_ids] + data_streams[self.key_sample_ids] = item[self.key_sample_ids] # Load and stream the image ids. img_id = item[self.key_image_ids] - data_dict[self.key_image_ids] = img_id + data_streams[self.key_image_ids] = img_id # Load the adequate image - only when required. if self.stream_images: - data_dict[self.key_images] = self.get_image(img_id) + data_streams[self.key_images] = self.get_image(img_id) # Return question. - data_dict[self.key_questions] = item[self.key_questions] + data_streams[self.key_questions] = item[self.key_questions] # Return answers. - data_dict[self.key_answers] = item[self.key_answers] - data_dict[self.key_full_answers] = item[self.key_full_answers] + data_streams[self.key_answers] = item[self.key_answers] + data_streams[self.key_full_answers] = item[self.key_full_answers] # Return sample. - return data_dict + return data_streams def collate_fn(self, batch): """ - Combines a list of DataDict (retrieved with :py:func:`__getitem__`) into a batch. + Combines a list of DataStreams (retrieved with :py:func:`__getitem__`) into a batch. :param batch: list of individual samples to combine :type batch: list - :return: DataDict({'indices', 'images', 'images_ids','questions', 'answers', 'category_ids', 'image_sizes'}) + :return: DataStreams({'indices', 'images', 'images_ids','questions', 'answers', 'category_ids', 'image_sizes'}) """ # Collate indices. - data_dict = self.create_data_dict([sample[self.key_indices] for sample in batch]) + data_streams = self.create_data_streams([sample[self.key_indices] for sample in batch]) # Collate sample ids. - data_dict[self.key_sample_ids] = [item[self.key_sample_ids] for item in batch] + data_streams[self.key_sample_ids] = [item[self.key_sample_ids] for item in batch] # Stack images. - data_dict[self.key_image_ids] = [item[self.key_image_ids] for item in batch] + data_streams[self.key_image_ids] = [item[self.key_image_ids] for item in batch] if self.stream_images: - data_dict[self.key_images] = torch.stack([item[self.key_images] for item in batch]).type(torch.FloatTensor) + data_streams[self.key_images] = torch.stack([item[self.key_images] for item in batch]).type(torch.FloatTensor) # Collate lists/lists of lists. - data_dict[self.key_questions] = [item[self.key_questions] for item in batch] - data_dict[self.key_answers] = [item[self.key_answers] for item in batch] - data_dict[self.key_full_answers] = [item[self.key_full_answers] for item in batch] + data_streams[self.key_questions] = [item[self.key_questions] for item in batch] + data_streams[self.key_answers] = [item[self.key_answers] for item in batch] + data_streams[self.key_full_answers] = [item[self.key_full_answers] for item in batch] # Return collated dict. - return data_dict + return data_streams diff --git a/ptp/components/problems/image_text_to_class/vqa_med_2019.py b/ptp/components/tasks/image_text_to_class/vqa_med_2019.py similarity index 93% rename from ptp/components/problems/image_text_to_class/vqa_med_2019.py rename to ptp/components/tasks/image_text_to_class/vqa_med_2019.py index a739e4a..3181ae4 100644 --- a/ptp/components/problems/image_text_to_class/vqa_med_2019.py +++ b/ptp/components/tasks/image_text_to_class/vqa_med_2019.py @@ -29,16 +29,16 @@ import torch from torchvision import transforms -from ptp.components.problems.problem import Problem +from ptp.components.tasks.task import Task from ptp.data_types.data_definition import DataDefinition -from ptp.components.utils.io import save_nparray_to_csv_file +from ptp.components.mixins.io import save_nparray_to_csv_file from ptp.configuration.config_parsing import get_value_list_from_dictionary, get_value_from_dictionary -class VQAMED2019(Problem): +class VQAMED2019(Task): """ - Problem providing data associated with ImageCLEF VQA 2019 challenge. + Task providing data associated with ImageCLEF VQA 2019 challenge. The dataset consists of four splits: @@ -64,7 +64,7 @@ class VQAMED2019(Problem): """ def __init__(self, name, config): """ - Initializes problem object. Calls base constructor. Downloads the dataset if not present and loads the adequate files depending on the mode. + Initializes task object. Calls base constructor. Downloads the dataset if not present and loads the adequate files depending on the mode. :param name: Name of the component. @@ -73,7 +73,7 @@ def __init__(self, name, config): :param config: Dictionary of parameters (read from configuration ``.yaml`` file). """ # Call constructors of parent classes. - Problem.__init__(self, name, VQAMED2019, config) + Task.__init__(self, name, VQAMED2019, config) # (Eventually) download required packages. nltk.download('punkt') @@ -295,7 +295,7 @@ def __init__(self, name, config): self.dataset[self.ix[0]][self.key_answers] )) - # Check if we want the problem to calculate and export the weights. + # Check if we want the task to calculate and export the weights. self.export_sample_weights = self.config["export_sample_weights"] if self.export_sample_weights != '': self.calculate_and_export_sample_weights(self.export_sample_weights) @@ -336,9 +336,9 @@ def output_data_definitions(self): def __len__(self): """ - Returns the "size" of the "problem" (total number of samples). + Returns the "size" of the "task" (total number of samples). - :return: The size of the problem. + :return: The size of the task. """ return len(self.dataset) @@ -421,7 +421,7 @@ def calculate_and_export_sample_weights(self, filename): # Process filename. (path, name) = os.path.split(filename) if path == '': - # Use default problem folder as destination. + # Use default task folder as destination. path = self.data_folder else: path = os.path.expanduser(path) @@ -807,17 +807,17 @@ def __getitem__(self, index): :param index: index of the sample to return. :type index: int - :return: DataDict({'indices', 'images', 'images_ids','questions', 'answers', 'category_ids', 'image_sizes'}) + :return: DataStreams({'indices', 'images', 'images_ids','questions', 'answers', 'category_ids', 'image_sizes'}) """ # Get item. item = self.dataset[self.ix[index]] # Create the resulting sample (data dict). - data_dict = self.create_data_dict(index) + data_streams = self.create_data_streams(index) # Load and stream the image ids. img_id = item[self.key_image_ids] - data_dict[self.key_image_ids] = img_id + data_streams[self.key_image_ids] = img_id # Load the adequate image - only when required. if self.stream_images: @@ -831,10 +831,10 @@ def __getitem__(self, index): img, img_size = self.get_image(img_id, item["image_folder"]) # Image related variables. - data_dict[self.key_images] = img + data_streams[self.key_images] = img # Scale width and height to range (0,1). - data_dict[self.key_image_sizes] = img_size + data_streams[self.key_image_sizes] = img_size # Apply question transformations. preprocessed_question = item[self.key_questions] @@ -846,23 +846,23 @@ def __getitem__(self, index): if 'random_shuffle_words' in self.question_preprocessing: preprocessed_question = self.random_shuffle_words(preprocessed_question) # Return question. - data_dict[self.key_questions] = preprocessed_question + data_streams[self.key_questions] = preprocessed_question # Return answer. preprocessed_answer = item[self.key_answers] - data_dict[self.key_answers] = preprocessed_answer + data_streams[self.key_answers] = preprocessed_answer # Question category related variables. # Check if this is binary question. if self.predict_yes_no(item[self.key_questions]): - data_dict[self.key_category_ids] = 4 # Binary. - data_dict[self.key_category_names] = self.category_idx_to_word[4] + data_streams[self.key_category_ids] = 4 # Binary. + data_streams[self.key_category_names] = self.category_idx_to_word[4] else: - data_dict[self.key_category_ids] = item[self.key_category_ids] - data_dict[self.key_category_names] = self.category_idx_to_word[item[self.key_category_ids]] + data_streams[self.key_category_ids] = item[self.key_category_ids] + data_streams[self.key_category_names] = self.category_idx_to_word[item[self.key_category_ids]] # Return sample. - return data_dict + return data_streams def predict_yes_no(self, qtext): """ @@ -878,30 +878,30 @@ def predict_yes_no(self, qtext): def collate_fn(self, batch): """ - Combines a list of DataDict (retrieved with :py:func:`__getitem__`) into a batch. + Combines a list of DataStreams (retrieved with :py:func:`__getitem__`) into a batch. :param batch: list of individual samples to combine :type batch: list - :return: DataDict({'indices', 'images', 'images_ids','questions', 'answers', 'category_ids', 'image_sizes'}) + :return: DataStreams({'indices', 'images', 'images_ids','questions', 'answers', 'category_ids', 'image_sizes'}) """ # Collate indices. - data_dict = self.create_data_dict([sample[self.key_indices] for sample in batch]) + data_streams = self.create_data_streams([sample[self.key_indices] for sample in batch]) # Stack images. - data_dict[self.key_image_ids] = [item[self.key_image_ids] for item in batch] + data_streams[self.key_image_ids] = [item[self.key_image_ids] for item in batch] if self.stream_images: - data_dict[self.key_images] = torch.stack([item[self.key_images] for item in batch]).type(torch.FloatTensor) - data_dict[self.key_image_sizes] = torch.stack([item[self.key_image_sizes] for item in batch]).type(torch.FloatTensor) + data_streams[self.key_images] = torch.stack([item[self.key_images] for item in batch]).type(torch.FloatTensor) + data_streams[self.key_image_sizes] = torch.stack([item[self.key_image_sizes] for item in batch]).type(torch.FloatTensor) # Collate lists/lists of lists. - data_dict[self.key_questions] = [item[self.key_questions] for item in batch] - data_dict[self.key_answers] = [item[self.key_answers] for item in batch] + data_streams[self.key_questions] = [item[self.key_questions] for item in batch] + data_streams[self.key_answers] = [item[self.key_answers] for item in batch] # Stack categories. - data_dict[self.key_category_ids] = torch.tensor([item[self.key_category_ids] for item in batch]) - data_dict[self.key_category_names] = [item[self.key_category_names] for item in batch] + data_streams[self.key_category_ids] = torch.tensor([item[self.key_category_ids] for item in batch]) + data_streams[self.key_category_names] = [item[self.key_category_names] for item in batch] # Return collated dict. - return data_dict + return data_streams diff --git a/ptp/components/problems/image_to_class/__init__.py b/ptp/components/tasks/image_to_class/__init__.py similarity index 100% rename from ptp/components/problems/image_to_class/__init__.py rename to ptp/components/tasks/image_to_class/__init__.py diff --git a/ptp/components/problems/image_to_class/cifar_100.py b/ptp/components/tasks/image_to_class/cifar_100.py similarity index 90% rename from ptp/components/problems/image_to_class/cifar_100.py rename to ptp/components/tasks/image_to_class/cifar_100.py index a17f785..d74c187 100644 --- a/ptp/components/problems/image_to_class/cifar_100.py +++ b/ptp/components/tasks/image_to_class/cifar_100.py @@ -1,7 +1,7 @@ #!/usr/bin/env python3 # -*- coding: utf-8 -*- # -# Copyright (C) IBM Corporation 2018 +# Copyright (C) IBM Corporation 2019 # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -21,12 +21,12 @@ import torch from torchvision import datasets, transforms -from ptp.components.problems.problem import Problem +from ptp.components.tasks.task import Task from ptp.data_types.data_definition import DataDefinition -class CIFAR100(Problem): +class CIFAR100(Task): """ - Classic CIFAR-100 image classification problem. + Classic CIFAR-100 image classification task. Reference page: http://www.cs.toronto.edu/~kriz/cifar.html @@ -35,13 +35,13 @@ class CIFAR100(Problem): def __init__(self, name, config): """ - Initializes the problem. + Initializes the task. .. warning:: Resizing images might cause a significant slow down in batch generation. - :param name: Problem name. + :param name: Task name. :type name: str :param config: Dictionary of parameters (read from the configuration ``.yaml`` file). @@ -157,9 +157,9 @@ def __init__(self, name, config): def __len__(self): """ - Returns the "size" of the "problem" (total number of samples). + Returns the "size" of the "task" (total number of samples). - :return: The size of the problem. + :return: The size of the task. """ return len(self.dataset) @@ -186,7 +186,7 @@ def __getitem__(self, index): :param index: index of the sample to return. :type index: int - :return: ``DataDict({'images','targets'})``, with: + :return: ``DataStreams({'images','targets'})``, with: - images: Image, resized if ``self.resize`` is set, - targets: Index of the target class @@ -194,15 +194,15 @@ def __getitem__(self, index): # Get image and fine label id. image, fine_target = self.dataset.__getitem__(index) - # Return data_dict. - data_dict = self.create_data_dict(index) - data_dict[self.key_inputs] = image + # Return data_streams. + data_streams = self.create_data_streams(index) + data_streams[self.key_inputs] = image # Targets. - data_dict[self.key_coarse_targets] = self.fine_to_coarse_id_mapping[fine_target] - data_dict[self.key_fine_targets] = fine_target + data_streams[self.key_coarse_targets] = self.fine_to_coarse_id_mapping[fine_target] + data_streams[self.key_fine_targets] = fine_target # Labels. - data_dict[self.key_coarse_labels] = self.coarse_ix_to_word[self.fine_to_coarse_id_mapping[fine_target]] - data_dict[self.key_fine_labels] = self.fine_ix_to_word[fine_target] + data_streams[self.key_coarse_labels] = self.coarse_ix_to_word[self.fine_to_coarse_id_mapping[fine_target]] + data_streams[self.key_fine_labels] = self.fine_ix_to_word[fine_target] - #print(data_dict) - return data_dict + #print(data_streams) + return data_streams diff --git a/ptp/components/problems/image_to_class/mnist.py b/ptp/components/tasks/image_to_class/mnist.py similarity index 88% rename from ptp/components/problems/image_to_class/mnist.py rename to ptp/components/tasks/image_to_class/mnist.py index 5d9c3fc..2446868 100644 --- a/ptp/components/problems/image_to_class/mnist.py +++ b/ptp/components/tasks/image_to_class/mnist.py @@ -1,7 +1,7 @@ #!/usr/bin/env python3 # -*- coding: utf-8 -*- # -# Copyright (C) IBM Corporation 2018 +# Copyright (C) IBM Corporation 2019 # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -21,12 +21,12 @@ import torch from torchvision import datasets, transforms -from ptp.components.problems.problem import Problem +from ptp.components.tasks.task import Task from ptp.data_types.data_definition import DataDefinition -class MNIST(Problem): +class MNIST(Task): """ - Classic MNIST digit classification problem. + Classic MNIST digit classification task. Please see reference here: http://yann.lecun.com/exdb/mnist/ @@ -41,13 +41,13 @@ class MNIST(Problem): def __init__(self, name, config): """ - Initializes the MNIST problem. + Initializes the MNIST task. .. warning:: Resizing images might cause a significant slow down in batch generation. - :param name: Problem name. + :param name: Task name. :type name: str :param config: Dictionary of parameters (read from the configuration ``.yaml`` file). @@ -113,9 +113,9 @@ def __init__(self, name, config): def __len__(self): """ - Returns the "size" of the "problem" (total number of samples). + Returns the "size" of the "task" (total number of samples). - :return: The size of the problem. + :return: The size of the task. """ return len(self.dataset) @@ -140,7 +140,7 @@ def __getitem__(self, index): :param index: index of the sample to return. :type index: int - :return: ``DataDict({'images','targets'})``, with: + :return: ``DataStreams({'images','targets'})``, with: - images: Image, resized if ``self.resize`` is set, - targets: Index of the target class @@ -148,9 +148,9 @@ def __getitem__(self, index): # Get image and target. img, target = self.dataset.__getitem__(index) - # Return data_dict. - data_dict = self.create_data_dict(index) - data_dict[self.key_inputs] = img - data_dict[self.key_targets] = target - data_dict[self.key_labels] = self.ix_to_word[target.item()] - return data_dict + # Return data_streams. + data_streams = self.create_data_streams(index) + data_streams[self.key_inputs] = img + data_streams[self.key_targets] = target + data_streams[self.key_labels] = self.ix_to_word[target.item()] + return data_streams diff --git a/ptp/components/problems/problem.py b/ptp/components/tasks/task.py similarity index 78% rename from ptp/components/problems/problem.py rename to ptp/components/tasks/task.py index 81ba670..02d1da1 100644 --- a/ptp/components/problems/problem.py +++ b/ptp/components/tasks/task.py @@ -1,6 +1,6 @@ # -*- coding: utf-8 -*- # -# Copyright (C) IBM Corporation 2018-2019 +# Copyright (C) IBM Corporation 2019-2019 # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -20,14 +20,14 @@ from torch.utils.data import Dataset from ptp.components.component import Component -from ptp.data_types.data_dict import DataDict +from ptp.data_types.data_streams import DataStreams -class Problem(Component, Dataset): +class Task(Component, Dataset): """ - Class representing base class for all Problems. + Class representing base class for all Tasks. - Inherits from :py:class:`torch.utils.data.Dataset` as all subclasses will represent a problem with an associated dataset,\ + Inherits from :py:class:`torch.utils.data.Dataset` as all subclasses will represent a task with an associated dataset,\ and the `worker` will use :py:class:`torch.utils.data.DataLoader` to generate batches. Implements features & attributes used by all subclasses. @@ -36,7 +36,7 @@ class Problem(Component, Dataset): def __init__(self, name, class_type, config): """ - Initializes problem object: + Initializes task object: - calls base class constructors. - sets key_indices variable (used for storing indices of samples) @@ -46,7 +46,7 @@ def __init__(self, name, class_type, config): >>> self.curriculum_config = {} - :param name: Problem name. + :param name: Task name. :type name: str :param class_type: Class type of the component. @@ -56,7 +56,7 @@ def __init__(self, name, class_type, config): .. note:: - It is likely to encounter a case where the model needs a parameter value only known when the problem has been + It is likely to encounter a case where the model needs a parameter value only known when the task has been instantiated, like the size of a vocabulary set or the number of marker bits. The user can pass those values in this app_state. All objects will be able to access it later: @@ -77,9 +77,9 @@ def __init__(self, name, class_type, config): def summarize_io(self, priority = -1): """ - Summarizes the problem by showing its name, type and output definitions. + Summarizes the task by showing its name, type and output definitions. - :param priority: Problem priority (DEFAULT: -1) + :param priority: Task priority (DEFAULT: -1) :return: Summary as a str. @@ -91,49 +91,49 @@ def summarize_io(self, priority = -1): summary_str += ' {}: {}, {}, {}\n'.format(key, value.dimensions, value.types, value. description) return summary_str - def __call__(self, data_dict): + def __call__(self, data_streams): """ - Method responsible for processing the data dict. Empty for all problem-derived classes. + Method responsible for processing the data dict. Empty for all task-derived classes. - :param data_dict: :py:class:`ptp.utils.DataDict` object containing both input data to be proces and that will be extended by the results. + :param data_streams: :py:class:`ptp.utils.DataStreams` object containing both input data to be proces and that will be extended by the results. """ pass def input_data_definitions(self): """ Function returns a dictionary with definitions of input data that are required by the component. - As there is assumption made (problems do not accept inputs) it returns empty dictionary. + As there is assumption made (tasks do not accept inputs) it returns empty dictionary. :return: Empty dictionary. """ return {} - def create_data_dict(self, index, data_definitions = None): + def create_data_streams(self, index, data_definitions = None): """ - Returns a :py:class:`ptp.utils.DataDict` object with keys created on the \ - problem data_definitions and empty values (None). + Returns a :py:class:`ptp.utils.DataStreams` object with keys created on the \ + task data_definitions and empty values (None). :param data_definitions: Data definitions that will be used (DEFAULT: None, meaninng that self.output_data_definitions() will be used) - :return: new :py:class:`ptp.utils.DataDict` object. + :return: new :py:class:`ptp.utils.DataStreams` object. """ # Use self.output_data_definitions() if required data_definitions = data_definitions if data_definitions is not None else self.output_data_definitions() # Add index - just in case. This key is required! if self.key_indices not in data_definitions: data_definitions[self.key_indices] = None - data_dict = DataDict({key: None for key in data_definitions.keys()}) + data_streams = DataStreams({key: None for key in data_definitions.keys()}) # Set index. - data_dict[self.key_indices] = index - return data_dict + data_streams[self.key_indices] = index + return data_streams def collate_fn(self, batch): """ Generates a batch of samples from a list of individuals samples retrieved by :py:func:`__getitem__`. - The method calls :py:func:`torch.utils.data.dataloader.default_collate` for every item in data_dict ! + The method calls :py:func:`torch.utils.data.dataloader.default_collate` for every item in data_streams ! .. note:: @@ -146,14 +146,14 @@ def collate_fn(self, batch): override this default :py:func:`collate_fn`. - :param batch: List of :py:class:`ptp.utils.DataDict` retrieved by :py:func:`__getitem__`, each containing \ + :param batch: List of :py:class:`ptp.utils.DataStreams` retrieved by :py:func:`__getitem__`, each containing \ tensors, numbers, dicts or lists. :type batch: list - :return: DataDict containing the created batch. + :return: DataStreams containing the created batch. """ - return DataDict({key: torch.utils.data.dataloader.default_collate([sample[key] for sample in batch]) for key in batch[0]}) + return DataStreams({key: torch.utils.data.dataloader.default_collate([sample[key] for sample in batch]) for key in batch[0]}) def initialize_epoch(self, epoch): @@ -203,7 +203,7 @@ def curriculum_learning_initialize(self, curriculum_config): def curriculum_learning_update_params(self, episode, epoch): """ - Updates problem parameters according to curriculum learning. + Updates task parameters according to curriculum learning. .. note:: diff --git a/ptp/components/problems/text_to_class/__init__.py b/ptp/components/tasks/text_to_class/__init__.py similarity index 100% rename from ptp/components/problems/text_to_class/__init__.py rename to ptp/components/tasks/text_to_class/__init__.py diff --git a/ptp/components/problems/text_to_class/dummy_language_identification.py b/ptp/components/tasks/text_to_class/dummy_language_identification.py similarity index 94% rename from ptp/components/problems/text_to_class/dummy_language_identification.py rename to ptp/components/tasks/text_to_class/dummy_language_identification.py index 4840228..e06c63d 100644 --- a/ptp/components/problems/text_to_class/dummy_language_identification.py +++ b/ptp/components/tasks/text_to_class/dummy_language_identification.py @@ -18,19 +18,19 @@ import os -import ptp.components.utils.io as io +import ptp.components.mixins.io as io class DummyLanguageIdentification(LanguageIdentification): """ - Simple Language identification (classification) problem. + Simple Language identification (classification) task. Data taken from the _example. .. _example: https://pytorch.org/tutorials/beginner/nlp/deep_learning_tutorial.html """ def __init__(self, name, config): """ - Initializes the problem object. Calls base constructor and generates the files, if not present. + Initializes the task object. Calls base constructor and generates the files, if not present. :param name: Name of the component. diff --git a/ptp/components/problems/text_to_class/language_identification.py b/ptp/components/tasks/text_to_class/language_identification.py similarity index 77% rename from ptp/components/problems/text_to_class/language_identification.py rename to ptp/components/tasks/text_to_class/language_identification.py index 5bfdc90..e7f142b 100644 --- a/ptp/components/problems/text_to_class/language_identification.py +++ b/ptp/components/tasks/text_to_class/language_identification.py @@ -14,18 +14,18 @@ __author__ = "Tomasz Kornuta" -from ptp.components.problems.problem import Problem +from ptp.components.tasks.task import Task from ptp.data_types.data_definition import DataDefinition -class LanguageIdentification(Problem): +class LanguageIdentification(Task): """ - Language identification (classification) problem. + Language identification (classification) task. """ def __init__(self, name, class_type, config): """ - Initializes problem object. Calls base constructor. + Initializes task object. Calls base constructor. :param name: Name of the component. @@ -34,7 +34,7 @@ def __init__(self, name, class_type, config): :param config: Dictionary of parameters (read from configuration ``.yaml`` file). """ # Call constructors of parent classes. - Problem.__init__(self, name, class_type, config) + Task.__init__(self, name, class_type, config) # Set key mappings. self.key_inputs = self.stream_keys["inputs"] @@ -60,9 +60,9 @@ def output_data_definitions(self): def __len__(self): """ - Returns the "size" of the "problem" (total number of samples). + Returns the "size" of the "task" (total number of samples). - :return: The size of the problem. + :return: The size of the task. """ return len(self.inputs) @@ -74,11 +74,11 @@ def __getitem__(self, index): :param index: index of the sample to return. :type index: int - :return: ``DataDict({'inputs','targets'})`` + :return: ``DataStreams({'inputs','targets'})`` """ - # Return data_dict. - data_dict = self.create_data_dict(index) - data_dict[self.key_inputs] = self.inputs[index] - data_dict[self.key_targets] = self.targets[index] - return data_dict + # Return data_streams. + data_streams = self.create_data_streams(index) + data_streams[self.key_inputs] = self.inputs[index] + data_streams[self.key_targets] = self.targets[index] + return data_streams diff --git a/ptp/components/problems/text_to_class/wily_language_identification.py b/ptp/components/tasks/text_to_class/wily_language_identification.py similarity index 91% rename from ptp/components/problems/text_to_class/wily_language_identification.py rename to ptp/components/tasks/text_to_class/wily_language_identification.py index c6ebb2a..6ab56a5 100644 --- a/ptp/components/problems/text_to_class/wily_language_identification.py +++ b/ptp/components/tasks/text_to_class/wily_language_identification.py @@ -16,13 +16,13 @@ import os -import ptp.components.utils.io as io +import ptp.components.mixins.io as io from .language_identification import LanguageIdentification class WiLYLanguageIdentification(LanguageIdentification): """ - Language identification (classification) problem. + Language identification (classification) task. Using WiLI-2018 benchmark _dataset taken from the paper: Thoma, Martin. "The WiLI benchmark dataset for written language identification." arXiv preprint arXiv:1801.07779 (2018). (_arxiv) The dataset contains sentences from 235 languages. @@ -32,7 +32,7 @@ class WiLYLanguageIdentification(LanguageIdentification): """ def __init__(self, name, config): """ - Initializes problem object. Calls base constructor. Downloads the dataset if not present and loads the adequate files depending on the mode. + Initializes task object. Calls base constructor. Downloads the dataset if not present and loads the adequate files depending on the mode. :param name: Name of the component. diff --git a/ptp/components/problems/text_to_class/wily_ngram_language_modeling.py b/ptp/components/tasks/text_to_class/wily_ngram_language_modeling.py similarity index 86% rename from ptp/components/problems/text_to_class/wily_ngram_language_modeling.py rename to ptp/components/tasks/text_to_class/wily_ngram_language_modeling.py index 7ffb256..64c84e5 100644 --- a/ptp/components/problems/text_to_class/wily_ngram_language_modeling.py +++ b/ptp/components/tasks/text_to_class/wily_ngram_language_modeling.py @@ -16,14 +16,14 @@ import os -import ptp.components.utils.io as io -from ptp.components.problems.problem import Problem +import ptp.components.mixins.io as io +from ptp.components.tasks.task import Task from ptp.data_types.data_definition import DataDefinition -class WiLYNGramLanguageModeling(Problem): +class WiLYNGramLanguageModeling(Task): """ - N-gram Language Modeling problem. + N-gram Language Modeling task. By default it is using sentences from the WiLI benchmark _dataset taken from the paper: Thoma, Martin. "The WiLI benchmark dataset for written language identification." arXiv preprint arXiv:1801.07779 (2018). (_arxiv) .. _dataset: https://zenodo.org/record/841984 @@ -31,7 +31,7 @@ class WiLYNGramLanguageModeling(Problem): """ def __init__(self, name, config): """ - Initializes problem object. Calls base constructor. + Initializes task object. Calls base constructor. :param name: Name of the component. @@ -40,7 +40,7 @@ def __init__(self, name, config): :param config: Dictionary of parameters (read from configuration ``.yaml`` file). """ # Call constructors of parent classes. - Problem.__init__(self, name, WiLYNGramLanguageModeling, config) + Task.__init__(self, name, WiLYNGramLanguageModeling, config) # Set key mappings. self.key_inputs = self.stream_keys["inputs"] @@ -123,9 +123,9 @@ def output_data_definitions(self): def __len__(self): """ - Returns the "size" of the "problem" (total number of samples). + Returns the "size" of the "task" (total number of samples). - :return: The size of the problem. + :return: The size of the task. """ return len(self.ngrams) @@ -137,12 +137,12 @@ def __getitem__(self, index): :param index: index of the sample to return. :type index: int - :return: ``DataDict({'inputs','targets'})`` + :return: ``DataStreams({'inputs','targets'})`` """ - # Return data_dict. - data_dict = self.create_data_dict(index) - data_dict[self.key_inputs] = ' '.join(self.ngrams[index][:self.context]) - data_dict[self.key_targets] = self.ngrams[index][-1] # Last word - #print("problem: context = {} target = {}".format(data_dict[self.key_inputs], data_dict[self.key_targets])) - return data_dict + # Return data_streams. + data_streams = self.create_data_streams(index) + data_streams[self.key_inputs] = ' '.join(self.ngrams[index][:self.context]) + data_streams[self.key_targets] = self.ngrams[index][-1] # Last word + #print("task: context = {} target = {}".format(data_streams[self.key_inputs], data_streams[self.key_targets])) + return data_streams diff --git a/ptp/components/problems/text_to_text/__init__.py b/ptp/components/tasks/text_to_text/__init__.py similarity index 100% rename from ptp/components/problems/text_to_text/__init__.py rename to ptp/components/tasks/text_to_text/__init__.py diff --git a/ptp/components/problems/text_to_text/translation_pairs.py b/ptp/components/tasks/text_to_text/translation_pairs.py similarity index 87% rename from ptp/components/problems/text_to_text/translation_pairs.py rename to ptp/components/tasks/text_to_text/translation_pairs.py index 3f72b87..5663ca4 100644 --- a/ptp/components/problems/text_to_text/translation_pairs.py +++ b/ptp/components/tasks/text_to_text/translation_pairs.py @@ -22,13 +22,13 @@ from nltk.tokenize import WhitespaceTokenizer -import ptp.components.utils.io as io +import ptp.components.mixins.io as io from ptp.configuration import ConfigurationError -from ptp.components.problems.problem import Problem +from ptp.components.tasks.task import Task from ptp.data_types.data_definition import DataDefinition -class TranslationPairs(Problem): +class TranslationPairs(Task): """ Bilingual sentence pairs from http://www.manythings.org/anki/. Only some pairs are included here, but many more are available on the website. @@ -48,7 +48,7 @@ def __init__(self, name, config): :param config: Dictionary of parameters (read from configuration ``.yaml`` file). """ # Call constructor of parent classes. - Problem.__init__(self, name, TranslationPairs, config) + Task.__init__(self, name, TranslationPairs, config) # Set streams key mappings. self.key_sources = self.stream_keys["sources"] @@ -59,12 +59,12 @@ def __init__(self, name, config): # Get dataset. if (self.config['dataset'] is None) or (self.config['dataset'] not in ["eng-fra", "eng-pol"]): - raise ConfigurationError("Problem supports only 'dataset' options: 'eng-fra', 'eng-pol'") + raise ConfigurationError("Task supports only 'dataset' options: 'eng-fra', 'eng-pol'") dataset = self.config['dataset'] # Get (sub)set: train/valid/test. if (self.config['subset'] is None) or (self.config['subset'] not in ['train', 'valid', 'test']): - raise ConfigurationError("Problem supports one 'subset' options: 'train', 'valid', 'test' ") + raise ConfigurationError("Task supports one 'subset' options: 'train', 'valid', 'test' ") subset = self.config['subset'] # Extract source and target language name @@ -72,7 +72,7 @@ def __init__(self, name, config): self.lang_target = self.config['dataset'].split('-')[1] - # Names of files used by this problem. + # Names of files used by this task. filenames = [ self.lang_source + ".train.txt", self.lang_target + ".train.txt", @@ -189,9 +189,9 @@ def normalizeString(self, s): def __len__(self): """ - Returns the "size" of the "problem" (total number of samples). + Returns the "size" of the "task" (total number of samples). - :return: The size of the problem. + :return: The size of the task. """ return self.dataset_length @@ -203,29 +203,29 @@ def __getitem__(self, index): :param index: index of the sample to return. :type index: int - :return: ``DataDict({'indices', sources','targets'})`` + :return: ``DataStreams({'indices', sources','targets'})`` """ - # Return data_dict. - data_dict = self.create_data_dict(index) - data_dict[self.key_sources] = self.sentences_source[index] - data_dict[self.key_targets] = self.sentences_target[index] - return data_dict + # Return data_streams. + data_streams = self.create_data_streams(index) + data_streams[self.key_sources] = self.sentences_source[index] + data_streams[self.key_targets] = self.sentences_target[index] + return data_streams def collate_fn(self, batch): """ Generates a batch of samples from a list of individuals samples retrieved by :py:func:`__getitem__`. - :param batch: List of :py:class:`ptp.utils.DataDict` retrieved by :py:func:`__getitem__` + :param batch: List of :py:class:`ptp.utils.DataStreams` retrieved by :py:func:`__getitem__` :type batch: list - :return: DataDict containing the created batch. + :return: DataStreams containing the created batch. """ # Collate indices. - data_dict = self.create_data_dict([sample[self.key_indices] for sample in batch]) + data_streams = self.create_data_streams([sample[self.key_indices] for sample in batch]) # Collate sources. - data_dict[self.key_sources] = [sample[self.key_sources] for sample in batch] - data_dict[self.key_targets] = [sample[self.key_targets] for sample in batch] - return data_dict + data_streams[self.key_sources] = [sample[self.key_sources] for sample in batch] + data_streams[self.key_targets] = [sample[self.key_targets] for sample in batch] + return data_streams diff --git a/ptp/components/problems/text_to_text/wikitext_language_modeling.py b/ptp/components/tasks/text_to_text/wikitext_language_modeling.py similarity index 79% rename from ptp/components/problems/text_to_text/wikitext_language_modeling.py rename to ptp/components/tasks/text_to_text/wikitext_language_modeling.py index b71002d..5618b1a 100644 --- a/ptp/components/problems/text_to_text/wikitext_language_modeling.py +++ b/ptp/components/tasks/text_to_text/wikitext_language_modeling.py @@ -18,17 +18,17 @@ from nltk.tokenize import WhitespaceTokenizer -import ptp.components.utils.io as io +import ptp.components.mixins.io as io from ptp.configuration import ConfigurationError -from ptp.components.problems.problem import Problem +from ptp.components.tasks.task import Task from ptp.data_types.data_definition import DataDefinition -class WikiTextLanguageModeling(Problem): +class WikiTextLanguageModeling(Task): """ - Language modeling problem using WikiText-2 (_dataset2) / WikiText-103 (_dataset103) datasets, featured at the Salesforce _website. + Language modeling task using WikiText-2 (_dataset2) / WikiText-103 (_dataset103) datasets, featured at the Salesforce _website. - Problem downloads the files, loads the file associated with a given subset (train/valid/test), concatenates all sencentes and tokenizes them using NLTK's WhitespaceTokenizer. + Task downloads the files, loads the file associated with a given subset (train/valid/test), concatenates all sencentes and tokenizes them using NLTK's WhitespaceTokenizer. Resulting tokens are then passed to samples (source/target) as list of tokens of a given length (set by the user in configuration file). @@ -53,7 +53,7 @@ def __init__(self, name, config): :param config: Dictionary of parameters (read from configuration ``.yaml`` file). """ # Call constructor of parent classes. - Problem.__init__(self, name, WikiTextLanguageModeling, config) + Task.__init__(self, name, WikiTextLanguageModeling, config) # Set streams key mappings. self.key_sources = self.stream_keys["sources"] @@ -64,12 +64,12 @@ def __init__(self, name, config): # Get dataset. if (self.config['dataset'] is None) or (self.config['dataset'] not in ["wikitext-2", "wikitext-103"]): - raise ConfigurationError("Problem supports two 'dataset' options: 'wikitext-2', 'wikitext-103' ") + raise ConfigurationError("Task supports two 'dataset' options: 'wikitext-2', 'wikitext-103' ") dataset = self.config['dataset'] # Get (sub)set: train/valid/test. if (self.config['subset'] is None) or (self.config['subset'] not in ['train', 'valid', 'test']): - raise ConfigurationError("Problem supports three 'subset' options: 'train', 'valid', 'test' ") + raise ConfigurationError("Task supports three 'subset' options: 'train', 'valid', 'test' ") subset = self.config['subset'] # Check if file with tokenized words exists. @@ -78,7 +78,7 @@ def __init__(self, name, config): if not io.check_files_existence(self.data_folder, filename_tokenized_words): # If not, we must generate (and save it) using source files. - # Names of files used by this problem. + # Names of files used by this task. filenames = ["wiki.train.tokens", "wiki.valid.tokens", "wiki.test.tokens"] # Initialize dataset if files do not exist. @@ -143,9 +143,9 @@ def output_data_definitions(self): def __len__(self): """ - Returns the "size" of the "problem" (total number of samples). + Returns the "size" of the "task" (total number of samples). - :return: The size of the problem. + :return: The size of the task. """ return self.dataset_length @@ -157,30 +157,30 @@ def __getitem__(self, index): :param index: index of the sample to return. :type index: int - :return: ``DataDict({'indices', sources','targets'})`` + :return: ``DataStreams({'indices', sources','targets'})`` """ - # Return data_dict. - data_dict = self.create_data_dict(index) - data_dict[self.key_sources] = self.tokens[index:index+self.sentence_length] - data_dict[self.key_targets] = self.tokens[index+1:index+self.sentence_length+1] # target is "shifted" by 1. - #print("problem: index = {} source = {} target = {}".format(index, data_dict[self.key_sources], data_dict[self.key_targets])) - return data_dict + # Return data_streams. + data_streams = self.create_data_streams(index) + data_streams[self.key_sources] = self.tokens[index:index+self.sentence_length] + data_streams[self.key_targets] = self.tokens[index+1:index+self.sentence_length+1] # target is "shifted" by 1. + #print("task: index = {} source = {} target = {}".format(index, data_streams[self.key_sources], data_streams[self.key_targets])) + return data_streams def collate_fn(self, batch): """ Generates a batch of samples from a list of individuals samples retrieved by :py:func:`__getitem__`. - :param batch: List of :py:class:`ptp.utils.DataDict` retrieved by :py:func:`__getitem__` + :param batch: List of :py:class:`ptp.utils.DataStreams` retrieved by :py:func:`__getitem__` :type batch: list - :return: DataDict containing the created batch. + :return: DataStreams containing the created batch. """ # Collate indices. - data_dict = self.create_data_dict([sample[self.key_indices] for sample in batch]) + data_streams = self.create_data_streams([sample[self.key_indices] for sample in batch]) # Collate sources. - data_dict[self.key_sources] = [sample[self.key_sources] for sample in batch] - data_dict[self.key_targets] = [sample[self.key_targets] for sample in batch] - return data_dict + data_streams[self.key_sources] = [sample[self.key_sources] for sample in batch] + data_streams[self.key_targets] = [sample[self.key_targets] for sample in batch] + return data_streams diff --git a/ptp/components/transforms/__init__.py b/ptp/components/transforms/__init__.py index 6200892..6aa27c3 100644 --- a/ptp/components/transforms/__init__.py +++ b/ptp/components/transforms/__init__.py @@ -1,11 +1,11 @@ -from .concatenation import Concatenation +from .concatenate_tensor import ConcatenateTensor from .list_to_tensor import ListToTensor from .reduce_tensor import ReduceTensor from .reshape_tensor import ReshapeTensor __all__ = [ - 'Concatenation', + 'ConcatenateTensor', 'ListToTensor', 'ReduceTensor', 'ReshapeTensor', diff --git a/ptp/components/transforms/concatenation.py b/ptp/components/transforms/concatenate_tensor.py similarity index 88% rename from ptp/components/transforms/concatenation.py rename to ptp/components/transforms/concatenate_tensor.py index 03080a9..023d73e 100644 --- a/ptp/components/transforms/concatenation.py +++ b/ptp/components/transforms/concatenate_tensor.py @@ -22,7 +22,7 @@ from ptp.data_types.data_definition import DataDefinition -class Concatenation(Component): +class ConcatenateTensor(Component): """ Class responsible for concatenation of list of input tensors into a single tensor. @@ -40,7 +40,7 @@ def __init__(self, name, config): """ # Call constructors of parent classes. - Component.__init__(self, name, Concatenation, config) + Component.__init__(self, name, ConcatenateTensor, config) # Get key mappings. self.key_outputs = self.stream_keys["outputs"] @@ -86,19 +86,19 @@ def output_data_definitions(self): } - def __call__(self, data_dict): + def __call__(self, data_streams): """ Encodes "inputs" in the format of a single tensor. - Stores reshaped tensor in "outputs" field of in data_dict. + Stores reshaped tensor in "outputs" field of in data_streams. - :param data_dict: :py:class:`ptp.utils.DataDict` object containing (among others): + :param data_streams: :py:class:`ptp.utils.DataStreams` object containing (among others): - "inputs": expected input field containing tensor [BATCH_SIZE x ...] - "outputs": added output field containing tensor [BATCH_SIZE x ...] """ # Get inputs to be concatentated. - inputs = [data_dict[stream_key] for stream_key in self.input_stream_keys] + inputs = [data_streams[stream_key] for stream_key in self.input_stream_keys] #print("{}: input shape: {}, device: {}\n".format(self.name, [input.shape for input in inputs], [input.device for input in inputs])) @@ -107,5 +107,5 @@ def __call__(self, data_dict): outputs = torch.cat(inputs, dim=self.dim) # Create the returned dict. - data_dict.extend({self.key_outputs: outputs}) + data_streams.publish({self.key_outputs: outputs}) diff --git a/ptp/components/transforms/list_to_tensor.py b/ptp/components/transforms/list_to_tensor.py index 0f12fb5..d6bccab 100644 --- a/ptp/components/transforms/list_to_tensor.py +++ b/ptp/components/transforms/list_to_tensor.py @@ -81,19 +81,19 @@ def output_data_definitions(self): } - def __call__(self, data_dict): + def __call__(self, data_streams): """ Encodes "inputs" in the format of a single tensor. - Stores reshaped tensor in "outputs" field of in data_dict. + Stores reshaped tensor in "outputs" field of in data_streams. - :param data_dict: :py:class:`ptp.utils.DataDict` object containing (among others): + :param data_streams: :py:class:`ptp.utils.DataStreams` object containing (among others): - "inputs": expected input field containing tensor [BATCH_SIZE x ...] - "outputs": added output field containing tensor [BATCH_SIZE x ...] """ # Get inputs to be encoded. - inputs = data_dict[self.key_inputs] + inputs = data_streams[self.key_inputs] # Change to tensor. if self.num_inputs_dims == 1: @@ -133,5 +133,5 @@ def __call__(self, data_dict): outputs = torch.stack(stacked_tensor_lst) # Create the returned dict. - data_dict.extend({self.key_outputs: outputs}) + data_streams.publish({self.key_outputs: outputs}) diff --git a/ptp/components/transforms/reduce_tensor.py b/ptp/components/transforms/reduce_tensor.py index d36ee25..b688679 100644 --- a/ptp/components/transforms/reduce_tensor.py +++ b/ptp/components/transforms/reduce_tensor.py @@ -113,22 +113,22 @@ def output_data_definitions(self): } - def __call__(self, data_dict): + def __call__(self, data_streams): """ Encodes "inputs" in the format of a single tensor. - Stores reshaped tensor in "outputs" field of in data_dict. + Stores reshaped tensor in "outputs" field of in data_streams. - :param data_dict: :py:class:`ptp.utils.DataDict` object containing (among others): + :param data_streams: :py:class:`ptp.utils.DataStreams` object containing (among others): - "inputs": expected input field containing tensor [BATCH_SIZE x ...] - "outputs": added output field containing tensor [BATCH_SIZE x ...] """ # Get inputs to be encoded. - inputs = data_dict[self.key_inputs] + inputs = data_streams[self.key_inputs] outputs = self.reduction(inputs, self.dim, self.keepdim) # Create the returned dict. - data_dict.extend({self.key_outputs: outputs}) + data_streams.publish({self.key_outputs: outputs}) diff --git a/ptp/components/transforms/reshape_tensor.py b/ptp/components/transforms/reshape_tensor.py index 6a78dd4..acb894c 100644 --- a/ptp/components/transforms/reshape_tensor.py +++ b/ptp/components/transforms/reshape_tensor.py @@ -74,22 +74,22 @@ def output_data_definitions(self): } - def __call__(self, data_dict): + def __call__(self, data_streams): """ Encodes "inputs" in the format of a single tensor. - Stores reshaped tensor in "outputs" field of in data_dict. + Stores reshaped tensor in "outputs" field of in data_streams. - :param data_dict: :py:class:`ptp.utils.DataDict` object containing (among others): + :param data_streams: :py:class:`ptp.utils.DataStreams` object containing (among others): - "inputs": expected input field containing tensor [BATCH_SIZE x ...] - "outputs": added output field containing tensor [BATCH_SIZE x ...] """ # Get inputs to be encoded. - inputs = data_dict[self.key_inputs] + inputs = data_streams[self.key_inputs] # Reshape. outputs = inputs.view(self.output_dims) # Create the returned dict. - data_dict.extend({self.key_outputs: outputs}) + data_streams.publish({self.key_outputs: outputs}) diff --git a/ptp/components/utils/word_mappings.py b/ptp/components/utils/word_mappings.py deleted file mode 100644 index 2f7e3a5..0000000 --- a/ptp/components/utils/word_mappings.py +++ /dev/null @@ -1,162 +0,0 @@ -#!/usr/bin/env python3 -# -*- coding: utf-8 -*- -# -# Copyright (C) tkornuta, IBM Corporation 2019 -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -__author__ = "Tomasz Kornuta" - -import os -import csv - - -def generate_word_mappings_from_source_files(logger, folder, source_files): - """ - Load list of files (containing raw text) and creates (word:index) mappings from all words (tokens). - Indexing starts from 0. - - :param logger: Logger object. - - :param folder: Relative path to to the folder. - :type folder: str - - :param source_files: Source files (separated by commas) - - :return: Dictionary with (word:index) mappings - """ - # Check if there are any source files to load. - if len(source_files) == 0: - logger.error("Cannot create dictionary: list of vocabulary source files is empty, please provide comma separated list of files to be processed") - exit(-1) - - # Get absolute path. - folder = os.path.expanduser(folder) - - # Dictionary word_to_ix maps each word in the vocab to a unique integer. - word_to_ix = {} - - for filename in source_files.split(','): - # filename + path. - fn = folder+ '/' + filename - if not os.path.exists(fn): - logger.warning("Cannot load tokens files from '{}' because file does not exist".format(fn)) - continue - # File exists, try to parse. - content = open(fn).read() - # Parse tokens. - for word in content.split(): - # If new token. - if word not in word_to_ix: - word_to_ix[word] = len(word_to_ix) - - logger.info("Generated mappings of size {}".format(len(word_to_ix))) - return word_to_ix - - -def load_word_mappings_from_csv_file(logger, folder, filename): - """ - Loads (word:index) mappings from csv file. - - .. warning:: - There is an assumption that file will contain key:value pairs (no content checking for now!) - - :param logger: Logger object. - - :param folder: Relative path to to the folder. - :type folder: str - - :param filename: File with encodings (absolute path + filename). - - :return: dictionary with word:index keys - """ - file_path = os.path.join(os.path.expanduser(folder), filename) - - if not os.path.exists(file_path): - logger.warning("Cannot load word mappings from '{}' because the file does not exist".format(file_path)) - - with open(file_path, mode='rt') as csvfile: - # Check the presence of the header. - sniffer = csv.Sniffer() - first_bytes = str(csvfile.read(256)) - has_header = sniffer.has_header(first_bytes) - # Rewind. - csvfile.seek(0) - reader = csv.reader(csvfile) - # Skip the header row. - if has_header: - next(reader) - # Read the remaining rows. - word_to_ix = {rows[0]:int(rows[1]) for rows in reader} - - logger.info("Loaded mappings of size {}".format(len(word_to_ix))) - return word_to_ix - - -def save_word_mappings_to_csv_file(logger, folder, filename, word_to_ix, fieldnames = ["word","index"]): - """ - Saves (word:index) mappings dictionary to a file. - - :param logger: Logger object. - - :param folder: Relative path to to the folder. - :type folder: str - - :param filename: Name of file with encodings. - - :param word_to_ix: Dictionary with word:index mappings to be saved. - - """ - # Expand path. - folder = os.path.expanduser(folder) - # Make sure directory exists. - os.makedirs(os.path.dirname(folder +'/'), exist_ok=True) - - file_path = os.path.join(folder, filename) - - with open(file_path, mode='w+') as csvfile: - writer = csv.DictWriter(csvfile, fieldnames=fieldnames) - # Create header. - writer.writeheader() - - # Write word-index pairs. - for (k,v) in word_to_ix.items(): - #print("{} : {}".format(k,v)) - writer.writerow({fieldnames[0]:k, fieldnames[1]: v}) - - logger.info("Saved mappings of size {} to file '{}'".format(len(word_to_ix), file_path)) - -def pad_trunc_list(l: list, length: int, padding_value = 0, eos_value = None): - """ - Will apply padding / clipping to list to meet requested length. - Works on the list in-place. - - :param l: List to manipulate - - :param length: Target length - - :param padding_value: Value to fill when padding. Default is int(0). - - :return: None - """ - if len(l) < length: - if eos_value is not None: - l.append(eos_value) - l.extend([padding_value]*(length-len(l))) - - elif len(l) > length: - #print("pad_trunc_list to cat!: {}".format(len(l))) - #exit(1) - del l[length:] - if eos_value is not None: - l[length-1] = eos_value diff --git a/ptp/components/viewers/__init__.py b/ptp/components/viewers/__init__.py index 62bdca5..0d35164 100644 --- a/ptp/components/viewers/__init__.py +++ b/ptp/components/viewers/__init__.py @@ -1,7 +1,7 @@ -from .image_to_class_viewer import ImageToClassViewer +from .image_viewer import ImageViewer from .stream_viewer import StreamViewer __all__ = [ - 'ImageToClassViewer', + 'ImageViewer', 'StreamViewer', ] diff --git a/ptp/components/viewers/image_to_class_viewer.py b/ptp/components/viewers/image_viewer.py similarity index 91% rename from ptp/components/viewers/image_to_class_viewer.py rename to ptp/components/viewers/image_viewer.py index aacea46..b5f08f4 100644 --- a/ptp/components/viewers/image_to_class_viewer.py +++ b/ptp/components/viewers/image_viewer.py @@ -24,7 +24,7 @@ from ptp.data_types.data_definition import DataDefinition -class ImageToClassViewer(Component): +class ImageViewer(Component): """ Utility for displaying contents image along with label and prediction (a single sample from the batch). """ @@ -41,7 +41,7 @@ def __init__(self, name, config): """ # Call constructors of parent classes. - Component.__init__(self, name, ImageToClassViewer, config) + Component.__init__(self, name, ImageViewer, config) # Get default key mappings. self.key_indices = self.stream_keys["indices"] @@ -75,21 +75,21 @@ def output_data_definitions(self): return { } - def __call__(self, data_dict): + def __call__(self, data_streams): """ Shows a sample from the batch. - :param data_dict: :py:class:`ptp.utils.DataDict` object. + :param data_streams: :py:class:`ptp.utils.DataStreams` object. """ # Use worker interval. if self.app_state.episode % self.app_state.args.logging_interval == 0: # Get inputs - indices = data_dict[self.key_indices] - images = data_dict[self.key_images] - labels = data_dict[self.key_labels] - answers = data_dict[self.key_answers] + indices = data_streams[self.key_indices] + images = data_streams[self.key_images] + labels = data_streams[self.key_labels] + answers = data_streams[self.key_answers] # Get sample number. if self.sample_number == -1: diff --git a/ptp/components/viewers/stream_viewer.py b/ptp/components/viewers/stream_viewer.py index 7c538ba..e44fb4c 100644 --- a/ptp/components/viewers/stream_viewer.py +++ b/ptp/components/viewers/stream_viewer.py @@ -72,19 +72,19 @@ def output_data_definitions(self): return { } - def __call__(self, data_dict): + def __call__(self, data_streams): """ Encodes batch, or, in fact, only one field of batch ("inputs"). - Stores result in "outputs" field of data_dict. + Stores result in "outputs" field of data_streams. - :param data_dict: :py:class:`ptp.utils.DataDict` object containing (among others) "indices". + :param data_streams: :py:class:`ptp.utils.DataStreams` object containing (among others) "indices". """ # Use worker interval. if self.app_state.episode % self.app_state.args.logging_interval == 0: # Get indices. - indices = data_dict[self.key_indices] + indices = data_streams[self.key_indices] # Get sample number. if self.sample_number == -1: @@ -97,8 +97,8 @@ def __call__(self, data_dict): absent_streams = [] disp_str = "Showing selected streams for sample {} (index: {}):\n".format(sample_number, indices[sample_number]) for stream_key in self.input_stream_keys: - if stream_key in data_dict.keys(): - disp_str += " '{}': {}\n".format(stream_key, data_dict[stream_key][sample_number]) + if stream_key in data_streams.keys(): + disp_str += " '{}': {}\n".format(stream_key, data_streams[stream_key][sample_number]) else: absent_streams.append(stream_key) diff --git a/ptp/configuration/config_interface.py b/ptp/configuration/config_interface.py index e9d256d..fca103b 100644 --- a/ptp/configuration/config_interface.py +++ b/ptp/configuration/config_interface.py @@ -1,7 +1,7 @@ #!/usr/bin/env python3 # -*- coding: utf-8 -*- # -# Copyright (C) IBM Corporation 2018 +# Copyright (C) IBM Corporation 2019 # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -223,7 +223,7 @@ def add_default_params(self, default_params: dict): .. note:: This method should be used by the objects necessitating default values \ - (problems, models, workers etc.). + (tasks, models, workers etc.). :param default_params: Dictionary containing `default` values. :type default_params: dict diff --git a/ptp/configuration/config_parsing.py b/ptp/configuration/config_parsing.py index 6b4cced..2dac705 100644 --- a/ptp/configuration/config_parsing.py +++ b/ptp/configuration/config_parsing.py @@ -1,7 +1,7 @@ #!/usr/bin/env python3 # -*- coding: utf-8 -*- # -# Copyright (C) IBM Corporation 2018 +# Copyright (C) IBM Corporation 2019 # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/ptp/configuration/config_registry.py b/ptp/configuration/config_registry.py index 57c2e57..ae74630 100644 --- a/ptp/configuration/config_registry.py +++ b/ptp/configuration/config_registry.py @@ -1,7 +1,7 @@ #!/usr/bin/env python3 # -*- coding: utf-8 -*- # -# Copyright (C) IBM Corporation 2018 +# Copyright (C) IBM Corporation 2019 # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -36,7 +36,7 @@ class ConfigRegistry(Mapping, metaclass=MetaSingletonABC): """ Registry singleton for the parameters loaded from the configuration files. - Registers `default` values (coming from workers, models, problems, etc) as well as \ + Registers `default` values (coming from workers, models, tasks, etc) as well as \ `config` values loaded by the user for a particular experiment. Parameters can be read from the registry by indexing. @@ -102,7 +102,7 @@ def add_default_params(self, default_params: dict): .. note:: This method should be used by the objects necessitating default values \ - (problems, models, workers etc.). + (tasks, models, workers etc.). :param default_params: Dictionary containing default values. :type default_params: dict diff --git a/ptp/data_types/__init__.py b/ptp/data_types/__init__.py index 31f0ee2..466d65b 100644 --- a/ptp/data_types/__init__.py +++ b/ptp/data_types/__init__.py @@ -1,7 +1,7 @@ -from .data_dict import DataDict +from .data_streams import DataStreams from .data_definition import DataDefinition __all__ = [ - 'DataDict', + 'DataStreams', 'DataDefinition', ] diff --git a/ptp/data_types/data_dict.py b/ptp/data_types/data_streams.py similarity index 78% rename from ptp/data_types/data_dict.py rename to ptp/data_types/data_streams.py index b05778a..6a203f8 100644 --- a/ptp/data_types/data_dict.py +++ b/ptp/data_types/data_streams.py @@ -15,29 +15,29 @@ # See the License for the specific language governing permissions and # limitations under the License. -__author__ = "Vincent Marois, Tomasz Kornuta" +__author__ = "Tomasz Kornuta, Vincent Marois" import torch import collections -class DataDict(collections.abc.MutableMapping): +class DataStreams(collections.abc.MutableMapping): """ - Mapping: A container object that supports arbitrary key lookups and implements the methods ``__getitem__``, \ ``__iter__`` and ``__len__``. - Mutable objects can change their value but keep their id() -> ease modifying existing keys' value. - DataDict: Dict used for storing batches of data by problems. + DataStreams: Dict used for storing batches of data by tasks. - **This is the main object class used to share data between all components through a worker, starting from problem to loss and visualization.** + **This is the main object class used to share data between all components through a worker, starting from task to loss and visualization.** """ def __init__(self, *args, **kwargs): """ - DataDict constructor. Can be initialized in different ways: + DataStreams constructor. Can be initialized in different ways: - >>> data_dict = DataDict() - >>> data_dict = DataDict({'inputs': torch.tensor(), 'targets': numpy.ndarray()}) + >>> data_streams = DataStreams() + >>> data_streams = DataStreams({'inputs': torch.tensor(), 'targets': numpy.ndarray()}) >>> # etc. :param args: Used to pass a non-keyworded, variable-length argument list. @@ -62,41 +62,41 @@ def __setitem__(self, key, value, addkey=False): .. warning:: `addkey` is set to ``False`` by default as setting it to ``True`` removes the constraints of the\ - ``DataDict`` and enables it to become mutable. + ``DataStreams`` and enables it to become mutable. """ if not addkey and key not in self.keys(): - msg = 'Cannot modify a non-existing key "{}" in DataDict'.format(key) + msg = 'Cannot modify a non-existing key "{}" in DataStreams'.format(key) raise KeyError(msg) else: self.__dict__[key] = value - def extend(self, dict_to_add): + def publish(self, dict_to_add): """ - Extends a :py:class:`ptp.utils.DataDict` object by adding (keys,values) from data_definitions. + Publishes a new data streams - extends data stream object by adding (keys,values) from data_definitions. .. warning:: This is in-place operation, i.e. extends existing object, does not return a new one. - :param data_dict: :py:class:`ptp.utils.DataDict` object to be extended. + :param data_streams: :py:class:`ptp.utils.DataStreams` object to be extended. :param data_definitions: key-value pairs. """ for (key,value) in dict_to_add.items(): if key in self.keys(): - msg = "Cannot extend DataDict, as {} already present in its keys".format(key) + msg = "Cannot extend DataStreams, as {} already present in its keys".format(key) raise KeyError(msg) # Call setitem with "additional argument". self.__setitem__(key, value, addkey=True) - def reinitialize(self, dict_to_leave): + def reinitialize(self, streams_to_leave): """ - Removes all keys (and associated values) from DatDict EXCEPT the ones passed in ``dict_to_leave``. + Removes all streams (keys and associated values) from DataStreams EXCEPT the ones passed in ``streams_to_leave``. """ # Keys to remove. - rem_keys = [key for key in self.keys() if key not in dict_to_leave.keys()] + rem_keys = [key for key in self.keys() if key not in streams_to_leave.keys()] # Leave index. if 'index' in rem_keys: rem_keys.remove('index') @@ -133,7 +133,7 @@ def __delitem__(self, key, delkey=False): """ if not delkey: - msg = 'Cannot delete key "{}" from DataDict'.format(key) + msg = 'Cannot delete key "{}" from DataStreams'.format(key) raise KeyError(msg) else: del self.__dict__[key] @@ -146,7 +146,7 @@ def __len__(self): def __str__(self): """ - :return: A simple Dict representation of ``DataDict``. + :return: A simple Dict representation of ``DataStreams``. """ return str(self.__dict__) @@ -156,7 +156,7 @@ def __repr__(self): :return: Echoes class, id, & reproducible representation in the Read–Eval–Print Loop. """ - return '{}, DataDict({})'.format(super(DataDict, self).__repr__(), self.__dict__) + return '{}, DataStreams({})'.format(super(DataStreams, self).__repr__(), self.__dict__) def to(self, device=None, keys_to_move=None, non_blocking=False): diff --git a/ptp/utils/__init__.py b/ptp/utils/__init__.py index 14361fa..f3c98f3 100644 --- a/ptp/utils/__init__.py +++ b/ptp/utils/__init__.py @@ -1,5 +1,5 @@ from .app_state import AppState -from .data_dict_parallel import DataDictParallel +from .data_streams_parallel import DataStreamsParallel from .globals_facade import GlobalsFacade from .key_mappings_facade import KeyMappingsFacade from .samplers import kFoldRandomSampler @@ -12,7 +12,7 @@ __all__ = [ 'AppState', - 'DataDictParallel', + 'DataStreamsParallel', 'GlobalsFacade', 'KeyMappingsFacade', 'kFoldRandomSampler', diff --git a/ptp/utils/data_dict_parallel.py b/ptp/utils/data_streams_parallel.py similarity index 81% rename from ptp/utils/data_dict_parallel.py rename to ptp/utils/data_streams_parallel.py index b35a8c3..367933b 100644 --- a/ptp/utils/data_dict_parallel.py +++ b/ptp/utils/data_streams_parallel.py @@ -23,10 +23,10 @@ from torch.nn.parallel.parallel_apply import parallel_apply -from ptp.data_types.data_dict import DataDict +from ptp.data_types.data_streams import DataStreams -def datadict_scatter(inputs, target_gpus, dim=0): +def data_streams_scatter(inputs, target_gpus, dim=0): r""" Slices tensors into approximately equal chunks and distributes them across given GPUs. Duplicates @@ -41,7 +41,7 @@ def scatter_map(obj): return list(map(list, zip(*map(scatter_map, obj)))) if isinstance(obj, dict) and len(obj) > 0: return list(map(type(obj), zip(*map(scatter_map, obj.items())))) - if isinstance(obj, DataDict) and len(obj) > 0: + if isinstance(obj, DataStreams) and len(obj) > 0: return list(map(type(obj), zip(*map(scatter_map, obj.items())))) # Return "unscattered" object for all GPUs. # This seems to be the cause of the issue for SentenceEmbeddings! @@ -59,10 +59,10 @@ def scatter_map(obj): scatter_map = None -def datadict_scatter_kwargs(inputs, kwargs, target_gpus, dim=0): +def data_streams_scatter_kwargs(inputs, kwargs, target_gpus, dim=0): r"""Scatter with support for kwargs dictionary""" - inputs = datadict_scatter(inputs, target_gpus, dim) if inputs else [] - kwargs = datadict_scatter(kwargs, target_gpus, dim) if kwargs else [] + inputs = data_streams_scatter(inputs, target_gpus, dim) if inputs else [] + kwargs = data_streams_scatter(kwargs, target_gpus, dim) if kwargs else [] if len(inputs) < len(kwargs): inputs.extend([() for _ in range(len(kwargs) - len(inputs))]) elif len(kwargs) < len(inputs): @@ -72,7 +72,7 @@ def datadict_scatter_kwargs(inputs, kwargs, target_gpus, dim=0): return inputs, kwargs -def datadict_gather(outputs, target_device, dim=0): +def data_streams_gather(outputs, target_device, dim=0): r""" Gathers tensors from different GPUs on a specified device (-1 means the CPU). @@ -84,7 +84,7 @@ def gather_map(outputs): if out is None: return None - if isinstance(out, DataDict): + if isinstance(out, DataStreams): if not all((len(out) == len(d) for d in outputs)): raise ValueError('All dicts must have the same number of keys') return type(out)(((k, gather_map([d[k] for d in outputs])) @@ -106,23 +106,23 @@ def gather_map(outputs): gather_map = None -class DataDictParallel(torch.nn.DataParallel): +class DataStreamsParallel(torch.nn.DataParallel): """ - Modified DataParallel wrapper enabling operation on DataDicts. + Modified DataParallel wrapper enabling operation on DataStreamss. .. warning: Compatible with PyTorch v1.0.1 !! """ def __init__(self, module, device_ids=None, output_device=None, dim=0): - super(DataDictParallel, self).__init__(module, device_ids, output_device, dim) + super(DataStreamsParallel, self).__init__(module, device_ids, output_device, dim) def forward(self, *inputs, **kwargs): """ - Performs "parallelized forward" pass by scattering batch into several batches, distributing models on different GPUs, performing parallel pass and gathering results into a single (returned) DataDict. + Performs "parallelized forward" pass by scattering batch into several batches, distributing models on different GPUs, performing parallel pass and gathering results into a single (returned) DataStreams. ..warning: - As the "external" operations are changing inputs to tuple of DataDicts, extension of main DataDict must be done "outside" of this method. + As the "external" operations are changing inputs to tuple of DataStreamss, extension of main DataStreams must be done "outside" of this method. """ # Simple processing. @@ -135,7 +135,7 @@ def forward(self, *inputs, **kwargs): # Preprocessing: get only the inputs important for to the wrapped model (optimization). inputs_tuple = [] for i, item in enumerate(inputs): - input_dict = DataDict({key: value for key,value in item.items() if key in self.module.input_data_definitions().keys()}) + input_dict = DataStreams({key: value for key,value in item.items() if key in self.module.input_data_definitions().keys()}) inputs_tuple.append(input_dict) # Convert to tuple. inputs_tuple = tuple(inputs_tuple) @@ -152,7 +152,7 @@ def forward(self, *inputs, **kwargs): # Gather tuple. This cannot be done "in place"! gathered_tuple = self.gather(inputs_tuple, self.output_device) - # Return 0-th tuple, i.e. a single DataDict on device 0. + # Return 0-th tuple, i.e. a single DataStreams on device 0. return gathered_tuple[0] @@ -160,13 +160,13 @@ def replicate(self, module, device_ids): return replicate(module, device_ids) def scatter(self, inputs, kwargs, device_ids): - return datadict_scatter_kwargs(inputs, kwargs, device_ids, dim=self.dim) + return data_streams_scatter_kwargs(inputs, kwargs, device_ids, dim=self.dim) def parallel_apply(self, replicas, inputs, kwargs): return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)]) def gather(self, outputs, output_device): - return datadict_gather(outputs, output_device, dim=self.dim) + return data_streams_gather(outputs, output_device, dim=self.dim) def add_statistics(self, stat_col): """ @@ -177,16 +177,16 @@ def add_statistics(self, stat_col): self.module.add_statistics(stat_col) - def collect_statistics(self, stat_col, data_dict): + def collect_statistics(self, stat_col, data_streams): """ Collects statistics for the wrapped model. :param stat_col: :py:class:`ptp.utils.StatisticsCollector`. - :param data_dict: ``DataDict`` containing inputs, targets etc. - :type data_dict: :py:class:`ptp.core_types.DataDict` + :param data_streams: ``DataStreams`` containing inputs, targets etc. + :type data_streams: :py:class:`ptp.data_types.DataStreams` """ - self.module.collect_statistics(stat_col, data_dict) + self.module.collect_statistics(stat_col, data_streams) def add_aggregators(self, stat_agg): diff --git a/ptp/utils/statistics_aggregator.py b/ptp/utils/statistics_aggregator.py index f756472..1dd5f18 100644 --- a/ptp/utils/statistics_aggregator.py +++ b/ptp/utils/statistics_aggregator.py @@ -1,7 +1,7 @@ #!/usr/bin/env python3 # -*- coding: utf-8 -*- # -# Copyright (C) IBM Corporation 2018 +# Copyright (C) IBM Corporation 2019 # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -58,7 +58,7 @@ def add_aggregator(self, key, formatting): should be based on an existing statistics collected by the :py:class:`ptp.utils.StatisticsCollector` \ (e.g. added by :py:func:`StatisticsCollector.add_statistics()` and collected by \ :py:func:`ptp.models.Model.collect_statistics()` or \ - :py:func:`ptp.models.Problem.collect_statistics()`. + :py:func:`ptp.models.Task.collect_statistics()`. :type key: str :param formatting: Formatting that will be used when logging and exporting to CSV. diff --git a/ptp/utils/statistics_collector.py b/ptp/utils/statistics_collector.py index 5367b4e..e2e8a14 100644 --- a/ptp/utils/statistics_collector.py +++ b/ptp/utils/statistics_collector.py @@ -1,7 +1,7 @@ #!/usr/bin/env python3 # -*- coding: utf-8 -*- # -# Copyright (C) IBM Corporation 2018 +# Copyright (C) IBM Corporation 2019 # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/ptp/workers/offline_trainer.py b/ptp/workers/offline_trainer.py index 11a783e..2c7f893 100644 --- a/ptp/workers/offline_trainer.py +++ b/ptp/workers/offline_trainer.py @@ -52,7 +52,7 @@ def setup_experiment(self): - Sets up the terminal conditions (loss threshold, episodes & epochs (optional) limits). """ - # Call base method to parse all command line arguments, load configuration, create problems and model etc. + # Call base method to parse all command line arguments, load configuration, create tasks and model etc. super(OfflineTrainer, self).setup_experiment() # In this trainer Partial Validation is mandatory, hence interval must be > 0. @@ -163,12 +163,12 @@ def run_experiment(self): self.app_state.epoch += 1 self.logger.info('Starting next epoch: {}\n{}'.format(self.app_state.epoch, '='*80)) - # Inform the problem managers that epoch has started. + # Inform the task managers that epoch has started. self.training.initialize_epoch() self.validation.initialize_epoch() - # Apply curriculum learning - change Problem parameters. - self.curric_done = self.training.problem.curriculum_learning_update_params( + # Apply curriculum learning - change Task parameters. + self.curric_done = self.training.task.curriculum_learning_update_params( 0 if self.app_state.episode < 0 else self.app_state.episode, self.app_state.epoch) @@ -245,8 +245,8 @@ def run_experiment(self): # 6. Validate and (optionally) save the model. if self.partial_validation_interval > 0 and (self.app_state.episode % self.partial_validation_interval) == 0: - # Clear the validation batch from all items aside of the ones originally returned by the problem. - self.validation.batch.reinitialize(self.validation.problem.output_data_definitions()) + # Clear the validation batch from all items aside of the ones originally returned by the task. + self.validation.batch.reinitialize(self.validation.task.output_data_definitions()) # Perform validation. self.validate_on_batch(self.validation.batch) # Do not save the model: OfflineTrainer uses the full set to determine whether to save or not. @@ -269,7 +269,7 @@ def run_experiment(self): self.aggregate_all_statistics(self.training, self.pipeline, self.training_stat_col, self.training_stat_agg) self.export_all_statistics( self.training_stat_agg, '[Full Training]') - # Inform the training problem manager that the epoch has ended. + # Inform the training task manager that the epoch has ended. self.training.finalize_epoch() # Validate over the entire validation set. @@ -281,7 +281,7 @@ def run_experiment(self): # Save the pipeline using the latest validation statistics. self.pipeline.save(self.checkpoint_dir, training_status, validation_set_loss) - # Inform the validation problem manager that the epoch has ended. + # Inform the validation task manager that the epoch has ended. self.validation.finalize_epoch() # Terminal conditions. diff --git a/ptp/workers/online_trainer.py b/ptp/workers/online_trainer.py index 165c93e..5b09513 100644 --- a/ptp/workers/online_trainer.py +++ b/ptp/workers/online_trainer.py @@ -31,7 +31,7 @@ class OnlineTrainer(Trainer): ..note :: The ``OfflineTrainer`` is based on epochs. While an epoch can be defined for all finite-size datasets, \ - it makes less sense for problems which have a very large, almost infinite, dataset (like algorithmic \ + it makes less sense for tasks which have a very large, almost infinite, dataset (like algorithmic \ tasks, which generate random data on-the-fly). \ This is why this OnlineTrainer was implemented. Despite the fact it has the notion of epoch, it is more \ @@ -54,7 +54,7 @@ def setup_experiment(self): - Sets up the terminal conditions (loss threshold, episodes & epochs (optional) limits). """ - # Call base method to parse all command line arguments, load configuration, create problems and model etc. + # Call base method to parse all command line arguments, load configuration, create tasks and model etc. super(OnlineTrainer, self).setup_experiment() # In this trainer Partial Validation is mandatory, hence interval must be > 0. @@ -165,12 +165,12 @@ def run_experiment(self): self.app_state.epoch += 1 self.logger.info('Starting next epoch: {}\n{}'.format(self.app_state.epoch, '='*80)) - # Inform the problem managers that epoch has started. + # Inform the task managers that epoch has started. self.training.initialize_epoch() self.validation.initialize_epoch() - # Apply curriculum learning - change Problem parameters. - self.curric_done = self.training.problem.curriculum_learning_update_params( + # Apply curriculum learning - change Task parameters. + self.curric_done = self.training.task.curriculum_learning_update_params( 0 if self.app_state.episode < 0 else self.app_state.episode, self.app_state.epoch) @@ -248,8 +248,8 @@ def run_experiment(self): # 6. Validate and (optionally) save the model. if (self.app_state.episode % self.partial_validation_interval) == 0: - # Clear the validation batch from all items aside of the ones originally returned by the problem. - self.validation.batch.reinitialize(self.validation.problem.output_data_definitions()) + # Clear the validation batch from all items aside of the ones originally returned by the task. + self.validation.batch.reinitialize(self.validation.task.output_data_definitions()) # Perform validation. self.validate_on_batch(self.validation.batch) # Get loss. @@ -293,7 +293,7 @@ def run_experiment(self): # Epoch just ended! self.logger.info('End of epoch: {}\n{}'.format(self.app_state.epoch, '='*80)) - # Inform the problem managers that the epoch has ended. + # Inform the task managers that the epoch has ended. self.training.finalize_epoch() self.validation.finalize_epoch() @@ -317,8 +317,8 @@ def run_experiment(self): if self.validation_stat_col["episode"][-1] != self.app_state.episode: # We still must validate and try to save the model as it may performed better during this episode. - # Clear the validation batch from all items aside of the ones originally returned by the problem. - self.validation.batch.reinitialize(self.validation.problem.output_data_definitions()) + # Clear the validation batch from all items aside of the ones originally returned by the task. + self.validation.batch.reinitialize(self.validation.task.output_data_definitions()) # Perform validation. self.validate_on_batch(self.validation.batch) # Get loss. diff --git a/ptp/workers/processor.py b/ptp/workers/processor.py index 8d3bad4..2a201dd 100644 --- a/ptp/workers/processor.py +++ b/ptp/workers/processor.py @@ -1,7 +1,7 @@ #!/usr/bin/env python3 # -*- coding: utf-8 -*- # -# Copyright (C) IBM Corporation 2018 +# Copyright (C) IBM Corporation 2019 # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -27,7 +27,7 @@ from ptp.workers.worker import Worker -from ptp.application.problem_manager import ProblemManager +from ptp.application.task_manager import TaskManager from ptp.application.pipeline_manager import PipelineManager from ptp.utils.statistics_collector import StatisticsCollector @@ -144,7 +144,7 @@ def setup_individual_experiment(self): - Creates the pipeline consisting of many components - - Creates testing problem manager + - Creates testing task manager - Performs testing of compatibility of testing pipeline @@ -160,11 +160,11 @@ def setup_individual_experiment(self): print("Error: Couldn't retrieve the section '{}' from the loaded configuration".format(self.tsn)) exit(-1) - # Get testing problem type. + # Get testing task type. try: - _ = self.config_test['problem']['type'] + _ = self.config_test['task']['type'] except KeyError: - print("Error: Couldn't retrieve the problem 'type' from the '{}' section in the loaded configuration".format(self.tsn)) + print("Error: Couldn't retrieve the task 'type' from the '{}' section in the loaded configuration".format(self.tsn)) exit(-5) # Get pipeline section. @@ -219,25 +219,25 @@ def setup_individual_experiment(self): ################# TESTING PROBLEM ################# - # Build the used problem manager. - self.pm = ProblemManager(self.tsn, self.config_test) + # Build the used task manager. + self.pm = TaskManager(self.tsn, self.config_test) errors += self.pm.build() # check if the maximum number of episodes is specified, if not put a # default equal to the size of the dataset (divided by the batch size) # So that by default, we loop over the test set once. - problem_size_in_episodes = len(self.pm) + task_size_in_episodes = len(self.pm) if self.config_test["terminal_conditions"]["episode_limit"] == -1: # Overwrite the config value! - self.config_test['terminal_conditions'].add_config_params({'episode_limit': problem_size_in_episodes}) + self.config_test['terminal_conditions'].add_config_params({'episode_limit': task_size_in_episodes}) # Warn if indicated number of episodes is larger than an epoch size: - if self.config_test["terminal_conditions"]["episode_limit"] > problem_size_in_episodes: + if self.config_test["terminal_conditions"]["episode_limit"] > task_size_in_episodes: self.logger.warning('Indicated limit of number of episodes is larger than one epoch, reducing it.') # Overwrite the config value! - self.config_test['terminal_conditions'].add_config_params({'episode_limit': problem_size_in_episodes}) + self.config_test['terminal_conditions'].add_config_params({'episode_limit': task_size_in_episodes}) self.logger.info("Limiting the number of episodes to: {}".format( self.config_test["terminal_conditions"]["episode_limit"])) @@ -250,7 +250,7 @@ def setup_individual_experiment(self): # Show pipeline. summary_str = self.pipeline.summarize_all_components_header() - summary_str += self.pm.problem.summarize_io(self.tsn) + summary_str += self.pm.task.summarize_io(self.tsn) summary_str += self.pipeline.summarize_all_components() self.logger.info(summary_str) @@ -261,7 +261,7 @@ def setup_individual_experiment(self): # Handshake definitions. self.logger.info("Handshaking testing pipeline") - defs_testing = self.pm.problem.output_data_definitions() + defs_testing = self.pm.task.output_data_definitions() errors += self.pipeline.handshake(defs_testing) # Check errors. @@ -332,7 +332,7 @@ def initialize_statistics_collection(self): # Create statistics collector. self.stat_col = StatisticsCollector() self.add_statistics(self.stat_col) - self.pm.problem.add_statistics(self.stat_col) + self.pm.task.add_statistics(self.stat_col) self.pipeline.add_statistics(self.stat_col) # Create the csv file to store the statistics. self.pm_batch_stats_file = self.stat_col.initialize_csv_file(self.app_state.log_dir, self.tsn+'_statistics.csv') @@ -340,7 +340,7 @@ def initialize_statistics_collection(self): # Create statistics aggregator. self.stat_agg = StatisticsAggregator() self.add_aggregators(self.stat_agg) - self.pm.problem.add_aggregators(self.stat_agg) + self.pm.task.add_aggregators(self.stat_agg) self.pipeline.add_aggregators(self.stat_agg) # Create the csv file to store the statistic aggregations. # Will contain a single row with aggregated statistics. @@ -381,7 +381,7 @@ def run_experiment(self): # Reset the counter. self.app_state.episode = -1 - # Inform the problem manager that epoch has started. + # Inform the task manager that epoch has started. self.pm.initialize_epoch() for batch in self.pm.dataloader: @@ -407,7 +407,7 @@ def run_experiment(self): self.app_state.episode += 1 # End for. - # Inform the problem managers that the epoch has ended. + # Inform the task managers that the epoch has ended. self.pm.finalize_epoch() self.logger.info('\n' + '='*80) diff --git a/ptp/workers/test_data_dict_parallel.py b/ptp/workers/test_data_dict_parallel.py index d55a8ce..b41c985 100644 --- a/ptp/workers/test_data_dict_parallel.py +++ b/ptp/workers/test_data_dict_parallel.py @@ -5,15 +5,15 @@ import time -from ptp.components.problems.problem import Problem +from ptp.components.tasks.task import Task from ptp.components.models.model import Model -from ptp.data_types.data_dict import DataDict +from ptp.data_types.data_streams import DataStreams from ptp.data_types.data_definition import DataDefinition -from ptp.utils.data_dict_parallel import DataDictParallel +from ptp.utils.data_streams_parallel import DataStreamsParallel -class RandomDataset(Problem): +class RandomDataset(Task): def __init__(self, size, length): self.len = length @@ -21,11 +21,11 @@ def __init__(self, size, length): def __getitem__(self, index): - # Return data_dict. - data_dict = DataDict({"index": None}) - data_dict["index"] = self.data[index] + # Return data_streams. + data_streams = DataStreams({"index": None}) + data_streams["index"] = self.data[index] - return data_dict + return data_streams #return self.data[index] @@ -37,7 +37,7 @@ def output_data_definitions(self): def collate_fn(self, batch): print("Collate!") - return DataDict({key: torch.utils.data.dataloader.default_collate([sample[key] for sample in batch]) for key in batch[0]}) + return DataStreams({key: torch.utils.data.dataloader.default_collate([sample[key] for sample in batch]) for key in batch[0]}) class TestModel1(Model): @@ -114,8 +114,8 @@ def output_data_definitions(self): if torch.cuda.device_count() > 1: print("Let's use", torch.cuda.device_count(), "GPUs!") # dim = 0 [30, xxx] -> [10, ...], [10, ...], [10, ...] on 3 GPUs - model1 = DataDictParallel(model1) - model2 = DataDictParallel(model2) + model1 = DataStreamsParallel(model1) + model2 = DataStreamsParallel(model2) use_dataparallel = True # Move to desired device. model1.to(device) @@ -123,7 +123,7 @@ def output_data_definitions(self): print("DataParallel DONE!!") #time.sleep(2) - #datadict1 = {}#DataDict({"index":None,"output":None}) + #datadict1 = {}#DataStreams({"index":None,"output":None}) for datadict in rand_loader: print("!!!!! Got object from loader: {}".format(type(datadict))) datadict.to(device) diff --git a/ptp/workers/trainer.py b/ptp/workers/trainer.py index f690698..038543d 100644 --- a/ptp/workers/trainer.py +++ b/ptp/workers/trainer.py @@ -1,7 +1,7 @@ #!/usr/bin/env python3 # -*- coding: utf-8 -*- # -# Copyright (C) IBM Corporation 2018 +# Copyright (C) IBM Corporation 2019 # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -28,7 +28,7 @@ from ptp.workers.worker import Worker -from ptp.application.problem_manager import ProblemManager +from ptp.application.task_manager import TaskManager from ptp.application.pipeline_manager import PipelineManager from ptp.utils.statistics_collector import StatisticsCollector @@ -110,15 +110,15 @@ def setup_experiment(self): - Creates the pipeline consisting of many components - - Creates training problem manager + - Creates training task manager - Handles curriculum learning if indicated - - Creates validation problem manager + - Creates validation task manager - Set optimizer - - Performs testing of compatibility of both training and validation problems and created pipeline. + - Performs testing of compatibility of both training and validation tasks and created pipeline. """ # Call base method to parse all command line arguments and add default sections. @@ -173,11 +173,11 @@ def setup_experiment(self): print("Error: Couldn't retrieve the training section '{}' from the loaded configuration".format(tsn)) exit(-1) - # Get training problem type. + # Get training task type. try: - training_problem_type = self.config_training['problem']['type'] + training_task_type = self.config_training['task']['type'] except KeyError: - print("Error: Couldn't retrieve the problem 'type' from the training section '{}' in the loaded configuration".format(tsn)) + print("Error: Couldn't retrieve the task 'type' from the training section '{}' in the loaded configuration".format(tsn)) exit(-1) # Get validation section. @@ -190,11 +190,11 @@ def setup_experiment(self): print("Error: Couldn't retrieve the validation section '{}' from the loaded configuration".format(vsn)) exit(-1) - # Get validation problem type. + # Get validation task type. try: - _ = self.config_validation['problem']['type'] + _ = self.config_validation['task']['type'] except KeyError: - print("Error: Couldn't retrieve the problem 'type' from the validation section '{}' in the loaded configuration".format(vsn)) + print("Error: Couldn't retrieve the task 'type' from the validation section '{}' in the loaded configuration".format(vsn)) exit(-1) # Get pipeline section. @@ -224,7 +224,7 @@ def setup_experiment(self): time_str = '{0:%Y%m%d_%H%M%S}'.format(datetime.now()) if self.app_state.args.exptag != '': time_str = time_str + "_" + self.app_state.args.exptag - self.app_state.log_dir = path.expanduser(self.app_state.args.expdir) + '/' + training_problem_type + '/' + pipeline_name + '/' + time_str + '/' + self.app_state.log_dir = path.expanduser(self.app_state.args.expdir) + '/' + training_task_type + '/' + pipeline_name + '/' + time_str + '/' # Lowercase dir. self.app_state.log_dir = self.app_state.log_dir.lower() makedirs(self.app_state.log_dir, exist_ok=False) @@ -256,15 +256,15 @@ def setup_experiment(self): ################# TRAINING PROBLEM ################# - # Build training problem manager. - self.training = ProblemManager('training', self.config_training) + # Build training task manager. + self.training = TaskManager('training', self.config_training) errors += self.training.build() # parse the curriculum learning section in the loaded configuration. if 'curriculum_learning' in self.config_training: # Initialize curriculum learning - with values from loaded configuration. - self.training.problem.curriculum_learning_initialize(self.config_training['curriculum_learning']) + self.training.task.curriculum_learning_initialize(self.config_training['curriculum_learning']) # If the 'must_finish' key is not present in config then then it will be finished by default self.config_training['curriculum_learning'].add_default_params({'must_finish': True}) @@ -279,8 +279,8 @@ def setup_experiment(self): ################# VALIDATION PROBLEM ################# - # Build validation problem manager. - self.validation = ProblemManager('validation', self.config_validation) + # Build validation task manager. + self.validation = TaskManager('validation', self.config_validation) errors += self.validation.build() ###################### PIPELINE ###################### @@ -296,18 +296,18 @@ def setup_experiment(self): # Show pipeline. summary_str = self.pipeline.summarize_all_components_header() - summary_str += self.training.problem.summarize_io("training") - summary_str += self.validation.problem.summarize_io("validation") + summary_str += self.training.task.summarize_io("training") + summary_str += self.validation.task.summarize_io("validation") summary_str += self.pipeline.summarize_all_components() self.logger.info(summary_str) # Handshake definitions. self.logger.info("Handshaking training pipeline") - defs_training = self.training.problem.output_data_definitions() + defs_training = self.training.task.output_data_definitions() errors += self.pipeline.handshake(defs_training) self.logger.info("Handshaking validation pipeline") - defs_valid = self.validation.problem.output_data_definitions() + defs_valid = self.validation.task.output_data_definitions() errors += self.pipeline.handshake(defs_valid) # Check errors. @@ -414,8 +414,8 @@ def initialize_statistics_collection(self): """ - Initializes all ``StatisticsCollectors`` and ``StatisticsAggregators`` used by a given worker: \ - - For training statistics (adds the statistics of the model & problem), - - For validation statistics (adds the statistics of the model & problem). + - For training statistics (adds the statistics of the model & task), + - For validation statistics (adds the statistics of the model & task). - Creates the output files (csv). @@ -424,7 +424,7 @@ def initialize_statistics_collection(self): # Create statistics collector for training. self.training_stat_col = StatisticsCollector() self.add_statistics(self.training_stat_col) - self.training.problem.add_statistics(self.training_stat_col) + self.training.task.add_statistics(self.training_stat_col) self.pipeline.add_statistics(self.training_stat_col) # Create the csv file to store the training statistics. self.training_batch_stats_file = self.training_stat_col.initialize_csv_file(self.app_state.log_dir, 'training_statistics.csv') @@ -432,7 +432,7 @@ def initialize_statistics_collection(self): # Create statistics aggregator for training. self.training_stat_agg = StatisticsAggregator() self.add_aggregators(self.training_stat_agg) - self.training.problem.add_aggregators(self.training_stat_agg) + self.training.task.add_aggregators(self.training_stat_agg) self.pipeline.add_aggregators(self.training_stat_agg) # Create the csv file to store the training statistic aggregations. self.training_set_stats_file = self.training_stat_agg.initialize_csv_file(self.app_state.log_dir, 'training_set_agg_statistics.csv') @@ -441,7 +441,7 @@ def initialize_statistics_collection(self): # Create statistics collector for validation. self.validation_stat_col = StatisticsCollector() self.add_statistics(self.validation_stat_col) - self.validation.problem.add_statistics(self.validation_stat_col) + self.validation.task.add_statistics(self.validation_stat_col) self.pipeline.add_statistics(self.validation_stat_col) # Create the csv file to store the validation statistics. self.validation_batch_stats_file = self.validation_stat_col.initialize_csv_file(self.app_state.log_dir, 'validation_statistics.csv') @@ -449,7 +449,7 @@ def initialize_statistics_collection(self): # Create statistics aggregator for validation. self.validation_stat_agg = StatisticsAggregator() self.add_aggregators(self.validation_stat_agg) - self.validation.problem.add_aggregators(self.validation_stat_agg) + self.validation.task.add_aggregators(self.validation_stat_agg) self.pipeline.add_aggregators(self.validation_stat_agg) # Create the csv file to store the validation statistic aggregations. self.validation_set_stats_file = self.validation_stat_agg.initialize_csv_file(self.app_state.log_dir, 'validation_set_agg_statistics.csv') @@ -512,8 +512,8 @@ def validate_on_batch(self, valid_batch): Additionally logs results (to files, TensorBoard) and handles visualization. - :param valid_batch: data batch generated by the problem and used as input to the model. - :type valid_batch: ``DataDict`` + :param valid_batch: data batch generated by the task and used as input to the model. + :type valid_batch: ``DataStreams`` :return: Validation loss. diff --git a/ptp/workers/worker.py b/ptp/workers/worker.py index 958c5d7..cd0dd3d 100644 --- a/ptp/workers/worker.py +++ b/ptp/workers/worker.py @@ -1,7 +1,7 @@ #!/usr/bin/env python3 # -*- coding: utf-8 -*- # -# Copyright (C) IBM Corporation 2018 +# Copyright (C) IBM Corporation 2019 # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -219,17 +219,17 @@ def run_experiment(self): """ - def collect_all_statistics(self, problem_mgr, pipeline_mgr, data_dict, stat_col): + def collect_all_statistics(self, task_mgr, pipeline_mgr, data_streams, stat_col): """ Function that collects statistics - :param pipeline: Pipeline containing both problem and list of components. + :param pipeline: Pipeline containing both task and list of components. :type pipeline: ``configuration.pipeline.Pipeline`` - :param problem_mgr: Problem manager. + :param task_mgr: Task manager. - :param data_dict: contains the batch of samples to pass through the pipeline. - :type data_dict: ``DataDict`` + :param data_streams: contains the batch of samples to pass through the pipeline. + :type data_streams: ``DataStreams`` :param stat_col: statistics collector used for logging accuracy etc. :type stat_col: ``StatisticsCollector`` @@ -241,20 +241,20 @@ def collect_all_statistics(self, problem_mgr, pipeline_mgr, data_dict, stat_col) stat_col['epoch'] = self.app_state.epoch # Collect rest of statistics. - problem_mgr.problem.collect_statistics(stat_col, data_dict) - pipeline_mgr.collect_statistics(stat_col, data_dict) + task_mgr.task.collect_statistics(stat_col, data_streams) + pipeline_mgr.collect_statistics(stat_col, data_streams) - def aggregate_all_statistics(self, problem_mgr, pipeline_mgr, stat_col, stat_agg): + def aggregate_all_statistics(self, task_mgr, pipeline_mgr, stat_col, stat_agg): """ Aggregates the collected statistics. Exports the aggregations to logger, csv and TB. \ Empties statistics collector for the next episode. - :param pipeline: Pipeline containing both problem and list of components. + :param pipeline: Pipeline containing both task and list of components. :type pipeline: ``configuration.pipeline.Pipeline`` - :param problem_mgr: Problem manager. + :param task_mgr: Task manager. :param stat_col: ``StatisticsCollector`` object. @@ -266,7 +266,7 @@ def aggregate_all_statistics(self, problem_mgr, pipeline_mgr, stat_col, stat_agg stat_agg.aggregators['episode'] = self.app_state.episode stat_agg.aggregators['episodes_aggregated'] = len(stat_col['episode']) # Aggregate rest of statistics. - problem_mgr.problem.aggregate_statistics(stat_col, stat_agg) + task_mgr.task.aggregate_statistics(stat_col, stat_agg) pipeline_mgr.aggregate_statistics(stat_col, stat_agg) diff --git a/setup.py b/setup.py index f37e232..5148d36 100644 --- a/setup.py +++ b/setup.py @@ -1,4 +1,4 @@ -# Copyright (C) IBM Corporation 2018-2019 +# Copyright (C) IBM Corporation 2019-2019 # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -136,7 +136,7 @@ # project page. What does your project relate to? # # Note that this is a string of words separated by whitespace, not a list. - keywords='machine learning neural nets pytorch pipeline component problem model', # Optional + keywords='machine learning neural nets pytorch pipeline component task model', # Optional # You can just specify package directories manually here if your project is # simple. Or you can use find_packages(). diff --git a/tests/__init__.py b/tests/__init__.py index 9471702..235448b 100644 --- a/tests/__init__.py +++ b/tests/__init__.py @@ -3,15 +3,15 @@ from .application.samplers_tests import TestkFoldRandomSampler, TestkFoldWeightedRandomSampler from .components.component_tests import TestComponent -from .components.problems.clevr_tests import TestCLEVR -from .components.problems.gqa_tests import TestGQA -from .components.problems.problem_tests import TestProblem +from .components.tasks.clevr_tests import TestCLEVR +from .components.tasks.gqa_tests import TestGQA +from .components.tasks.task_tests import TestTask from .configuration.config_interface_tests import TestConfigInterface from .configuration.config_registry_tests import TestConfigRegistry from .configuration.handshaking_tests import TestHandshaking -from .data_types.data_dict_tests import TestDataDict +from .data_types.data_streams_tests import TestDataStreams from .data_types.data_definition_tests import TestDataDefinition from .utils.app_state_tests import TestAppState @@ -26,13 +26,13 @@ # Components 'TestComponent', 'TestGQA', - 'TestProblem', + 'TestTask', # Configuration 'TestConfigRegistry', 'TestConfigInterface', 'TestHandshaking', # DataTypes - 'TestDataDict', + 'TestDataStreams', 'TestDataDefinition', # Utils 'TestAppState', diff --git a/tests/application/pipeline_tests.py b/tests/application/pipeline_tests.py index 42c89f1..42cccd0 100644 --- a/tests/application/pipeline_tests.py +++ b/tests/application/pipeline_tests.py @@ -40,7 +40,7 @@ def test_create_component_full_type(self): config.add_default_params({ 'bow_encoder' : { - 'type': 'ptp.components.text.bow_encoder.BOWEncoder', + 'type': 'ptp.components.language.bow_encoder.BOWEncoder', 'priority': 1.1 } }) diff --git a/tests/application/sampler_factory_tests.py b/tests/application/sampler_factory_tests.py index 7a79bd3..69829c2 100644 --- a/tests/application/sampler_factory_tests.py +++ b/tests/application/sampler_factory_tests.py @@ -24,8 +24,8 @@ from ptp.configuration.config_interface import ConfigInterface from ptp.application.sampler_factory import SamplerFactory -# Problem. -class TestProblemMockup(object): +# Task. +class TestTaskMockup(object): def __len__(self): return 50 @@ -42,7 +42,7 @@ def test_create_subset_random_sampler_range(self): config.add_default_params({'type': 'SubsetRandomSampler', 'indices': indices}) # Create the sampler. - sampler = SamplerFactory.build(TestProblemMockup(), config, "training") + sampler = SamplerFactory.build(TestTaskMockup(), config, "training") # Check number of samples. self.assertEqual(len(sampler), 20) @@ -55,7 +55,7 @@ def test_create_subset_random_sampler_range_str(self): config.add_default_params({'type': 'SubsetRandomSampler', 'indices': range_str}) # Create the sampler. - sampler = SamplerFactory.build(TestProblemMockup(), config, "training") + sampler = SamplerFactory.build(TestTaskMockup(), config, "training") # Check number of samples. self.assertEqual(len(sampler), 20) @@ -69,7 +69,7 @@ def test_create_subset_random_sampler_list_of_indices(self): config.add_default_params({'type': 'SubsetRandomSampler', 'indices': yaml_list}) # Create the sampler. - sampler = SamplerFactory.build(TestProblemMockup(), config, "training") + sampler = SamplerFactory.build(TestTaskMockup(), config, "training") # Check number of samples. self.assertEqual(len(sampler), 4) @@ -88,7 +88,7 @@ def test_create_subset_random_sampler_file(self): config.add_default_params({'type': 'SubsetRandomSampler', 'indices': filename}) # Create the sampler. - sampler = SamplerFactory.build(TestProblemMockup(), config, "training") + sampler = SamplerFactory.build(TestTaskMockup(), config, "training") # Check number of samples. self.assertEqual(len(sampler), 5) diff --git a/tests/components/component_tests.py b/tests/components/component_tests.py index 9aa48a4..48a9d98 100644 --- a/tests/components/component_tests.py +++ b/tests/components/component_tests.py @@ -19,16 +19,16 @@ import unittest from ptp.components.component import Component -from ptp.components.problems.problem import Problem +from ptp.components.tasks.task import Task from ptp.data_types.data_definition import DataDefinition from ptp.configuration.config_interface import ConfigInterface -class MockupProblem (Problem): +class MockupTask (Task): """ - Mockup problem class. + Mockup task class. """ def __init__(self, name, config): - Problem.__init__(self, name, None, config) + Task.__init__(self, name, None, config) def output_data_definitions(self): return { @@ -51,33 +51,33 @@ def __init__(self, *args, **kwargs): # Overwrite abc abstract methods. MockupComponent.__abstractmethods__=set() - MockupProblem.__abstractmethods__=set() + MockupTask.__abstractmethods__=set() # Create mocked-up component. config = ConfigInterface() - self.problem = MockupProblem("test_problem", config) + self.task = MockupTask("test_task", config) self.component = MockupComponent("test_component", config) - def test_create_data_dict_key_present(self): + def test_create_data_streams_key_present(self): """ Tests whether the created data dict contains required keys. """ - data_dict = self.problem.create_data_dict(1) + data_streams = self.task.create_data_streams(1) # Check presence of index. - self.assertEqual(data_dict['indices'], 1) # Even if we didn't explicitly indicated that in definitions! - self.assertEqual(data_dict['inputs'], None) + self.assertEqual(data_streams['indices'], 1) # Even if we didn't explicitly indicated that in definitions! + self.assertEqual(data_streams['inputs'], None) # And targets is not present (yet)... with self.assertRaises(KeyError): - data_dict['targets'] + data_streams['targets'] - def test_extend_data_dict_key_present(self): + def test_extend_data_streams_key_present(self): """ Tests whether the created data dict contains required keys. """ - data_dict = self.problem.create_data_dict(1) - # Extend data_dict. - data_dict.extend({"targets": 3}) + data_streams = self.task.create_data_streams(1) + # Extend data_streams. + data_streams.publish({"targets": 3}) # Check presence of all "streams". - self.assertEqual(data_dict['indices'], 1) # Even if we didn't explicitly indicated that in definitions! - self.assertEqual(data_dict['inputs'], None) - self.assertEqual(data_dict['targets'], 3) + self.assertEqual(data_streams['indices'], 1) # Even if we didn't explicitly indicated that in definitions! + self.assertEqual(data_streams['inputs'], None) + self.assertEqual(data_streams['targets'], 3) def test_global_set_get(self): """ Tests setting and getting global value. """ diff --git a/tests/components/problems/clevr_tests.py b/tests/components/tasks/clevr_tests.py similarity index 88% rename from tests/components/problems/clevr_tests.py rename to tests/components/tasks/clevr_tests.py index 791bf40..0dc38e7 100644 --- a/tests/components/problems/clevr_tests.py +++ b/tests/components/tasks/clevr_tests.py @@ -20,8 +20,8 @@ from unittest.mock import MagicMock, patch from os import path -from ptp.components.utils.io import check_file_existence -from ptp.components.problems.image_text_to_class.clevr import CLEVR +from ptp.components.mixins.io import check_file_existence +from ptp.components.tasks.image_text_to_class.clevr import CLEVR from ptp.configuration.config_interface import ConfigInterface @@ -60,11 +60,11 @@ def test_training_set(self): 'split': 'train', 'answer': 'yes', 'question': 'Are there more big green things than large purple shiny cubes?'}] # Mock up the load_dataset method. - with patch( "ptp.components.problems.image_text_to_class.clevr.CLEVR.load_dataset", MagicMock( side_effect = [ dataset_content ] )): + with patch( "ptp.components.tasks.image_text_to_class.clevr.CLEVR.load_dataset", MagicMock( side_effect = [ dataset_content ] )): clevr = CLEVR("clevr_training", config["clevr_training"]) # Mock up the get_image method. - with patch( "ptp.components.problems.image_text_to_class.clevr.CLEVR.get_image", MagicMock( side_effect = [ "0" ] )): + with patch( "ptp.components.tasks.image_text_to_class.clevr.CLEVR.get_image", MagicMock( side_effect = [ "0" ] )): sample = clevr[0] # Check sample. @@ -106,11 +106,11 @@ def test_validation_set(self): 'question_index': 0, 'image_filename': 'CLEVR_val_000000.png', 'question_family_index': 39, 'split': 'val', 'answer': 'no', 'question': 'Are there any other things that are the same shape as the big metallic object?'}] # Mock up the load_dataset method. - with patch( "ptp.components.problems.image_text_to_class.clevr.CLEVR.load_dataset", MagicMock( side_effect = [ dataset_content ] )): + with patch( "ptp.components.tasks.image_text_to_class.clevr.CLEVR.load_dataset", MagicMock( side_effect = [ dataset_content ] )): clevr = CLEVR("clevr_validation", config["clevr_validation"]) # Mock up the get_image method. - with patch( "ptp.components.problems.image_text_to_class.clevr.CLEVR.get_image", MagicMock( side_effect = [ "0" ] )): + with patch( "ptp.components.tasks.image_text_to_class.clevr.CLEVR.get_image", MagicMock( side_effect = [ "0" ] )): sample = clevr[0] # Check sample. @@ -149,11 +149,11 @@ def test_test_set(self): dataset_content = [{'image_index': 0, 'split': 'test', 'image_filename': 'CLEVR_test_000000.png', 'question_index': 0, 'question': 'Is there anything else that is the same shape as the small brown matte object?'}] # Mock up the load_dataset method. - with patch( "ptp.components.problems.image_text_to_class.clevr.CLEVR.load_dataset", MagicMock( side_effect = [ dataset_content ] )): + with patch( "ptp.components.tasks.image_text_to_class.clevr.CLEVR.load_dataset", MagicMock( side_effect = [ dataset_content ] )): clevr = CLEVR("clevr_test", config["clevr_test"]) # Mock up the get_image method. - with patch( "ptp.components.problems.image_text_to_class.clevr.CLEVR.get_image", MagicMock( side_effect = [ "0" ] )): + with patch( "ptp.components.tasks.image_text_to_class.clevr.CLEVR.get_image", MagicMock( side_effect = [ "0" ] )): sample = clevr[0] # Check sample. diff --git a/tests/components/problems/gqa_tests.py b/tests/components/tasks/gqa_tests.py similarity index 74% rename from tests/components/problems/gqa_tests.py rename to tests/components/tasks/gqa_tests.py index 7174f71..762066f 100644 --- a/tests/components/problems/gqa_tests.py +++ b/tests/components/tasks/gqa_tests.py @@ -20,8 +20,8 @@ from unittest.mock import MagicMock, patch from os import path -from ptp.components.utils.io import check_file_existence -from ptp.components.problems.image_text_to_class.gqa import GQA +from ptp.components.mixins.io import check_file_existence +from ptp.components.tasks.image_text_to_class.gqa import GQA from ptp.configuration.config_interface import ConfigInterface @@ -43,24 +43,24 @@ def test_training_0_split(self): if False: #check_file_existence(path.expanduser('~/data/gqa/questions1.2/train_all_questions'),'train_all_questions_0.json'): # Create object. - problem = GQA("gqa_training_0", config["gqa_training_0"]) + task = GQA("gqa_training_0", config["gqa_training_0"]) # Check dataset size. - self.assertEqual(len(problem), 1430536) + self.assertEqual(len(task), 1430536) # Get sample. - sample = problem[0] + sample = task[0] else: processed_dataset_content = [ {'sample_ids': '07333408', 'image_ids': '2375429', 'questions': 'What is on the white wall?', 'answers': 'pipe', 'full_answers': 'The pipe is on the wall.'} ] # Mock up the load_dataset method. - with patch( "ptp.components.problems.image_text_to_class.gqa.GQA.load_dataset", MagicMock( side_effect = [ processed_dataset_content ] )): - problem = GQA("gqa_training_0", config["gqa_training_0"]) + with patch( "ptp.components.tasks.image_text_to_class.gqa.GQA.load_dataset", MagicMock( side_effect = [ processed_dataset_content ] )): + task = GQA("gqa_training_0", config["gqa_training_0"]) # Mock up the get_image method. - with patch( "ptp.components.problems.image_text_to_class.gqa.GQA.get_image", MagicMock( side_effect = [ "0" ] )): - sample = problem[0] + with patch( "ptp.components.tasks.image_text_to_class.gqa.GQA.get_image", MagicMock( side_effect = [ "0" ] )): + sample = task[0] # Check sample. self.assertEqual(sample['indices'], 0) @@ -86,24 +86,24 @@ def test_validation_split(self): if False: #check_file_existence(path.expanduser('~/data/gqa/questions1.2'),'val_all_questions.json'): # Create object. - problem = GQA("gqa_validation", config["gqa_validation"]) + task = GQA("gqa_validation", config["gqa_validation"]) # Check dataset size. - self.assertEqual(len(problem), 2011853) + self.assertEqual(len(task), 2011853) # Get sample. - sample = problem[0] + sample = task[0] else: processed_dataset_content = [ {'sample_ids': '05451384', 'image_ids': '2382986', 'questions': 'Are there blankets under the brown cat?', 'answers': 'no', 'full_answers': 'No, there is a towel under the cat.'} ] # Mock up the load_dataset method. - with patch( "ptp.components.problems.image_text_to_class.gqa.GQA.load_dataset", MagicMock( side_effect = [ processed_dataset_content ] )): - problem = GQA("gqa_validation", config["gqa_validation"]) + with patch( "ptp.components.tasks.image_text_to_class.gqa.GQA.load_dataset", MagicMock( side_effect = [ processed_dataset_content ] )): + task = GQA("gqa_validation", config["gqa_validation"]) # Mock up the get_image method. - with patch( "ptp.components.problems.image_text_to_class.gqa.GQA.get_image", MagicMock( side_effect = [ "0" ] )): - sample = problem[0] + with patch( "ptp.components.tasks.image_text_to_class.gqa.GQA.get_image", MagicMock( side_effect = [ "0" ] )): + sample = task[0] # Check sample. self.assertEqual(sample['indices'], 0) @@ -129,24 +129,24 @@ def test_test_dev_split(self): if False: #check_file_existence(path.expanduser('~/data/gqa/questions1.2'),'testdev_all_questions.json'): # Create object. - problem = GQA("gqa_testdev", config["gqa_testdev"]) + task = GQA("gqa_testdev", config["gqa_testdev"]) # Check dataset size. - self.assertEqual(len(problem), 172174) + self.assertEqual(len(task), 172174) # Get sample. - sample = problem[0] + sample = task[0] else: processed_dataset_content = [ {'sample_ids': '20968379', 'image_ids': 'n288870', 'questions': 'Do the shorts have dark color?', 'answers': 'yes', 'full_answers': 'Yes, the shorts are dark.'} ] # Mock up the load_dataset method. - with patch( "ptp.components.problems.image_text_to_class.gqa.GQA.load_dataset", MagicMock( side_effect = [ processed_dataset_content ] )): - problem = GQA("gqa_testdev", config["gqa_testdev"]) + with patch( "ptp.components.tasks.image_text_to_class.gqa.GQA.load_dataset", MagicMock( side_effect = [ processed_dataset_content ] )): + task = GQA("gqa_testdev", config["gqa_testdev"]) # Mock up the get_image method. - with patch( "ptp.components.problems.image_text_to_class.gqa.GQA.get_image", MagicMock( side_effect = [ "0" ] )): - sample = problem[0] + with patch( "ptp.components.tasks.image_text_to_class.gqa.GQA.get_image", MagicMock( side_effect = [ "0" ] )): + sample = task[0] # Check sample. self.assertEqual(sample['indices'], 0) @@ -172,24 +172,24 @@ def test_test_split(self): if False: #check_file_existence(path.expanduser('~/data/gqa/questions1.2'),'test_all_questions.json'): # Create object. - problem = GQA("gqa_test", config["gqa_test"]) + task = GQA("gqa_test", config["gqa_test"]) # Check dataset size. - self.assertEqual(len(problem), 1340048) + self.assertEqual(len(task), 1340048) # Get sample. - sample = problem[0] + sample = task[0] else: processed_dataset_content = [ {'sample_ids': '201971873', 'image_ids': 'n15740', 'questions': 'Is the blanket to the right of a pillow?', 'answers': '', 'full_answers': ''} ] # Mock up the load_dataset method. - with patch( "ptp.components.problems.image_text_to_class.gqa.GQA.load_dataset", MagicMock( side_effect = [ processed_dataset_content ] )): - problem = GQA("gqa_test", config["gqa_test"]) + with patch( "ptp.components.tasks.image_text_to_class.gqa.GQA.load_dataset", MagicMock( side_effect = [ processed_dataset_content ] )): + task = GQA("gqa_test", config["gqa_test"]) # Mock up the get_image method. - with patch( "ptp.components.problems.image_text_to_class.gqa.GQA.get_image", MagicMock( side_effect = [ "0" ] )): - sample = problem[0] + with patch( "ptp.components.tasks.image_text_to_class.gqa.GQA.get_image", MagicMock( side_effect = [ "0" ] )): + sample = task[0] # Check sample. self.assertEqual(sample['indices'], 0) diff --git a/tests/components/problems/problem_tests.py b/tests/components/tasks/task_tests.py similarity index 65% rename from tests/components/problems/problem_tests.py rename to tests/components/tasks/task_tests.py index 8fdb38c..2e16289 100644 --- a/tests/components/problems/problem_tests.py +++ b/tests/components/tasks/task_tests.py @@ -18,17 +18,17 @@ import unittest -from ptp.components.problems.problem import Problem +from ptp.components.tasks.task import Task from ptp.data_types.data_definition import DataDefinition from ptp.configuration.config_interface import ConfigInterface -class MockupProblem (Problem): +class MockupTask (Task): """ - Mockup problem class. + Mockup task class. """ def __init__(self, name, config): - Problem.__init__(self, name, None, config) + Task.__init__(self, name, None, config) def output_data_definitions(self): return { @@ -37,24 +37,24 @@ def output_data_definitions(self): } -class TestProblem(unittest.TestCase): +class TestTask(unittest.TestCase): def __init__(self, *args, **kwargs): - super(TestProblem, self).__init__(*args, **kwargs) + super(TestTask, self).__init__(*args, **kwargs) # Overwrite abc abstract methods. - MockupProblem.__abstractmethods__=set() - # Create mocked-up problem. + MockupTask.__abstractmethods__=set() + # Create mocked-up task. config = ConfigInterface() - self.problem = MockupProblem("test", config) + self.task = MockupTask("test", config) - def test_crete_data_dict_key_present(self): + def test_crete_data_streams_key_present(self): """ Tests whether the created data dict contains required keys. """ - data_dict = self.problem.create_data_dict(1) + data_streams = self.task.create_data_streams(1) # Check presence of index. - self.assertEqual(data_dict['indices'], 1) # Even if we didn't explicitly indicated that in definitions! - self.assertEqual(data_dict['inputs'], None) - self.assertEqual(data_dict['targets'], None) + self.assertEqual(data_streams['indices'], 1) # Even if we didn't explicitly indicated that in definitions! + self.assertEqual(data_streams['inputs'], None) + self.assertEqual(data_streams['targets'], None) #if __name__ == "__main__": # unittest.main() \ No newline at end of file diff --git a/tests/data_types/data_dict_tests.py b/tests/data_types/data_streams_tests.py similarity index 64% rename from tests/data_types/data_dict_tests.py rename to tests/data_types/data_streams_tests.py index dfa9045..dd5f6f4 100644 --- a/tests/data_types/data_dict_tests.py +++ b/tests/data_types/data_streams_tests.py @@ -18,46 +18,46 @@ import unittest -from ptp.data_types.data_dict import DataDict +from ptp.data_types.data_streams import DataStreams -class TestDataDict(unittest.TestCase): +class TestDataStreams(unittest.TestCase): def __init__(self, *args, **kwargs): - super(TestDataDict, self).__init__(*args, **kwargs) + super(TestDataStreams, self).__init__(*args, **kwargs) data_definitions = { 'inputs': None, 'targets': None } # Create object. - self.data_dict = DataDict(data_definitions) + self.data_streams = DataStreams(data_definitions) def test_keys_present(self): """ Tests whether the original keys are present and can be retrieved/modified. """ - self.assertTrue('inputs' in self.data_dict.keys() ) - self.assertTrue('targets' in self.data_dict.keys() ) + self.assertTrue('inputs' in self.data_streams.keys() ) + self.assertTrue('targets' in self.data_streams.keys() ) # Check initial value. - self.assertEqual(self.data_dict['inputs'], None) + self.assertEqual(self.data_streams['inputs'], None) # Modify and retrieve. - self.data_dict['inputs'] = 1.2 - self.assertEqual(self.data_dict['inputs'], 1.2) + self.data_streams['inputs'] = 1.2 + self.assertEqual(self.data_streams['inputs'], 1.2) def test_keys_absent(self): """ Tests whether absent keys are really absent and cannot be simply added. """ with self.assertRaises(KeyError): - a = self.data_dict["predictions"] + a = self.data_streams["predictions"] with self.assertRaises(KeyError): - self.data_dict["predictions"] = 12 + self.data_streams["predictions"] = 12 def test_keys_extend(self): """ Tests whether append works as expected. """ # Cannot add existing key. with self.assertRaises(KeyError): - self.data_dict.extend( {"inputs": 1.5 } ) + self.data_streams.publish( {"inputs": 1.5 } ) # Can add new key. - self.data_dict.extend( {"predictions": 12 } ) - self.assertEqual(self.data_dict['predictions'], 12) + self.data_streams.publish( {"predictions": 12 } ) + self.assertEqual(self.data_streams['predictions'], 12)