model with modified data set reader

[junkyul]

forward receives

        fields = {
            "wid": wid_field,                                   # label
            "title": title_field,                               # label
            "types": type_field,                                # multi label for one hot
            "sentence": sentence_field,                         # text
            "sentence_left": sentence_left_field,               # text
            "sentence_right": sentence_right_field,             # text
            "mention": mention_surface_field,                   # meta
            "mention_normalized": mention_normalized_field,     # meta
            "coherences": coherences_field                      # multi label
        }

• all are padded & batched tensors except meta field.

• Use pre-trained embedding, GloVe

• Embeddings with One Hot Vector from MultiLabels (match dimensions + 1 for UNK)

• DenseSparseAdam encountered numerical error

• initializer in config how?

• regularizer in config how?

[junkyul]

model summary

[junkyul]

• non-trainable GloVe text embedding; can be reused?

• embedding dropout
  biattentive_classification_network_elmo.jsonnet
  "embedding_dropout": 0.5,
  self.word_embedding_dropout_left = torch.nn.Dropout(word_embedding_dropout)

• configs for Embedding

  Embedding {
               num_embeddings: int,
               embedding_dim: int,
               projection_dim: int = None,
               weight: torch.FloatTensor = None,
               padding_index: int = None,
               trainable: bool = True,
               max_norm: float = None,
               norm_type: float = 2.,
               scale_grad_by_freq: bool = False,
               sparse: bool = False,
               vocab_namespace: str = None,
               pretrained_file: str = None
  }
  

  vocab_namespace passed, Embedding may extend_vocab

embedding usage?

[junkyul]

• loss

torch.nn.CrossEntropyLoss()

pass score and target per each example (batched)
cross entropy

candidates can be only 1! 100% accuracy guaranteed by crosswiki strong candidates.
loss cannot process length 1 classification.

is it possible to enforce the length of candidates be at least 2? (if 1 pad unknown WID as a faulty class)
min_padding_length for character embedding.
allenai/allennlp#1954

just add @@unknown@@ if a length is one and give probability zero prior

• get probability values from loss computation (combined with prior do prediction)
  while training, prior is not required; it is part of inference (irrelevant to the loss function)
  how to pull out those numbers?
  https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/loss.py
  The losses are averaged across observations for each minibatch.

https://github.com/pytorch/pytorch/blob/master/torch/nn/functional.py
write a custom loss class that returns probabilities

log_softmaxandnll_loss

def cross_entropy(input, target, weight=None, size_average=None, ignore_index=-100,
                  reduce=None, reduction='mean'):
    if size_average is not None or reduce is not None:
        reduction = _Reduction.legacy_get_string(size_average, reduce)
    return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)

def log_softmax(input, dim=None, _stacklevel=3, dtype=None):
    # type: (Tensor, Optional[int], int, Optional[int]) -> Tensor
    if dim is None:
        dim = _get_softmax_dim('log_softmax', input.dim(), _stacklevel)
    if dtype is None:
        ret = input.log_softmax(dim)
    else:
        ret = input.log_softmax(dim, dtype=dtype)
    return ret

softmax( score ) --> normalized btw 0 and 1
and
log

def nll_loss(input, target, weight=None, size_average=None, ignore_index=-100,
             reduce=None, reduction='mean'):
    # type: (Tensor, Tensor, Optional[Tensor], Optional[bool], int, Optional[bool], str) -> Tensor
    if size_average is not None or reduce is not None:
        reduction = _Reduction.legacy_get_string(size_average, reduce)
    dim = input.dim()
    if dim < 2:
        raise ValueError('Expected 2 or more dimensions (got {})'.format(dim))

    if input.size(0) != target.size(0):
        raise ValueError('Expected input batch_size ({}) to match target batch_size ({}).'
                         .format(input.size(0), target.size(0)))
    if dim == 2:
        ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
    elif dim == 4:
        ret = torch._C._nn.nll_loss2d(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
    else:
        # dim == 3 or dim > 4
        n = input.size(0)
        c = input.size(1)
        out_size = (n,) + input.size()[2:]
        if target.size()[1:] != input.size()[2:]:
            raise ValueError('Expected target size {}, got {}'.format(
                out_size, target.size()))
        input = input.contiguous().view(n, c, 1, -1)
        target = target.contiguous().view(n, 1, -1)
        reduction_enum = _Reduction.get_enum(reduction)
        if reduction != 'none':
            ret = torch._C._nn.nll_loss2d(
                input, target, weight, reduction_enum, ignore_index)
        else:
            out = torch._C._nn.nll_loss2d(
                input, target, weight, reduction_enum, ignore_index)
            ret = out.view(out_size)
    return ret

[junkyul]

• GloVe embedding

token vocab built from training data does not match pre-trained glove embedding. (non-english tokens)

allennlp.common.checks.ConfigurationError: "No embeddings of correct dimension found; you probably misspecified your embedding_dim parameter, or didn't pre-populate your Vocabulary"

still, load pre-trained embedding?
without extending embedding with the data from instances; it raises error due to mismatch of words; many token words from instances does not exist in pre-trained vocabulary