Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ConvE in config.yaml? #228

Closed
Filco306 opened this issue Sep 25, 2021 · 13 comments
Closed

ConvE in config.yaml? #228

Filco306 opened this issue Sep 25, 2021 · 13 comments

Comments

@Filco306
Copy link

Hello (again, haha)

I am currently running some experiments with your package; I love it. However, I am concerned about the validity of one of my experiments. I performed an AxSearchJob on one of my models (which I have named RIE2). In my config.yaml from one of the runs, I get a conve setting, which is strange to me. I don't know why it is there, and I am concerned that it might be, in the worst case scenario, a ConvE-model training in the background (and not my model implementation).

Now, I have tried to make sure that my worries are not legitimate by introducing exceptions in all parts of the ConvE code, if it is executed, and it seems to run through without a problem. In other words, it does not seem like ConvE is used, since no code from the ConvE-part of the code seems to be used. Also, loading the model will load an RIE2-model.

I would simply like to know; why is there a ConvE part being created in this config.yaml? Even if I erase that part and re-run the experiment, it comes back into the new config.yaml-file for the new experiment. I do not believe there is an issue here, but I would just like to make sure with a 100 % certainty.

Thank you in advance, and thanks again for a very nice package!

The config file is as seen below:

1vsAll:
  class_name: TrainingJob1vsAll
KvsAll:
  class_name: TrainingJobKvsAll
  label_smoothing: -0.10110532511025666
  query_types:
    _po: true
    s_o: false
    sp_: true
RIE2:
  class_name: RIE2
  entity_embedder:
    +++: +++
    dropout: 0.13526539038866758
    normalize:
      p: 2.0
    regularize_weight: 1.4196422325008177e-05
    type: lookup_embedder
  l_norm: 1.0
  relation_embedder:
    +++: +++
    dropout: 0.1788316359743476
    normalize:
      p: -1.0
    regularize_weight: 0.0002284745422735363
    type: lookup_embedder
  round_dim: false
  tuneable_doubletan: false
ax_search:
  class_name: AxSearchJob
  num_sobol_trials: 30
  num_trials: 30
  parameter_constraints: []
  parameters:
  - name: model
    type: choice
    values:
    - RIE2
    - reciprocal_relations_model
  - is_ordered: true
    name: train.batch_size
    type: choice
    values:
    - 128
    - 256
    - 512
    - 1024
  - name: train.type
    type: fixed
    value: negative_sampling
  - name: train.optimizer
    type: choice
    values:
    - Adam
    - Adagrad
  - name: train.loss
    type: fixed
    value: kl
  - bounds:
    - 0.0003
    - 1.0
    log_scale: true
    name: train.optimizer_args.lr
    type: range
  - name: train.lr_scheduler
    type: fixed
    value: ReduceLROnPlateau
  - name: train.lr_scheduler_args.mode
    type: fixed
    value: max
  - name: train.lr_scheduler_args.factor
    type: fixed
    value: 0.95
  - name: train.lr_scheduler_args.threshold
    type: fixed
    value: 0.0001
  - bounds:
    - 0
    - 10
    name: train.lr_scheduler_args.patience
    type: range
  - is_ordered: true
    name: lookup_embedder.dim
    type: choice
    values:
    - 128
    - 256
    - 512
  - name: lookup_embedder.initialize
    type: choice
    values:
    - xavier_normal_
    - xavier_uniform_
    - normal_
    - uniform_
  - name: lookup_embedder.initialize_args.normal_.mean
    type: fixed
    value: 0.0
  - bounds:
    - 1.0e-05
    - 1.0
    log_scale: true
    name: lookup_embedder.initialize_args.normal_.std
    type: range
  - bounds:
    - -1.0
    - -1.0e-05
    name: lookup_embedder.initialize_args.uniform_.a
    type: range
  - name: lookup_embedder.initialize_args.xavier_uniform_.gain
    type: fixed
    value: 1.0
  - name: lookup_embedder.initialize_args.xavier_normal_.gain
    type: fixed
    value: 1.0
  - is_ordered: true
    name: lookup_embedder.regularize
    type: choice
    values:
    - ''
    - l3
    - l2
    - l1
  - name: lookup_embedder.regularize_args.weighted
    type: choice
    values:
    - true
    - false
  - bounds:
    - 1.0e-20
    - 0.1
    log_scale: true
    name: RIE2.entity_embedder.regularize_weight
    type: range
  - bounds:
    - 1.0e-20
    - 0.1
    log_scale: true
    name: RIE2.relation_embedder.regularize_weight
    type: range
  - bounds:
    - -0.5
    - 0.5
    name: RIE2.entity_embedder.dropout
    type: range
  - bounds:
    - -0.5
    - 0.5
    name: RIE2.relation_embedder.dropout
    type: range
  - bounds:
    - -0.3
    - 0.3
    name: KvsAll.label_smoothing
    type: range
  - is_ordered: true
    name: RIE2.l_norm
    type: choice
    values:
    - 1.0
    - 2.0
  - name: RIE2.entity_embedder.normalize.p
    type: choice
    values:
    - -1.0
    - 2.0
  - name: RIE2.relation_embedder.normalize.p
    type: choice
    values:
    - -1.0
    - 2.0
  - name: negative_sampling.implementation
    type: fixed
    value: spo
  sobol_seed: 0
console:
  format: {}
  quiet: false
conve: # <--- HERE IS THE CONVE PART
  2D_aspect_ratio: 2 # <--- HERE IS THE CONVE PART
  class_name: ConvE # <--- HERE IS THE CONVE PART
  convolution_bias: true # <--- HERE IS THE CONVE PART
  entity_embedder: # <--- HERE IS THE CONVE PART
    +++: +++ # <--- HERE IS THE CONVE PART
    dropout: 0.2 # <--- HERE IS THE CONVE PART
    type: lookup_embedder # <--- HERE IS THE CONVE PART
  feature_map_dropout: 0.2 # <--- HERE IS THE CONVE PART
  filter_size: 3 # <--- HERE IS THE CONVE PART
  padding: 0 # <--- HERE IS THE CONVE PART
  projection_dropout: 0.3 # <--- HERE IS THE CONVE PART
  relation_embedder: # <--- HERE IS THE CONVE PART
    +++: +++ # <--- HERE IS THE CONVE PART
    dropout: 0.2 # <--- HERE IS THE CONVE PART
    type: lookup_embedder # <--- HERE IS THE CONVE PART
  round_dim: false # <--- HERE IS THE CONVE PART
  stride: 1 # <--- HERE IS THE CONVE PART
dataset:
  +++: +++
  files:
    +++: +++
    entity_ids:
      filename: entity_ids.del
      type: map
    entity_strings:
      filename: entity_strings.del
      type: idmap
    relation_ids:
      filename: relation_ids.del
      type: map
    relation_strings:
      filename: relation_ids.del
      type: map
    test:
      filename: test.del
      size: 3134
      split_type: test
      type: triples
    test_without_unseen:
      filename: test_without_unseen.del
      size: 2924
      split_type: test
      type: triples
    train:
      filename: train.del
      size: 86835
      split_type: train
      type: triples
    train_sample:
      filename: train_sample.del
      size: 3034
      split_type: train
      type: triples
    valid:
      filename: valid.del
      size: 3034
      split_type: valid
      type: triples
    valid_without_unseen:
      filename: valid_without_unseen.del
      size: 2824
      split_type: valid
      type: triples
  name: wnrr
  num_entities: 40943
  num_relations: 11
  pickle: true
entity_ranking:
  chunk_size: -1
  class_name: EntityRankingJob
  filter_splits:
  - train
  - valid
  filter_with_test: true
  hits_at_k_s:
  - 1
  - 3
  - 10
  - 50
  - 100
  - 200
  - 300
  - 400
  - 500
  - 1000
  metrics_per:
    argument_frequency: false
    head_and_tail: false
    relation_type: true
  tie_handling: rounded_mean_rank
eval:
  batch_size: 256
  num_workers: 0
  pin_memory: false
  split: valid
  trace_level: epoch
  type: entity_ranking
grid_search:
  class_name: GridSearchJob
  parameters:
    +++: +++
  run: true
import:
- RIE2
- reciprocal_relations_model
job:
  device: cuda
  type: train
lookup_embedder:
  class_name: LookupEmbedder
  dim: 128
  dropout: 0.0
  initialize: xavier_uniform_
  initialize_args:
    +++: +++
    normal_:
      mean: 0.0
      std: 0.00043470941774473033
    uniform_:
      a: -0.4313754705902655
    xavier_normal_:
      gain: 1.0
    xavier_uniform_:
      gain: 1.0
  normalize:
    p: -1.0
  pretrain:
    ensure_all: false
    model_filename: ''
  regularize: lp
  regularize_args:
    +++: +++
    p: 1
    weighted: true
  regularize_weight: 0.0
  round_dim_to: []
  sparse: false
manual_search:
  class_name: ManualSearchJob
  configurations: []
  run: true
model: RIE2
modules:
- kge.model.embedder
- kge.model
- kge.job
negative_sampling:
  class_name: TrainingJobNegativeSampling
  filtering:
    implementation: fast_if_available
    o: false
    p: false
    s: false
    split: ''
  frequency:
    smoothing: 1
  implementation: triple
  num_samples:
    o: -1
    p: 0
    s: 3
  sampling_type: uniform
  shared: false
  shared_type: default
  with_replacement: true
random_seed:
  default: -1
  numba: -1
  numpy: -1
  python: -1
  torch: -1
reciprocal_relations_model:
  base_model:
    +++: +++
    type: RIE2
  class_name: ReciprocalRelationsModel
search:
  device_pool: []
  num_workers: 1
  on_error: abort
  type: ax_search
train:
  abort_on_nan: true
  auto_correct: true
  batch_size: 256
  checkpoint:
    every: 5
    keep: 3
    keep_init: true
  loss: kl
  loss_arg: .nan
  lr_scheduler: ReduceLROnPlateau
  lr_scheduler_args:
    +++: +++
    factor: 0.95
    mode: max
    patience: 2
    threshold: 0.0001
  lr_warmup: 0
  max_epochs: 400
  num_workers: 0
  optimizer:
    +++: +++
    default:
      args:
        +++: +++
        lr: 0.4969193773628293
      type: Adagrad
  pin_memory: false
  split: train
  subbatch_auto_tune: false
  subbatch_size: -1
  trace_level: epoch
  type: negative_sampling
  visualize_graph: false
training_loss:
  class_name: TrainingLossEvaluationJob
user:
  +++: +++
valid:
  early_stopping:
    patience: 10
    threshold:
      epochs: 50
      metric_value: 0.05
  every: 5
  metric: mean_reciprocal_rank_filtered_with_test
  metric_expr: float("nan")
  metric_max: true
  split: valid
  trace_level: epoch

The new ConvE-file looks like this (so it should not be run, which it does not seem to be).

import torch
from torch import Tensor
import math

from kge import Config, Dataset
from kge.model.kge_model import RelationalScorer, KgeModel


class ConvEScorer(RelationalScorer):
    r"""Implementation of the ConvE KGE scorer.
    Must be used with ReciprocalRelationsModel."""

    def __init__(self, config: Config, dataset: Dataset, configuration_key=None):
        raise Exception("This model should not be used at all! This is the error! ")
        

    def score_emb(self, s_emb, p_emb, o_emb, combine: str):
        raise Exception("This model should not be used at all! This is the error! ")
        


class ConvE(KgeModel):
    r"""Implementation of the ConvE KGE model."""

    def __init__(
        self,
        config: Config,
        dataset: Dataset,
        configuration_key=None,
        init_for_load_only=False,
    ):
        raise Exception("This model should not be used at all! This is the error! ")
        

    def score_spo(self, s: Tensor, p: Tensor, o: Tensor, direction=None) -> Tensor:
        raise Exception("This model should not be used at all! This is the error! ")
@rufex2001
Copy link
Member

rufex2001 commented Sep 26, 2021 via email

@Filco306
Copy link
Author

I generate it using kge start config.yaml, where config.yaml is the file above. If I remove the part about ConvE, it will still be in the new file again which I do not understand why is the case.

@Filco306
Copy link
Author

Filco306 commented Sep 26, 2021 •
edited
Loading

And no, I do not base my model on ConvE in any way. I did however copy over some settings for the hyperparameter search from the ICLR config files, but I looked through that and I cannot see the ConvE-name anywhere in that file either.

@Filco306
Copy link
Author

I think I have figured it out through some backtracing. When I am using the ReciprocalRelationsModel, it will import the ConvE-model

See the reciprocal_relations.yaml:

import: [conve]

reciprocal_relations_model:
  class_name: ReciprocalRelationsModel
  base_model:
    type: conve
    +++: +++

Is this a desired feature? This means that ConvE will always be imported when the reciprocal_relations_model is used. If I understand it correctly, the reciprocal relations model is meant to use different embeddings when predicting subject and object.

In other words, this gets imported automatically, but I believe my model overrides this setting, so the ConvE-model is not used. Please correct me if I am wrong.

This is my hyperparameter search job:

1vsAll:
  class_name: TrainingJob1vsAll
KvsAll:
  class_name: TrainingJobKvsAll
  label_smoothing: 0.0
  query_types:
    _po: true
    s_o: false
    sp_: true
RIE2:
  class_name: RIE2
  entity_embedder:
    +++: +++
    type: lookup_embedder
  l_norm: 1.0
  relation_embedder:
    +++: +++
    type: lookup_embedder
  tuneable_doubletan: false
ax_search:
  class_name: AxSearchJob
  num_sobol_trials: 30
  num_trials: 30
  parameter_constraints: []
  parameters:
  - name: model
    type: fixed
    value: reciprocal_relations_model
  - is_ordered: true
    name: train.batch_size
    type: choice
    values:
    - 128
    - 256
  - name: train.type
    type: fixed
    value: KvsAll
  - name: train.optimizer
    type: choice
    values:
    - Adam
    - Adagrad
  - name: train.loss
    type: fixed
    value: kl
  - bounds:
    - 0.0003
    - 1.0
    log_scale: true
    name: train.optimizer_args.lr
    type: range
  - name: train.lr_scheduler
    type: fixed
    value: ReduceLROnPlateau
  - name: train.lr_scheduler_args.mode
    type: fixed
    value: max
  - name: train.lr_scheduler_args.factor
    type: fixed
    value: 0.95
  - name: train.lr_scheduler_args.threshold
    type: fixed
    value: 0.0001
  - bounds:
    - 0
    - 10
    name: train.lr_scheduler_args.patience
    type: range
  - is_ordered: true
    name: lookup_embedder.dim
    type: choice
    values:
    - 128
    - 256
    - 512
  - name: lookup_embedder.initialize
    type: choice
    values:
    - xavier_normal_
    - xavier_uniform_
    - normal_
    - uniform_
  - name: lookup_embedder.initialize_args.normal_.mean
    type: fixed
    value: 0.0
  - bounds:
    - 1.0e-05
    - 1.0
    log_scale: true
    name: lookup_embedder.initialize_args.normal_.std
    type: range
  - bounds:
    - -1.0
    - -1.0e-05
    name: lookup_embedder.initialize_args.uniform_.a
    type: range
  - name: lookup_embedder.initialize_args.xavier_uniform_.gain
    type: fixed
    value: 1.0
  - name: lookup_embedder.initialize_args.xavier_normal_.gain
    type: fixed
    value: 1.0
  - is_ordered: true
    name: lookup_embedder.regularize
    type: choice
    values:
    - ''
    - l3
    - l2
    - l1
  - name: lookup_embedder.regularize_args.weighted
    type: choice
    values:
    - true
    - false
  - bounds:
    - 1.0e-20
    - 0.1
    log_scale: true
    name: RIE2.entity_embedder.regularize_weight
    type: range
  - bounds:
    - 1.0e-20
    - 0.1
    log_scale: true
    name: RIE2.relation_embedder.regularize_weight
    type: range
  - bounds:
    - -0.5
    - 0.5
    name: RIE2.entity_embedder.dropout
    type: range
  - bounds:
    - -0.5
    - 0.5
    name: RIE2.relation_embedder.dropout
    type: range
  - bounds:
    - -0.3
    - 0.3
    name: KvsAll.label_smoothing
    type: range
  sobol_seed: 0
console:
  format: {}
  quiet: false
conve:
  2D_aspect_ratio: 2
  class_name: ConvE
  convolution_bias: true
  entity_embedder:
    +++: +++
    dropout: 0.2
    type: lookup_embedder
  feature_map_dropout: 0.2
  filter_size: 3
  padding: 0
  projection_dropout: 0.3
  relation_embedder:
    +++: +++
    dropout: 0.2
    type: lookup_embedder
  round_dim: false
  stride: 1
dataset:
  +++: +++
  files:
    +++: +++
    entity_ids:
      filename: entity_ids.del
      type: map
    entity_strings:
      filename: entity_ids.del
      type: map
    relation_ids:
      filename: relation_ids.del
      type: map
    relation_strings:
      filename: relation_ids.del
      type: map
    test:
      filename: test.del
      type: triples
    train:
      filename: train.del
      type: triples
    valid:
      filename: valid.del
      type: triples
  name: wnrr
  num_entities: -1
  num_relations: -1
  pickle: true
entity_ranking:
  chunk_size: -1
  class_name: EntityRankingJob
  filter_splits:
  - train
  - valid
  filter_with_test: true
  hits_at_k_s:
  - 1
  - 3
  - 10
  - 50
  - 100
  - 200
  - 300
  - 400
  - 500
  - 1000
  metrics_per:
    argument_frequency: false
    head_and_tail: false
    relation_type: true
  tie_handling: rounded_mean_rank
eval:
  batch_size: 16
  num_workers: 0
  pin_memory: false
  split: valid
  trace_level: epoch
  type: entity_ranking
grid_search:
  class_name: GridSearchJob
  parameters:
    +++: +++
  run: true
import:
- RIE2
- reciprocal_relations_model
job:
  device: cuda:1
  type: search
lookup_embedder:
  class_name: LookupEmbedder
  dim: 100
  dropout: 0.0
  initialize: normal_
  initialize_args:
    +++: +++
  normalize:
    p: -1.0
  pretrain:
    ensure_all: false
    model_filename: ''
  regularize: lp
  regularize_args:
    +++: +++
    p: 2
    weighted: false
  regularize_weight: 0.0
  round_dim_to: []
  sparse: false
manual_search:
  class_name: ManualSearchJob
  configurations: []
  run: true
model: ''
modules:
- kge.job
- kge.model.embedder
- kge.model
negative_sampling:
  class_name: TrainingJobNegativeSampling
  filtering:
    implementation: fast_if_available
    o: false
    p: false
    s: false
    split: ''
  frequency:
    smoothing: 1
  implementation: batch
  num_samples:
    o: -1
    p: 0
    s: 3
  sampling_type: uniform
  shared: false
  shared_type: default
  with_replacement: true
random_seed:
  default: -1
  numba: -1
  numpy: -1
  python: -1
  torch: -1
reciprocal_relations_model:
  base_model:
    +++: +++
    type: RIE2
  class_name: ReciprocalRelationsModel
search:
  device_pool: []
  num_workers: 1
  on_error: abort
  type: ax_search
train:
  abort_on_nan: true
  auto_correct: true
  batch_size: 100
  checkpoint:
    every: 5
    keep: 3
    keep_init: true
  loss: kl
  loss_arg: .nan
  lr_scheduler: ''
  lr_scheduler_args:
    +++: +++
  lr_warmup: 0
  max_epochs: 400
  num_workers: 0
  optimizer:
    +++: +++
    default:
      args:
        +++: +++
      type: Adagrad
  pin_memory: false
  split: train
  subbatch_auto_tune: false
  subbatch_size: -1
  trace_level: epoch
  type: KvsAll
  visualize_graph: false
training_loss:
  class_name: TrainingLossEvaluationJob
user:
  +++: +++
valid:
  early_stopping:
    patience: 10
    threshold:
      epochs: 50
      metric_value: 0.05
  every: 5
  metric: mean_reciprocal_rank_filtered_with_test
  metric_expr: float("nan")
  metric_max: true
  split: valid
  trace_level: epoch

@Filco306
Copy link
Author

As a clarification; If I understand it correctly, this means that the reciprocal relations model is asymmetric, meaning that a prediction of a head is not equal to the prediction of a tail? This has significant meaning in the implementation of my model, so it would be great to know :) Thanks!

@rufex2001
Copy link
Member

rufex2001 commented Sep 26, 2021 via email

@Filco306
Copy link
Author

Filco306 commented Sep 26, 2021 •
edited
Loading

Exactly, just as I understood. Thank you @rufex2001 . Let me know if there is something off with the fact that ConvE is imported; otherwise I will assume it is correct :)

@rufex2001
Copy link
Member

rufex2001 commented Sep 26, 2021 via email

@Filco306
Copy link
Author

Another question while we are at it; the metrics posted in the README for the datasets; which one of the outputs is it? Is it mrr, mrr_filtered or mrr_filtered_with_test that is documented in the README? Thanks again for all your responses!!

@Filco306
Copy link
Author

Yes, that is my understanding as well; but my model is still used; i.e., the ConvE architecture is not used.

@Filco306
Copy link
Author

And thanks again for the clarifications!

@rufex2001
Copy link
Member

rufex2001 commented Sep 26, 2021 via email

@Filco306
Copy link
Author

Great, thank you! Will close this now. Thank you for helping me sort this out, but let me know if you believe any issues might be caused by this import anyway. Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@rufex2001 @Filco306