Skip to content

Commit

Permalink
Updated dynet to use initialization
Browse files Browse the repository at this point in the history
  • Loading branch information
@neubig
neubig committed Oct 11, 2016
1 parent 25b5490 commit b0041d7
Show file tree
Hide file tree
Showing 2 changed files with 7 additions and 4 deletions.
7 changes: 5 additions & 2 deletions dynet/model.cc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -111,7 +111,8 @@ LookupParameterStorage::LookupParameterStorage(unsigned n, const Dim& d) : dim(d
all_grads.device = all_values.device = default_device;
default_device->allocate_tensor(DeviceMempool::PS, all_values);
default_device->allocate_tensor(DeviceMempool::PS, all_grads);
TensorTools::Zero(all_values);
ParameterInitGlorot init(true);
init.initialize_params(all_values);
initialize_lookups();
}

Expand Down Expand Up @@ -192,7 +193,9 @@ void ParameterInitConst::initialize_params(Tensor & values) const {
}

void ParameterInitGlorot::initialize_params(Tensor & values) const {
float my_scale = sqrt(6) / sqrt(values.d.sum_dims());
int dims = 0, dim_len = values.d.nd-(lookup?1:0);
for(int i = 0; i < dim_len; ++i) dims += values.d[i];
float my_scale = sqrt(6) / sqrt(dims);
TensorTools::RandomizeUniform(values, -my_scale, my_scale);
}

Expand Down
4 changes: 2 additions & 2 deletions dynet/model.h
View file
Original file line number Diff line number Diff line change
Expand Up @@ -196,10 +196,10 @@ struct ParameterInitConst : public ParameterInit {
};

struct ParameterInitGlorot : public ParameterInit {
ParameterInitGlorot() {}
ParameterInitGlorot(bool is_lookup = false) : lookup(is_lookup) {}
virtual void initialize_params(Tensor & values) const override;
private:
float cnst;
bool lookup;
};

struct ParameterInitSaxe : public ParameterInit {
Expand Down

2 comments on commit b0041d7

@radsimu
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should make less, but more consistent commits. Do your work in a separate branch and when a feature is finished, merge to master as a single squashed commit.

@neubig
Copy link
Contributor Author

@neubig neubig commented on b0041d7 Oct 12, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the past few days we were in a bit of a transition stage, so things were moving quickly, but in the future we'll probably do this. Feel free to use the dynet v1 branch if you'd like something that will be stable long term: https://github.com/clab/dynet/tree/v1

Please sign in to comment.