Add `from_params` to `Linear` & `Conv` #7525

crcrpar · 2019-06-16T16:14:43Z

This addresses #7042 & #7041 by showing draft implementations of some connection links that are commonly used.

The aim of this PR is easing initializing links from weights that are not easy to generate chainer.Initializer\s or that are trained beforehand in some way (e.g. word embedding vectors or a weight extracted from a pretrained model).
Currently, users can pass those weights to initialW/initial_bias if they are numpy.ndarray as a variant of chainer.initializers.Constant, but I don't think this manner intuitive.

crcrpar · 2019-07-02T03:57:58Z

I think this depends on #7531 somewhat.

Any thoughts, @emcastillo?

emcastillo · 2019-07-03T01:55:24Z

AFAIK #7531 just makes the initializers to allow chainerx arrays as arguments and disable the explicit fallback for random routines as chainerx now supports random (through chainerx internal fallback).

In this PR the proposed from_params could take any chainerx array as input and as long as chainerx supports the operations being done in that method there shouldn't be any problem.

crcrpar · 2019-07-03T05:54:42Z

Sorry for the lack of explanation.

The motivation of this PR is to ease Link initialization with some specific ndarrays, for instance, initializing L.EmbedID with some publicly available embedding weight.
So, the method added by this PR takes :ref:ndarray W & optional b as its arguments and passes them to the corresponding initializer-related arguments (initialW and initial_bias) of the constructor the class.

emcastillo · 2019-07-08T01:37:58Z

Sorry for the delay.
Yes, if the from_params end up relying on initializers when calling the constructor, chainer x support is going to be needed.

This was already merged in #7687 so there shouldn't be issues now.

crcrpar · 2019-07-10T11:15:16Z

Thanks for the info.

I'll resume working.

crcrpar · 2019-08-12T12:20:00Z

Apologize for having been slow to work.
I'm done with this, so could you give me a review?

emcastillo · 2019-08-19T01:10:40Z

tests/chainer_tests/links_tests/connection_tests/test_convolution_nd.py

+        'cuda:0',
+    ]
+}))
+class TestConvolutionNDFromParams(unittest.TestCase):


Instead of adding a new Test case,
How about using the from_params in the create_link of the ConvolutionND(testing.LinkTestCase) main test

Sure.

Is this applicable to the other test cases?

yes,
rebase with the master branch so the new tests appear :)

You can ignore this suggestion if you dont feel comfortable with this change

emcastillo

You will need to rebase because some of the convolution tests changed some days ago.
Thanks!!

emcastillo

Please fix the syntax error, once it is done I think it will be ready for CI.

Thanks!

errors

emcastillo · 2019-08-21T02:46:29Z

tests/chainer_tests/links_tests/connection_tests/test_convolution_2d.py

+        assert link2.W.shape == link1.W.shape
+        assert link2.b.shape == link2.b.shape
+        assert link2.stride == link1.stride
+        assert link2.pad = link1.pad


Syntax error here, should be ==

Thanks, accessed.

emcastillo · 2019-08-22T01:31:40Z

Another syntax error:
/home/travis/build/chainer/chainer/tests/chainer_tests/links_tests/connection_tests/test_convolution_2d.py:117:17: F821 undefined name 'L'
😅

emcastillo · 2019-08-22T13:14:53Z

Fix travis errors please.
I dont know if you can run the tests locally using pytest, but it would help a lot 😃

crcrpar · 2019-08-22T14:42:02Z

I'm sorry, but did the ci run as we expect the last time?

emcastillo · 2019-08-22T14:48:32Z

there are several errors on travis

emcastillo · 2019-08-23T00:58:42Z

tests/chainer_tests/links_tests/connection_tests/test_convolution_nd.py

+
+        link2 = convolution_nd.ConvolutionND.from_params(
+            link1.W, link1.b,
+            stride=self.stride, pad=self.pad, groups=self.group)


groups=self.groups

crcrpar · 2019-08-23T01:09:33Z

there are several errors on travis

I meant to say that some configuration of log display seemed to be changed since just there was only kind of "test failed" message first.

emcastillo · 2019-08-23T01:24:50Z

chainer/links/connection/convolution_2d.py

+
+        link = cls(
+            in_channels, out_channels, (kw, kh), stride, pad, nobias,
+            initialW=variable.as_array(W), initial_bias=variable.as_array(b),


this code fails when arrays other than numpy are passed as initializers.

Running your branch in my environment shows

Base test method: TestConvolution2D_use_chainerx_false__chainerx_device_None__use_cuda_true__cuda_device_0__use_cudnn_always__cudnn_deterministic_false__autotune_false__cudnn_fast_batch_normalization_false__use_ideep_never.test_from_params E Test parameters: E W_dtype: <class 'numpy.float64'> E x_dtype: <class 'numpy.float64'> E E E (caused by) E TypeError: invalid type of initializer: <class 'cupy.core.core.ndarray'>

We can allow to use cupy/chainerx as a constant initializer (shouldnt be difficult)
but let us discuss it first!

I just submitted a PR to tackle this issue #8022

emcastillo · 2019-08-23T01:25:27Z

tests/chainer_tests/links_tests/connection_tests/test_linear.py

@@ -187,4 +187,31 @@ def test_invalid_size(self):
            self.link(chainer.Variable(self.x))


+@testing.parameterize(*testing.product({
+    'nobias': [True, False],
+    'device': [


You can remove the devices, they are not being used in the test

emcastillo · 2019-08-23T01:25:37Z

tests/chainer_tests/links_tests/connection_tests/test_embed_id.py

+        '@cupy:0',
+        'native:0',
+        'cuda:0',
+    ]


emcastillo · 2019-10-03T01:28:53Z

Check CI errors

crcrpar · 2019-10-03T09:35:38Z

ah, sphinx syntax error...

emcastillo

LGTM

emcastillo · 2019-10-07T01:35:23Z

Jenkins, test this please

crcrpar · 2019-10-07T01:51:28Z

Ah...
I'll fix this kind of failures once GPUs available


2019-10-07 10:46:34.260067 STDOUT 2327] | =================================== FAILURES ===================================
-- | --
2019-10-07 10:46:34.260069 STDOUT 2327] | _ TestConvolution2D_use_chainerx_false__chainerx_device_None__use_cuda_true__cuda_device_1__use_cudnn_always__cudnn_deterministic_false__autotune_false__cudnn_fast_batch_normalization_false__use_ideep_never_param_0_{W_dtype=float16, x_dtype=float16}.test_from_params _
2019-10-07 10:46:34.260334 STDOUT 2327] |  
2019-10-07 10:46:34.260336 STDOUT 2327] | self = <chainer.testing._bundle.TestConvolution2D_use_chainerx_false__chainerx_device_None__use_cuda_true__cuda_device_1__use...fast_batch_normalization_false__use_ideep_never_param_0_{W_dtype=float16, x_dtype=float16} testMethod=test_from_params>
2019-10-07 10:46:34.260339 STDOUT 2327] | backend_config = <BackendConfig use_chainerx=False chainerx_device=None use_cuda=True cuda_device=1 use_cudnn='always' cudnn_deterministic=False autotune=False cudnn_fast_batch_normalization=False use_ideep='never'>
2019-10-07 10:46:34.260342 STDOUT 2327] |  
2019-10-07 10:46:34.260343 STDOUT 2327] | def test_from_params(self, backend_config):
2019-10-07 10:46:34.260344 STDOUT 2327] | link1 = self.create_link(self.generate_params())
2019-10-07 10:46:34.260346 STDOUT 2327] | link1.to_device(backend_config.device)
2019-10-07 10:46:34.260347 STDOUT 2327] | link2 = links.Convolution2D.from_params(
2019-10-07 10:46:34.260348 STDOUT 2327] | >           link1.W, link1.b, stride=self.stride, pad=self.pad)
2019-10-07 10:46:34.260350 STDOUT 2327] |  
2019-10-07 10:46:34.260351 STDOUT 2327] | chainer/tests/chainer_tests/links_tests/connection_tests/test_convolution_2d.py:118:
2019-10-07 10:46:34.260352 STDOUT 2327] | _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
2019-10-07 10:46:34.260353 STDOUT 2327] | usr/local/lib/python3.5/dist-packages/chainer/links/connection/convolution_2d.py:214: in from_params
2019-10-07 10:46:34.260355 STDOUT 2327] | dilate=dilate, groups=groups)
2019-10-07 10:46:34.260356 STDOUT 2327] | usr/local/lib/python3.5/dist-packages/chainer/links/connection/convolution_2d.py:134: in __init__
2019-10-07 10:46:34.260575 STDOUT 2327] | self._initialize_params(in_channels)
2019-10-07 10:46:34.260578 STDOUT 2327] | usr/local/lib/python3.5/dist-packages/chainer/links/connection/convolution_2d.py:168: in _initialize_params
2019-10-07 10:46:34.260579 STDOUT 2327] | self.W.initialize(W_shape)
2019-10-07 10:46:34.260581 STDOUT 2327] | usr/local/lib/python3.5/dist-packages/chainer/variable.py:1799: in initialize
2019-10-07 10:46:34.260582 STDOUT 2327] | self.initializer, shape, xp, device=device)
2019-10-07 10:46:34.260584 STDOUT 2327] | usr/local/lib/python3.5/dist-packages/chainer/initializers/__init__.py:69: in generate_array
2019-10-07 10:46:34.260585 STDOUT 2327] | initializer(array)
2019-10-07 10:46:34.260586 STDOUT 2327] | usr/local/lib/python3.5/dist-packages/chainer/initializers/constant.py:58: in __call__
2019-10-07 10:46:34.260587 STDOUT 2327] | array, self.fill_value.astype(array.dtype, copy=False))
2019-10-07 10:46:34.260588 STDOUT 2327] | cupy/core/core.pyx:318: in cupy.core.core.ndarray.astype
2019-10-07 10:46:34.260590 STDOUT 2327] | ???
2019-10-07 10:46:34.260598 STDOUT 2327] | cupy/core/core.pyx:389: in cupy.core.core.ndarray.astype
2019-10-07 10:46:34.260600 STDOUT 2327] | ???
2019-10-07 10:46:34.260601 STDOUT 2327] | cupy/core/_kernel.pyx:828: in cupy.core._kernel.ufunc.__call__
2019-10-07 10:46:34.260602 STDOUT 2327] | ???

chainer-ci · 2019-10-07T02:36:54Z

Jenkins CI test (for commit 9ea1ca8, target branch master) failed with status FAILURE.

emcastillo · 2019-10-29T03:20:12Z

Jenkins, test this please

chainer-ci · 2019-10-29T04:22:44Z

Jenkins CI test (for commit 8566494, target branch master) succeeded!

chainer-ci · 2019-12-05T14:50:44Z

Jenkins CI test (for commit 8566494, target branch master) succeeded!

niboshi assigned emcastillo Jun 17, 2019

crcrpar changed the title ~~Add from_params to Linear & Conv~~ [RFC] Add from_params to Linear & Conv Jun 17, 2019

crcrpar marked this pull request as ready for review August 10, 2019 01:00

crcrpar force-pushed the feature/from_params branch from 376d2d9 to b97903b Compare August 12, 2019 12:18

emcastillo reviewed Aug 19, 2019

View reviewed changes

emcastillo added cat:enhancement Implementation that does not break interfaces. st:awaiting-author State indicating that response is needed from contributors, often authors of pull request. labels Aug 19, 2019

emcastillo added this to the v7.0.0b3 milestone Aug 19, 2019

crcrpar force-pushed the feature/from_params branch from b97903b to 1324b04 Compare August 21, 2019 00:05

emcastillo previously approved these changes Aug 21, 2019

View reviewed changes

emcastillo reviewed Aug 21, 2019

View reviewed changes

asi1024 removed this from the v7.0.0b3 milestone Aug 22, 2019

crcrpar force-pushed the feature/from_params branch from d1003cc to ecdac19 Compare August 22, 2019 11:16

emcastillo reviewed Aug 23, 2019

View reviewed changes

emcastillo requested changes Aug 23, 2019

View reviewed changes

crcrpar added 7 commits October 2, 2019 17:01

add tests

cc69025

documentation

8795d86

Fix tests and groups (conv2d)

126a6d0

support N-D convolution

0156d0c

parameterize device tests

ed0dfc0

Move from_params test tp TestConvoluion?D

5ea24cc

fix syntax

5e28c5f

crcrpar force-pushed the feature/from_params branch from ecdac19 to 5e28c5f Compare October 2, 2019 08:02

crcrpar added 2 commits October 2, 2019 17:39

remove unused device parameterization

9c55e02

Initialize weight if not yet

2948e11

remote wrong use of to_device

cd1878c

crcrpar added 2 commits October 3, 2019 19:30

indent

83cab98

fix sphinx error

9ea1ca8

crcrpar requested a review from emcastillo October 4, 2019 10:42

emcastillo approved these changes Oct 7, 2019

View reviewed changes

crcrpar added 2 commits October 23, 2019 17:42

Merge branch 'master' into feature/from_params

35936a3

skip some tests

8566494

emcastillo added this to the v7.0.0 milestone Oct 31, 2019

emcastillo added st:test-and-merge State indicating that pull request is approved by a reviewer and can be merged after CI passes. and removed st:awaiting-author State indicating that response is needed from contributors, often authors of pull request. labels Oct 31, 2019

mergify bot merged commit e189205 into chainer:master Oct 31, 2019

emcastillo changed the title ~~[RFC] Add from_params to Linear & Conv~~ Add from_params to Linear & Conv Dec 5, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `from_params` to `Linear` & `Conv` #7525

Add `from_params` to `Linear` & `Conv` #7525

crcrpar commented Jun 16, 2019

crcrpar commented Jul 2, 2019 •

edited

Loading

emcastillo commented Jul 3, 2019

crcrpar commented Jul 3, 2019

emcastillo commented Jul 8, 2019

crcrpar commented Jul 10, 2019

crcrpar commented Aug 12, 2019

emcastillo Aug 19, 2019 •

edited

Loading

crcrpar Aug 19, 2019

emcastillo Aug 20, 2019

emcastillo left a comment •

edited

Loading

emcastillo left a comment •

edited

Loading

emcastillo Aug 21, 2019

crcrpar Aug 21, 2019

emcastillo commented Aug 22, 2019

emcastillo commented Aug 22, 2019

crcrpar commented Aug 22, 2019

emcastillo commented Aug 22, 2019

emcastillo Aug 23, 2019

crcrpar commented Aug 23, 2019

emcastillo Aug 23, 2019 •

edited

Loading

emcastillo Aug 23, 2019

emcastillo Aug 23, 2019

emcastillo Aug 23, 2019 •

edited

Loading

emcastillo commented Oct 3, 2019

crcrpar commented Oct 3, 2019

emcastillo left a comment

emcastillo commented Oct 7, 2019

crcrpar commented Oct 7, 2019

chainer-ci commented Oct 7, 2019

emcastillo commented Oct 29, 2019

chainer-ci commented Oct 29, 2019

chainer-ci commented Dec 5, 2019

Add from_params to Linear & Conv #7525

Add from_params to Linear & Conv #7525

Conversation

crcrpar commented Jun 16, 2019

crcrpar commented Jul 2, 2019 • edited Loading

emcastillo commented Jul 3, 2019

crcrpar commented Jul 3, 2019

emcastillo commented Jul 8, 2019

crcrpar commented Jul 10, 2019

crcrpar commented Aug 12, 2019

emcastillo Aug 19, 2019 • edited Loading

Choose a reason for hiding this comment

crcrpar Aug 19, 2019

Choose a reason for hiding this comment

emcastillo Aug 20, 2019

Choose a reason for hiding this comment

emcastillo left a comment • edited Loading

Choose a reason for hiding this comment

emcastillo left a comment • edited Loading

Choose a reason for hiding this comment

emcastillo Aug 21, 2019

Choose a reason for hiding this comment

crcrpar Aug 21, 2019

Choose a reason for hiding this comment

emcastillo commented Aug 22, 2019

emcastillo commented Aug 22, 2019

crcrpar commented Aug 22, 2019

emcastillo commented Aug 22, 2019

emcastillo Aug 23, 2019

Choose a reason for hiding this comment

crcrpar commented Aug 23, 2019

emcastillo Aug 23, 2019 • edited Loading

Choose a reason for hiding this comment

emcastillo Aug 23, 2019

Choose a reason for hiding this comment

emcastillo Aug 23, 2019

Choose a reason for hiding this comment

emcastillo Aug 23, 2019 • edited Loading

Choose a reason for hiding this comment

emcastillo commented Oct 3, 2019

crcrpar commented Oct 3, 2019

emcastillo left a comment

Choose a reason for hiding this comment

emcastillo commented Oct 7, 2019

crcrpar commented Oct 7, 2019

chainer-ci commented Oct 7, 2019

emcastillo commented Oct 29, 2019

chainer-ci commented Oct 29, 2019

chainer-ci commented Dec 5, 2019

Add `from_params` to `Linear` & `Conv` #7525

Add `from_params` to `Linear` & `Conv` #7525

crcrpar commented Jul 2, 2019 •

edited

Loading

emcastillo Aug 19, 2019 •

edited

Loading

emcastillo left a comment •

edited

Loading

emcastillo left a comment •

edited

Loading

emcastillo Aug 23, 2019 •

edited

Loading

emcastillo Aug 23, 2019 •

edited

Loading