Storage from_file method #1821

vlasenkov · 2017-06-16T12:38:30Z

Resolves #884

torch/csrc/generic/StorageMethods.cpp

+    return NULL;
+  }
+#if defined(__APPLE__)
+  shared_mem = 0;


torch/csrc/generic/StorageMethods.cpp

+  const char *filename;
+  Py_ssize_t size = 0;
+  int shared = 0, shared_mem = 0;
+  static char *kwlist[] = {"filename", "shared", "size", "shared_mem", NULL};


torch/csrc/generic/StorageMethods.cpp

+  Py_ssize_t size = 0;
+  int shared = 0, shared_mem = 0;
+  static char *kwlist[] = {"filename", "shared", "size", "shared_mem", NULL};
+  if (!PyArg_ParseTupleAndKeywords(args, keywds, "s|pnp", kwlist,


test/test_torch.py

+        def check(filename, share_mem):
+            s1 = torch.FloatStorage.from_file(filename, True, size, share_mem)
+            t1 = torch.FloatTensor(s1).copy_(torch.randn(size))
+            t1_copy = t1.new().resize_as_(t1).copy_(t1)


test/test_torch.py

+            s2 = torch.FloatStorage.from_file(filename, True, size)
+            t2 = torch.FloatTensor(s2)
+            t2_copy = t2.new().resize_as_(t2).copy_(t2)
+            self.assertEqual(t1_copy, t2_copy, 0)


test/test_torch.py

+            self.assertEqual(t1, t2, 0)
+
+            if sys.startswith('win'):
+                os.remove(filename)


test/test_torch.py

+    def test_from_file(self):
+        size = 10000
+        filename = 'testPytorchStorageFromFile'
+        sys = platform.system()


test/test_torch.py

+
+        # check shm_open + open
+        if sys != 'Darwin':  # OS X has no /dev/shm
+            check(filename, True)


test/test_torch.py

+            t1 = torch.FloatTensor(s1).copy_(torch.randn(size))
+            t1_copy = t1.new().resize_as_(t1).copy_(t1)
+            if share_mem and sys == 'Linux':
+                filename = os.path.join('/dev/shm', filename)


apaszke · 2017-06-16T13:43:29Z

Actually now I think it should use libshm even now. Otherwise the unlinking semantics are going to change, and later commits will be breaking.

vlasenkov · 2017-06-16T13:45:48Z

So, I'm removing shared_mem for now to save at least ordinary mmap functionality

apaszke · 2017-06-16T13:51:15Z

Sounds good

vlasenkov · 2017-06-16T13:54:20Z

where should be Storage docs stored? In the _torch_docs.py, or some new _storage_docs.py?

apaszke · 2017-06-16T13:56:21Z

I think you need to create _storage_docs.py. We only document functions implemented in Python at the moment

vlasenkov · 2017-06-16T16:34:41Z

@apaszke, I've added docs. ready for review/merge

apaszke · 2017-06-16T22:34:37Z

Thank you!

…2b732d (pytorch#17264) Summary: Pull Request resolved: pytorch#17264 Previous import was 822d8df0a2a32233c6022f50a158817a0f19bdc7 Included changes: - **[4c091e0](onnx/onnx@4c091e0)**: Support defined ONNX_ML in parent cmake files (pytorch#1821) <Lu Fang> - **[57372f3](onnx/onnx@57372f3)**: Delete OpsetVersionConverter.md which is a duplicate of VersionConverter.md (pytorch#1818) <Prasanth Pulavarthi> - **[ab1c57e](onnx/onnx@ab1c57e)**: [ONNXIFI]Add extension to be implementable (pytorch#1796) <Rui Zhu> - **[b92eee8](onnx/onnx@b92eee8)**: Revert "Implement Op Annotation's for ONNX (pytorch#1648)" (pytorch#1812) <Ke Zhang> - **[61f1e9e](onnx/onnx@61f1e9e)**: Enable ONNX_ML by default (pytorch#1810) <Shinichiro Hamaji> - **[4f064a1](onnx/onnx@4f064a1)**: fix Greater and Less doc (pytorch#1811) <Guoliang Hua> - **[0628582](onnx/onnx@0628582)**: Implement Op Annotation's for ONNX (pytorch#1648) <Armen> - **[ad9d2f7](onnx/onnx@ad9d2f7)**: Versioning doc update for Opset 9 (pytorch#1805) <Vinitra Swamy> - **[e71e3be](onnx/onnx@e71e3be)**: add dilation case for ConvTranspose op (pytorch#1797) <Randy> Differential Revision: D14135024 fbshipit-source-id: 9ee7d39a5efea5d9b3c12ac3e6acc32ae83c1d0e

…2b732d (pytorch#17264) Summary: Pull Request resolved: pytorch#17264 Previous import was 822d8df0a2a32233c6022f50a158817a0f19bdc7 Included changes: - **[4c091e0](onnx/onnx@4c091e0)**: Support defined ONNX_ML in parent cmake files (pytorch#1821) <Lu Fang> - **[57372f3](onnx/onnx@57372f3)**: Delete OpsetVersionConverter.md which is a duplicate of VersionConverter.md (pytorch#1818) <Prasanth Pulavarthi> - **[ab1c57e](onnx/onnx@ab1c57e)**: [ONNXIFI]Add extension to be implementable (pytorch#1796) <Rui Zhu> - **[b92eee8](onnx/onnx@b92eee8)**: Revert "Implement Op Annotation's for ONNX (pytorch#1648)" (pytorch#1812) <Ke Zhang> - **[61f1e9e](onnx/onnx@61f1e9e)**: Enable ONNX_ML by default (pytorch#1810) <Shinichiro Hamaji> - **[4f064a1](onnx/onnx@4f064a1)**: fix Greater and Less doc (pytorch#1811) <Guoliang Hua> - **[0628582](onnx/onnx@0628582)**: Implement Op Annotation's for ONNX (pytorch#1648) <Armen> - **[ad9d2f7](onnx/onnx@ad9d2f7)**: Versioning doc update for Opset 9 (pytorch#1805) <Vinitra Swamy> - **[e71e3be](onnx/onnx@e71e3be)**: add dilation case for ConvTranspose op (pytorch#1797) <Randy> Reviewed By: yinghai Differential Revision: D14135024 fbshipit-source-id: 1e4f9dda89abf48994798d080dd5d58207a6e4b6

vlasenkov added 2 commits June 15, 2017 20:48

storage.from_file

35173d3

tests

8a1e4b0

apaszke reviewed Jun 16, 2017

View reviewed changes

soumith added the in progress label Jun 16, 2017

rm shm

1675898

vlasenkov added 4 commits June 16, 2017 17:42

storage docs

68c7b6a

storage docs

6a11203

storage docs

8dea581

fix flake8 errors

8904d95

apaszke approved these changes Jun 16, 2017

View reviewed changes

apaszke merged commit 3cecdf8 into pytorch:master Jun 16, 2017

soumith removed the in progress label Jun 16, 2017

vlasenkov deleted the storage-from-file branch July 16, 2017 13:30

houseroad mentioned this pull request Feb 19, 2019

Automatic update of fbcode/onnx to 4c091e048ca42682d63ccd3c1811560bc12b732d #17264

Closed

ezyang added the open source label Jun 24, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Storage from_file method #1821

Storage from_file method #1821

vlasenkov commented Jun 16, 2017

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

apaszke commented Jun 16, 2017

vlasenkov commented Jun 16, 2017

apaszke commented Jun 16, 2017

vlasenkov commented Jun 16, 2017

apaszke commented Jun 16, 2017

vlasenkov commented Jun 16, 2017

apaszke commented Jun 16, 2017

Storage from_file method #1821

Storage from_file method #1821

Conversation

vlasenkov commented Jun 16, 2017

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

apaszke commented Jun 16, 2017

vlasenkov commented Jun 16, 2017

apaszke commented Jun 16, 2017

vlasenkov commented Jun 16, 2017

apaszke commented Jun 16, 2017

vlasenkov commented Jun 16, 2017

apaszke commented Jun 16, 2017