Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

added nn.RecordBytesDecoder with its test #5475

Merged
merged 51 commits into from
Aug 3, 2021
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
91d46d9
add RecordBytesDecoder
Kevin-XiongC Jul 13, 2021
ef49841
Merge branch 'master' into dev-bytedecoder
Kevin-XiongC Jul 13, 2021
d15ce0b
Merge branch 'master' into dev-bytedecoder
Kevin-XiongC Jul 13, 2021
20e1b9d
Merge branch 'master' into dev-bytedecoder
Kevin-XiongC Jul 14, 2021
530a70f
add docs for OFRecordBytesDecoder
Kevin-XiongC Jul 14, 2021
315650b
Merge branch 'master' into dev-bytedecoder
Kevin-XiongC Jul 14, 2021
762f142
Merge branch 'master' into dev-bytedecoder
Kevin-XiongC Jul 14, 2021
2fa054c
Merge branch 'master' into dev-bytedecoder
Kevin-XiongC Jul 15, 2021
130b098
Merge branch 'master' into dev-bytedecoder
Kevin-XiongC Jul 15, 2021
406719f
update
Kevin-XiongC Jul 15, 2021
25a24f8
Merge branch 'dev-bytedecoder' of https://github.com/Oneflow-Inc/onef…
Kevin-XiongC Jul 15, 2021
66645b2
Merge branch 'master' into dev-bytedecoder
Kevin-XiongC Jul 16, 2021
9dfd781
Merge branch 'master' into dev-bytedecoder
Kevin-XiongC Jul 19, 2021
17910cf
Merge branch 'master' into dev-bytedecoder
Kevin-XiongC Jul 19, 2021
78ea746
Merge branch 'master' into dev-bytedecoder
Kevin-XiongC Jul 21, 2021
02f1ace
Merge branch 'master' into dev-bytedecoder
Kevin-XiongC Jul 22, 2021
e7384bb
Merge branch 'master' into dev-bytedecoder
Kevin-XiongC Jul 22, 2021
ddb4132
Merge branch 'master' into dev-bytedecoder
Kevin-XiongC Jul 23, 2021
3dc2ecc
Merge branch 'master' into dev-bytedecoder
Kevin-XiongC Jul 23, 2021
49c0505
Merge branch 'master' into dev-bytedecoder
Kevin-XiongC Jul 23, 2021
0ae3078
reformat the code
Kevin-XiongC Jul 23, 2021
c444542
resovlved conflits
Kevin-XiongC Jul 27, 2021
7c1b671
resolve conflicts
Kevin-XiongC Jul 27, 2021
5277d04
resolving conflicts
Kevin-XiongC Jul 27, 2021
e85f4e5
Merge branch 'master' into dev-bytedecoder
Kevin-XiongC Jul 27, 2021
840e793
Merge branch 'master' into dev-bytedecoder
Kevin-XiongC Jul 27, 2021
8a4d6fb
Merge branch 'master' into dev-bytedecoder
Kevin-XiongC Jul 27, 2021
71de7a5
format the code
Kevin-XiongC Jul 27, 2021
b620383
Delete randperm.py
Kevin-XiongC Jul 27, 2021
3c9c0c7
Merge branch 'master' into dev-bytedecoder
Kevin-XiongC Jul 27, 2021
8167831
Merge branch 'master' into dev-bytedecoder
Kevin-XiongC Jul 28, 2021
2767351
Merge branch 'master' into dev-bytedecoder
Kevin-XiongC Jul 31, 2021
2c7e12b
format
Kevin-XiongC Jul 31, 2021
fabcf63
Merge branch 'master' into dev-bytedecoder
oneflow-ci-bot Aug 1, 2021
5e2c5d2
auto format by CI
oneflow-ci-bot Aug 1, 2021
c4128dc
Update test_dataset.py
Kevin-XiongC Aug 2, 2021
8f0c15d
Merge branch 'master' into dev-bytedecoder
oneflow-ci-bot Aug 2, 2021
fb22fbb
auto format by CI
oneflow-ci-bot Aug 2, 2021
da5c7ca
Merge branch 'master' into dev-bytedecoder
oneflow-ci-bot Aug 2, 2021
73960ab
Merge branch 'master' into dev-bytedecoder
oneflow-ci-bot Aug 2, 2021
865ab3e
The path in doctest fixed?
Kevin-XiongC Aug 2, 2021
66855f9
Merge branch 'master' into dev-bytedecoder
oneflow-ci-bot Aug 2, 2021
179f11f
Update test_dataset.py
Kevin-XiongC Aug 2, 2021
f12a922
Merge branch 'master' into dev-bytedecoder
oneflow-ci-bot Aug 2, 2021
f7e1eed
auto format by CI
oneflow-ci-bot Aug 2, 2021
0cee9eb
Merge branch 'master' into dev-bytedecoder
oneflow-ci-bot Aug 2, 2021
96a0ce1
Merge branch 'master' into dev-bytedecoder
oneflow-ci-bot Aug 2, 2021
8afce93
Merge branch 'master' into dev-bytedecoder
oneflow-ci-bot Aug 2, 2021
e18451d
refine doctest
Kevin-XiongC Aug 3, 2021
0efc7df
refine doctest
Kevin-XiongC Aug 3, 2021
1545654
Merge branch 'master' into dev-bytedecoder
oneflow-ci-bot Aug 3, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions oneflow/python/nn/modules/dataset.py
Original file line number Diff line number Diff line change
Expand Up @@ -621,6 +621,21 @@ def __init__(self, shape: Sequence[int], dtype: flow.dtype, alignment: int):
def forward(self, input):
return self._op(input)[0]

@oneflow_export("nn.OFRecordBytesDecoder")
duchengyao marked this conversation as resolved.
Show resolved Hide resolved
@experimental_api
class OFRecordBytesDecoder(Module):
def __init__(self, blob_name: str,name: Optional[str] = None):
super().__init__()
self._op = (
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

可能需要移到functor。不太确定,帮你加另一个reviewer

flow.builtin_op("ofrecord_bytes_decoder",name)
.Input("in")
.Output("out")
.Attr("name", blob_name)
.Build()
)

def forward(self, input):
return self._op(input)[0]

if __name__ == "__main__":
import doctest
Expand Down
24 changes: 24 additions & 0 deletions oneflow/python/test/modules/test_dataset.py
Original file line number Diff line number Diff line change
Expand Up @@ -315,6 +315,30 @@ def test_coco_reader(test_case):
np.array_equal(segm_index_list[i], sample["poly_index"])
)

@flow.unittest.skip_unless_1n1d()
@unittest.skipIf(
not flow.unittest.env.eager_execution_enabled(),
".numpy() doesn't work in lazy mode",
)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

去掉空行

class TestOFRecordBytesDecoder(flow.unittest.TestCase):
def test_OFRecordBytesDecoder(test_case):
batch_size = 16
record_reader = flow.nn.OfrecordReader(
"dataset/imagenette/ofrecord",
batch_size=batch_size,
part_name_suffix_length=5,
)
val_record = record_reader()

bytesdecoder_img = flow.nn.OFRecordBytesDecoder("encoded")

image_raw_buffer = bytesdecoder_img(val_record)

image_raw_buffer_nd = image_raw_buffer.numpy()[0]
gt_np = cv2.imread("dataset/imagenette/ofrecord/gt_tensor_buffer_image.png")
img = cv2.imdecode(image_raw_buffer_nd,cv2.IMREAD_COLOR)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

在build目录下make of_format

test_case.assertTrue(np.array_equal(img, gt_np))

if __name__ == "__main__":
unittest.main()