Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Dev flow.utils.data part1 #5406

Merged
merged 65 commits into from Jul 14, 2021
Merged

Dev flow.utils.data part1 #5406

merged 65 commits into from Jul 14, 2021

Conversation

Flowingsun007
Copy link
Contributor

@Flowingsun007 Flowingsun007 commented Jul 6, 2021 •
edited

dataloader最小实现（part1），原始pr见：#5074

  • 支持单进程dataloader，numpy/tensor Dataset的数据集加载；
  • test case直接在oneflow/oneflow/python/test/dataloader/下，运行python3 xxx.py即可

Flowingsun007 and others added 30 commits June 10, 2021 11:15
@Flowingsun007
refine and add test case
eb694ab
@Flowingsun007
Merge branch 'master' of https://github.com/Oneflow-Inc/oneflow
813f379
@Flowingsun007
Merge branch 'master' of https://github.com/Oneflow-Inc/oneflow
0aae06b
@Flowingsun007
Merge branch 'master' of https://github.com/Oneflow-Inc/oneflow
4a38ccd
@Flowingsun007
Merge branch 'master' of https://github.com/Oneflow-Inc/oneflow
b58b849
@Flowingsun007
Merge branch 'master' of https://github.com/Oneflow-Inc/oneflow
b5e151a
@Flowingsun007
support ellipsis type slice
0ff1651
@Flowingsun007
refine
1276e65
@Flowingsun007
refine
b9066f9
@Flowingsun007
support slice assign ellipsis type
8f81967
@Flowingsun007
refine
8f8cee2
@Flowingsun007
Merge branch 'master' into dev_fix_slice_bug
a79dcdf
@Flowingsun007
Merge branch 'master' of https://github.com/Oneflow-Inc/oneflow
81a400e
@Flowingsun007
Merge branch 'master' into dev_fix_slice_bug
e9e2fa4
@Flowingsun007
Merge branch 'master' of https://github.com/Oneflow-Inc/oneflow
dbdcf18
@Flowingsun007
Merge branch 'master' into dev_fix_slice_bug
b11243f
@oneflow-ci-bot
Merge branch 'master' into dev_fix_slice_bug
9c7185b
@Flowingsun007
register fn to localtensor
c8c78fb
@Flowingsun007
Merge branch 'dev_fix_slice_bug' of https://github.com/Oneflow-Inc/on…
f565929 
…eflow into dev_fix_slice_bug
@Flowingsun007
Merge branch 'dev_fix_slice_bug'
ebc25f0
@Flowingsun007
Merge branch 'master' of https://github.com/Oneflow-Inc/oneflow
475bbff
@Flowingsun007
Merge branch 'master' of https://github.com/Oneflow-Inc/oneflow
b69f554
@Flowingsun007
Merge branch 'master' of https://github.com/Oneflow-Inc/oneflow
e8cd9e3
@Flowingsun007
Merge branch 'master' of https://github.com/Oneflow-Inc/oneflow
a500a6d
@Flowingsun007
Merge branch 'master' of https://github.com/Oneflow-Inc/oneflow
5387b8f
@Flowingsun007
Merge branch 'master' of https://github.com/Oneflow-Inc/oneflow
756b0ed
@Flowingsun007
Merge branch 'master' of https://github.com/Oneflow-Inc/oneflow
34e9fd5
@Flowingsun007
Merge branch 'master' of https://github.com/Oneflow-Inc/oneflow
a5d67ac
@Flowingsun007
Merge branch 'master' of https://github.com/Oneflow-Inc/oneflow
e547b4b
@Flowingsun007
Merge branch 'master' of https://github.com/Oneflow-Inc/oneflow
756e537
wyg1997
wyg1997 reviewed Jul 14, 2021
View reviewed changes
oneflow/python/utils/data/dataloader.py Show resolved Hide resolved
oneflow/python/utils/data/dataloader.py Outdated Show resolved Hide resolved
oneflow/python/utils/data/dataset.py Outdated Show resolved Hide resolved
oneflow/python/utils/data/sampler.py Outdated Show resolved Hide resolved
wyg1997
wyg1997 approved these changes Jul 14, 2021
View reviewed changes
oneflow/python/test/dataloader/test_numpy_dataset.py


class ScpDataset(Data.Dataset):
def __init__(self, chunksize=200, dim=81, length=2000):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

length弄少一点？设置成10这种数量级的？

@Flowingsun007 Flowingsun007 requested review from oneflow-ci-bot and removed request for oneflow-ci-bot July 14, 2021 08:02
@oneflow-ci-bot oneflow-ci-bot removed their request for review July 14, 2021 08:26
@oneflow-ci-bot oneflow-ci-bot self-requested a review July 14, 2021 08:26
@oneflow-ci-bot oneflow-ci-bot requested review from oneflow-ci-bot and removed request for oneflow-ci-bot July 14, 2021 11:08
@oneflow-ci-bot oneflow-ci-bot requested review from oneflow-ci-bot and removed request for oneflow-ci-bot July 14, 2021 14:43
@oneflow-ci-bot oneflow-ci-bot merged commit 30a3727 into master Jul 14, 2021
@oneflow-ci-bot oneflow-ci-bot deleted the dev_flow.utils.data_part1 branch July 14, 2021 22:39
@chengtbf
Copy link
Contributor

  • test case直接在oneflow/oneflow/python/test/dataloader/下，运行python3 xxx.py即可

@Flowingsun007 @jackalcooper @wyg1997

test/dataloader 这个路径，是不是没有加到 CI 里，导致其实测试不了？其实 modules 下面有 test_dataset.py 测试了，可以放在那个里面。

@jackalcooper
Copy link
Collaborator

  • test case直接在oneflow/oneflow/python/test/dataloader/下，运行python3 xxx.py即可

@Flowingsun007 @jackalcooper @wyg1997

test/dataloader 这个路径，是不是没有加到 CI 里，导致其实测试不了？其实 modules 下面有 test_dataset.py 测试了，可以放在那个里面。

是的，ci不会运行这个目录下的脚本

@wyg1997
Copy link
Contributor

wyg1997 commented Jul 15, 2021

dataloader部分要单独分个文件夹吧，后面各种类型和参数的dataloader都要加单测的

@jackalcooper
Copy link
Collaborator

最近老出现这个问题，我之后加个ci步骤，不允许在test目录下随意增加子目录

@Flowingsun007
Copy link
Contributor Author

最近老出现这个问题，我之后加个ci步骤，不允许在test目录下随意增加子目录

这个dataloader应该要在test下独立目录吧？麻烦晟航帮忙操作下呗

@Flowingsun007 Flowingsun007 mentioned this pull request Jul 16, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Ldpe2G Ldpe2G Ldpe2G left review comments

@wyg1997 wyg1997 wyg1997 approved these changes

@leaves-zwx leaves-zwx Awaiting requested review from leaves-zwx

@ScXfjiang ScXfjiang Awaiting requested review from ScXfjiang

Assignees
No one assigned
Labels
automerge eager enhancement feature interface
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
6 participants
@Flowingsun007 @Ldpe2G @chengtbf @jackalcooper @wyg1997 @oneflow-ci-bot