Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Dataset refactor #807

Merged
merged 34 commits into from Mar 22, 2024
Merged

Dataset refactor #807

merged 34 commits into from Mar 22, 2024

Conversation

wangxingjun778
Copy link
Collaborator

  1. Add compatible HF datasets wrapper
  2. Add compatible HF file utils wrapper
  3. Add create dataset func
  4. Add APIs: list_repo_tree, get_dataset_infos
  5. Fix read_csv and to_csv escapechar

@wangxingjun778
update
e34c638
@wangxingjun778
ok Merge branch 'master' of github.com:modelscope/modelscope into dev…
91f3481 
…/dataset_refactor
@wangxingjun778
add main entry in ms_dataset
2f0d702
@wangxingjun778
add main entry in ms_dataset
3c77cca
@wangxingjun778
update func get_data_patterns import
64e0aa4
@wangxingjun778
modify return_config_only
c10fda5
@wangxingjun778
update
f57a2c9
@wangxingjun778
update
76475c5
@wangxingjun778
update
f90d214
@wangxingjun778
modify return_config_only to dataset_info_only
32e21da
@wangxingjun778
udpate version for test
732984f
@wangxingjun778
del get_logger(__name__)
b978553
@wangxingjun778
fix py script loading
a130f0b
@wangxingjun778
fix loading py and without py
b99985b
@wangxingjun778
add subset support
d7038a3
@wangxingjun778
add hf_datasets_util; refine list_repo_tree_ms; fix private datasets …
ecacc30 
…loading issue
@wangxingjun778
update version to rc5
948bed6
@wangxingjun778
fix and support preview for dataset_info_only mode
7cb839b
@wangxingjun778
fix urlencode
fc13737
@wangxingjun778
update to rc7
53b6382
@wangxingjun778
1. loading of dataset_infos.json is deprecated; 2. add some ut
b5f75ca
@wangxingjun778
update version
1f8cda0
@wangxingjun778
add escapechar for read_csv and to_csv
014770a
@wangxingjun778
add params: Source=SDK
c1f7dad
@wangxingjun778
add create_dataset func
5b97201
@wangxingjun778
overwrite _get_paths_info
ecddc5a
@wangxingjun778
ok Merge branch 'master' of github.com:modelscope/modelscope into dev…
a4f4584 
…/dataset_refactor
@wangxingjun778
update & version
8830a07
@wangxingjun778
update
fb3a025
@wangxingjun778
update list_repo_tree name
02e48b1
liuyhwangyh
liuyhwangyh reviewed Mar 22, 2024
View reviewed changes
modelscope/hub/api.py Outdated Show resolved Hide resolved
@wenmengzhou wenmengzhou merged commit 1a66f06 into master Mar 22, 2024
1 of 3 checks passed
tastelikefeet added a commit that referenced this pull request Mar 22, 2024
@tastelikefeet
Merge commit '1a66f069c432e43865775affbb23f77ee5c52f53' into release/…
9247ff4 
…1.13

* commit '1a66f069c432e43865775affbb23f77ee5c52f53':
  Dataset refactor (#807)
  fix download file with spical name as 'Image+Title.png' (#805)
  Fix error "modelscope attributeerror: 'dict' object has no attribute 'task_name' " (#800)
  move doc to classroom (#802)
  add ViViT-demo (#796)
  chore: Formal LICENSE content (#799)
  upload marigold monocular depth estimation core files (#703)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@liuyhwangyh liuyhwangyh liuyhwangyh left review comments

@wenmengzhou wenmengzhou wenmengzhou approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@wangxingjun778 @liuyhwangyh @wenmengzhou