feat: public api for dataset #179

Jiei-S · 2023-08-31T02:45:47Z

概要

データセットをPython SDKから登録できる - dataset（tag, visibility）
https://github.com/fastlabel/roadmap/issues/2

申し送り

dataset_objects 関連の API は後続の Issue で対応（仕様 Fix 待ちのため本件先行して進める）
organization 関連の仕様は対応しない（現時点でプラットフォーム側でサポートされていないため）

テスト

Create

正常 - tags（通常）、visibility（workspace）

dataset = client.create_dataset(
    name="object-detection5",
    license="The MIT License",
    tags=["cat", "dog"],
    visibility="workspace",
)

{'createdAt': '2023-09-01T03:16:33.110Z',
 'id': '54b759d2-b273-42e7-976e-6748a765576d',
 'license': 'The MIT License',
 'name': 'object-detection5',
 'tags': ['cat', 'dog'],
 'updatedAt': '2023-09-01T03:16:33.110Z',
 'visibility': 'workspace'}

正常 - tags（通常）、visibility（public）

dataset = client.create_dataset(
    name="object-detection6",
    license="The MIT License",
    tags=["cat", "dog"],
    visibility="public",
)

{'createdAt': '2023-09-01T03:22:03.953Z',
 'id': '1b08cb78-70e6-4767-ac4c-48559b6268a9',
 'license': 'The MIT License',
 'name': 'object-detection6',
 'tags': ['cat', 'dog'],
 'updatedAt': '2023-09-01T03:22:03.953Z',
 'visibility': 'public'}

正常 - tags（なし）、visibility（なし）

dataset = client.create_dataset(
    name="object-detection7",
    license="The MIT License",
)

{'createdAt': '2023-09-01T03:26:08.207Z',
 'id': '512c4041-4963-4223-bd7b-791eb13d06c2',
 'license': 'The MIT License',
 'name': 'object-detection7',
 'tags': [],
 'updatedAt': '2023-09-01T03:26:08.207Z',
 'visibility': 'workspace'}

異常 - tags（文字数超過）

dataset = client.create_dataset(
    name="object-detection8",
    license="The MIT License",
    tags=["cat", "dogdogdogdogdogdogdog"],
)

fastlabel.exceptions.FastLabelInvalidException: <Response [400]> Dataset tag is too long. Please use less than 20 characters.

異常 - tags（件数超過）

dataset = client.create_dataset(
    name="object-detection8",
    license="The MIT License",
    tags=["motorcycle", "airplane", "car", "truck", "bus", "bicycle"],
)

fastlabel.exceptions.FastLabelInvalidException: <Response [400]> Dataset tags are too many. Please use less than 5 tags.

異常 - tags（重複）

dataset = client.create_dataset(
    name="object-detection8",
    license="The MIT License",
    tags=["motorcycle", "airplane", "car", "bus", "bus"],
)

fastlabel.exceptions.FastLabelInvalidException: <Response [409]> Duplicate error.

異常 - visibility（値不正）

dataset = client.create_dataset(
    name="object-detection8",
    license="The MIT License",
    visibility="test",
)

fastlabel.exceptions.FastLabelInvalidException: <Response [400]> {'params.visibility': {'message': 'Could not match the union against any of the items. Issues: [{"params.visibility":{"message":"should be one of the following; [\'workspace\']","value":"test"}},{"params.visibility":{"message":"should be one of the following; [\'public\']","value":"test"}}]', 'value': 'test'}}

Find

正常

dataset = client.find_dataset(dataset_id="b6e9f16a-b234-45ef-9bca-60400532495c")

{'createdAt': '2023-09-01T03:49:52.535Z',
 'id': 'b6e9f16a-b234-45ef-9bca-60400532495c',
 'license': 'The MIT License',
 'name': 'object-detection8',
 'tags': ['airplane', 'bus', 'car', 'motorcycle', 'truck'],
 'updatedAt': '2023-09-01T03:49:52.535Z',
 'visibility': 'workspace'}

異常 - Not Found

dataset = client.find_dataset(dataset_id="test")

fastlabel.exceptions.FastLabelInvalidException: <Response [404]> Not found error.

Get

正常 - フィルタなし

datasets = client.get_datasets()

[{'createdAt': '2023-09-01T03:49:52.535Z',
  'id': 'b6e9f16a-b234-45ef-9bca-60400532495c',
  'license': 'The MIT License',
  'name': 'object-detection8',
  'tags': ['airplane', 'bus', 'car', 'motorcycle', 'truck'],
  'updatedAt': '2023-09-01T03:49:52.535Z',
  'visibility': 'workspace'},
 {'createdAt': '2023-09-01T03:26:08.207Z',
  'id': '512c4041-4963-4223-bd7b-791eb13d06c2',
  'license': 'The MIT License',
  'name': 'object-detection7',
  'tags': [],
  'updatedAt': '2023-09-01T03:26:08.207Z',
  'visibility': 'workspace'},
 {'createdAt': '2023-09-01T03:22:03.953Z',
  'id': '1b08cb78-70e6-4767-ac4c-48559b6268a9',
  'license': 'The MIT License',
  'name': 'object-detection6',
  'tags': ['cat', 'dog'],
  'updatedAt': '2023-09-01T03:22:03.953Z',
  'visibility': 'public'},
 {'createdAt': '2023-09-01T03:16:33.110Z',
  'id': '54b759d2-b273-42e7-976e-6748a765576d',
  'license': 'The MIT License',
  'name': 'object-detection5',
  'tags': ['cat', 'dog'],
  'updatedAt': '2023-09-01T03:16:33.110Z',
  'visibility': 'workspace'},
 {'createdAt': '2023-09-01T02:57:42.404Z',
  'id': '5c6f3f11-6fe2-4cc1-a2d5-5da989db2b06',
  'name': 'test4',
  'tags': [],
  'updatedAt': '2023-09-01T02:57:42.404Z',
  'visibility': 'workspace'},
 {'createdAt': '2023-09-01T02:57:27.066Z',
  'id': 'a90a6542-8ba4-4098-a165-bd682fcdef9b',
  'license': 'Attribution Assurance License',
  'name': 'test3',
  'tags': ['test3'],
  'updatedAt': '2023-09-01T02:57:27.066Z',
  'visibility': 'workspace'},
 {'createdAt': '2023-09-01T02:41:34.470Z',
  'id': 'eb3a13fd-1ba4-467c-add2-b59743b5d0d3',
  'name': 'test2',
  'tags': [],
  'updatedAt': '2023-09-01T02:41:34.470Z',
  'visibility': 'workspace'},
 {'createdAt': '2023-09-01T02:41:20.289Z',
  'id': '5b9c9d9b-2b26-4de3-a31d-6c24953463c8',
  'license': '1-clause BSD License',
  'name': 'test1',
  'tags': ['test1'],
  'updatedAt': '2023-09-01T02:41:20.289Z',
  'visibility': 'public'}]

正常 - フィルタ（tags）ヒットあり

datasets = client.get_datasets(
    tags=["airplane", "test1"],
)

[{'createdAt': '2023-09-01T03:49:52.535Z',
  'id': 'b6e9f16a-b234-45ef-9bca-60400532495c',
  'license': 'The MIT License',
  'name': 'object-detection8',
  'tags': ['airplane', 'bus', 'car', 'motorcycle', 'truck'],
  'updatedAt': '2023-09-01T03:49:52.535Z',
  'visibility': 'workspace'},
 {'createdAt': '2023-09-01T02:41:20.289Z',
  'id': '5b9c9d9b-2b26-4de3-a31d-6c24953463c8',
  'license': '1-clause BSD License',
  'name': 'test1',
  'tags': ['test1'],
  'updatedAt': '2023-09-01T02:41:20.289Z',
  'visibility': 'public'}]

正常 - フィルタ（tags）ヒットなし

datasets = client.get_datasets(
    tags=["あいうえお"],
)

[]

正常 - フィルタ（visibility）ヒットあり

datasets = client.get_datasets(
    visibility="public",
)

[{'createdAt': '2023-09-01T03:22:03.953Z',
  'id': '1b08cb78-70e6-4767-ac4c-48559b6268a9',
  'license': 'The MIT License',
  'name': 'object-detection6',
  'tags': ['cat', 'dog'],
  'updatedAt': '2023-09-01T03:22:03.953Z',
  'visibility': 'public'},
 {'createdAt': '2023-09-01T02:41:20.289Z',
  'id': '5b9c9d9b-2b26-4de3-a31d-6c24953463c8',
  'license': '1-clause BSD License',
  'name': 'test1',
  'tags': ['test1'],
  'updatedAt': '2023-09-01T02:41:20.289Z',
  'visibility': 'public'}]

datasets = client.get_datasets(
    visibility="workspace",
)

[{'createdAt': '2023-09-01T03:49:52.535Z',
  'id': 'b6e9f16a-b234-45ef-9bca-60400532495c',
  'license': 'The MIT License',
  'name': 'object-detection8',
  'tags': ['airplane', 'bus', 'car', 'motorcycle', 'truck'],
  'updatedAt': '2023-09-01T03:49:52.535Z',
  'visibility': 'workspace'},
 {'createdAt': '2023-09-01T03:26:08.207Z',
  'id': '512c4041-4963-4223-bd7b-791eb13d06c2',
  'license': 'The MIT License',
  'name': 'object-detection7',
  'tags': [],
  'updatedAt': '2023-09-01T03:26:08.207Z',
  'visibility': 'workspace'},
 {'createdAt': '2023-09-01T03:16:33.110Z',
  'id': '54b759d2-b273-42e7-976e-6748a765576d',
  'license': 'The MIT License',
  'name': 'object-detection5',
  'tags': ['cat', 'dog'],
  'updatedAt': '2023-09-01T03:16:33.110Z',
  'visibility': 'workspace'},
 {'createdAt': '2023-09-01T02:57:42.404Z',
  'id': '5c6f3f11-6fe2-4cc1-a2d5-5da989db2b06',
  'name': 'test4',
  'tags': [],
  'updatedAt': '2023-09-01T02:57:42.404Z',
  'visibility': 'workspace'},
 {'createdAt': '2023-09-01T02:57:27.066Z',
  'id': 'a90a6542-8ba4-4098-a165-bd682fcdef9b',
  'license': 'Attribution Assurance License',
  'name': 'test3',
  'tags': ['test3'],
  'updatedAt': '2023-09-01T02:57:27.066Z',
  'visibility': 'workspace'},
 {'createdAt': '2023-09-01T02:41:34.470Z',
  'id': 'eb3a13fd-1ba4-467c-add2-b59743b5d0d3',
  'name': 'test2',
  'tags': [],
  'updatedAt': '2023-09-01T02:41:34.470Z',
  'visibility': 'workspace'}]

異常 - フィルタ不正（visibility）

datasets = client.get_datasets(
    visibility="test",
)

fastlabel.exceptions.FastLabelInvalidException: <Response [400]>

正常 - フィルタ（keyword）ヒットあり

datasets = client.get_datasets(
    keyword="test1",
)

[{'createdAt': '2023-09-01T02:41:20.289Z',
  'id': '5b9c9d9b-2b26-4de3-a31d-6c24953463c8',
  'license': '1-clause BSD License',
  'name': 'test1',
  'tags': ['test1'],
  'updatedAt': '2023-09-01T02:41:20.289Z',
  'visibility': 'public'}]

正常 - フィルタ（keyword）ヒットなし

datasets = client.get_datasets(
    keyword="あいうえお",
)

[]

正常 - フィルタ（マルチ）ヒットなし

datasets = client.get_datasets(
    keyword="あいうえお",
    tags=["cat"],
    visibility="workspace",
)

datasets = client.get_datasets(
    keyword="object-detection5",
    tags=["test"],
    visibility="workspace",
)

datasets = client.get_datasets(
    keyword="object-detection5",
    tags=["cat"],
    visibility="public",
)

[]

正常 - フィルタ（マルチ）全て一致

datasets = client.get_datasets(
    keyword="object-detection5",
    tags=["cat"],
    visibility="workspace",
)

[{'createdAt': '2023-09-01T03:16:33.110Z',
  'id': '54b759d2-b273-42e7-976e-6748a765576d',
  'license': 'The MIT License',
  'name': 'object-detection5',
  'tags': ['cat', 'dog'],
  'updatedAt': '2023-09-01T03:16:33.110Z',
  'visibility': 'workspace'}]

正常 - フィルタ（マルチ）一部一致

datasets = client.get_datasets(
    tags=["cat", "dog", "airplane"], # object-detection5, object-detection6, object-detection8 一致
    visibility="public", # object-detection6 一致
)

[{'createdAt': '2023-09-01T03:22:03.953Z',
  'id': '1b08cb78-70e6-4767-ac4c-48559b6268a9',
  'license': 'The MIT License',
  'name': 'object-detection6',
  'tags': ['cat', 'dog'],
  'updatedAt': '2023-09-01T03:22:03.953Z',
  'visibility': 'public'}]

datasets = client.get_datasets(
    tags=["cat", "dog", "airplane"], # object-detection5, object-detection6, object-detection8 一致
    visibility="workspace", # object-detection5, object-detection8 一致
)

[{'createdAt': '2023-09-01T03:49:52.535Z',
  'id': 'b6e9f16a-b234-45ef-9bca-60400532495c',
  'license': 'The MIT License',
  'name': 'object-detection8',
  'tags': ['airplane', 'bus', 'car', 'motorcycle', 'truck'],
  'updatedAt': '2023-09-01T03:49:52.535Z',
  'visibility': 'workspace'},
 {'createdAt': '2023-09-01T03:16:33.110Z',
  'id': '54b759d2-b273-42e7-976e-6748a765576d',
  'license': 'The MIT License',
  'name': 'object-detection5',
  'tags': ['cat', 'dog'],
  'updatedAt': '2023-09-01T03:16:33.110Z',
  'visibility': 'workspace'}]

Update

dataset = client.update_dataset(
    dataset_id="1b08cb78-70e6-4767-ac4c-48559b6268a9",
    name="object-detection6-update",
    tags=["update"],
)

{'createdAt': '2023-09-01T03:22:03.953Z',
 'id': '1b08cb78-70e6-4767-ac4c-48559b6268a9',
 'license': 'The MIT License',
 'name': 'object-detection6-update',
 'tags': ['update'],
 'updatedAt': '2023-09-01T04:54:47.000Z',
 'visibility': 'public'}

（空の場合 ignore）

dataset = client.update_dataset(
    dataset_id="1b08cb78-70e6-4767-ac4c-48559b6268a9",
    name="object-detection6-update",
    tags=[],
)

{'createdAt': '2023-09-01T03:22:03.953Z',
 'id': '1b08cb78-70e6-4767-ac4c-48559b6268a9',
 'license': 'The MIT License',
 'name': 'object-detection6-update',
 'tags': ['update'],
 'updatedAt': '2023-09-01T04:54:47.000Z',
 'visibility': 'public'}

Delete

client.delete_dataset(dataset_id="1b08cb78-70e6-4767-ac4c-48559b6268a9") # object-detection6-update

feat: dataset api - create

fda7ec6

Jiei-S self-assigned this Aug 31, 2023

Jiei-S added 4 commits August 31, 2023 11:51

feat: dataset api - update

f7048bf

feat: dataset api - get

335c706

feat: dataset api - resolve conflict

c952235

feat: dataset api - mod README

fd0881f

Jiei-S marked this pull request as ready for review September 1, 2023 05:28

Jiei-S requested a review from ueno-k-work September 1, 2023 05:28

feat: dataset api - remove unnecessary comment

c06a66b

ueno-k-work approved these changes Sep 4, 2023

View reviewed changes

ueno-k-work merged commit f480b4e into main Sep 4, 2023

ueno-k-work deleted the feature/dataset-api branch September 4, 2023 02:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: public api for dataset #179

feat: public api for dataset #179

Uh oh!

Jiei-S commented Aug 31, 2023 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

feat: public api for dataset #179

feat: public api for dataset #179

Uh oh!

Conversation

Jiei-S commented Aug 31, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

概要

申し送り

関連

テスト

Create

正常 - tags（通常）、visibility（workspace）

正常 - tags（通常）、visibility（public）

正常 - tags（なし）、visibility（なし）

異常 - tags（文字数超過）

異常 - tags（件数超過）

異常 - tags（重複）

異常 - visibility（値不正）

Find

正常

異常 - Not Found

Get

正常 - フィルタなし

正常 - フィルタ（tags）ヒットあり

正常 - フィルタ（tags）ヒットなし

正常 - フィルタ（visibility）ヒットあり

異常 - フィルタ不正（visibility）

正常 - フィルタ（keyword）ヒットあり

正常 - フィルタ（keyword）ヒットなし

正常 - フィルタ（マルチ）ヒットなし

正常 - フィルタ（マルチ）全て一致

正常 - フィルタ（マルチ）一部一致

Update

Delete

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Jiei-S commented Aug 31, 2023 •

edited

Loading