Skip to content

[SN-120] Include examples of using composite mask in image projects #1431

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Feb 26, 2024
Merged

[SN-120] Include examples of using composite mask in image projects #1431

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
8ec7c52
update title
Feb 21, 2024
592c932
Include composite masks examples for the image notebook
Feb 21, 2024
d8fdf2b
Merge branch 'develop' into ao_sn120_updates_2
Feb 21, 2024
a1223c6
latest updates
Feb 21, 2024
24fc0a9
removed how to fetch single instance masks
Feb 22, 2024
3443557
incorporated Gabe's suggestions
Feb 26, 2024
b8e39ce
incorporated Gabe's suggestions
Feb 26, 2024
3eedc62
incorporated Gabe's suggestions
Feb 26, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
208 changes: 131 additions & 77 deletions examples/annotation_import/image.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,10 +85,12 @@
"metadata": {},
"source": [
"import uuid\n",
"from PIL import Image\n",
"import requests\n",
"import base64\n",
"import labelbox as lb\n",
"import labelbox.types as lb_types"
"import labelbox.types as lb_types\n",
"from io import BytesIO\n"
],
"cell_type": "code",
"outputs": [],
Expand Down Expand Up @@ -500,47 +502,27 @@
{
"metadata": {},
"source": [
"### Segmentation Mask"
"### Composite mask upload using different mask tools from the project's ontology\n",
"This example shows how to assigned different annotations (mask instances) from a composite mask using different mask tools"
],
"cell_type": "markdown"
},
{
"metadata": {},
"source": [
"### Raster Segmentation (Byte string array)\n",
"url = \"https://storage.googleapis.com/labelbox-datasets/image_sample_data/raster_seg.png\"\n",
"response = requests.get(url)\n",
"# First we need to extract all the unique colors from the composite mask\n",
"def extract_rgb_colors_from_url(image_url):\n",
" response = requests.get(image_url)\n",
" img = Image.open(BytesIO(response.content))\n",
"\n",
"mask_data = lb.types.MaskData(im_bytes=response.content) # You can also use \"url\" instead of img_bytes to pass the PNG mask url.\n",
"mask_annotation = lb_types.ObjectAnnotation(\n",
" name=\"mask\",\n",
" value=lb_types.Mask(\n",
" mask=mask_data,\n",
" color=(255, 255, 255))\n",
")\n",
"\n",
"# NDJSON using instanceURI, or bytes array.\n",
"mask_annotation_ndjson = {\n",
" \"name\": \"mask\",\n",
" \"classifications\": [],\n",
" \"mask\": {\n",
" \t\"instanceURI\": url,\n",
" \t\"colorRGB\": (255, 255, 255)\n",
" }\n",
"}\n",
"\n",
"#Using bytes array.\n",
"response = requests.get(url)\n",
"im_bytes = base64.b64encode(response.content).decode('utf-8')\n",
" colors = set()\n",
" for x in range(img.width):\n",
" for y in range(img.height):\n",
" pixel = img.getpixel((x, y))\n",
" if pixel[:3] != (0,0,0):\n",
" colors.add(pixel[:3]) # Get only the RGB values\n",
"\n",
"mask_annotation_ndjson = {\n",
" \"name\": \"mask\",\n",
" \"classifications\": [],\n",
" \"mask\": {\n",
" \t\"imBytes\": im_bytes,\n",
" \"colorRGB\": (255, 255, 255)\n",
" }\n",
" }"
" return colors"
],
"cell_type": "code",
"outputs": [],
Expand All @@ -549,40 +531,98 @@
{
"metadata": {},
"source": [
"### Segmentation mask with nested classification "
],
"cell_type": "markdown"
},
{
"metadata": {},
"source": [
"url_2 = \"https://storage.googleapis.com/labelbox-datasets/image_sample_data/raster_seg_with_subclass.png\"\n",
"response = requests.get(url_2)\n",
"mask_data = lb_types.MaskData(im_bytes=response.content)\n",
"\n",
"# Python annotation\n",
"mask_with_text_subclass_annotation = lb_types.ObjectAnnotation(\n",
" name = \"mask_with_text_subclass\", # must match your ontology feature\"s name\n",
" value=lb_types.Mask(\n",
" mask=mask_data,\n",
" color=(255, 255, 255)),\n",
" classifications=[\n",
" lb_types.ClassificationAnnotation(\n",
" name=\"sub_free_text\",\n",
" value=lb_types.Text(answer=\"free text answer\")\n",
" )]\n",
")\n",
"cp_mask_url = \"https://storage.googleapis.com/labelbox-datasets/image_sample_data/composite_mask.png\"\n",
"colors = extract_rgb_colors_from_url(cp_mask_url)\n",
"response = requests.get(cp_mask_url)\n",
"\n",
"# NDJSON using instanceURI, bytes array is not fully supported.\n",
"mask_with_text_subclass_ndjson = {\n",
" \"name\": \"mask_with_text_subclass\",\n",
" \"mask\": {\"instanceURI\": url_2,\n",
" \"colorRGB\": (255, 255, 255)},\n",
" \"classifications\":[{\n",
" \"name\": \"sub_free_text\",\n",
" \"answer\": \"free text answer\"\n",
" }]\n",
"}"
"mask_data = lb.types.MaskData(im_bytes=response.content) # You can also use \"url\" instead of img_bytes to pass the PNG mask url.\n",
"rgb_colors_for_mask_with_text_subclass_tool = [(73, 39, 85), (111, 87, 176), (23, 169, 254)]\n",
"\n",
"cp_mask = []\n",
"for color in colors:\n",
" # We are assigning the color related to the mask_with_text_subclass tool by identifying the unique RGB colors\n",
" if color in rgb_colors_for_mask_with_text_subclass_tool:\n",
" cp_mask.append(\n",
" lb_types.ObjectAnnotation(\n",
" name = \"mask_with_text_subclass\", # must match your ontology feature\"s name\n",
" value=lb_types.Mask(\n",
" mask=mask_data,\n",
" color=color),\n",
" classifications=[\n",
" lb_types.ClassificationAnnotation(\n",
" name=\"sub_free_text\",\n",
" value=lb_types.Text(answer=\"free text answer sample\")\n",
" )]\n",
" )\n",
" )\n",
" else:\n",
" # Create ObjectAnnotation for other masks\n",
" cp_mask.append(\n",
" lb_types.ObjectAnnotation(\n",
" name=\"mask\",\n",
" value=lb_types.Mask(\n",
" mask=mask_data,\n",
" color=color\n",
" )\n",
" )\n",
" )\n",
"\n",
"\n",
"# NDJSON using instanceURI, or bytes array - use one of the two options\n",
"cp_mask_ndjson = []\n",
"\n",
"for color in colors:\n",
" if color in rgb_colors_for_mask_with_text_subclass_tool:\n",
" cp_mask_ndjson.append({\n",
" \"name\": \"mask_with_text_subclass\",\n",
" \"mask\": {\"instanceURI\": cp_mask_url,\n",
" \"colorRGB\": color },\n",
" \"classifications\":[{\n",
" \"name\": \"sub_free_text\",\n",
" \"answer\": \"free text answer\"\n",
" }]\n",
" }\n",
" )\n",
" else:\n",
" cp_mask_ndjson.append({\n",
" \"name\": \"mask\",\n",
" \"classifications\": [],\n",
" \"mask\": {\n",
" \"instanceURI\": cp_mask_url,\n",
" \"colorRGB\": color\n",
" }\n",
" }\n",
" )\n",
"\n",
"\n",
"#Using bytes array.\n",
"response = requests.get(cp_mask_url)\n",
"im_bytes = base64.b64encode(response.content).decode('utf-8')\n",
"for color in colors:\n",
" if color in rgb_colors_for_mask_with_text_subclass_tool:\n",
" cp_mask_ndjson.append({\n",
" \"name\": \"mask_with_text_subclass\",\n",
" \"mask\": {\"instanceURI\": im_bytes,\n",
" \"colorRGB\": color },\n",
" \"classifications\":[{\n",
" \"name\": \"sub_free_text\",\n",
" \"answer\": \"free text answer\"\n",
" }]\n",
" }\n",
" )\n",
" else:\n",
" cp_mask_ndjson.append({\n",
" \"name\": \"mask\",\n",
" \"classifications\": [],\n",
" \"mask\": {\n",
" \"instanceURI\": im_bytes,\n",
" \"colorRGB\": color\n",
" }\n",
" }\n",
" )\n",
"\n",
"\n"
],
"cell_type": "code",
"outputs": [],
Expand Down Expand Up @@ -709,7 +749,7 @@
"metadata": {},
"source": [
"# send a sample image as batch to the project\n",
"global_key = \"2560px-Kitano_Street_Kobe01s5s41102.jpeg\"\n",
"global_key = \"2560px-Kitano_Street_Kobe01s5s4110.jpeg\"\n",
"\n",
"test_img_url = {\n",
" \"row_data\":\n",
Expand All @@ -722,8 +762,25 @@
"task = dataset.create_data_rows([test_img_url])\n",
"task.wait_till_done()\n",
"\n",
"print(f\"Failed data rows: {task.failed_data_rows}\")\n",
"print(f\"Errors: {task.errors}\")\n",
"print(f\"Failed data rows: {task.failed_data_rows}\")"
"\n",
"if task.errors:\n",
" for error in task.errors:\n",
" if 'Duplicate global key' in error['message'] and dataset.row_count == 0:\n",
" # If the global key already exists in the workspace the dataset will be created empty, so we can delete it.\n",
" print(f\"Deleting empty dataset: {dataset}\")\n",
" dataset.delete()\n",
"\n"
],
"cell_type": "code",
"outputs": [],
"execution_count": null
},
{
"metadata": {},
"source": [
"print(dataset)"
],
"cell_type": "code",
"outputs": [],
Expand Down Expand Up @@ -897,14 +954,13 @@
" bbox_annotation,\n",
" bbox_with_radio_subclass_annotation,\n",
" polygon_annotation,\n",
" mask_annotation,\n",
" mask_with_text_subclass_annotation,\n",
" point_annotation,\n",
" polyline_annotation,\n",
" bbox_source,\n",
" bbox_target,\n",
" relationship,\n",
"]\n",
" relationship\n",
"] + cp_mask\n",
"\n",
"label.append(\n",
" lb_types.Label(data=lb_types.ImageData(global_key=global_key),\n",
" annotations=annotations))"
Expand Down Expand Up @@ -934,14 +990,13 @@
" bbox_annotation_ndjson,\n",
" bbox_with_radio_subclass_ndjson,\n",
" polygon_annotation_ndjson,\n",
" mask_annotation_ndjson,\n",
" mask_with_text_subclass_ndjson,\n",
" point_annotation_ndjson,\n",
" polyline_annotation_ndjson,\n",
" bbox_source_ndjson,\n",
" bbox_target_ndjson,\n",
" relationship_ndjson, ## Only supported for MAL imports\n",
"]\n",
"] + cp_mask_ndjson\n",
"\n",
"for annotation in annotations:\n",
" annotation.update({\n",
" \"dataRow\": {\n",
Expand Down Expand Up @@ -1018,8 +1073,7 @@
{
"metadata": {},
"source": [
"# project.delete()\n",
"# dataset.delete()"
"# project.delete()"
],
"cell_type": "code",
"outputs": [],
Expand Down
45 changes: 0 additions & 45 deletions examples/exports/export_data.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -849,51 +849,6 @@
"cell_type": "code",
"outputs": [],
"execution_count": null
},
{
"metadata": {},
"source": [
"## How to access a `mask` URL \n",
"\n",
"Annotations of the kind `ImageSegmentationMask` and `VideoSegmentationMask` can only be present in labels made on image or video data rows, respectively. In order to access the mask data, you must pass your Labelbox API key stored in `client.headers` in an API request.\n",
"\n",
"When you grab a URL from the mask annotation in the export, the `project_id` and `feature_id` will already be in place. Here, we provide the framework for structuring a URL with any project ID and feature ID."
],
"cell_type": "markdown"
},
{
"metadata": {},
"source": [
"# Provide a project ID and feature ID. Alternatively, replace the entire mask_url with a URL grabbed from your export.\n",
"project_id = \"\"\n",
"feature_id = \"\"\n",
"\n",
"mask_url = f\"https://api.labelbox.com/api/v1/projects/{project_id}/annotations/{feature_id}/index/1/mask\""
],
"cell_type": "code",
"outputs": [],
"execution_count": null
},
{
"metadata": {},
"source": [
"# Make the API request\n",
"req = urllib.request.Request(mask_url, headers=client.headers)"
],
"cell_type": "code",
"outputs": [],
"execution_count": null
},
{
"metadata": {},
"source": [
"# Print the image of the mask\n",
"image = Image.open(urllib.request.urlopen(req))\n",
"image\n"
],
"cell_type": "code",
"outputs": [],
"execution_count": null
}
]
}