Added support for Cartoon Set #436

sklan · 2019-04-06T08:11:11Z

I added support for the Cartoon Set datasets.
I also fixed a spelling mistake in download_and_prepare.py.
Gist: https://gist.github.com/sklan/3aa0a3cc9036224b0e93ca01dc6d66f5

us

Your checksum file is empty. Please add --register_checksums parameter to download_and_prepare script and create fill it. Check this [link](Your checksum file is empty. Please add --register_checksums parameter to download_and_prepare script.).

sklan · 2019-04-06T10:37:20Z

@us Since, a google account is needed to download the dataset. The dataset needs to be added manually. So, no checksum.

ChanchalKumarMaji

Why not "Heavy Implementation" using BuilderConfigs ? I see similar features being exposed in both the classes.

ChanchalKumarMaji

Where is the fake_examples = 3+30 ?

sklan · 2019-04-06T16:59:29Z

@ChanchalKumarMaji

Why not "Heavy Implementation" using BuilderConfigs ? I see similar features being exposed in both the classes.

Since, the directory structure is slightly different I am unsure if I can use BuilderConfig to handle it. If it can be done I would require some pointers.

Where is the fake_examples = 3+30 ?

I forgot to commit them. Added them now.

us · 2019-04-06T18:02:15Z

@us Since, a google account is needed to download the dataset. The dataset needs to be added manually. So, no checksum.

sorry i missed it.

ChanchalKumarMaji

@sklan
I can see the codes in both the classes are same. So, just merge them into one and create a BuilderConfig class. You can see this, this, this, etc..

Just describe your directory structure here if it is possible, I mean how they differ or which folders each contains ?

sklan · 2019-04-06T22:01:38Z

@ChanchalKumarMaji

For the 100k version the files are divided into subfolders from 0-9 with each subfolder containing 10,000 images.
So I had to add an extra for loop to _generate_examples in the CartoonSet100k version.
Now, that I look at it again I realise it might actually be trivial. I'll switch to BuilderConfig.

ChanchalKumarMaji · 2019-04-07T06:12:39Z

@ChanchalKumarMaji

For the 100k version the files are divided into subfolders from 0-9 with each subfolder containing 10,000 images.
So I had to add an extra for loop to _generate_examples in the CartoonSet100k version.
Now, that I look at it again I realise it might actually be trivial. I'll switch to BuilderConfig.

use self.builder_config.name to distinguish by using a simple if-else statements.

sklan · 2019-04-07T09:40:29Z

@ChanchalKumarMaji
I have updated to BuilderConfig.

use self.builder_config.name to distinguish by using a simple if-else statements.

I figured out a way to do that without an if-else statement.
I am however unsure about the tests. Could you have a look?

tensorflow_datasets/image/cartoonset.py

tensorflow_datasets/image/cartoonset_test.py

ChanchalKumarMaji

Thanks @sklan .

ChanchalKumarMaji · 2019-04-09T19:55:03Z

@rsepassi @cyfra @Conchylicultor , I think this dataset is ready, please see. Thanks.

us · 2019-07-03T14:11:12Z

@sklan can you solve the conflicts,

I think it's forgotten pr. Check please @Conchylicultor @cyfra @pierrot0

sklan · 2019-07-03T17:56:19Z

@sklan can you solve the conflicts,

I think it's forgotten pr. Check please @Conchylicultor @cyfra @pierrot0

Yea sure I'll look into fixing the merge conflicts

cyfra · 2019-07-29T13:31:55Z

Seems that kokoro is failing wiith:

E ImportError: dlopen: cannot load any more object with static TLS
E It seems that scikit-image has not been built correctly.
E
E Your install of scikit-image appears to be broken.
E Try re-installing the package following the instructions at:
E https://scikit-image.org/docs/stable/install.html
E Tried importing %s but failed. See setup.py extras_require. The dataset you are trying to use may have additional dependencies.

sklan · 2019-07-30T20:59:34Z

Seems that kokoro is failing wiith:

E ImportError: dlopen: cannot load any more object with static TLS
E It seems that scikit-image has not been built correctly.
E
E Your install of scikit-image appears to be broken.
E Try re-installing the package following the instructions at:
E https://scikit-image.org/docs/stable/install.html
E Tried importing %s but failed. See setup.py extras_require. The dataset you are trying to use may have additional dependencies.

I'll look into it

us · 2019-07-31T06:37:33Z

tensorflow_datasets/image/cartoonset.py

+        name, dtype = file.split('.')
+        if dtype == 'png':
+          image = tfds.core.lazy_imports.skimage.io.imread(
+              path + '/' + name + '.png')


You should need to os.path.join(path, name), because / is not dynamic type.

us · 2019-07-31T06:38:41Z

tensorflow_datasets/image/cartoonset_test.py

+  DATASET_CLASS = cartoonset.Cartoonset
+  BUILDER_CONFIG_NAMES_TO_TEST = ["cartoonset100k"]
+  SPLITS = {
+      "train": 30,  # Fake training examples


30 test images is too much 3-4 is enough for testing.

Yes - there are far too many test files in this PR.

cyfra · 2020-02-08T21:39:15Z

tensorflow_datasets/image/cartoonset_test.py

+  DATASET_CLASS = cartoonset.Cartoonset
+  BUILDER_CONFIG_NAMES_TO_TEST = ["cartoonset100k"]
+  SPLITS = {
+      "train": 30,  # Fake training examples


Yes - there are far too many test files in this PR.

cyfra · 2020-02-08T21:39:29Z

tensorflow_datasets/image/cartoonset.py

+
+import tensorflow as tf
+
+import tensorflow_datasets as tfds


import tensorflow.compat.v2 as tf

cyfra · 2020-02-08T21:40:29Z

tensorflow_datasets/image/cartoonset.py

+    # There is no predefined train/val/test split for this dataset.
+    path = dl_manager.manual_dir
+    if not tf.io.gfile.exists(path):
+      msg = 'You must download the dataset files manually and place them in: '


You must also set MANUAL_DOWNLOAD_INSTRUCTIONS field (see other datasets like c4)

cyfra · 2020-02-08T21:40:37Z

tensorflow_datasets/image/cartoonset.py

+    return [
+        tfds.core.SplitGenerator(
+            name=tfds.Split.TRAIN,
+            num_shards=2,


remove num_Shards

cyfra · 2020-02-08T21:44:15Z

tensorflow_datasets/image/cartoonset.py

+        features_dict = dict()
+        name, dtype = file.split('.')
+        if dtype == 'png':
+          image = tfds.core.lazy_imports.skimage.io.imread(


you have to read using tf.gfile (to be compatible with non-local filesystems):

with tf.io.gfile.GFile(os.path.join(root, fname), "rb") as png_f: mask = tfds.core.lazy_imports.cv2.imdecode( np.fromstring(png_f.read(), dtype=np.uint8), flags=0)

sklan added 3 commits April 6, 2019 13:15

spelling

f027b28

add cartoon set

fe87f38

PEP

fd0e55b

googlebot added the cla: yes Author has signed CLA label Apr 6, 2019

us suggested changes Apr 6, 2019

View reviewed changes

ChanchalKumarMaji reviewed Apr 6, 2019

View reviewed changes

ChanchalKumarMaji suggested changes Apr 6, 2019

View reviewed changes

Add Fake Example

70b4965

ChanchalKumarMaji suggested changes Apr 6, 2019

View reviewed changes

Update to BuilderConfig

ee4ca35

ChanchalKumarMaji suggested changes Apr 8, 2019

View reviewed changes

tensorflow_datasets/image/cartoonset.py Outdated Show resolved Hide resolved

tensorflow_datasets/image/cartoonset.py Outdated Show resolved Hide resolved

tensorflow_datasets/image/cartoonset_test.py Show resolved Hide resolved

making requested changes

3808e3d

ChanchalKumarMaji approved these changes Apr 9, 2019

View reviewed changes

Conchylicultor added the dataset request Request for a new dataset to be added label Apr 25, 2019

Merge branch 'master' into master

b43b563

us suggested changes Jul 31, 2019

View reviewed changes

cyfra added the tfds:please_review TFDS team: please review this PR. label Feb 8, 2020

cyfra suggested changes Feb 8, 2020

View reviewed changes

cyfra removed the tfds:please_review TFDS team: please review this PR. label Feb 8, 2020

cyfra added the author:please_respond Author - please respond to the recent comments. label Feb 8, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added support for Cartoon Set #436

Added support for Cartoon Set #436

sklan commented Apr 6, 2019 •

edited

Loading

us left a comment

sklan commented Apr 6, 2019

ChanchalKumarMaji left a comment

ChanchalKumarMaji left a comment

sklan commented Apr 6, 2019 •

edited

Loading

us commented Apr 6, 2019

ChanchalKumarMaji left a comment •

edited

Loading

sklan commented Apr 6, 2019

ChanchalKumarMaji commented Apr 7, 2019

sklan commented Apr 7, 2019

ChanchalKumarMaji left a comment

ChanchalKumarMaji commented Apr 9, 2019

us commented Jul 3, 2019

sklan commented Jul 3, 2019

cyfra commented Jul 29, 2019

sklan commented Jul 30, 2019

us Jul 31, 2019

us Jul 31, 2019

cyfra Feb 8, 2020

cyfra Feb 8, 2020

cyfra Feb 8, 2020

cyfra Feb 8, 2020

cyfra Feb 8, 2020

cyfra Feb 8, 2020

Added support for Cartoon Set #436

Are you sure you want to change the base?

Added support for Cartoon Set #436

Conversation

sklan commented Apr 6, 2019 • edited Loading

us left a comment

Choose a reason for hiding this comment

sklan commented Apr 6, 2019

ChanchalKumarMaji left a comment

Choose a reason for hiding this comment

ChanchalKumarMaji left a comment

Choose a reason for hiding this comment

sklan commented Apr 6, 2019 • edited Loading

us commented Apr 6, 2019

ChanchalKumarMaji left a comment • edited Loading

Choose a reason for hiding this comment

sklan commented Apr 6, 2019

ChanchalKumarMaji commented Apr 7, 2019

sklan commented Apr 7, 2019

ChanchalKumarMaji left a comment

Choose a reason for hiding this comment

ChanchalKumarMaji commented Apr 9, 2019

us commented Jul 3, 2019

sklan commented Jul 3, 2019

cyfra commented Jul 29, 2019

sklan commented Jul 30, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sklan commented Apr 6, 2019 •

edited

Loading

sklan commented Apr 6, 2019 •

edited

Loading

ChanchalKumarMaji left a comment •

edited

Loading