RetinaNet TPU Compatibility [looking for a long term fix instead of #1021]

[LukeWood]

We can enable this in predict().

This might be a big performance delta, but is much lower priority than TPU compatibility.

[bhack]

Don't we want to use this ticket to investigate more #1021 (comment)?

[LukeWood]

Sure, we can further investigate.

[bhack]

@ianstenbit if you can share your gs:// pre-populated it could be useful as re-run the extraction is super-slow.

[ianstenbit]

@bhack sorry for the delay on this.
I've moved this data to data_dir="gs://keras-cv/datasets", where you should have access. I've also updated my repro colab to reflect the change.

Also -- extraction on GCS hangs (see tensorflow/datasets#4115), so make sure to pass download=False to your tfds.load call.

[bhack]

@ianstenbit Thanks

[bhack]

I don't think that the public url is gs://keras-cv/datasets. What we need to use?

[ianstenbit]

When I pull the public URL of e.g the dataset info, I am seeing this:

https://storage.googleapis.com/keras-cv/datasets/voc/2007/4.0.0/dataset_info.json

So I suppose you could try https://storage.googleapis.com/keras-cv/datasets -- but I would expect the gs version to work, so I'm not sure

[bhack]

What is the value of:

Once public access has been granted, Copy URL appears for each object in the public access column. You can click this button to get the public URL for the object.

[bhack]

It is not working:

RuntimeError: PermissionDeniedError: Failed to construct dataset voc: Error executing an HTTP request: HTTP response code 401 with body '{
  "error": {
    "code": 401,
    "message": "Anonymous caller does not have storage.objects.get access to the Google Cloud Storage object. Permission 'storage.objects.get' denied on resource (or it may not exist).",
    "errors": [
      {
        "message": "Anonymous caller does not have storage.objects.get access to the Google Cloud Storage object. Permission 'storage.objects.get' denied on resource (or it may not exist).",
        "domain": "global",
        "reason": "required",
        '
	 when reading metadata of gs://kerascv-dataset/dataset/voc/2007

Can you navigate it in a incognito browser tab?

[ianstenbit]

Yes, I am able to reach it when I don't have an auth token (e.g. incognito mode)

Are you not able to reach those files in incognito mode?

[ianstenbit]

One other thing you can do is load+extract the VOC dataset locally and then upload the resulting tfds folder to a bucket that you have control over. (That's how I got these records in GCS in the first place, since extraction fails on TFDS.load for gcs)

[bhack]

One other thing you can do is load+extract the VOC dataset locally and then upload the resulting tfds folder to a bucket that you have control over. (That's how I got these records in GCS in the first place, since extraction fails on TFDS.load for gcs)

Yes I know but it is faster if I can reuse your own.

https://storage.googleapis.com/keras-cv/datasets/voc/ in incognito mode:

<Error>
<Code>AccessDenied</Code>
<Message>Access denied.</Message>
<Details>
Anonymous caller does not have storage.objects.get access to the Google Cloud Storage object. Permission 'storage.objects.get' denied on resource (or it may not exist).
</Details>
</Error>

[ianstenbit]

Interesting. That works from my work device as well as my personal device.

I'm thinking that tfds.load probably requires object list permission in addition to access to the underlying files. I don't have the power to give that permission on any of these GCS buckets, so I'm not sure if there's anything else I can do.

FWIW you shouldn't need VOC specifically to repro the issue.
You could create a fake bbox dataset with this method and use that as train_ds and eval_ds.

If you'd like I can update the repro colab to use that instead of VOC

[bhack]

Thank you but I cannot find a free TPU anymore now..

[bhack]

Also on tf-nightly I still see:

(0) INVALID_ARGUMENT: {{function_node __inference_train_function_133447}} TF to XLA legalization failed: <unknown>:0: error: loc(fused["SigmoidGrad:", "gradient_tape/FocalLoss/Sigmoid/SigmoidGrad"]): 'mhlo.constant' op result #0 must be statically shaped tensor of floating-point or pred (AKA boolean or 1-bit integer) or 8/16/32/64-bit signless integer or 8/16/32/64-bit unsigned integer or complex type with 32-bit float or 64-bit float elements values, but got 'tensor<?x?x20xf32>'
<unknown>:0: note: loc(fused["SigmoidGrad:", "gradient_tape/FocalLoss/Sigmoid/SigmoidGrad"]): see current operation: %5996 = "mhlo.constant"() {value = dense<1.000000e+00> : tensor<?x?x20xf32>} : () -> tensor<?x?x20xf32>

I cannot dump on the local filesystem or GCS:

os.environ["TF_DUMP_GRAPH_PREFIX"]= "gs://keras_cv/dataset/" # or /tmp/
os.environ["TF_XLA_FLAGS"]="--tf_xla_clustering_debug --tf_xla_auto_jit=2"
os.environ["XLA_FLAGS"]="--xla_dump_hlo_as_text --xla_dump_to=/tmp/generated"

@smit-hinsu How we could dump on Colab TPU (I still see tensorflow/tensorflow#49702)?

[smit-hinsu]

Sorry, I don't have an idea on that.

[bhack]

@smit-hinsu Do you know someone who could help on this failed legalization?

[bhack]

We have a problem with:
tensorflow/tensorflow@76079c0#diff-20844a653ebc1cb9b4b2adb26672259647f08dc1e78a06458d2e753a1449336bR2070

But in our case we have tensor<?x?x20xf32> in the sigmoid grad where the constrain is #0 must be statically shaped tensor

keras-cv/keras_cv/losses/focal.py

Lines 81 to 82 in d2d56d6

	if self.from_logits:
	y_pred = tf.nn.sigmoid(y_pred)

[bhack]

Tracking this upstream at tensorflow/tensorflow#58645

[tanzhenyu]

Done through recent series of refactoring