min/max values of transformed metadata

[pnezis]

Assume I am applying the string_to_int function on a categorical column. In the transformed schema this column is mapped to INT where the min and max entries of the domain are the min and max values of INT

"domain": {
      "ints": {
            "isCategorical": false,
            "max": "9223372036854775807",
            "min": "-9223372036854775808"
        }
 },

I would expect instead the actual min (-1) and max (size(vocab)-1) values of this transformed column. Is there any workaround for this?

[KesterTong]

Currently not, but I am working on setting this information automatically. Should be done on 2-3 weeks.

[KesterTong]

Update: made some progress here, ETA is still 2-3 weeks

[buffxz]

Is there any updates about this issues? When can we expect we have more informative meta data about the dataset ?

can we also have the quantile values?

[KesterTong]

This is fixed at head, but it will be about 1 month before the next release of TFT on PyPI.

Regarding other metadata, the situation of using string_to_int is special because we already know from the string_to_int calculation, what the range is.

For other cases, we don't have code to calculate these ranges. In principal this is straightforward but the internal code isn't setup to calculate arbitrary statistics for the outputs. We plan to add this but it's not clear how we will do it at this point.

[katsiapis]

@pnezis Tensorflow Transform 0.3.1 and 0.3.1 have been released and contain several enhancements, including (I believe) support for your initial request.

@buffxz regarding Quantiles, please see our response in #29

@KesterTong or @elmer-garduno, could we apply the appropriate milestone (0.3.0?) to this Issue and perhaps Close it (if we've fully answered all the questions)?

[KesterTong]

This was fixed in 0.3.0 but we've added it to the 0.3.1 milestone since we didn't create a milestone for 0.3.0. To answer the original question, as of version 0.3.0, we will infer min and max values (based on the vocab size) and set them in the transformed schema, but only for the output of string_to_int. If you do any transformations after string_to_int we will not provide min and max values.