Obtaining inverse transform values from tft.transform

[agonojo]

I'm throughly confused by the tft transform output.
I've read dozens and dozens of documentation pages and I've even tried (dumbly) to explore the saved_model.pb graph to see if I could find the constants computed during pre_processing with

tft.scale_to_z_score(outputs[key], name='z_scale_'+key) #for the sake of an example we can pretend key = 'height'

how can I obtain the std and mean computed during the AnalyzeAndTransformDataset step for my numeric column "height" ?

I clearly see in my transformed_data files that it's been transformed. In particular, this is important for my target prediction (regression problem). I feel silly. Someone please point me in the right direction?

If it helps, I'm using the census_v2 code as an example where the only major difference in code is our model architecture and loss function (mine is custom)

[agonojo]

If it's supposed to be saving something in my transform_model.variables or in the variables/ folder, it isn't.

[agonojo]

Ah, it seems my model is not returning this either which I realize could give me those stats..
pre_transform_statistics_path <- not being created

[zoyahav]

These constants are being saved in the graph as constants used for transformations.
If you wanted to see these in the trainer, then you could also save those statistics as output features of your preprocessing_fn.

def preprocessing_fn(inputs):
  height = inputs['height']

  scaled_height = tft.scale_to_z_score(height, name='z_scale_height')
  height_mean = tft.mean(height)
  height_var = tft.var(height)
  
  batch_size = tf.shape(input=height)[0]
  def feature_from_scalar(value):
    return tf.tile(tf.expand_dims(value, 0), multiples=[batch_size])

  return {
    'scaled_height': scaled_height,
    'height_mean': feature_from_scalar(height_mean),
    'height_var': feature_from_scalar(height_var),
  }

{pre|post}_transform_statistics_path is where statistics are stored about your data that can be used for your analysis, not so much using the contents during training or anything like that.

[agonojo]

Thank you so much for such a timely response Zoyahav. Indeed, I had checked the graph via:

pre_stats = tf.saved_model.load('/folder_path/tftransform_tmp/')

#print(pre_stats.graph.as_graph_def())
print(pre_stats.graph.as_graph_element("z_scale_heaight/mean_and_var/Const"))

but none of the constants were actually floats that look like a variance or mean. (They were either int64 values or floats = 1.0).

If they really should be stored at all, they're definitely not being saved on my end, either in the graph or in any output folders though my transformed data does appear to have undergone the transformations. I am testing to see if the few NaN values (which I imputed in a prior step) might be causing an issue.

But overall, having to compute something beam is already calculating to perform z scaling seems redundant and inefficient.. I figured they'd be present in the output object or graph but I can't seem to get them. Regardless, I can proceed as you're suggesting. Perhaps this is something a future update could provide? or even just an option to reverse a transformation. Say, for example, I scale my numeric target. My prediction will be scaled. I have no idea what that means in my normal scale since I can't reverse the transformation without those statistics.

Now that aside, do you find it concerning at all that my process isn't actually outputting {pre|post}_transform_statistics folders? I would like those values

[agonojo]

Also, I also included this in my Pipeline following decode. I'll run again and update if I can get something working.
| 'GenerateStatistics' >> stats_api.GenerateStatistics(stats_options) | 'WriteStatsOutput' >> stats_api.WriteStatisticsToTFRecord( output_path))

[agonojo]

These constants are being saved in the graph as constants used for transformations.
If you wanted to see these in the trainer, then you could also save those statistics as output features of your preprocessing_fn.

def preprocessing_fn(inputs):
  height = inputs['height']

  scaled_height = tft.scale_to_z_score(height, name='z_scale_height')
  height_mean = tft.mean(height)
  height_var = tft.var(height)
  
  batch_size = tf.shape(input=height)[0]
  def feature_from_scalar(value):
    return tf.tile(tf.expand_dims(value, 0), multiples=[batch_size])

  return {
    'scaled_height': scaled_height,
    'height_mean': feature_from_scalar(height_mean),
    'height_var': feature_from_scalar(height_var),
  }

{pre|post}_transform_statistics_path is where statistics are stored about your data that can be used for your analysis, not so much using the contents during training or anything like that.

Side note, I was unable to get this to work. My preprocessing function would return an empty dict:

Traceback (most recent call last):
  File "tf_transform-2.py", line 1009, in <module>
    test_size=TEST_SIZE
  File "tf_transform-2.py", line 904, in main
    transform_data(TRAIN_FILES, TEST_FILES, working_dir)
  File "tf_transform-2.py", line 332, in transform_data
    raw_dataset | tft_beam.AnalyzeAndTransformDataset(preprocessing_fn))
  File "/home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/apache_beam/transforms/ptransform.py", line 562, in __ror__
    result = p.apply(self, pvalueish, label)
  File "/home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/apache_beam/pipeline.py", line 655, in apply
    pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
  File "/home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/apache_beam/runners/runner.py", line 198, in apply
    return m(transform, input, options)
  File "/home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/apache_beam/runners/runner.py", line 228, in apply_PTransform
    return transform.expand(input)
  File "/home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow_transform/beam/impl.py", line 1029, in expand
    dataset | 'AnalyzeDataset' >> AnalyzeDataset(self._preprocessing_fn))
  File "/home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/apache_beam/transforms/ptransform.py", line 998, in __ror__
    return self.transform.__ror__(pvalueish, self.label)
  File "/home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/apache_beam/transforms/ptransform.py", line 562, in __ror__
    result = p.apply(self, pvalueish, label)
  File "/home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/apache_beam/pipeline.py", line 612, in apply
    return self.apply(transform, pvalueish)
  File "/home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/apache_beam/pipeline.py", line 655, in apply
    pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
  File "/home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/apache_beam/runners/runner.py", line 198, in apply
    return m(transform, input, options)
  File "/home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/apache_beam/runners/runner.py", line 228, in apply_PTransform
    return transform.expand(input)
  File "/home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow_transform/beam/impl.py", line 976, in expand
    None, input_metadata))
  File "/home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow_transform/beam/impl.py", line 835, in expand
    raise ValueError('The preprocessing function returned an empty dict')
ValueError: The preprocessing function returned an empty dict

[zoyahav]

Does your preprocessing_fn return an empty dict? (perhaps look if there's an issue with indentation or something like that)
In the snippet above it doesn't, so this error shouldn't occur.