Transforming date values

[Malonl]

Hi.

I have a use case where I want to use date features as input values for a predictive model. I need to transform the date features to be useful.
For example, I need to know the difference between two dates (for example, just the difference in days between 01-04-2019 and 16-04-2019, but the dates can also be months or years apart).
Or just getting the day of the month, the month itself or the year (i.e. for 16-04-2019, getting 16, 4 and 2019 as seperate values).

My question is if it is possible to do this within TFX and if not, is this a feature that is coming up?
It would be important for my use case because the transform needs to be done in the graph format so that I can serve the model with the transformations inside the pipeline.
Otherwise I would need to add something that can do this for me outside of TFX.

Thanks in advance!

Martijn

[Efaq]

Same issue here!

[ruoyu90]

IIUC this is about converting 1 column into several features. You can use the transform component to do this by putting your logic into preprocessing_fn like our chicago taxi example.

[Harshini-Gadige]

For more information on using preprocessing_fn within Tensorflow Transform, please check Preprocessing function example here.

[Efaq]

@ruoyu90 I think the issue goes through graph operations with dates, more than just converting one column into three. So:

• task: given two date columns, generate a new column with the difference between them in days.
• if we could use some python library (datetime for example), it would be trivial. Without the library, we would need to implement the knowledge about the calendar (number of days in each month etc)
• I believe we cannot use a conventional python library because if we use it, the transformation would not be written to the graph, and thus we would not be able to have it at serving time.

If there is no way such operations to the graph, then as mentioned above we would need to implement a piece of pipeline transforming the data both before training and before serving, outside of the graph.

Does it make sense, or am I missing something here?

[chanshah]

@tensorflow/transform-team please take a look.

[KesterTong]

Everything @Efaq Says above is correct: this should be done with Transform and must be done with TensorFlow ops (not py_func) so that it can be done in serving.

Please file an issue in the tensorflow/transform repo, and we can further discuss the exact functionality required.

[Malonl]

Thanks for all of your replies. I will post an issue in the transform repo.

[ian-hensel-apex]

Any update on this?

[dmitra79]

Any update on these features? They really would be great to have!

[pritamdodeja]

Since tfx isn't supporting this, here is a workaround. You can use https://www.tensorflow.org/addons/api_docs/python/tfa/text/parse_time to parse the string to unix date time, and then derive temporal features from the number of seconds since 1970. I would almost be tempted to have a utility function that takes as input a list of columns that are date time and transforms those features to be unix time. I'll likely need to write this, and when I do, I'll post it here.