Slow performance when computing stats for moderately large data set

[AndreaPi]

Hi,

I like this project a lot, and thanks for releasing it! I see the potential to save a lot of time when I first receive new datasets. However, I have issues with performance.

OS: the notebook is running in a Docker container based on https://hub.docker.com/r/tensorflow/tensorflow/.

Hardware:

GPUs 16X NVIDIA® Tesla V100
GPU Memory 512GB total
CPU Dual Intel Xeon Platinum
8168, 2.7 GHz, 24-cores
System Memory 1.5TB

It takes me >8 hours (30900 s) to compute the statistics for a ~100 dataset, with file sizes from 0.5 MB to 300 MB, with a median of 70 MB. It's true that the Docker container introduces some overhead, but given the specs of my hardware, I think it's too much. Any tips on how to speed up computations, w/o changing hardware (i.e., no cloud)? For example, if there was an option to compute statistics in a dataframe, rather than in a protocol buffer, one could use Modin to speedup pandas computations with minimal changes to code.

PS if I use a GPU container instead, with

$ docker run --runtime=nvidia -it -p 8888:8888 tensorflow/tensorflow:latest-gpu
should I see a speedup?

[vincentteyssier]

Running into the same problem. How many CPU are available in the cgroup docker is assigned to?
Since you have a lot of RAM available I would concatenate files if that works for your use case. I noticed a usual rate of 1:10 between RAM and size of dataset to compute stats on.... so if you concatenate up to 1Gb, it should fit in 10Gb of RAM...

[paulgc]

@AndreaPi
Thanks for the feedback. We recently released TFDV 0.11 with some performance improvements (~2x improvement on benchmark datasets). We are continuing to work on more improvements.

Regarding further performance improvement, Beam is adding support for Flink runner which would allow you to run statistics generation across a cluster or using multiple processes within a single machine.

[AndreaPi]

@paulgc thanks! I just updated my Dockerfile and I'm now using TFDV 0.11. In my use case, I need to generate statistics for multiple CSV files, all having the same features. I think there's a way to speedup tfdv.generate_statistics_from_csv by passing the schema as an input, rather than having TFDV infer it for each file. Is that right? Or is the schema only used by tfdv.validate_statistics? If also tfdv.generate_statistics_from_csv can benefit by having a schema passed as an input, can you show me how to do it? The only argument of tfdv.generate_statistics_from_csv which seems to be relevant is stats_options, but the API documentation doesn't tell me a lot about it:

https://www.tensorflow.org/tfx/data_validation/api_docs/python/tfdv/generate_statistics_from_csv

[paulgc]

@AndreaPi The website is not updated with the latest documentation. You can pass a schema as part of the stats_options and set the infer_type_from_schema flag in stats_options to be True. You can find the updated documentation here.

[AndreaPi]

@paulgc thanks a lot. I’m at NeurIPS these days, so I can’t test my use case now, but I will once I’m back, next week. Thanks again

[Harshini-Gadige]

@AndreaPi Any update on this ?

[AndreaPi]

@hgadig sorry I forgot to update. I’m on a business trip at the moment, I promise I’ll update by this time next week. There is indeed an improvement. I think we’re down to 2 h & 1/2, from > 8 h originally. I’ can’t share the dataset unfortunately, but I will share the Jupyter notebook so can see the timings.

[paulgc]

Thanks @AndreaPi.

Also note that in the latest release, we have added support for generating statistics from a pandas Dataframe, which can exploit multi-processing on a single machine. If your CSV data fits in memory, you can use the generate_statistics_from_dataframe method and set n_jobs=-1 to use all the cores in your machine.

[AndreaPi]

@paulgc in order to use generate_statistics_from_dataframe, do I need to load the whole dataset in memory, or is it ok if each CSV file separately fits in memory? I have plenty of RAM, so I should be fine in both cases, but I'd like to know, just for my own understanding.

[paulgc]

@AndreaPi Currently we don't have a way to merge statistics computed from individual files to obtain the overall statistics. Hence, you would have to load the whole dataset in memory when using generate_statistics_from_dataframe method.