Can we force columns by using col.types and does it do what it supposed to do?

[wendycwong]

H2O-3 parsing optimizes the memory use and will try to find the smaller unit to store a column of data. There is no rounding/truncation in this process.

However, if an user wants to store integers as a double, can this be done using col.types argument when calling h2o.import_file?

[wendycwong]

Okay, I have done the experiment. I have a frame that contains the following:

It is saved as a CSV file. Next, I do an import_file specified both columns to be real

csv_df = h2o.import_file("/Users/wendycwong/temp/df_parquet.csv", col_types=col_types)
print(csv_df.types)

here is the result:

Note that the second column is still parsed as integer.

The only way to force a real column is to this:

[wendycwong]

I am going to add a flag force_col_types to make sure that Parquet schema column types are used if force_col_types is set to true.

For other parsers, users can set the force_col_types and then force the column types to be the same as is specified in col_types.

The reason this flag is needed for numeric columns are because H2O-3 tries to optimize memory use by using the minimum memory needed to store each column. For example, if your column is declared double but only contains integer, H2O-3 will store your column as a column of integers. There is no truncation or rounding. Hence, there is no precision lost.

However, if you want to specify your own numeric columns to be integer/double, you need to set force_col_types to be true and say so in col_types.

For parquet files, if you set force_col_types = true, the column types in parquet schema will be used in determining the final column types. You will miss out on the memory optimization provided by H2O-3.

[wendycwong]

For integer/double columns, they are specified as numeric in col_types. The parser will get the final say to parse it as integer or double. Hence, the force_col_types will not deal with this case.