How to handle named variables

[AndrewC19]

Hi,

I have been working with the causal-learn package for a couple of weeks now and I was wondering if there is a plan to add support for named variables?

As I understand it, at the moment data is generally assumed to take the form of a 2d np.array where each column contains a value for a particular variable X1, X2, ..., XN. I can quite easily take a csv file and convert it to this format but, in doing so, I lose the names of my columns. This relies on me having to keep track of which variable (X1, X2, ..., XN) corresponds to which named column of my csv data, which makes tasks such as visualising the output quite difficult.

I have been using the to_pyplot method to draw the discovered DAG and passing it the labels. However, this relies on the order being preserved (simply replacing X1 with the first label, X2 with the second, and so on). This has tripped me a couple of times when comparing two graphs produced from similar data where the variables do not align.

I am not sure what the solution is here, but it would be really useful if it were possible to attach actual variable names to the graph rather than doing this manually after computation. One possibility would be to support pandas dataframes for reading in data.

[kunwuz]

Hi Andrew, thanks for your interest, and we are very grateful for your question. Just a quick confirmation, regarding the to_pyplot method you mentioned, did you mean GraphUtils.to_pydot(cg.G, labels=[“A”, “B”, “C”])?

[AndrewC19]

Hi, yes, that's what I mean. Apologies for the typo!

[kunwuz]

Thanks for the clarification! we will get back to you with potential solutions quickly :)

[AndrewC19]

Great, thanks. Let me know if any further information would be helpful.

[AndrewC19]

Hi again, I have encountered another similar issue that may influence your solution.

If my csv of data contains columns for variables of different types -- say three continuous variables for age, height, and weight, and categorical variables for gender and blood group -- loading them into a numpy array forces you to encode them as the same type. This means that if my categorical variables are encoded as ints they will all be cast to floats, or if they are encoded as strings, all of my continuous data will be cast to strings.

This is problematic for dealing with categorical variables automatically (I would like to apply a different conditional independence test depending on this). Ideally, I would treat all floats as continuous and then use a heuristic to determine if my integers should be treated as continuous or categories. However, all of my integers are being converted to floats so I have no reliable way of differentiating them.

I'm not sure if there is a way to encode the type of the variable in causal-learn already. If so, please could you show me how?

[kunwuz]

Hi Andrew, thanks for raising this. Regarding mixed type variables, we are currently integrating two solutions, i.e., conditional Gaussian and degenerate Gaussian. Please kindly refer to this issue for more information.

Since all of our methods are based on NumPy.array as input, is it possible to differentiate continuous/categorical variables based on column/variable indices?

Regarding the label of the variable, IMO the most convenient way to do it is using our provided functions. Currently, we do not support input as pandas.DataFrame and perhaps we would like to avoid extra dependencies. Besides the function we provided previously (i.e., GraphUtils.to_pydot(cg.G, labels=[“A”, “B”, “C”])), we have just added a label parameter in the draw_pydot_graph() function, i.e., cg.draw_pydot_graph(labels=[“A”, “B”, “C”]). We hope it could be more convenient.

Moreover, if users would like to transfer pandas.DataFrame (e.g., df) to numpy.array while preserving labels, cg.draw_pydot_graph(labels=list(df.columns)) or GraphUtils.to_pydot(cg.G, labels=list(df.columns)) should work.

[AndrewC19]

Hi again, thanks for the response.

With respect to preserving labels, your proposed solution is effectively what I am currently using i.e. pass a list of the column titles to the drawing function. This works fine in most cases, but if you don't use all of the data in your csv/dataframe or the order of your csv/dataframe changes, it becomes challenging to keep track of which variable is which. I have resorted to sorting my data columns alphabetically before starting causal discovery and removing all unused columns to avoid this.

With respect to differentiating between variables in a numpy array, the issue is that numpy arrays do not allow you to have different columns with different type. For example, this code:

mixed_data_array = np.array([[1, 2, 3], [1.1, 2.2, 3.3], [100, 200, 300]]).T
print(mixed_data_array)

produces the following output:

array([[  1. ,   1.1, 100. ],
       [  2. ,   2.2, 200. ],
       [  3. ,   3.3, 300. ]])

Now if I check the data type of each column, I get float each time and not the original data type (int, float, int, respectively). I can't find an elegant solution for this problem with numpy arrays. Even if I go back and cast the datatype of each column using astype, it does not update.

[tofuwen]

Hi Andrew,

Will this PR help you in your case?

#42

@AndrewC19

[tofuwen]

And btw, Andrew, it seems that you have another question regarding data types --- it would be great if you can open another issue for each different question. :) It will make tracking issues easier and make sure we address all the issues easier. :)

[AndrewC19]

Thanks and my apologies, it was intended as a response to @kunwuz's question (is it possible to differentiate continuous/categorical variables based on column/variable indices?).

[tofuwen]

Oh, cool! Totally understand it! :)

Hmm, I am also curious how to do causal-learn with both categorical features and numerical features, let me tag some experts here to help. :)

cc @kunwuz @chenweiDelight @Biwei-Huang

[kunwuz]

Thanks for the discussion! They are really helpful and inspiring.

Here is a brief summary of our discussions before and some suggestions from our team members.

Regarding the question, from the algorithmic side, we could use PC with KCI or GES with the generalized score to deal with mixed data types. Also, there will be more methods available in the future (#31).

Regarding the representation of mixed data type, currently, we only support numpy.array as input, which needs some post/pre-processing from users to track the type of variables. We are working on extending our input to other types of data, e.g., pandas.DataFrame. Please kindly refer to this issue for future updates (#43).