Metadata for columns and/or DataFrames

[tshort]

Should we leave room for metadata on structures? Frank Harrell's Hmisc package allows units and labels to be attached to data.frame columns.

People may want to attach other metadata like experimenter name or a DataFrame comment.

We could add a meta Dict to the DataFrame, the colindex, and/or at the DataVec level.

[HarlanH]

Yes, I think this could be useful. At the DataVec level, we will need meta-data for factor-like behavior (#6). And some of the other things you suggest make sense too. On the other hand, we probably want to rely less on arbitrary attributes, like R, and more on types, when there's the possibility of doing so.

[houshuang]

I would love to see this, wrote about my wish for better support for things like questionnaires with the code book integrated with the code here (towards the bottom): http://reganmian.net/blog/2013/10/02/likert-graphs-in-r-embedding-metadata-for-easier-plotting/...

Of course also raises the issue about serialization.

[johnmyleswhite]

If we can make this work without performance degradation, I'm in.

[nalimilan]

Standardizing on a few meta-data attributes like variable label and unit would be wonderful. In R, Harrell's Hmisc offers this feature, but unfortunately very few package use it since it's not standard at all. OTC, SAS has built-in support for variable labels, which are used e.g. to label tables and plot axes automatically. Stata also has this concept, and even allows associating longer "notes" to variables, to explicit their meaning.

More specialized attributes like question names would be useful, if there was an easy way for a separate package to create and use them.

[johnmyleswhite]

Adding units should be trivial, especially if Julia settles on a standard unit package soon. What are the variable labels for: descriptions of the columns to supplement the brief names?

[nalimilan]

Yeah, variable labels are just the readable, complete name of the variable, as opposed to the abbreviated form that is practical to type (no spaces, no special characters...) but often cryptic and ugly which is used for variable names. The most typical use of variable labels is when you want to provide a good default for axes labels, like "Annual GDP growth", rather than "GDPG". They could also be useful to describe the contents of a database, with a function like Hmisc's describe() [1] or SAS's proc contents.

1: http://www.inside-r.org/packages/cran/Hmisc/docs/describe

[HarlanH]

Yes, I agree with all of this. A long, human-readable Name (which could be
leveraged for axes labels by plotting routines), Units, and
Level-of-measurement would be very helpful. Possibly also Domain.

LoM could be very handy for statistical modeling routines and the creation
of appropriate model matrices (or the throwing of warnings). I never
intended PooledDataVector to be equivalent to Factor -- it's a
representational optimization. Would be much better for statistical
routines to look for Nominal or Ordinal types and act appropriately, even
if the underlying type is a non-pooled integer or string.

On Sun, Oct 6, 2013 at 11:54 AM, Milan Bouchet-Valat <
notifications@github.com> wrote:

Yeah, variable labels are just the readable, complete name of the
variable, as opposed to the abbreviated form that is practical to type (no
spaces, no special characters...) but often cryptic and ugly which is used
for variable names. The most typical use of variable labels is when you
want to provide a good default for axes labels, like "Annual GDP growth",
rather than "GDPG". They could also be useful to describe the contents of a
database, with a function like Hmisc's describe() [1] or SAS's proc
contents.

1: http://www.inside-r.org/packages/cran/Hmisc/docs/describe

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/35#issuecomment-25770479
.

[houshuang]

All great ideas. long name, LoM (for example I'd love to indicate that something is a likert-item, which is more specific than just categorical), etc. Not sure what is meant by domain? Units of course useful for measurements. An open ended comment field would be great for code book stuff (how data is collected, coded etc) - I could see some great ways of showing this, especially in the web view.

Not sure how this would fit in, but in my R code, I also have the concept of grouping columns - for example having five groups of questions.

Also curious about how we serialize this - we can't just spit this out into CSV again. What's DataFrame's "native" format for storing all this metadata? Ideally it would be something that was compatible with other tools as well. HDF5?

[johnmyleswhite]

I think separating Likert scales from other categorical variables might be too specific for something as generic as DataFrames: what functions would apply to them that don't apply to other categorical variables?

I believe domain is meant in the math sense of "allowable, but not necessarily present, values for entries in this column".

We once had grouped columns, but they were dropped because they proved difficult to maintain. They need to added back in, but it takes a good chunk of work to do.

Serialization is kind of a nightmare. I think HDF5 may work, but that's a question for people with more expertise than I have in our current serialization infrastructure.

[houshuang]

I was thinking of LoM as user defined, for example I might want to graph
likert-scales differently from a demographic categorical variable... But
this isn't super-important.

Stian

On Mon, Oct 7, 2013 at 11:57 AM, John Myles White
notifications@github.comwrote:

I think separating Likert scales from other categorical variables might be
too specific for something as generic as DataFrames: what functions would
apply to them that don't apply to other categorical variables?

I believe domain is meant in the math sense of "allowable, but not
necessarily present, values for entries in this column".

We once had grouped columns, but they were dropped because they proved
difficult to maintain. They need to added back in, but it takes a good
chunk of work to do.

Serialization is kind of a nightmare. I think HDF5 may work, but that's a
question for people with more expertise than I have in our current
serialization infrastructure.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/35#issuecomment-25820799
.

http://reganmian.net/blog -- Random Stuff that Matters

[johnmyleswhite]

You could actually do that already: you'd just make a DataArray{LikertResponse}, where LikertResponse is a custom type. This is one of the virtues of our approach to NA: you can create a DataArray for any type in Julia, not just those we've built into the system.

[houshuang]

I think serialization becomes an important issue - many of these things can
probably be done already by subclassing DataFrame etc (and whether it's
better to extend DataFrame or subclass it becomes a design question),
however the key question is how I can setup my data the way I want it (with
full names, groups, etc), and then store it for future analysis by other
scripts...

On Mon, Oct 7, 2013 at 12:04 PM, John Myles White
notifications@github.comwrote:

You could actually do that already: you'd just make a
DataArray{LikertResponse}, where LikertResponse is a custom type. This is
one of the virtues of our approach to NA: you can create a DataArray for
any type in Julia, not just those we've built into the system.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/35#issuecomment-25821376
.

http://reganmian.net/blog -- Random Stuff that Matters

[nalimilan]

If you add support for arbitrary meta-data attributes to DataFrames, it will be easy for separate packages to mark some columns as grouped using a group index. No need to hardcode support for every specific feature - just make it easy to extend.

[johnmyleswhite]

What would arbitrary metadata consist of? A Dict called metadata that people can do anything with?

[nalimilan]

Sure, a Dict containing vectors with one value per column, or even just a DataFrame, since attributes would all have the same length. Only standard attributes would have a pre-specified type, others would be free. Of course setters and getters would make the whole process transparent.