Enable environment variable expansion in crail-site.conf

[raduioanstoica]

No description provided.

[raduioanstoica]

Sometimes it is easier to set dynamic configuration options through environment variables. This allows reusing the same Crail configuration file for similar deployments where only non-Crail specific parameters are different (such as the user name, HADOOP/SPARK/CRAIL deployment directories, hostnames).

[animeshtrivedi]

Hey Radu, can you please open a JIRA (http://crail.incubator.apache.org/community/) for this pull request for discussion. We just do code review on github.

Also, what happens if a environment variable is not set? Is it possible to do some sort of sanity checks and throw an exception? Because if the variable is not set then probably it is bug (?)

[asqasq]

Small comment, otherwise looks fine with me.

[asqasq]

Can you add braces to the if-else like every where else in the file?

[raduioanstoica]

@animeshtrivedi

• If an environment variable is not set, a warning is printed and the option is set to an empty string (""). An exception will probably follow in case the option is not optional. Throwing an exception right away is probably better. We should do the same everywhere (e.g. when using CRAIL_HOME).
• Related to how configuration options are handled, several things could be improved: 1) specify for each parameter whether it is optional or not & data type + automatic validation; 2) print warnings for options that were set but are not used (to catch typos, obsolete old variable names); 3) update automatically crail-site.conf.template so users do not have to search in the code. Not sure though which is the best way to do this.
• I'll try to open a JIRA thread.

@asqasq
Sure, good point.

[animeshtrivedi]

Looks OK to me too.

@raduioanstoica yes, all of those issue need improvement in Crail code, specially unused or obsolete parameter names. I think we can have a look at other project and see how they do it.

I am not sure what you mean by (3) update automatically [...]?

[raduioanstoica]

At 3) I mean to extract the options directly from the code (from the CrailConstants-style classes) and update the crail-site.conf.template file in /conf. The options would be ideally commented out and include the default values and comments (if any). It would be much easier for a user to see all configuration options in one place.

There are many ways to do this. I guess we could start by having a look at Hadoop / Spark to see if they do something similar.

[asqasq]

Looks good to me

[PepperJo]

Looks good to me. See discussion https://issues.apache.org/jira/browse/CRAIL-11