Support meteor.var or .env for secrets

[ravisuhag]

Instead of meteor_prefix variable accept a .var or .env for accepting secrets. or maybe support both. If not provided, meteor should also promote users for secrets in the command line before running.

[burntcarrot]

Could we use salt for loading env vars in populateData()? populateData() returns a map[string]string as of now, and salt uses a Loader (viper wrapper) AFAIK.

https://github.com/odpf/meteor/blob/09db3663770352943a17e79235ca6d8f0a1532fd/recipe/envs.go#L12-L14

https://github.com/odpf/meteor/blob/09db3663770352943a17e79235ca6d8f0a1532fd/recipe/reader.go#L12-L24

[burntcarrot]

However, template is being used in the recipe reader for applying the parsed template to reader.data:

https://github.com/odpf/meteor/blob/09db3663770352943a17e79235ca6d8f0a1532fd/recipe/reader.go#L54

So, I don't think changing the method signature for populateData() would be possible, as template execution is dependent upon the reader's data. Is there any way we can possibly write values to the buffer through salt?

https://github.com/odpf/meteor/blob/14dab680e7cc2ae422fe842dd46acca988e6ec51/recipe/reader.go#L53-L62

[ravisuhag]

@burntcarrot Salt does not have a package to serve this need at the moment. Feel free to put a proposal.

[StewartJingga]

So, I don't think changing the method signature for populateData() would be possible, as template execution is dependent upon the reader's data.

@burntcarrot There are two parts when reading recipe from .yaml to memory (recipe struct):

1. collecting data (right it is only via meteor_prefix
2. and applying the data to the recipe

This proposed feature/enhancement will only focus on part 1 which is collecting data. At this stage it is totally easy to extend.