backend.parameters is depreciated.

[mayankbansal89]

Hi,

I was reading the earlier issues about backend.parameters being depreciated and that you are planning to remove it from rmr 2.1 .
I think there is a case for still keeping the backend.parameters option. I agree with your view that on a properly set cluster it is not required, but there are issues security and access restrictions which sometimes hinder figuring out a wrongly set cluster and an erroneous code. In these cases backend.parameters helps you figure it out.
Recently I had a issue where only 1 reducer was being spawned, it turned out to be a wrongly configured cluster which I was able to guess only after specifying backend.parameters and reduce tasks to a specific number.
So I would really appreciate if you keep that option. It is helpful.

[piccolbo]

The feature is "deprecated" to be precise. The good news is that we didn't axe it in 2.1 so it's there, still deprecated. The way you are using it, that's what it's there for, and it wouldn't be deprecated if that's all that people did with it. But it's a hole in the abstraction layer and it's dangerous. Some people jumped in it and started changing separators, tampering with the format and all sort of other stuff that rmr uses internally to implement its features. It's like using a compiler and then tampering with the assembly code a bit. The first time the compiler changes, or if you even change an option in the compiler call, or maybe even a line in the program, and the thing breaks apart. So you may say once people are warned about it, they are grown ups, they should know better. Why penalize people who use the feature the way it's intended? Unfortunately it's not that simple. I've seen on line rmr programs where most of the lines were arguments to backend.parameters. Those things get posted and other people learn that that's the way to write programs and then they complain here that rmr doesn't work. And besides getting a lot of avoidable and difficult support work, one also gets the "rmr breaks a lot" reputation, instead of "rmr breaks a lot if you misuse it". But the plan is to replace backend.parameters with a more restricted set of options that will capture the safe and common uses of backend parameters. There was already an issue concerning the number of mapper and reducers open in the old repo, I will reopen it here. You can open additional issues if you have other uses of backend.parameters that you feel are important and safe. Thanks for your feedback.

[piccolbo]

I opened issue #10 on the main topic raised here. If there are more important and acceptable uses of backend parameters we can open additional ones.