HDDS-4601. envtoconf broken for .conf and few other formats

[adoroszlai]

What changes were proposed in this pull request?

Fix envtoconf for some output formats that are currently broken. These are not used in Ozone, nor covered by tests. The problem may be accidentally triggered by specific environment variable names, eg. HBASE_CONF_DIR or similar.

https://issues.apache.org/jira/browse/HDDS-4601

How was this patch tested?

Test:

cd hadoop-ozone/dist/target/ozone-1.1.0-SNAPSHOT
OZONE_RUNNER_VERSION=20200625-1
docker run -it --rm -e DUMMY.CONF_key=value -v $(pwd)/libexec:/opt/hadoop/libexec apache/ozone:0.5.0 bash -c 'ls ${HADOOP_CONF_DIR}/dummy.conf'
docker run -it --rm -e DUMMY.SH_key=value -v $(pwd)/libexec:/opt/hadoop/libexec apache/ozone:1.0.0 bash -c 'ls ${HADOOP_CONF_DIR}/dummy.sh'
docker run -it --rm -e DUMMY.ENV_key=value -v $(pwd)/libexec/transformation.py:/opt/transformation.py apache/hadoop:3 bash -c 'ls ${HADOOP_CONF_DIR}/dummy.env'
docker run -it --rm -e DUMMY.CFG_key=value -v $(pwd):/opt/hadoop apache/ozone-runner:${OZONE_RUNNER_VERSION} bash -c 'ls ${HADOOP_CONF_DIR}/dummy.cfg'

Output:

/etc/hadoop/dummy.conf
/etc/hadoop/dummy.sh
/etc/hadoop/dummy.env
/etc/hadoop/dummy.cfg

[adoroszlai]

(Answering the part of your comment from #1667 relevant to this PR.)

The change itself in transformation.py is to go through the items instead of the collection means that we are transforming something different in those cases and we do it similarly as the one case you mentioned working well before this change also, I am not sure though, how this solves the problem.

@fapifta Here's a Python REPL session to show what's going on:

>>> props = {"key": "value"}
>>> props
{'key': 'value'}
>>> props.keys()
['key']
>>> props.values()
['value']
>>> props.items()
[('key', 'value')]
>>> for x in props: print x
...
key
>>> for x in props.keys(): print x
...
key
>>> for x, y in props: print "{}={}".format(x, y)
...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: too many values to unpack
>>> for x, y in props.items(): print "{}={}".format(x, y)
...
key=value
>>>

props is a map (dictionary). for x in props iterates keys, same as for x in props.keys(). for x, y in props also iterates keys (in props), but tries to extract key-value pairs, which fails with ValueError, since each key is only a single item. for x, y in props.items() correctly iterates key-value pairs.

Already used correctly in to_properties:

ozone/hadoop-ozone/dist/src/main/dockerlibexec/transformation.py

Lines 81 to 85 in cbe8651

	def to_properties(content):
	"""transform to properties"""
	result = ""
	props = process_properties(content)
	for key, val in props.items():

[fapifta]

Thank you for making it clear what the exact problem is.
Looks good to me, +1

[adoroszlai]

Thanks @fapifta for the review.