AMBARI-24261. logfeeder fail to start due to missing conf file post l…

[aonishuk]

Upgraded above cluster from Ambari 2.6.2 to Ambari 2.7.0 and logsearch also from 2.6.2 to 2.7.0.0

In ambari 2.6.2 we had created file "kafka-output.json" under "/usr/lib/ambari-logsearch-logfeeder/conf" so that logsearch data is pushed to kafka topic.

Looks like Ambari upgrade renames the conf dir to "conf-old" as below and due to which kafka-output.json file in missing from the new config dir.

[oleewere]

sounds invalid, we never had /usr/lib/ambari-logsearch-logfeeder/conf before (it was /etc/ambari-logsearch-logfeeder/conf in 2.6).
if you remove the generated keys + checkpoints folder copy, from the old logfeeder config location, logfeeder wont work with SSL and logs will be duplicated, as it will re-parse all the logs.

I would recommend to copy the keys + checkpoints folders to $LOGFEEDER_CONF_SOURCE , then delete just those folders if they were existed before. (in order to not copy it again in an another yum install or upgrade)

that means rm on $LOGFEEDER_CONF in preinstall is ok, but you can keep the copy of keys + checkpoint folders part, but there you need to remove those after the copy (so do an rm of the folders after the cp command)

[asfgit]

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/Ambari-Github-PullRequest-Builder/3009/
Test PASSed.

[oleewere]

@aonishuk , update: can you close these PRs as that won't really solve the problem ? (i will start to fix that) i guess https://community.hortonworks.com/articles/57099/hdp-logs-push-to-kafka-topic-using-ambari-logsearc.html was done for adding the kafka-output.json, the problem is that: it's a relative path, and before that was (or could be) located in /etc/ambari-logsearch-logfeeder/conf, but that won't be anyomore on he classpath, so not deleting that folder won't solve the problem. (we won't find the custom json files) - and we should not add /etc/ambari-logsearch-logfeeder/conf to the classpath as other old input json files could be incompatible with logfeeder 2.7.0

so i think the solution would be more complicated, we need to get the custom json files from the old logfeeder.properties file, we can keep the folder copy to conf-old, and based on the names we can copy the custom json to the new /usr/lib/ambari-logsearch-logfeeder/conf folder. other than that we probably need to update the checkpoint folder in the upgrade catalog if we want to delete /etc/ambari-logsearch-logfeeder/conf folder. (or if we keep it then we wont need that)

[aonishuk]

@oleewere thanks for answer. Was looking forward to your response on this.
Closing the PR.