METRON-565: apps/metron/enrichment/indexed directory path does not get created for metron cluster deployed via Ambari #365
Conversation
…t created for metron cluster deployed via Ambari
| @@ -246,7 +246,7 @@ bolt.hdfs.file.system.url={{ default_fs }} | |||
| bolt.hdfs.wip.file.path=/paloalto/wip | |||
| bolt.hdfs.finished.file.path=/paloalto/rotated | |||
| bolt.hdfs.compression.codec.class=org.apache.hadoop.io.compress.SnappyCodec | |||
| index.hdfs.output={{ metron_apps_enrichment_dir }} | |||
There was a problem hiding this comment.
I agree that this works, but could we do
index.hdfs.output={{ metron_apps_enrichment_dir }}/indexing
or even defined in the config, e.g.
index.hdfs.output={{ metron_indexing_hdfs_dir }}/indexing
I don't have a particularly strong opinion on constructing it in the params vs metron-env, but we should be using metron_apps_enrichment_dir since we already calculate it in the params anyway.
There was a problem hiding this comment.
But that param/value is hidden from the user right?
There was a problem hiding this comment.
That is a fair point, since we do expose the metron-env in Ambari. I'm willing to leave it separately, although it does make me wonder if there's a better way to handle configs like that in general. Having said that, I don't see a good way around it based on how Ambari needs things defined.
|
The Travis failure appears to be an unrelated profiler integration test error, although I'd like @mmiklavc to double-check. Assuming it is unrelated, I'm +1, by inspection. Thanks for picking this up! |
|
@nickwallen can you take a look and verify? |
|
Yes, appears to be an intermittent timing problem. If you re-run, I assume it would pass. In METRON-575 I changed how these tests are validated. I assume that would help prevent such a thing from happening in the future. |
…t created for metron cluster deployed via Ambari
|
Ended up making a few additional changes here. First, got rid of the /apps/metron/enrichment directory entirely. With indexing being split off into its own topology, this no longer made sense to live there. Furthermore, "indexed" was the only thing that lived in the enrichment directory in hdfs. The indexed doc path will now live in /apps/metron/indexing/indexed to keep consistent with the dir structure we used in the past. In addition, I've now exposed the directory as a property the user can both see and modify in the advanced properties section in Ambari.
Lastly, I updated the quickdev environment to reflect this change as well. |
|
I'm +1 on this, good stuff. @justinleet - you still gtg with the additional changes? |
|
Yep, I'm +1, too. |
This PR addresses https://issues.apache.org/jira/browse/METRON-565
Modified the parameters file to point to the correct path for indexing.
Testing
#364 (comment)