Bug 1538560 - [RFE]rename the project name mux-undefined #1065

nhosoi · 2018-04-05T07:02:16Z

mux-undefined ==> .orphaned
mux.sh: if there are any tests which create project "mux-undefined"
just remove that project checking and creation.

richm · 2018-04-05T11:54:49Z

docs/mux-logging-service.md

-use a namespace of `mux-undefined`, so the log records will be in an index
-named something like `project.mux-undefined.$mux-undefined_uuid.YYYY.MM.DD`
+use a namespace of `.orphaned`, so the log records will be in an index
+named something like `project..orphaned.$mux-undefined_uuid.YYYY.MM.DD`


.orphaned isn't a namespace - @jcantrill - what is the full index name for orphaned records? .orphaned.orphan-uuid.YYYY.MM.DD?

https://github.com/ViaQ/fluent-plugin-viaq_data_model/blob/master/lib/fluent/plugin/filter_viaq_data_model.rb#L146
https://github.com/ViaQ/fluent-plugin-viaq_data_model/blob/master/lib/fluent/plugin/filter_viaq_data_model.rb#L400-L403
Based on the above code, it'd be .orphaned.YYYY.MM.DD?

richm · 2018-04-05T11:55:07Z

docs/mux-logging-service.md

@@ -277,7 +277,7 @@ of the cluster.  It is expected that the record has the `@timestamp` field and
 is already formatted for the ViaQ data model.  If the tag matches
 `project.namespacename.**`, mux extracts the `namespacename` and passes this to
 the k8s meta filter plugin in order to get the namespace uuid.  If the tag does
-not match, mux uses a namespacename of `mux-undefined`.  Except for the k8s
+not match, mux uses a namespacename of `.orphaned`.  Except for the k8s


.orphaned isn't a namespace

richm · 2018-04-05T11:55:23Z

docs/mux-logging-service.md

@@ -360,7 +360,7 @@ The next stage in the pipeline is this filter:

 This examines the tag to see if it is of the form
 `project.namespacename.whatever`, and sets the field `mux_namespace_name` to
-the value of `namespacename` if it is in that form, or `mux-undefined`
+the value of `namespacename` if it is in that form, or `.orphaned`


.orphaned isn't a namespace

richm · 2018-04-05T11:56:12Z

fluentd/configs.d/input-post-forward-mux.conf

      mux_need_k8s_meta ${(record['namespace_uuid'] || record.fetch('kubernetes', Hash.new)['namespace_id'].nil?) ? "true" : "false"}
-      kubernetes {"namespace_name":"${record['namespace_name'] || (tag_parts[0] == 'project' && tag_parts[1]) || ENV['MUX_UNDEFINED_NAMESPACE'] || 'mux-undefined'}","namespace_id":"${record['namespace_uuid'] || record.fetch('kubernetes', Hash.new)['namespace_id']}"}
+      kubernetes {"namespace_name":"${record['namespace_name'] || (tag_parts[0] == 'project' && tag_parts[1]) || ENV['MUX_UNDEFINED_NAMESPACE'] || '.orphaned'}","namespace_id":"${record['namespace_uuid'] || record.fetch('kubernetes', Hash.new)['namespace_id']}"}


@jcantrill what is the namespace_id for .orphaned?

fluent-plugin-kubernetes_metadata_filter/README.md:

orphaned_namespace_name - The namespace to associate with records where the namespace can not be determined (default: .orphaned)

orphaned_namespace_id - The namespace id to associate with records where the namespace can not be determined (default: orphaned)

Could there be a possibility record.fetch('kubernetes', Hash.new)['namespace_id'] is nil? If so, we should add another default for orphane?

kubernetes {"namespace_name":"${record['namespace_name'] || (tag_parts[0] == 'project' && tag_parts[1]) || ENV['MUX_UNDEFINED_NAMESPACE'] || '.orphaned'}","namespace_id":"${record['namespace_uuid'] || record.fetch('kubernetes', Hash.new)['namespace_id'] || 'orphaned'}"}

Could there be a possibility record.fetch('kubernetes', Hash.new)['namespace_id'] is nil? If so, we should add another default for orphane?

Yes. If we cannot determine the namespace_id in any other way, we should use orphaned.

https://github.com/openshift/origin-aggregated-logging/blob/master/fluentd/configs.d/input-post-forward-mux.conf#L87-L88
If mux_namespace_name is ENV["MUX_UNDEFINED_NAMESPACE"] or ".orphaned", do we want to set the tag to "mux" to skip "k8s meta"?

yes, good idea

richm · 2018-04-05T11:57:00Z

test/mux.sh

        espod=$es_pod
        # make sure this namespace exists
-        os::cmd::try_until_success "oc get project mux-undefined" 2>&1 | artifact_out
+        os::cmd::try_until_success "oc get project .orphaned" 2>&1 | artifact_out


I thought this didn't work?

Right. It fails...

we don't need this check anymore

richm · 2018-04-20T14:14:45Z

/retest

The logs which failed to associate with the namespace/project will be indexed into the ES index file .orphaned.YYYY.MM.DD instead of the project.mux-undefined.*. In rewrite_tag_filter (input-post-forward-mux.conf), if mux_namespace_name is .orphaned, we will skip the effort to get k8s_metadata.

nhosoi · 2018-04-23T01:03:52Z

Note: This pr is supposed to be released with openshift/openshift-ansible#7792.

richm · 2018-04-23T02:06:44Z

does this PR work without the corresponding ansible change? If so, what happens?

nhosoi · 2018-04-23T05:38:46Z

The corresponding openshift-ansible is for eliminating openshift_logging_mux_default_namespaces. Functionality-wise, there is no impact with or without the openshift-ansible pr7792, I'd think.

richm · 2018-04-23T14:10:50Z

/lgtm

richm · 2018-04-23T14:23:00Z

There is a backport 3.9 - do we need this in 3.9?

nhosoi · 2018-04-23T14:34:38Z

There is a backport 3.9 - do we need this in 3.9?

I set it just because the target release of bz1538560 is 3.9.z. Probably, we could reset it to 3.10?

richm · 2018-04-23T14:42:47Z

I set it just because the target release of bz1538560 is 3.9.z. Probably, we could reset it to 3.10?

Yes, I think so. The other related change to rename logging to openshift-logging is also in 3.10 and is not backported.

nhosoi · 2018-04-23T14:56:53Z

Reset the target release of bz. Removing backport/3.9, as well...

openshift-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Apr 5, 2018

richm reviewed Apr 5, 2018

View reviewed changes

nhosoi force-pushed the bz1538560 branch from 1cfb015 to 070a9a7 Compare April 20, 2018 01:43

nhosoi force-pushed the bz1538560 branch 2 times, most recently from 1e07f63 to 46faaf8 Compare April 20, 2018 23:49

nhosoi force-pushed the bz1538560 branch from 46faaf8 to 9e4a97e Compare April 20, 2018 23:50

nhosoi mentioned this pull request Apr 23, 2018

Bug 1538560 - [RFE]rename the project name mux-undefined openshift/openshift-ansible#7792

Merged

nhosoi changed the title ~~[WIP] Bug 1538560 - [RFE]rename the project name mux-undefined~~ Bug 1538560 - [RFE]rename the project name mux-undefined Apr 23, 2018

openshift-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 23, 2018

nhosoi added the backport/3.9 backport this PR to release-3.9 label Apr 23, 2018

openshift-ci-robot assigned richm Apr 23, 2018

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Apr 23, 2018

richm added the backport/es5.x label Apr 23, 2018

nhosoi removed the backport/3.9 backport this PR to release-3.9 label Apr 23, 2018

openshift-merge-robot merged commit fb557c7 into openshift:master Apr 23, 2018

richm mentioned this pull request Jun 20, 2018

5x into master #1214

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bug 1538560 - [RFE]rename the project name mux-undefined #1065

Bug 1538560 - [RFE]rename the project name mux-undefined #1065

nhosoi commented Apr 5, 2018

richm Apr 5, 2018

nhosoi Apr 20, 2018

richm Apr 5, 2018

richm Apr 5, 2018

richm Apr 5, 2018

nhosoi Apr 20, 2018

richm Apr 20, 2018

nhosoi Apr 20, 2018

richm Apr 20, 2018

richm Apr 5, 2018

nhosoi Apr 20, 2018

richm Apr 20, 2018

richm commented Apr 20, 2018

nhosoi commented Apr 23, 2018

richm commented Apr 23, 2018

nhosoi commented Apr 23, 2018

richm commented Apr 23, 2018

richm commented Apr 23, 2018

nhosoi commented Apr 23, 2018

richm commented Apr 23, 2018

nhosoi commented Apr 23, 2018

Bug 1538560 - [RFE]rename the project name mux-undefined #1065

Bug 1538560 - [RFE]rename the project name mux-undefined #1065

Conversation

nhosoi commented Apr 5, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

richm commented Apr 20, 2018

nhosoi commented Apr 23, 2018

richm commented Apr 23, 2018

nhosoi commented Apr 23, 2018

richm commented Apr 23, 2018

richm commented Apr 23, 2018

nhosoi commented Apr 23, 2018

richm commented Apr 23, 2018

nhosoi commented Apr 23, 2018