Backports needed for gnocchi #1
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Looks all good. |
|
@dtroyer |
|
I am OK with the set of backports, I would ask that you keep the original commit message and add a pointer to the source of each commit and to the story directly in the commit messages so that linkage can be traced in the future. Github PR messages are not kept in the repo and all of that information is left behind in a git clone for example. If changes to the original commit were necessary add a mention of that also. The practice when cherry-picking/backporting changes in OpenStack is to keep the Commit-ID the same also as that allows Gerrit to do some useful things. Since these repos are not in Gerrit we lose that bit but for the sake of consistency I would advise we keep the original Comit-ID also. |
This removes the deprecated Gnocchi dispatcher and replaces it by its equivalent publisher. Change-Id: Ie44baf20ccb8de5794f5f0c3d4717f7e56afa63b (cherry picked from commit 228517e) original commit 83ffaff StoryBoard: “Various backports to core openstack components to pick up fixes” https://storyboard.openstack.org/#!/story/2003407
ujson is faster than jsonutils, and we do not need any fancy feature jsonutils might offer. This also has the benefit of removing a big dependency on Ceilometer. Change-Id: I24bf08d0fa6ccc34beef0a0c34a47bf2fa266e3e (cherry picked from commit 886dc38) original commit 7b959b3 StoryBoard: “Various backports to core openstack components to pick up fixes” https://storyboard.openstack.org/#!/story/2003407
don't use lambda to filter. this is 25% faster but really negligible. Change-Id: Ib76470fc3a5cdfc79e497d568147b66c08e6dba1 (cherry picked from commit 992efae) original commit 8e3ba4d StoryBoard: “Various backports to core openstack components to pick up fixes” https://storyboard.openstack.org/#!/story/2003407
ujson has not had any active maintenance for the last 12 months; switch to using json module instead. Change-Id: I39027b534e94b3f877d881647a7c843183f60f92 Closes-Bug: 1737989 (cherry picked from commit 5ebc250) original commit 05050e4 StoryBoard: “Various backports to core openstack components to pick up fixes” https://storyboard.openstack.org/#!/story/2003407
we shouldn't waste our time looping through resources and searching metrics to find a match. just build hashmap and find the resource based on metric. this removes fnmatch functionality because we don't match on wildcards and the code itself actually requires metric names to be explicit to create_metrics Change-Id: I2398247270217759c876ab5a9b60038dad79a9d3 (cherry picked from commit e516e82) original commit 8099a94 StoryBoard: “Various backports to core openstack components to pick up fixes” https://storyboard.openstack.org/#!/story/2003407
we shouldn't be wasting time doing the same thing over and over in loop Change-Id: Iaa5aeb5b91004945a24fab105c8b4884918c1644 (cherry picked from commit 7275dd5) original commit 40317d7 StoryBoard: “Various backports to core openstack components to pick up fixes” https://storyboard.openstack.org/#!/story/2003407
don't bother with a running count to build stats Change-Id: I6c40e49046115512d5858a6e1348629e10d6a42f Conflicts: ceilometer/publisher/gnocchi.py (cherry picked from commit c473be4) original commit 02e8ac2 StoryBoard: “Various backports to core openstack components to pick up fixes” https://storyboard.openstack.org/#!/story/2003407
we run through samples and format samples and resources to what gnocchi expects. this simplifies that process a little: - don't needlessly sort and group by metric name - build a full resource model iff resource needs to be created Change-Id: I2ac25b3b0978eed664c500e645bae2d1b4ae6781 (cherry picked from commit e58fe24) original commit 5b216fa StoryBoard: “Various backports to core openstack components to pick up fixes” https://storyboard.openstack.org/#!/story/2003407
Once I delete resource manually by gnocchi client, the resource won't be created by the ceilometer until 600s passed， because the resource has cached in the memcached. Change-Id: I601e39b4c4782276daba3f9d1b7be92f09efabfc Closes-Bug: #1718570 (cherry picked from commit 7838fe0) original commit 02a012660aebe171627de0922783a9da8be8fddf StoryBoard: “Various backports to core openstack components to pick up fixes” https://storyboard.openstack.org/#!/story/2003407
only the attributes of a resource may change so there's no reason to capture the entire resource in cache. this means we don't need to cache if no attributes. this also changes code so we only compute attribute_hash once. Change-Id: I1363e0b0dc6e0957a7e933536b67737bd3496cbd (cherry picked from commit 64f36fc) original commit 236cfbeea6ab40c94af5f64c5843ce4b62f4d677 StoryBoard: “Various backports to core openstack components to pick up fixes” https://storyboard.openstack.org/#!/story/2003407
This creates two archive policies "ceilometer-low" and "ceilometer-low-rate" in Gnocchi. And use them to create metrics in Gnocchi. The archive policies settings can now be configured in gnocchi_resources.yaml. This requires upcoming release of coming (gnocchixyz/gnocchi#655) Change-Id: Ib804ef4652af9d40338ad970690f6e92f12ed636 Conflicts: ceilometer/publisher/data/gnocchi_resources.yaml doc/source/admin/telemetry-measurements.rst (cherry picked from commit bb4692d) original commit b96b16928ca84bd1c4025c3f996a65e447a8920f StoryBoard: “Various backports to core openstack components to pick up fixes” https://storyboard.openstack.org/#!/story/2003407
We can't create archive policies in publisher init because if it fail because Gnocchi is not yet ready the publisher fail to load. Ceilometer will work, but samples will go to nowhere. This change creates/checks archive policies when we publish a sample for the very first time. If that fail because Gnocchi is not ready, this will just retry next time Ceilometer will publish samples. Closes-Bug: #1752420 Change-Id: Ib6b4da54592ad99a4e6561a73473b6c7ec73a21f (cherry picked from commit 717e968) original commit 14906f8672282b21467d705b316f0b8ec88c6bee StoryBoard: “Various backports to core openstack components to pick up fixes” https://storyboard.openstack.org/#!/story/2003407
event is now an object. Change-Id: I22fbb797dc9061240068ff2b4e0b3a2d76463248 (cherry picked from commit 27064b6) original commit 0f14afe StoryBoard: “Various backports to core openstack components to pick up fixes” https://storyboard.openstack.org/#!/story/2003407
Ceilometer creates metrics that will never get measures (service not polled, notification not configured). This is because we create all metrics during resource creation. That was done sometimes ago because Gnocchi was not provide metrics creating during batching process. But since a while now, we also create metrics (if missing) during the batch processing. This change removes the creation of metrics during resource creation. All metrics with measures got created during the batch call. This stops to have metric without measures in Gnocchi. Change-Id: I3f3b805722b11029aefdb76bae93413cb32ecff6 (cherry picked from commit 10cd6ba) original commit 826ba35c6eb9900bb0a557f6e4f06f7d1b9bd394 StoryBoard: “Various backports to core openstack components to pick up fixes” https://storyboard.openstack.org/#!/story/2003407
dchavoll
force-pushed
the
backports_needed_for_gnocchi
branch
from
10cd6ba
to
e9c65b1
Compare
|
LGTM Thanks for the commit message updates. Note that the 'cherry-pick' commit SHAs are for an older internal repo and are not publicly available, in the future let's label those as internal or something so people do not expect they will be visible. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
These 14 commits are backports to starlingX-staging, that are required to enable delivery of the gnocchi feature.
This activity corresponds to the following task in StoryBoard:
“[Enhancement] Various backports to core openstack components to pick up fixes”
https://storyboard.openstack.org/#!/story/2003407
The commits, all cleanly backported are:
228517e Replace the Gnocchi dispatcher by a publisher
886dc38 Replace jsonutils by ujson
992efae simplify cache generation
5ebc250 Replace ujson with json
e516e82 use hashmap to quickly find matching resource def
7275dd5 compute sample as dictionary once
c473be4 fix gnocchi stats logging
e58fe24 simplify gnocchi batch push setup
7838fe0 Cached should not be used when creating gnocchi resources
64f36fc set cache with attributes only
bb4692d gnocchi: configure archive policies on Ceilo side
717e968 Don't use gnocchiclient during publisher init
27064b6 fix gnocchi publisher
10cd6ba gnocchi: don't create metrics with resource
Testing: Code compiles successfully