From ebc737f0129e0d93ff94a125ea7370936a7f535e Mon Sep 17 00:00:00 2001 From: Leighton Date: Tue, 5 Nov 2019 13:44:34 -0800 Subject: [PATCH 001/109] changes --- LICENSE | 223 +++---------------------------- azure_monitor/CHANGELOG.md | 8 ++ azure_monitor/examples/server.py | 2 +- azure_monitor/examples/trace.py | 2 +- 4 files changed, 31 insertions(+), 204 deletions(-) create mode 100644 azure_monitor/CHANGELOG.md diff --git a/LICENSE b/LICENSE index fea9e74..3d8b93b 100644 --- a/LICENSE +++ b/LICENSE @@ -1,202 +1,21 @@ - Apache License - Version 2.0, January 2004 - http://www.apache.org/licenses/ - - TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION - - 1. Definitions. - - "License" shall mean the terms and conditions for use, reproduction, - and distribution as defined by Sections 1 through 9 of this document. - - "Licensor" shall mean the copyright owner or entity authorized by - the copyright owner that is granting the License. - - "Legal Entity" shall mean the union of the acting entity and all - other entities that control, are controlled by, or are under common - control with that entity. For the purposes of this definition, - "control" means (i) the power, direct or indirect, to cause the - direction or management of such entity, whether by contract or - otherwise, or (ii) ownership of fifty percent (50%) or more of the - outstanding shares, or (iii) beneficial ownership of such entity. - - "You" (or "Your") shall mean an individual or Legal Entity - exercising permissions granted by this License. - - "Source" form shall mean the preferred form for making modifications, - including but not limited to software source code, documentation - source, and configuration files. - - "Object" form shall mean any form resulting from mechanical - transformation or translation of a Source form, including but - not limited to compiled object code, generated documentation, - and conversions to other media types. - - "Work" shall mean the work of authorship, whether in Source or - Object form, made available under the License, as indicated by a - copyright notice that is included in or attached to the work - (an example is provided in the Appendix below). - - "Derivative Works" shall mean any work, whether in Source or Object - form, that is based on (or derived from) the Work and for which the - editorial revisions, annotations, elaborations, or other modifications - represent, as a whole, an original work of authorship. For the purposes - of this License, Derivative Works shall not include works that remain - separable from, or merely link (or bind by name) to the interfaces of, - the Work and Derivative Works thereof. - - "Contribution" shall mean any work of authorship, including - the original version of the Work and any modifications or additions - to that Work or Derivative Works thereof, that is intentionally - submitted to Licensor for inclusion in the Work by the copyright owner - or by an individual or Legal Entity authorized to submit on behalf of - the copyright owner. For the purposes of this definition, "submitted" - means any form of electronic, verbal, or written communication sent - to the Licensor or its representatives, including but not limited to - communication on electronic mailing lists, source code control systems, - and issue tracking systems that are managed by, or on behalf of, the - Licensor for the purpose of discussing and improving the Work, but - excluding communication that is conspicuously marked or otherwise - designated in writing by the copyright owner as "Not a Contribution." - - "Contributor" shall mean Licensor and any individual or Legal Entity - on behalf of whom a Contribution has been received by Licensor and - subsequently incorporated within the Work. - - 2. Grant of Copyright License. Subject to the terms and conditions of - this License, each Contributor hereby grants to You a perpetual, - worldwide, non-exclusive, no-charge, royalty-free, irrevocable - copyright license to reproduce, prepare Derivative Works of, - publicly display, publicly perform, sublicense, and distribute the - Work and such Derivative Works in Source or Object form. - - 3. Grant of Patent License. Subject to the terms and conditions of - this License, each Contributor hereby grants to You a perpetual, - worldwide, non-exclusive, no-charge, royalty-free, irrevocable - (except as stated in this section) patent license to make, have made, - use, offer to sell, sell, import, and otherwise transfer the Work, - where such license applies only to those patent claims licensable - by such Contributor that are necessarily infringed by their - Contribution(s) alone or by combination of their Contribution(s) - with the Work to which such Contribution(s) was submitted. If You - institute patent litigation against any entity (including a - cross-claim or counterclaim in a lawsuit) alleging that the Work - or a Contribution incorporated within the Work constitutes direct - or contributory patent infringement, then any patent licenses - granted to You under this License for that Work shall terminate - as of the date such litigation is filed. - - 4. Redistribution. You may reproduce and distribute copies of the - Work or Derivative Works thereof in any medium, with or without - modifications, and in Source or Object form, provided that You - meet the following conditions: - - (a) You must give any other recipients of the Work or - Derivative Works a copy of this License; and - - (b) You must cause any modified files to carry prominent notices - stating that You changed the files; and - - (c) You must retain, in the Source form of any Derivative Works - that You distribute, all copyright, patent, trademark, and - attribution notices from the Source form of the Work, - excluding those notices that do not pertain to any part of - the Derivative Works; and - - (d) If the Work includes a "NOTICE" text file as part of its - distribution, then any Derivative Works that You distribute must - include a readable copy of the attribution notices contained - within such NOTICE file, excluding those notices that do not - pertain to any part of the Derivative Works, in at least one - of the following places: within a NOTICE text file distributed - as part of the Derivative Works; within the Source form or - documentation, if provided along with the Derivative Works; or, - within a display generated by the Derivative Works, if and - wherever such third-party notices normally appear. The contents - of the NOTICE file are for informational purposes only and - do not modify the License. You may add Your own attribution - notices within Derivative Works that You distribute, alongside - or as an addendum to the NOTICE text from the Work, provided - that such additional attribution notices cannot be construed - as modifying the License. - - You may add Your own copyright statement to Your modifications and - may provide additional or different license terms and conditions - for use, reproduction, or distribution of Your modifications, or - for any such Derivative Works as a whole, provided Your use, - reproduction, and distribution of the Work otherwise complies with - the conditions stated in this License. - - 5. Submission of Contributions. Unless You explicitly state otherwise, - any Contribution intentionally submitted for inclusion in the Work - by You to the Licensor shall be under the terms and conditions of - this License, without any additional terms or conditions. - Notwithstanding the above, nothing herein shall supersede or modify - the terms of any separate license agreement you may have executed - with Licensor regarding such Contributions. - - 6. Trademarks. This License does not grant permission to use the trade - names, trademarks, service marks, or product names of the Licensor, - except as required for reasonable and customary use in describing the - origin of the Work and reproducing the content of the NOTICE file. - - 7. Disclaimer of Warranty. Unless required by applicable law or - agreed to in writing, Licensor provides the Work (and each - Contributor provides its Contributions) on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or - implied, including, without limitation, any warranties or conditions - of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A - PARTICULAR PURPOSE. You are solely responsible for determining the - appropriateness of using or redistributing the Work and assume any - risks associated with Your exercise of permissions under this License. - - 8. Limitation of Liability. In no event and under no legal theory, - whether in tort (including negligence), contract, or otherwise, - unless required by applicable law (such as deliberate and grossly - negligent acts) or agreed to in writing, shall any Contributor be - liable to You for damages, including any direct, indirect, special, - incidental, or consequential damages of any character arising as a - result of this License or out of the use or inability to use the - Work (including but not limited to damages for loss of goodwill, - work stoppage, computer failure or malfunction, or any and all - other commercial damages or losses), even if such Contributor - has been advised of the possibility of such damages. - - 9. Accepting Warranty or Additional Liability. While redistributing - the Work or Derivative Works thereof, You may choose to offer, - and charge a fee for, acceptance of support, warranty, indemnity, - or other liability obligations and/or rights consistent with this - License. However, in accepting such obligations, You may act only - on Your own behalf and on Your sole responsibility, not on behalf - of any other Contributor, and only if You agree to indemnify, - defend, and hold each Contributor harmless for any liability - incurred by, or claims asserted against, such Contributor by reason - of your accepting any such warranty or additional liability. - - END OF TERMS AND CONDITIONS - - APPENDIX: How to apply the Apache License to your work. - - To apply the Apache License to your work, attach the following - boilerplate notice, with the fields enclosed by brackets "[]" - replaced with your own identifying information. (Don't include - the brackets!) The text should be enclosed in the appropriate - comment syntax for the file format. We also recommend that a - file or class name and description of purpose be included on the - same "printed page" as the copyright notice for easier - identification within third-party archives. - - Copyright [yyyy] [name of copyright owner] - - Licensed under the Apache License, Version 2.0 (the "License"); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. - \ No newline at end of file + MIT License + + Copyright (c) Microsoft Corporation. + + Permission is hereby granted, free of charge, to any person obtaining a copy + of this software and associated documentation files (the "Software"), to deal + in the Software without restriction, including without limitation the rights + to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + copies of the Software, and to permit persons to whom the Software is + furnished to do so, subject to the following conditions: + + The above copyright notice and this permission notice shall be included in all + copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + SOFTWARE diff --git a/azure_monitor/CHANGELOG.md b/azure_monitor/CHANGELOG.md new file mode 100644 index 0000000..d19f53f --- /dev/null +++ b/azure_monitor/CHANGELOG.md @@ -0,0 +1,8 @@ +# Changelog + +## Unreleased + +## 0.1a.0 +Released 2019-11-05 + +- Initial release diff --git a/azure_monitor/examples/server.py b/azure_monitor/examples/server.py index 1437e69..4325335 100644 --- a/azure_monitor/examples/server.py +++ b/azure_monitor/examples/server.py @@ -22,7 +22,7 @@ @app.route("/") def hello(): - with trace.tracer().start_span("parent"): + with trace.tracer().start_as_current_span("parent"): requests.get("https://www.wikipedia.org/wiki/Rabbit") return "hello" diff --git a/azure_monitor/examples/trace.py b/azure_monitor/examples/trace.py index 76f5507..41f1a12 100644 --- a/azure_monitor/examples/trace.py +++ b/azure_monitor/examples/trace.py @@ -11,5 +11,5 @@ SimpleExportSpanProcessor(AzureMonitorSpanExporter()) ) -with tracer.start_span("hello") as span: +with tracer.start_as_current_span("hello") as span: print("Hello, World!") From c7916f257030a0154fe1cf039e0700d20c6fb8b6 Mon Sep 17 00:00:00 2001 From: Leighton Date: Tue, 5 Nov 2019 13:50:12 -0800 Subject: [PATCH 002/109] codeowners --- .github/CODEOWNERS | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 .github/CODEOWNERS diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS new file mode 100644 index 0000000..a61e72c --- /dev/null +++ b/.github/CODEOWNERS @@ -0,0 +1,5 @@ +# Code owners file. +# This file controls who is tagged for review for any given pull request. + +# For anything not explicitly taken by someone else: +* @hectorhdzg @lzchen From 99b2128b37b964bdd73df2f9b76a329fe984a55a Mon Sep 17 00:00:00 2001 From: Leighton Date: Tue, 5 Nov 2019 14:01:16 -0800 Subject: [PATCH 003/109] remove changelog --- azure_monitor/CHANGELOG.md | 8 -------- 1 file changed, 8 deletions(-) delete mode 100644 azure_monitor/CHANGELOG.md diff --git a/azure_monitor/CHANGELOG.md b/azure_monitor/CHANGELOG.md deleted file mode 100644 index d19f53f..0000000 --- a/azure_monitor/CHANGELOG.md +++ /dev/null @@ -1,8 +0,0 @@ -# Changelog - -## Unreleased - -## 0.1a.0 -Released 2019-11-05 - -- Initial release From 3c777c1834746440f23f39fd217dba9757607dc9 Mon Sep 17 00:00:00 2001 From: Leighton Date: Wed, 6 Nov 2019 14:38:01 -0800 Subject: [PATCH 004/109] update --- azure_monitor/setup.cfg | 4 ++-- azure_monitor/src/azure_monitor/version.py | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/azure_monitor/setup.cfg b/azure_monitor/setup.cfg index 465ed0b..e769fe9 100644 --- a/azure_monitor/setup.cfg +++ b/azure_monitor/setup.cfg @@ -9,11 +9,11 @@ author = Microsoft author_email = appinsightssdk@microsoft.com url = https://github.com/microsoft/opentelemetry-exporters-python platforms = any -license = Apache-2.0 +license = MIT classifiers = Development Status :: 3 - Alpha Intended Audience :: Developers - License :: OSI Approved :: Apache Software License + License :: OSI Approved :: MIT License Programming Language :: Python Programming Language :: Python :: 3 Programming Language :: Python :: 3.4 diff --git a/azure_monitor/src/azure_monitor/version.py b/azure_monitor/src/azure_monitor/version.py index 9092865..7a5fe00 100644 --- a/azure_monitor/src/azure_monitor/version.py +++ b/azure_monitor/src/azure_monitor/version.py @@ -1,3 +1,3 @@ # Copyright (c) Microsoft Corporation. All rights reserved. # Licensed under the MIT License. -__version__ = "0.1.dev0" +__version__ = "0.2.dev0" From f71cbebe3f480de9daaf5eadc3d21d9dd00acc55 Mon Sep 17 00:00:00 2001 From: Spencer Miller Date: Tue, 12 Nov 2019 13:59:25 -0600 Subject: [PATCH 005/109] added requests example --- azure_monitor/examples/requests.py | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) create mode 100644 azure_monitor/examples/requests.py diff --git a/azure_monitor/examples/requests.py b/azure_monitor/examples/requests.py new file mode 100644 index 0000000..ad4e4a9 --- /dev/null +++ b/azure_monitor/examples/requests.py @@ -0,0 +1,18 @@ +import requests +from azure_monitor import AzureMonitorSpanExporter +from opentelemetry import trace +from opentelemetry.ext import http_requests +from opentelemetry.sdk.trace import Tracer +from opentelemetry.sdk.trace.export import SimpleExportSpanProcessor + +trace.set_preferred_tracer_implementation(Tracer) + +http_requests.enable(trace.tracer()) +span_processor = SimpleExportSpanProcessor( + AzureMonitorSpanExporter(instrumentation_key="") +) +trace.tracer().add_span_processor(span_processor) + +with trace.tracer().start_as_current_span("parent"): + response = requests.get("", timeout=5) + From f56d52edb82c1a864dc57e145cdcd0381d708875 Mon Sep 17 00:00:00 2001 From: Spencer Miller Date: Fri, 15 Nov 2019 10:13:29 -0600 Subject: [PATCH 006/109] added copyright and spacing --- azure_monitor/examples/requests.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/azure_monitor/examples/requests.py b/azure_monitor/examples/requests.py index ad4e4a9..c59663b 100644 --- a/azure_monitor/examples/requests.py +++ b/azure_monitor/examples/requests.py @@ -1,4 +1,7 @@ +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. import requests + from azure_monitor import AzureMonitorSpanExporter from opentelemetry import trace from opentelemetry.ext import http_requests From c92d538124c6ed74941fedcdbf951949937ff5ae Mon Sep 17 00:00:00 2001 From: Leighton Date: Fri, 15 Nov 2019 12:55:43 -0800 Subject: [PATCH 007/109] Fix imports --- azure_monitor/examples/client.py | 8 +++++--- azure_monitor/examples/{requests.py => request.py} | 0 azure_monitor/examples/server.py | 6 ++++-- azure_monitor/examples/trace.py | 5 +++-- azure_monitor/setup.cfg | 1 + 5 files changed, 13 insertions(+), 7 deletions(-) rename azure_monitor/examples/{requests.py => request.py} (100%) diff --git a/azure_monitor/examples/client.py b/azure_monitor/examples/client.py index fa78dd9..cc2ae84 100644 --- a/azure_monitor/examples/client.py +++ b/azure_monitor/examples/client.py @@ -11,8 +11,10 @@ trace.set_preferred_tracer_implementation(lambda T: Tracer()) tracer = trace.tracer() http_requests.enable(tracer) -span_processor = BatchExportSpanProcessor(AzureMonitorSpanExporter()) -tracer.add_span_processor(span_processor) +span_processor = BatchExportSpanProcessor( + AzureMonitorSpanExporter(instrumentation_key="") +) +trace.tracer().add_span_processor(span_processor) -response = requests.get(url="http://127.0.0.1:5000/") +response = requests.get(url="http://127.0.0.1:8080/") span_processor.shutdown() diff --git a/azure_monitor/examples/requests.py b/azure_monitor/examples/request.py similarity index 100% rename from azure_monitor/examples/requests.py rename to azure_monitor/examples/request.py diff --git a/azure_monitor/examples/server.py b/azure_monitor/examples/server.py index 4325335..cf2583f 100644 --- a/azure_monitor/examples/server.py +++ b/azure_monitor/examples/server.py @@ -13,7 +13,9 @@ trace.set_preferred_tracer_implementation(lambda T: Tracer()) http_requests.enable(trace.tracer()) -span_processor = BatchExportSpanProcessor(AzureMonitorSpanExporter()) +span_processor = BatchExportSpanProcessor( + AzureMonitorSpanExporter(instrumentation_key="") +) trace.tracer().add_span_processor(span_processor) app = flask.Flask(__name__) @@ -28,5 +30,5 @@ def hello(): if __name__ == "__main__": - app.run(debug=True) + app.run(host='localhost', port=8080, threaded=True) span_processor.shutdown() diff --git a/azure_monitor/examples/trace.py b/azure_monitor/examples/trace.py index 41f1a12..ae490b0 100644 --- a/azure_monitor/examples/trace.py +++ b/azure_monitor/examples/trace.py @@ -7,9 +7,10 @@ trace.set_preferred_tracer_implementation(lambda T: Tracer()) tracer = trace.tracer() -tracer.add_span_processor( - SimpleExportSpanProcessor(AzureMonitorSpanExporter()) +span_processor = SimpleExportSpanProcessor( + AzureMonitorSpanExporter(instrumentation_key="") ) +trace.tracer().add_span_processor(span_processor) with tracer.start_as_current_span("hello") as span: print("Hello, World!") diff --git a/azure_monitor/setup.cfg b/azure_monitor/setup.cfg index e769fe9..d6b9775 100644 --- a/azure_monitor/setup.cfg +++ b/azure_monitor/setup.cfg @@ -29,6 +29,7 @@ packages=find_namespace: install_requires = opentelemetry-api opentelemetry-sdk + requests ~= 2.0 [options.packages.find] where = src From 52b7917f99ca71769564bc60ea36fb52224e49a1 Mon Sep 17 00:00:00 2001 From: Leighton Date: Fri, 22 Nov 2019 11:21:58 -0800 Subject: [PATCH 008/109] add props --- azure_monitor/src/azure_monitor/trace.py | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/azure_monitor/src/azure_monitor/trace.py b/azure_monitor/src/azure_monitor/trace.py index 7907b4a..4c3125a 100644 --- a/azure_monitor/src/azure_monitor/trace.py +++ b/azure_monitor/src/azure_monitor/trace.py @@ -134,8 +134,16 @@ def span_to_envelope(self, span): # noqa pylint: disable=too-many-branches data.type = "HTTP" # TODO if "http.url" in span.attributes: url = span.attributes["http.url"] + # data is the url + data.data = url + parse_url = urlparse(url) # TODO: error handling, probably put scheme as well - data.name = urlparse(url).netloc + # target matches authority (host:port) + data.target = parse_url.netloc + if "http.method" in span.attributes: + # name is METHOD/path + data.name = span.attributes["http.method"] \ + + "/" + parse_url.path if "http.status_code" in span.attributes: data.resultCode = str(span.attributes["http.status_code"]) else: # SpanKind.INTERNAL From d9a834e8a1fc713c8bcf0b5d4173cf348cc513df Mon Sep 17 00:00:00 2001 From: Leighton Date: Mon, 25 Nov 2019 08:51:14 -0800 Subject: [PATCH 009/109] set http --- azure_monitor/src/azure_monitor/trace.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/azure_monitor/src/azure_monitor/trace.py b/azure_monitor/src/azure_monitor/trace.py index 4c3125a..be10836 100644 --- a/azure_monitor/src/azure_monitor/trace.py +++ b/azure_monitor/src/azure_monitor/trace.py @@ -131,7 +131,9 @@ def span_to_envelope(self, span): # noqa pylint: disable=too-many-branches baseData=data, baseType="RemoteDependencyData" ) if span.kind in (SpanKind.CLIENT, SpanKind.PRODUCER): - data.type = "HTTP" # TODO + if "component" in span.attributes and \ + span.attributes["component"] == "http": + data.type = "HTTP" if "http.url" in span.attributes: url = span.attributes["http.url"] # data is the url From 5210a17a12a469ed1795cfa02d9da18a0b479e08 Mon Sep 17 00:00:00 2001 From: Leighton Date: Wed, 4 Dec 2019 14:14:11 -0800 Subject: [PATCH 010/109] add tests --- azure_monitor/src/azure_monitor/trace.py | 21 +- .../src/azure_monitor/{util.py => utils.py} | 10 + azure_monitor/tests/__init__.py | 2 + azure_monitor/tests/test_protocol.py | 48 ++ azure_monitor/tests/test_utils.py | 17 + azure_monitor/tests/trace/__init__.py | 2 + azure_monitor/tests/trace/test_trace.py | 446 ++++++++++++++++++ 7 files changed, 530 insertions(+), 16 deletions(-) rename azure_monitor/src/azure_monitor/{util.py => utils.py} (73%) create mode 100644 azure_monitor/tests/test_protocol.py create mode 100644 azure_monitor/tests/test_utils.py create mode 100644 azure_monitor/tests/trace/__init__.py create mode 100644 azure_monitor/tests/trace/test_trace.py diff --git a/azure_monitor/src/azure_monitor/trace.py b/azure_monitor/src/azure_monitor/trace.py index be10836..1c61fe7 100644 --- a/azure_monitor/src/azure_monitor/trace.py +++ b/azure_monitor/src/azure_monitor/trace.py @@ -6,7 +6,7 @@ import requests -from azure_monitor import protocol, util +from azure_monitor import protocol, utils from opentelemetry.sdk.trace.export import SpanExporter, SpanExportResult from opentelemetry.sdk.util import ns_to_iso_str from opentelemetry.trace import Span, SpanKind @@ -16,7 +16,7 @@ class AzureMonitorSpanExporter(SpanExporter): def __init__(self, **options): - self.options = util.Options(**options) + self.options = utils.Options(**options) if not self.options.instrumentation_key: raise ValueError("The instrumentation_key is not provided.") @@ -63,21 +63,10 @@ def export(self, spans): return SpanExportResult.FAILED_NOT_RETRYABLE - @staticmethod - def ns_to_duration(nanoseconds): - value = (nanoseconds + 500000) // 1000000 # duration in milliseconds - value, microseconds = divmod(value, 1000) - value, seconds = divmod(value, 60) - value, minutes = divmod(value, 60) - days, hours = divmod(value, 24) - return "{:d}.{:02d}:{:02d}:{:02d}.{:03d}".format( - days, hours, minutes, seconds, microseconds - ) - def span_to_envelope(self, span): # noqa pylint: disable=too-many-branches envelope = protocol.Envelope( iKey=self.options.instrumentation_key, - tags=dict(util.azure_monitor_context), + tags=dict(utils.azure_monitor_context), time=ns_to_iso_str(span.start_time), ) envelope.tags["ai.operation.id"] = "{:032x}".format( @@ -96,7 +85,7 @@ def span_to_envelope(self, span): # noqa pylint: disable=too-many-branches id="|{:032x}.{:016x}.".format( span.context.trace_id, span.context.span_id ), - duration=self.ns_to_duration(span.end_time - span.start_time), + duration=utils.ns_to_duration(span.end_time - span.start_time), responseCode="0", success=False, properties={}, @@ -123,7 +112,7 @@ def span_to_envelope(self, span): # noqa pylint: disable=too-many-branches span.context.trace_id, span.context.span_id ), resultCode="0", # TODO - duration=self.ns_to_duration(span.end_time - span.start_time), + duration=utils.ns_to_duration(span.end_time - span.start_time), success=True, # TODO properties={}, ) diff --git a/azure_monitor/src/azure_monitor/util.py b/azure_monitor/src/azure_monitor/utils.py similarity index 73% rename from azure_monitor/src/azure_monitor/util.py rename to azure_monitor/src/azure_monitor/utils.py index f885ba9..33a596d 100644 --- a/azure_monitor/src/azure_monitor/util.py +++ b/azure_monitor/src/azure_monitor/utils.py @@ -21,6 +21,16 @@ ), } +def ns_to_duration(nanoseconds): + value = (nanoseconds + 500000) // 1000000 # duration in milliseconds + value, microseconds = divmod(value, 1000) + value, seconds = divmod(value, 60) + value, minutes = divmod(value, 60) + days, hours = divmod(value, 24) + return "{:d}.{:02d}:{:02d}:{:02d}.{:03d}".format( + days, hours, minutes, seconds, microseconds + ) + class Options(BaseObject): _default = BaseObject( diff --git a/azure_monitor/tests/__init__.py b/azure_monitor/tests/__init__.py index e69de29..6fcf0de 100644 --- a/azure_monitor/tests/__init__.py +++ b/azure_monitor/tests/__init__.py @@ -0,0 +1,2 @@ +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. \ No newline at end of file diff --git a/azure_monitor/tests/test_protocol.py b/azure_monitor/tests/test_protocol.py new file mode 100644 index 0000000..927cb4d --- /dev/null +++ b/azure_monitor/tests/test_protocol.py @@ -0,0 +1,48 @@ +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. + +import unittest + +from azure_monitor import protocol + +class TestProtocol(unittest.TestCase): + def test_object(self): + data = protocol.BaseObject() + self.assertEqual(repr(data), '{}') + data.foo = 1 + self.assertEqual(data.foo, 1) + self.assertEqual(data['foo'], 1) + data['bar'] = 2 + self.assertEqual(data.bar, 2) + self.assertEqual(data['bar'], 2) + self.assertRaises(KeyError, lambda: data['baz']) + self.assertRaises(AttributeError, lambda: data.baz) + + def test_data(self): + data = protocol.Data() + self.assertIsNone(data.baseData) + self.assertIsNone(data.baseType) + + def test_envelope(self): + data = protocol.Envelope() + self.assertEqual(data.ver, 1) + + def test_event(self): + data = protocol.Event() + self.assertEqual(data.ver, 2) + + def test_exception_data(self): + data = protocol.ExceptionData() + self.assertEqual(data.ver, 2) + + def test_message(self): + data = protocol.Message() + self.assertEqual(data.ver, 2) + + def test_remote_dependency(self): + data = protocol.RemoteDependency() + self.assertEqual(data.ver, 2) + + def test_request(self): + data = protocol.Request() + self.assertEqual(data.ver, 2) diff --git a/azure_monitor/tests/test_utils.py b/azure_monitor/tests/test_utils.py new file mode 100644 index 0000000..6832c4f --- /dev/null +++ b/azure_monitor/tests/test_utils.py @@ -0,0 +1,17 @@ +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. + +import unittest + +from azure_monitor import utils + + +class TestUtils(unittest.TestCase): + def test_nanoseconds_to_duration(self): + ns_to_duration = utils.ns_to_duration + self.assertEqual(ns_to_duration(0), '0.00:00:00.000') + self.assertEqual(ns_to_duration(1000000), '0.00:00:00.001') + self.assertEqual(ns_to_duration(1000000000), '0.00:00:01.000') + self.assertEqual(ns_to_duration(60 * 1000000000), '0.00:01:00.000') + self.assertEqual(ns_to_duration(3600 * 1000000000), '0.01:00:00.000') + self.assertEqual(ns_to_duration(86400 * 1000000000), '1.00:00:00.000') diff --git a/azure_monitor/tests/trace/__init__.py b/azure_monitor/tests/trace/__init__.py new file mode 100644 index 0000000..6fcf0de --- /dev/null +++ b/azure_monitor/tests/trace/__init__.py @@ -0,0 +1,2 @@ +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. \ No newline at end of file diff --git a/azure_monitor/tests/trace/test_trace.py b/azure_monitor/tests/trace/test_trace.py new file mode 100644 index 0000000..2bcf055 --- /dev/null +++ b/azure_monitor/tests/trace/test_trace.py @@ -0,0 +1,446 @@ +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. + +import mock +import time +import unittest + +from azure_monitor.trace import AzureMonitorSpanExporter + +class TestAzureExporter(unittest.TestCase): + def test_ctor(self): + from azure_monitor.utils import Options + instrumentation_key = Options._default.instrumentation_key + Options._default.instrumentation_key = None + self.assertRaises(ValueError, lambda: AzureMonitorSpanExporter()) + Options._default.instrumentation_key = instrumentation_key + + def test_span_data_to_envelope(self): + from opentelemetry.trace import SpanKind + from opentelemetry.sdk.trace import Span + from opentelemetry.trace import SpanContext + + exporter = AzureMonitorSpanExporter( + instrumentation_key='12345678-1234-5678-abcd-12345678abcd' + ) + + parent_span = Span( + name='test', + context=SpanContext( + trace_id=36873507687745823477771305566750195431, + span_id=12030755672171557338, + ), + ) + + start_time = 1575494316027612800 + end_time = start_time + 1001000000 + + # SpanKind.CLIENT HTTP + span = Span( + name='test', + context=SpanContext( + trace_id=36873507687745823477771305566750195431, + span_id=12030755672171557337, + ), + parent=parent_span, + sampler=None, + trace_config=None, + resource=None, + attributes={ + 'component': 'http', + 'http.method': 'GET', + 'http.url': 'https://www.wikipedia.org/wiki/Rabbit', + 'http.status_code': 200, + }, + events=None, + links=None, + kind=SpanKind.CLIENT + ) + span.start_time = start_time + span.end_time = end_time + envelope = exporter.span_to_envelope(span) + self.assertEqual( + envelope.iKey, + '12345678-1234-5678-abcd-12345678abcd') + self.assertEqual( + envelope.name, + 'Microsoft.ApplicationInsights.RemoteDependency') + self.assertEqual( + envelope.tags['ai.operation.parentId'], + '|1bbd944a73a05d89eab5d3740a213ee7.a6f5d48acb4d31da.') + self.assertEqual( + envelope.tags['ai.operation.id'], + '1bbd944a73a05d89eab5d3740a213ee7') + self.assertEqual( + envelope.time, + '2019-12-04T21:18:36.027613Z') + self.assertEqual( + envelope.data.baseData.name, + 'GET//wiki/Rabbit') + self.assertEqual( + envelope.data.baseData.data, + 'https://www.wikipedia.org/wiki/Rabbit') + self.assertEqual( + envelope.data.baseData.target, + 'www.wikipedia.org') + self.assertEqual( + envelope.data.baseData.id, + '|1bbd944a73a05d89eab5d3740a213ee7.a6f5d48acb4d31d9.') + self.assertEqual( + envelope.data.baseData.resultCode, + '200') + self.assertEqual( + envelope.data.baseData.duration, + '0.00:00:01.001') + self.assertEqual( + envelope.data.baseData.type, + 'HTTP') + self.assertEqual( + envelope.data.baseType, + 'RemoteDependencyData') + + # SpanKind.CLIENT unknown type + span = Span( + name='test', + context=SpanContext( + trace_id=36873507687745823477771305566750195431, + span_id=12030755672171557337, + ), + parent=parent_span, + sampler=None, + trace_config=None, + resource=None, + attributes={}, + events=None, + links=None, + kind=SpanKind.CLIENT + ) + span.start_time = start_time + span.end_time = end_time + envelope = exporter.span_to_envelope(span) + self.assertEqual( + envelope.iKey, + '12345678-1234-5678-abcd-12345678abcd') + self.assertEqual( + envelope.name, + 'Microsoft.ApplicationInsights.RemoteDependency') + self.assertEqual( + envelope.tags['ai.operation.parentId'], + '|1bbd944a73a05d89eab5d3740a213ee7.a6f5d48acb4d31da.') + self.assertEqual( + envelope.tags['ai.operation.id'], + '1bbd944a73a05d89eab5d3740a213ee7') + self.assertEqual( + envelope.time, + '2019-12-04T21:18:36.027613Z') + self.assertEqual( + envelope.data.baseData.name, + 'test') + self.assertEqual( + envelope.data.baseData.id, + '|1bbd944a73a05d89eab5d3740a213ee7.a6f5d48acb4d31d9.') + self.assertEqual( + envelope.data.baseData.duration, + '0.00:00:01.001') + self.assertEqual( + envelope.data.baseData.type, + None) + self.assertEqual( + envelope.data.baseType, + 'RemoteDependencyData') + + # SpanKind.CLIENT missing method + span = Span( + name='test', + context=SpanContext( + trace_id=36873507687745823477771305566750195431, + span_id=12030755672171557337, + ), + parent=parent_span, + sampler=None, + trace_config=None, + resource=None, + attributes={ + 'component': 'http', + 'http.url': 'https://www.wikipedia.org/wiki/Rabbit', + 'http.status_code': 200, + }, + events=None, + links=None, + kind=SpanKind.CLIENT + ) + span.start_time = start_time + span.end_time = end_time + envelope = exporter.span_to_envelope(span) + self.assertEqual( + envelope.iKey, + '12345678-1234-5678-abcd-12345678abcd') + self.assertEqual( + envelope.name, + 'Microsoft.ApplicationInsights.RemoteDependency') + self.assertEqual( + envelope.tags['ai.operation.parentId'], + '|1bbd944a73a05d89eab5d3740a213ee7.a6f5d48acb4d31da.') + self.assertEqual( + envelope.tags['ai.operation.id'], + '1bbd944a73a05d89eab5d3740a213ee7') + self.assertEqual( + envelope.time, + '2019-12-04T21:18:36.027613Z') + self.assertEqual( + envelope.data.baseData.name, + 'test') + self.assertEqual( + envelope.data.baseData.data, + 'https://www.wikipedia.org/wiki/Rabbit') + self.assertEqual( + envelope.data.baseData.target, + 'www.wikipedia.org') + self.assertEqual( + envelope.data.baseData.id, + '|1bbd944a73a05d89eab5d3740a213ee7.a6f5d48acb4d31d9.') + self.assertEqual( + envelope.data.baseData.resultCode, + '200') + self.assertEqual( + envelope.data.baseData.duration, + '0.00:00:01.001') + self.assertEqual( + envelope.data.baseData.type, + 'HTTP') + self.assertEqual( + envelope.data.baseType, + 'RemoteDependencyData') + + # SpanKind.SERVER HTTP - 200 request + span = Span( + name='test', + context=SpanContext( + trace_id=36873507687745823477771305566750195431, + span_id=12030755672171557337, + ), + parent=parent_span, + sampler=None, + trace_config=None, + resource=None, + attributes={ + 'component': 'http', + 'http.method': 'GET', + 'http.path': '/wiki/Rabbit', + 'http.route': '/wiki/Rabbit', + 'http.url': 'https://www.wikipedia.org/wiki/Rabbit', + 'http.status_code': 200, + }, + events=None, + links=None, + kind=SpanKind.SERVER + ) + span.start_time = start_time + span.end_time = end_time + envelope = exporter.span_to_envelope(span) + self.assertEqual( + envelope.iKey, + '12345678-1234-5678-abcd-12345678abcd') + self.assertEqual( + envelope.name, + 'Microsoft.ApplicationInsights.Request') + self.assertEqual( + envelope.tags['ai.operation.parentId'], + '|1bbd944a73a05d89eab5d3740a213ee7.a6f5d48acb4d31da.') + self.assertEqual( + envelope.tags['ai.operation.id'], + '1bbd944a73a05d89eab5d3740a213ee7') + self.assertEqual( + envelope.tags['ai.operation.name'], + 'GET /wiki/Rabbit') + self.assertEqual( + envelope.time, + '2019-12-04T21:18:36.027613Z') + self.assertEqual( + envelope.data.baseData.id, + '|1bbd944a73a05d89eab5d3740a213ee7.a6f5d48acb4d31d9.') + self.assertEqual( + envelope.data.baseData.duration, + '0.00:00:01.001') + self.assertEqual( + envelope.data.baseData.responseCode, + '200') + self.assertEqual( + envelope.data.baseData.name, + 'GET /wiki/Rabbit') + self.assertEqual( + envelope.data.baseData.success, + True) + self.assertEqual( + envelope.data.baseData.url, + 'https://www.wikipedia.org/wiki/Rabbit') + self.assertEqual( + envelope.data.baseType, + 'RequestData') + + # SpanKind.SERVER HTTP - Failed request + span = Span( + name='test', + context=SpanContext( + trace_id=36873507687745823477771305566750195431, + span_id=12030755672171557337, + ), + parent=parent_span, + sampler=None, + trace_config=None, + resource=None, + attributes={ + 'component': 'http', + 'http.method': 'GET', + 'http.path': '/wiki/Rabbit', + 'http.route': '/wiki/Rabbit', + 'http.url': 'https://www.wikipedia.org/wiki/Rabbit', + 'http.status_code': 400, + }, + events=None, + links=None, + kind=SpanKind.SERVER + ) + span.start_time = start_time + span.end_time = end_time + envelope = exporter.span_to_envelope(span) + self.assertEqual( + envelope.iKey, + '12345678-1234-5678-abcd-12345678abcd') + self.assertEqual( + envelope.name, + 'Microsoft.ApplicationInsights.Request') + self.assertEqual( + envelope.tags['ai.operation.parentId'], + '|1bbd944a73a05d89eab5d3740a213ee7.a6f5d48acb4d31da.') + self.assertEqual( + envelope.tags['ai.operation.id'], + '1bbd944a73a05d89eab5d3740a213ee7') + self.assertEqual( + envelope.tags['ai.operation.name'], + 'GET /wiki/Rabbit') + self.assertEqual( + envelope.time, + '2019-12-04T21:18:36.027613Z') + self.assertEqual( + envelope.data.baseData.id, + '|1bbd944a73a05d89eab5d3740a213ee7.a6f5d48acb4d31d9.') + self.assertEqual( + envelope.data.baseData.duration, + '0.00:00:01.001') + self.assertEqual( + envelope.data.baseData.responseCode, + '400') + self.assertEqual( + envelope.data.baseData.name, + 'GET /wiki/Rabbit') + self.assertEqual( + envelope.data.baseData.success, + False) + self.assertEqual( + envelope.data.baseData.url, + 'https://www.wikipedia.org/wiki/Rabbit') + self.assertEqual( + envelope.data.baseType, + 'RequestData') + + # SpanKind.SERVER unknown type + span = Span( + name='test', + context=SpanContext( + trace_id=36873507687745823477771305566750195431, + span_id=12030755672171557337, + ), + parent=parent_span, + sampler=None, + trace_config=None, + resource=None, + attributes={ + 'component': 'http', + 'http.method': 'GET', + 'http.path': '/wiki/Rabbit', + 'http.route': '/wiki/Rabbit', + 'http.url': 'https://www.wikipedia.org/wiki/Rabbit', + 'http.status_code': 400, + }, + events=None, + links=None, + kind=SpanKind.SERVER + ) + span.start_time = start_time + span.end_time = end_time + envelope = exporter.span_to_envelope(span) + self.assertEqual( + envelope.iKey, + '12345678-1234-5678-abcd-12345678abcd') + self.assertEqual( + envelope.name, + 'Microsoft.ApplicationInsights.Request') + self.assertEqual( + envelope.tags['ai.operation.parentId'], + '|1bbd944a73a05d89eab5d3740a213ee7.a6f5d48acb4d31da.') + self.assertEqual( + envelope.tags['ai.operation.id'], + '1bbd944a73a05d89eab5d3740a213ee7') + self.assertEqual( + envelope.time, + '2019-12-04T21:18:36.027613Z') + self.assertEqual( + envelope.data.baseData.id, + '|1bbd944a73a05d89eab5d3740a213ee7.a6f5d48acb4d31d9.') + self.assertEqual( + envelope.data.baseData.duration, + '0.00:00:01.001') + self.assertEqual( + envelope.data.baseType, + 'RequestData') + + # SpanKind.INTERNAL + span = Span( + name='test', + context=SpanContext( + trace_id=36873507687745823477771305566750195431, + span_id=12030755672171557337, + ), + parent=None, + sampler=None, + trace_config=None, + resource=None, + attributes={'key1': 'value1'}, + events=None, + links=None, + kind=SpanKind.INTERNAL + ) + span.start_time = start_time + span.end_time = end_time + envelope = exporter.span_to_envelope(span) + self.assertEqual( + envelope.iKey, + '12345678-1234-5678-abcd-12345678abcd') + self.assertEqual( + envelope.name, + 'Microsoft.ApplicationInsights.RemoteDependency') + self.assertRaises( + KeyError, + lambda: envelope.tags['ai.operation.parentId']) + self.assertEqual( + envelope.tags['ai.operation.id'], + '1bbd944a73a05d89eab5d3740a213ee7') + self.assertEqual( + envelope.time, + '2019-12-04T21:18:36.027613Z') + self.assertEqual( + envelope.data.baseData.name, + 'test') + self.assertEqual( + envelope.data.baseData.duration, + '0.00:00:01.001') + self.assertEqual( + envelope.data.baseData.id, + '|1bbd944a73a05d89eab5d3740a213ee7.a6f5d48acb4d31d9.') + self.assertEqual( + envelope.data.baseData.type, + 'InProc') + self.assertEqual( + envelope.data.baseType, + 'RemoteDependencyData') From e5a6a7cd5f254dbafe168ab36a5aa7a47811e7e7 Mon Sep 17 00:00:00 2001 From: Leighton Date: Wed, 4 Dec 2019 14:15:38 -0800 Subject: [PATCH 011/109] rename --- azure_monitor/tests/trace/test_trace.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/azure_monitor/tests/trace/test_trace.py b/azure_monitor/tests/trace/test_trace.py index 2bcf055..c232f32 100644 --- a/azure_monitor/tests/trace/test_trace.py +++ b/azure_monitor/tests/trace/test_trace.py @@ -15,7 +15,7 @@ def test_ctor(self): self.assertRaises(ValueError, lambda: AzureMonitorSpanExporter()) Options._default.instrumentation_key = instrumentation_key - def test_span_data_to_envelope(self): + def test_span_to_envelope(self): from opentelemetry.trace import SpanKind from opentelemetry.sdk.trace import Span from opentelemetry.trace import SpanContext From 61bdfd28c92731057b20baa066f3d91337466f1c Mon Sep 17 00:00:00 2001 From: Leighton Date: Thu, 26 Dec 2019 12:17:25 -0800 Subject: [PATCH 012/109] fix examples --- azure_monitor/examples/client.py | 10 +++++----- azure_monitor/examples/request.py | 11 ++++++----- azure_monitor/examples/server.py | 23 +++++++++++++++-------- azure_monitor/examples/trace.py | 8 ++++---- 4 files changed, 30 insertions(+), 22 deletions(-) diff --git a/azure_monitor/examples/client.py b/azure_monitor/examples/client.py index cc2ae84..6607615 100644 --- a/azure_monitor/examples/client.py +++ b/azure_monitor/examples/client.py @@ -5,16 +5,16 @@ from azure_monitor import AzureMonitorSpanExporter from opentelemetry import trace from opentelemetry.ext import http_requests -from opentelemetry.sdk.trace import Tracer +from opentelemetry.sdk.trace import TracerSource from opentelemetry.sdk.trace.export import BatchExportSpanProcessor -trace.set_preferred_tracer_implementation(lambda T: Tracer()) -tracer = trace.tracer() -http_requests.enable(tracer) +trace.set_preferred_tracer_source_implementation(lambda T: TracerSource()) +tracer = trace.tracer_source().get_tracer(__name__) +http_requests.enable(trace.tracer_source()) span_processor = BatchExportSpanProcessor( AzureMonitorSpanExporter(instrumentation_key="") ) -trace.tracer().add_span_processor(span_processor) +trace.tracer_source().add_span_processor(span_processor) response = requests.get(url="http://127.0.0.1:8080/") span_processor.shutdown() diff --git a/azure_monitor/examples/request.py b/azure_monitor/examples/request.py index c59663b..975ac86 100644 --- a/azure_monitor/examples/request.py +++ b/azure_monitor/examples/request.py @@ -5,17 +5,18 @@ from azure_monitor import AzureMonitorSpanExporter from opentelemetry import trace from opentelemetry.ext import http_requests -from opentelemetry.sdk.trace import Tracer +from opentelemetry.sdk.trace import TracerSource from opentelemetry.sdk.trace.export import SimpleExportSpanProcessor -trace.set_preferred_tracer_implementation(Tracer) +trace.set_preferred_tracer_source_implementation(lambda T: TracerSource()) -http_requests.enable(trace.tracer()) +http_requests.enable(trace.tracer_source()) span_processor = SimpleExportSpanProcessor( AzureMonitorSpanExporter(instrumentation_key="") ) -trace.tracer().add_span_processor(span_processor) +trace.tracer_source().add_span_processor(span_processor) +tracer = trace.tracer_source().get_tracer(__name__) -with trace.tracer().start_as_current_span("parent"): +with tracer.start_as_current_span("parent"): response = requests.get("", timeout=5) diff --git a/azure_monitor/examples/server.py b/azure_monitor/examples/server.py index cf2583f..bdeb4f9 100644 --- a/azure_monitor/examples/server.py +++ b/azure_monitor/examples/server.py @@ -7,24 +7,31 @@ from opentelemetry import trace from opentelemetry.ext import http_requests from opentelemetry.ext.wsgi import OpenTelemetryMiddleware -from opentelemetry.sdk.trace import Tracer +from opentelemetry.sdk.trace import TracerSource from opentelemetry.sdk.trace.export import BatchExportSpanProcessor -trace.set_preferred_tracer_implementation(lambda T: Tracer()) +# The preferred tracer implementation must be set, as the opentelemetry-api +# defines the interface with a no-op implementation. +trace.set_preferred_tracer_source_implementation(lambda T: TracerSource()) +tracer = trace.tracer_source().get_tracer(__name__) -http_requests.enable(trace.tracer()) -span_processor = BatchExportSpanProcessor( - AzureMonitorSpanExporter(instrumentation_key="") -) -trace.tracer().add_span_processor(span_processor) +exporter = AzureMonitorSpanExporter(instrumentation_key="") +# SpanExporter receives the spans and send them to the target location. +span_processor = BatchExportSpanProcessor(exporter) +trace.tracer_source().add_span_processor(span_processor) + +# Integrations are the glue that binds the OpenTelemetry API and the +# frameworks and libraries that are used together, automatically creating +# Spans and propagating context as appropriate. +http_requests.enable(trace.tracer_source()) app = flask.Flask(__name__) app.wsgi_app = OpenTelemetryMiddleware(app.wsgi_app) @app.route("/") def hello(): - with trace.tracer().start_as_current_span("parent"): + with tracer.start_as_current_span("parent"): requests.get("https://www.wikipedia.org/wiki/Rabbit") return "hello" diff --git a/azure_monitor/examples/trace.py b/azure_monitor/examples/trace.py index ae490b0..031fc50 100644 --- a/azure_monitor/examples/trace.py +++ b/azure_monitor/examples/trace.py @@ -2,15 +2,15 @@ # Licensed under the MIT License. from azure_monitor import AzureMonitorSpanExporter from opentelemetry import trace -from opentelemetry.sdk.trace import Tracer +from opentelemetry.sdk.trace import TracerSource from opentelemetry.sdk.trace.export import SimpleExportSpanProcessor -trace.set_preferred_tracer_implementation(lambda T: Tracer()) -tracer = trace.tracer() +trace.set_preferred_tracer_source_implementation(lambda T: TracerSource()) span_processor = SimpleExportSpanProcessor( AzureMonitorSpanExporter(instrumentation_key="") ) -trace.tracer().add_span_processor(span_processor) +trace.tracer_source().add_span_processor(span_processor) +tracer = trace.tracer_source().get_tracer(__name__) with tracer.start_as_current_span("hello") as span: print("Hello, World!") From 1a88035ecda3b2903f33eef1656e46faccdf3cfc Mon Sep 17 00:00:00 2001 From: Leighton Date: Thu, 26 Dec 2019 13:29:40 -0800 Subject: [PATCH 013/109] fix ids, tess --- azure_monitor/src/azure_monitor/trace.py | 17 +++-- azure_monitor/tests/trace/test_trace.py | 97 ++++++++++++++++++++---- 2 files changed, 92 insertions(+), 22 deletions(-) diff --git a/azure_monitor/src/azure_monitor/trace.py b/azure_monitor/src/azure_monitor/trace.py index 1c61fe7..d8de38b 100644 --- a/azure_monitor/src/azure_monitor/trace.py +++ b/azure_monitor/src/azure_monitor/trace.py @@ -78,12 +78,12 @@ def span_to_envelope(self, span): # noqa pylint: disable=too-many-branches if parent: envelope.tags[ "ai.operation.parentId" - ] = "|{:032x}.{:016x}.".format(parent.trace_id, parent.span_id) + ] = "{:016x}".format(parent.span_id) if span.kind in (SpanKind.CONSUMER, SpanKind.SERVER): envelope.name = "Microsoft.ApplicationInsights.Request" data = protocol.Request( - id="|{:032x}.{:016x}.".format( - span.context.trace_id, span.context.span_id + id="{:016x}".format( + span.context.span_id ), duration=utils.ns_to_duration(span.end_time - span.start_time), responseCode="0", @@ -108,8 +108,8 @@ def span_to_envelope(self, span): # noqa pylint: disable=too-many-branches envelope.name = "Microsoft.ApplicationInsights.RemoteDependency" data = protocol.RemoteDependency( name=span.name, - id="|{:032x}.{:016x}.".format( - span.context.trace_id, span.context.span_id + id="{:016x}".format( + span.context.span_id ), resultCode="0", # TODO duration=utils.ns_to_duration(span.end_time - span.start_time), @@ -140,6 +140,9 @@ def span_to_envelope(self, span): # noqa pylint: disable=too-many-branches else: # SpanKind.INTERNAL data.type = "InProc" for key in span.attributes: + # This removes redundant data from ApplicationInsights + if key.startswith('http.'): + continue data.properties[key] = span.attributes[key] if span.links: links = [] @@ -149,8 +152,8 @@ def span_to_envelope(self, span): # noqa pylint: disable=too-many-branches "operation_Id": "{:032x}".format( link.context.trace_id ), - "id": "|{:032x}.{:016x}.".format( - link.context.trace_id, link.context.span_id + "id": "{:016x}".format( + link.context.span_id ), } ) diff --git a/azure_monitor/tests/trace/test_trace.py b/azure_monitor/tests/trace/test_trace.py index c232f32..e8697ed 100644 --- a/azure_monitor/tests/trace/test_trace.py +++ b/azure_monitor/tests/trace/test_trace.py @@ -16,9 +16,8 @@ def test_ctor(self): Options._default.instrumentation_key = instrumentation_key def test_span_to_envelope(self): - from opentelemetry.trace import SpanKind + from opentelemetry.trace import Link, SpanContext, SpanKind from opentelemetry.sdk.trace import Span - from opentelemetry.trace import SpanContext exporter = AzureMonitorSpanExporter( instrumentation_key='12345678-1234-5678-abcd-12345678abcd' @@ -67,7 +66,7 @@ def test_span_to_envelope(self): 'Microsoft.ApplicationInsights.RemoteDependency') self.assertEqual( envelope.tags['ai.operation.parentId'], - '|1bbd944a73a05d89eab5d3740a213ee7.a6f5d48acb4d31da.') + 'a6f5d48acb4d31da') self.assertEqual( envelope.tags['ai.operation.id'], '1bbd944a73a05d89eab5d3740a213ee7') @@ -85,7 +84,7 @@ def test_span_to_envelope(self): 'www.wikipedia.org') self.assertEqual( envelope.data.baseData.id, - '|1bbd944a73a05d89eab5d3740a213ee7.a6f5d48acb4d31d9.') + 'a6f5d48acb4d31d9') self.assertEqual( envelope.data.baseData.resultCode, '200') @@ -126,7 +125,7 @@ def test_span_to_envelope(self): 'Microsoft.ApplicationInsights.RemoteDependency') self.assertEqual( envelope.tags['ai.operation.parentId'], - '|1bbd944a73a05d89eab5d3740a213ee7.a6f5d48acb4d31da.') + 'a6f5d48acb4d31da') self.assertEqual( envelope.tags['ai.operation.id'], '1bbd944a73a05d89eab5d3740a213ee7') @@ -138,7 +137,7 @@ def test_span_to_envelope(self): 'test') self.assertEqual( envelope.data.baseData.id, - '|1bbd944a73a05d89eab5d3740a213ee7.a6f5d48acb4d31d9.') + 'a6f5d48acb4d31d9') self.assertEqual( envelope.data.baseData.duration, '0.00:00:01.001') @@ -180,7 +179,7 @@ def test_span_to_envelope(self): 'Microsoft.ApplicationInsights.RemoteDependency') self.assertEqual( envelope.tags['ai.operation.parentId'], - '|1bbd944a73a05d89eab5d3740a213ee7.a6f5d48acb4d31da.') + 'a6f5d48acb4d31da') self.assertEqual( envelope.tags['ai.operation.id'], '1bbd944a73a05d89eab5d3740a213ee7') @@ -198,7 +197,7 @@ def test_span_to_envelope(self): 'www.wikipedia.org') self.assertEqual( envelope.data.baseData.id, - '|1bbd944a73a05d89eab5d3740a213ee7.a6f5d48acb4d31d9.') + 'a6f5d48acb4d31d9') self.assertEqual( envelope.data.baseData.resultCode, '200') @@ -246,7 +245,7 @@ def test_span_to_envelope(self): 'Microsoft.ApplicationInsights.Request') self.assertEqual( envelope.tags['ai.operation.parentId'], - '|1bbd944a73a05d89eab5d3740a213ee7.a6f5d48acb4d31da.') + 'a6f5d48acb4d31da') self.assertEqual( envelope.tags['ai.operation.id'], '1bbd944a73a05d89eab5d3740a213ee7') @@ -258,7 +257,7 @@ def test_span_to_envelope(self): '2019-12-04T21:18:36.027613Z') self.assertEqual( envelope.data.baseData.id, - '|1bbd944a73a05d89eab5d3740a213ee7.a6f5d48acb4d31d9.') + 'a6f5d48acb4d31d9') self.assertEqual( envelope.data.baseData.duration, '0.00:00:01.001') @@ -312,7 +311,7 @@ def test_span_to_envelope(self): 'Microsoft.ApplicationInsights.Request') self.assertEqual( envelope.tags['ai.operation.parentId'], - '|1bbd944a73a05d89eab5d3740a213ee7.a6f5d48acb4d31da.') + 'a6f5d48acb4d31da') self.assertEqual( envelope.tags['ai.operation.id'], '1bbd944a73a05d89eab5d3740a213ee7') @@ -324,7 +323,7 @@ def test_span_to_envelope(self): '2019-12-04T21:18:36.027613Z') self.assertEqual( envelope.data.baseData.id, - '|1bbd944a73a05d89eab5d3740a213ee7.a6f5d48acb4d31d9.') + 'a6f5d48acb4d31d9') self.assertEqual( envelope.data.baseData.duration, '0.00:00:01.001') @@ -378,7 +377,7 @@ def test_span_to_envelope(self): 'Microsoft.ApplicationInsights.Request') self.assertEqual( envelope.tags['ai.operation.parentId'], - '|1bbd944a73a05d89eab5d3740a213ee7.a6f5d48acb4d31da.') + 'a6f5d48acb4d31da') self.assertEqual( envelope.tags['ai.operation.id'], '1bbd944a73a05d89eab5d3740a213ee7') @@ -387,7 +386,7 @@ def test_span_to_envelope(self): '2019-12-04T21:18:36.027613Z') self.assertEqual( envelope.data.baseData.id, - '|1bbd944a73a05d89eab5d3740a213ee7.a6f5d48acb4d31d9.') + 'a6f5d48acb4d31d9') self.assertEqual( envelope.data.baseData.duration, '0.00:00:01.001') @@ -437,10 +436,78 @@ def test_span_to_envelope(self): '0.00:00:01.001') self.assertEqual( envelope.data.baseData.id, - '|1bbd944a73a05d89eab5d3740a213ee7.a6f5d48acb4d31d9.') + 'a6f5d48acb4d31d9') self.assertEqual( envelope.data.baseData.type, 'InProc') self.assertEqual( envelope.data.baseType, 'RemoteDependencyData') + + # Attributes + span = Span( + name='test', + context=SpanContext( + trace_id=36873507687745823477771305566750195431, + span_id=12030755672171557337, + ), + parent=parent_span, + sampler=None, + trace_config=None, + resource=None, + attributes={ + 'component': 'http', + 'http.method': 'GET', + 'http.url': 'https://www.wikipedia.org/wiki/Rabbit', + 'http.status_code': 200, + 'test': 'asd' + }, + events=None, + links=None, + kind=SpanKind.CLIENT + ) + span.start_time = start_time + span.end_time = end_time + envelope = exporter.span_to_envelope(span) + self.assertEqual( + len(envelope.data.baseData.properties), 2) + self.assertEqual( + envelope.data.baseData.properties['component'], 'http') + self.assertEqual(envelope.data.baseData.properties['test'], 'asd') + + # Links + links = [] + links.append(Link(context=SpanContext( + trace_id=36873507687745823477771305566750195432, + span_id=12030755672171557338, + ))) + span = Span( + name='test', + context=SpanContext( + trace_id=36873507687745823477771305566750195431, + span_id=12030755672171557337, + ), + parent=parent_span, + sampler=None, + trace_config=None, + resource=None, + attributes={ + 'component': 'http', + 'http.method': 'GET', + 'http.url': 'https://www.wikipedia.org/wiki/Rabbit', + 'http.status_code': 200, + }, + events=None, + links=links, + kind=SpanKind.CLIENT + ) + span.start_time = start_time + span.end_time = end_time + envelope = exporter.span_to_envelope(span) + self.assertEqual( + len(envelope.data.baseData.properties), 2) + links_json = '[{"operation_Id": ' + \ + '"1bbd944a73a05d89eab5d3740a213ee8", "id": "a6f5d48acb4d31da"}]' + self.assertEqual(envelope.data.baseData.properties['_MS.links'], links_json) + + \ No newline at end of file From 3650497c6daab313c0fbae6ab42db5e4d99d3b77 Mon Sep 17 00:00:00 2001 From: Leighton Date: Thu, 26 Dec 2019 14:14:55 -0800 Subject: [PATCH 014/109] fix status, tests --- azure_monitor/src/azure_monitor/trace.py | 15 ++- azure_monitor/tests/trace/test_trace.py | 165 ++++++++++++++++++++++- 2 files changed, 175 insertions(+), 5 deletions(-) diff --git a/azure_monitor/src/azure_monitor/trace.py b/azure_monitor/src/azure_monitor/trace.py index d8de38b..e3a346a 100644 --- a/azure_monitor/src/azure_monitor/trace.py +++ b/azure_monitor/src/azure_monitor/trace.py @@ -10,6 +10,7 @@ from opentelemetry.sdk.trace.export import SpanExporter, SpanExportResult from opentelemetry.sdk.util import ns_to_iso_str from opentelemetry.trace import Span, SpanKind +from opentelemetry.trace.status import StatusCanonicalCode logger = logging.getLogger(__name__) @@ -87,7 +88,7 @@ def span_to_envelope(self, span): # noqa pylint: disable=too-many-branches ), duration=utils.ns_to_duration(span.end_time - span.start_time), responseCode="0", - success=False, + success=False, # Modify based off attributes or Status properties={}, ) envelope.data = protocol.Data( @@ -104,6 +105,8 @@ def span_to_envelope(self, span): # noqa pylint: disable=too-many-branches status_code = span.attributes["http.status_code"] data.responseCode = str(status_code) data.success = 200 <= status_code < 400 + elif span.status == StatusCanonicalCode.OK: + data.success = True else: envelope.name = "Microsoft.ApplicationInsights.RemoteDependency" data = protocol.RemoteDependency( @@ -111,9 +114,9 @@ def span_to_envelope(self, span): # noqa pylint: disable=too-many-branches id="{:016x}".format( span.context.span_id ), - resultCode="0", # TODO + resultCode="0", duration=utils.ns_to_duration(span.end_time - span.start_time), - success=True, # TODO + success=False, # Modify based off attributes or Status properties={}, ) envelope.data = protocol.Data( @@ -136,7 +139,11 @@ def span_to_envelope(self, span): # noqa pylint: disable=too-many-branches data.name = span.attributes["http.method"] \ + "/" + parse_url.path if "http.status_code" in span.attributes: - data.resultCode = str(span.attributes["http.status_code"]) + status_code = span.attributes["http.status_code"] + data.resultCode = str(status_code) + data.success = 200 <= status_code < 400 + elif span.status == StatusCanonicalCode.OK: + data.success = True else: # SpanKind.INTERNAL data.type = "InProc" for key in span.attributes: diff --git a/azure_monitor/tests/trace/test_trace.py b/azure_monitor/tests/trace/test_trace.py index e8697ed..2ff2184 100644 --- a/azure_monitor/tests/trace/test_trace.py +++ b/azure_monitor/tests/trace/test_trace.py @@ -17,6 +17,7 @@ def test_ctor(self): def test_span_to_envelope(self): from opentelemetry.trace import Link, SpanContext, SpanKind + from opentelemetry.trace.status import StatusCanonicalCode from opentelemetry.sdk.trace import Span exporter = AzureMonitorSpanExporter( @@ -510,4 +511,166 @@ def test_span_to_envelope(self): '"1bbd944a73a05d89eab5d3740a213ee8", "id": "a6f5d48acb4d31da"}]' self.assertEqual(envelope.data.baseData.properties['_MS.links'], links_json) - \ No newline at end of file + + # Status + span = Span( + name='test', + context=SpanContext( + trace_id=36873507687745823477771305566750195431, + span_id=12030755672171557337, + ), + parent=parent_span, + sampler=None, + trace_config=None, + resource=None, + attributes={ + 'component': 'http', + 'http.method': 'GET', + 'http.url': 'https://www.wikipedia.org/wiki/Rabbit', + 'http.status_code': 500, + }, + events=None, + links=None, + kind=SpanKind.SERVER + ) + span.start_time = start_time + span.end_time = end_time + envelope = exporter.span_to_envelope(span) + self.assertEqual( + envelope.data.baseData.responseCode, '500') + self.assertFalse(envelope.data.baseData.success) + + span = Span( + name='test', + context=SpanContext( + trace_id=36873507687745823477771305566750195431, + span_id=12030755672171557337, + ), + parent=parent_span, + sampler=None, + trace_config=None, + resource=None, + attributes={ + 'component': 'http', + 'http.method': 'GET', + 'http.url': 'https://www.wikipedia.org/wiki/Rabbit', + 'http.status_code': 500, + }, + events=None, + links=None, + kind=SpanKind.CLIENT + ) + span.start_time = start_time + span.end_time = end_time + envelope = exporter.span_to_envelope(span) + self.assertEqual( + envelope.data.baseData.resultCode, '500') + self.assertFalse(envelope.data.baseData.success) + + span = Span( + name='test', + context=SpanContext( + trace_id=36873507687745823477771305566750195431, + span_id=12030755672171557337, + ), + parent=parent_span, + sampler=None, + trace_config=None, + resource=None, + attributes={ + 'component': 'http', + 'http.method': 'GET', + 'http.url': 'https://www.wikipedia.org/wiki/Rabbit', + }, + events=None, + links=None, + kind=SpanKind.SERVER + ) + span.start_time = start_time + span.end_time = end_time + span.status = StatusCanonicalCode.OK + envelope = exporter.span_to_envelope(span) + self.assertEqual( + envelope.data.baseData.responseCode, '0') + self.assertTrue(envelope.data.baseData.success) + + span = Span( + name='test', + context=SpanContext( + trace_id=36873507687745823477771305566750195431, + span_id=12030755672171557337, + ), + parent=parent_span, + sampler=None, + trace_config=None, + resource=None, + attributes={ + 'component': 'http', + 'http.method': 'GET', + 'http.url': 'https://www.wikipedia.org/wiki/Rabbit', + }, + events=None, + links=None, + kind=SpanKind.CLIENT + ) + span.start_time = start_time + span.end_time = end_time + span.status = StatusCanonicalCode.OK + envelope = exporter.span_to_envelope(span) + self.assertEqual( + envelope.data.baseData.resultCode, '0') + self.assertTrue(envelope.data.baseData.success) + + span = Span( + name='test', + context=SpanContext( + trace_id=36873507687745823477771305566750195431, + span_id=12030755672171557337, + ), + parent=parent_span, + sampler=None, + trace_config=None, + resource=None, + attributes={ + 'component': 'http', + 'http.method': 'GET', + 'http.url': 'https://www.wikipedia.org/wiki/Rabbit', + }, + events=None, + links=None, + kind=SpanKind.SERVER + ) + span.start_time = start_time + span.end_time = end_time + span.status = StatusCanonicalCode.UNKNOWN + envelope = exporter.span_to_envelope(span) + self.assertEqual( + envelope.data.baseData.responseCode, '0') + self.assertFalse(envelope.data.baseData.success) + + span = Span( + name='test', + context=SpanContext( + trace_id=36873507687745823477771305566750195431, + span_id=12030755672171557337, + ), + parent=parent_span, + sampler=None, + trace_config=None, + resource=None, + attributes={ + 'component': 'http', + 'http.method': 'GET', + 'http.url': 'https://www.wikipedia.org/wiki/Rabbit', + }, + events=None, + links=None, + kind=SpanKind.CLIENT + ) + span.start_time = start_time + span.end_time = end_time + span.status = StatusCanonicalCode.UNKNOWN + envelope = exporter.span_to_envelope(span) + self.assertEqual( + envelope.data.baseData.resultCode, '0') + self.assertFalse(envelope.data.baseData.success) From 9652011fef1555d08967875c1af7c7be72613359 Mon Sep 17 00:00:00 2001 From: Leighton Date: Fri, 27 Dec 2019 13:51:08 -0800 Subject: [PATCH 015/109] use status --- azure_monitor/src/azure_monitor/trace.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/azure_monitor/src/azure_monitor/trace.py b/azure_monitor/src/azure_monitor/trace.py index e3a346a..b1f855c 100644 --- a/azure_monitor/src/azure_monitor/trace.py +++ b/azure_monitor/src/azure_monitor/trace.py @@ -87,7 +87,7 @@ def span_to_envelope(self, span): # noqa pylint: disable=too-many-branches span.context.span_id ), duration=utils.ns_to_duration(span.end_time - span.start_time), - responseCode="0", + responseCode=str(StatusCanonicalCode.OK.value), success=False, # Modify based off attributes or Status properties={}, ) @@ -114,7 +114,7 @@ def span_to_envelope(self, span): # noqa pylint: disable=too-many-branches id="{:016x}".format( span.context.span_id ), - resultCode="0", + resultCode=str(StatusCanonicalCode.OK.value), duration=utils.ns_to_duration(span.end_time - span.start_time), success=False, # Modify based off attributes or Status properties={}, From d019c9f9b02a9b6872903c36a4ec4d9dad61555a Mon Sep 17 00:00:00 2001 From: Leighton Date: Fri, 27 Dec 2019 14:06:43 -0800 Subject: [PATCH 016/109] usde span status --- azure_monitor/src/azure_monitor/trace.py | 4 ++-- azure_monitor/tests/trace/test_trace.py | 17 +++++++++++++++-- 2 files changed, 17 insertions(+), 4 deletions(-) diff --git a/azure_monitor/src/azure_monitor/trace.py b/azure_monitor/src/azure_monitor/trace.py index b1f855c..14da4cd 100644 --- a/azure_monitor/src/azure_monitor/trace.py +++ b/azure_monitor/src/azure_monitor/trace.py @@ -87,7 +87,7 @@ def span_to_envelope(self, span): # noqa pylint: disable=too-many-branches span.context.span_id ), duration=utils.ns_to_duration(span.end_time - span.start_time), - responseCode=str(StatusCanonicalCode.OK.value), + responseCode=str(span.status.value), success=False, # Modify based off attributes or Status properties={}, ) @@ -114,7 +114,7 @@ def span_to_envelope(self, span): # noqa pylint: disable=too-many-branches id="{:016x}".format( span.context.span_id ), - resultCode=str(StatusCanonicalCode.OK.value), + resultCode=str(span.status.value), duration=utils.ns_to_duration(span.end_time - span.start_time), success=False, # Modify based off attributes or Status properties={}, diff --git a/azure_monitor/tests/trace/test_trace.py b/azure_monitor/tests/trace/test_trace.py index 2ff2184..ab76f6f 100644 --- a/azure_monitor/tests/trace/test_trace.py +++ b/azure_monitor/tests/trace/test_trace.py @@ -56,6 +56,7 @@ def test_span_to_envelope(self): links=None, kind=SpanKind.CLIENT ) + span.status = StatusCanonicalCode.OK span.start_time = start_time span.end_time = end_time envelope = exporter.span_to_envelope(span) @@ -115,6 +116,7 @@ def test_span_to_envelope(self): links=None, kind=SpanKind.CLIENT ) + span.status = StatusCanonicalCode.OK span.start_time = start_time span.end_time = end_time envelope = exporter.span_to_envelope(span) @@ -169,6 +171,7 @@ def test_span_to_envelope(self): links=None, kind=SpanKind.CLIENT ) + span.status = StatusCanonicalCode.OK span.start_time = start_time span.end_time = end_time envelope = exporter.span_to_envelope(span) @@ -235,6 +238,7 @@ def test_span_to_envelope(self): links=None, kind=SpanKind.SERVER ) + span.status = StatusCanonicalCode.OK span.start_time = start_time span.end_time = end_time envelope = exporter.span_to_envelope(span) @@ -301,6 +305,7 @@ def test_span_to_envelope(self): links=None, kind=SpanKind.SERVER ) + span.status = StatusCanonicalCode.OK span.start_time = start_time span.end_time = end_time envelope = exporter.span_to_envelope(span) @@ -367,6 +372,7 @@ def test_span_to_envelope(self): links=None, kind=SpanKind.SERVER ) + span.status = StatusCanonicalCode.OK span.start_time = start_time span.end_time = end_time envelope = exporter.span_to_envelope(span) @@ -411,6 +417,7 @@ def test_span_to_envelope(self): links=None, kind=SpanKind.INTERNAL ) + span.status = StatusCanonicalCode.OK span.start_time = start_time span.end_time = end_time envelope = exporter.span_to_envelope(span) @@ -467,6 +474,7 @@ def test_span_to_envelope(self): links=None, kind=SpanKind.CLIENT ) + span.status = StatusCanonicalCode.OK span.start_time = start_time span.end_time = end_time envelope = exporter.span_to_envelope(span) @@ -502,6 +510,7 @@ def test_span_to_envelope(self): links=links, kind=SpanKind.CLIENT ) + span.status = StatusCanonicalCode.OK span.start_time = start_time span.end_time = end_time envelope = exporter.span_to_envelope(span) @@ -533,6 +542,7 @@ def test_span_to_envelope(self): links=None, kind=SpanKind.SERVER ) + span.status = StatusCanonicalCode.OK span.start_time = start_time span.end_time = end_time envelope = exporter.span_to_envelope(span) @@ -560,6 +570,7 @@ def test_span_to_envelope(self): links=None, kind=SpanKind.CLIENT ) + span.status = StatusCanonicalCode.OK span.start_time = start_time span.end_time = end_time envelope = exporter.span_to_envelope(span) @@ -586,6 +597,7 @@ def test_span_to_envelope(self): links=None, kind=SpanKind.SERVER ) + span.status = StatusCanonicalCode.OK span.start_time = start_time span.end_time = end_time span.status = StatusCanonicalCode.OK @@ -613,6 +625,7 @@ def test_span_to_envelope(self): links=None, kind=SpanKind.CLIENT ) + span.status = StatusCanonicalCode.OK span.start_time = start_time span.end_time = end_time span.status = StatusCanonicalCode.OK @@ -645,7 +658,7 @@ def test_span_to_envelope(self): span.status = StatusCanonicalCode.UNKNOWN envelope = exporter.span_to_envelope(span) self.assertEqual( - envelope.data.baseData.responseCode, '0') + envelope.data.baseData.responseCode, '2') self.assertFalse(envelope.data.baseData.success) span = Span( @@ -672,5 +685,5 @@ def test_span_to_envelope(self): span.status = StatusCanonicalCode.UNKNOWN envelope = exporter.span_to_envelope(span) self.assertEqual( - envelope.data.baseData.resultCode, '0') + envelope.data.baseData.resultCode, '2') self.assertFalse(envelope.data.baseData.success) From 04540e9041627d5f1d281cb33964bcd6e570b65c Mon Sep 17 00:00:00 2001 From: Victor Lima Date: Thu, 16 Jan 2020 15:49:41 -0300 Subject: [PATCH 017/109] Rewrite protocol classes to be more user friendly and use less size --- azure_monitor/src/azure_monitor/protocol.py | 303 +++++----- azure_monitor/src/azure_monitor/trace.py | 11 +- azure_monitor/src/azure_monitor/utils.py | 14 +- azure_monitor/tests/test_protocol.py | 23 +- azure_monitor/tests/trace/test_trace.py | 594 ++++++++------------ 5 files changed, 432 insertions(+), 513 deletions(-) diff --git a/azure_monitor/src/azure_monitor/protocol.py b/azure_monitor/src/azure_monitor/protocol.py index 0de20a9..54e1c48 100644 --- a/azure_monitor/src/azure_monitor/protocol.py +++ b/azure_monitor/src/azure_monitor/protocol.py @@ -1,61 +1,29 @@ # Copyright (c) Microsoft Corporation. All rights reserved. # Licensed under the MIT License. -class BaseObject(dict): - def __init__(self, *args, **kwargs): - super().__init__(*args, **kwargs) - for key in kwargs: - self[key] = kwargs[key] - - def __repr__(self): - tmp = {} - current = self - while True: - for item in current.items(): - if item[0] not in tmp: - tmp[item[0]] = item[1] - if ( - current._default # noqa pylint: disable=protected-access - == current - ): - break - current = current._default # noqa pylint: disable=protected-access - return repr(tmp) - - def __setattr__(self, name, value): - self[name] = value - - def __getattr__(self, name): - try: - return self[name] - except KeyError: - raise AttributeError( - "'{}' object has no attribute {}".format( - type(self).__name__, name - ) - ) - - def __getitem__(self, key): - if self._default is self: - return super().__getitem__(key) - if key in self: - return super().__getitem__(key) - return self._default[key] - - -BaseObject._default = BaseObject() # noqa pylint: disable=protected-access - - -class Data(BaseObject): - _default = BaseObject(baseData=None, baseType=None) - - def __init__(self, *args, **kwargs): - super().__init__(*args, **kwargs) - self.baseData = self.baseData # noqa pylint: disable=invalid-name - self.baseType = self.baseType # noqa pylint: disable=invalid-name - - -class DataPoint(BaseObject): - _default = BaseObject( + + +class Data: + __slots__ = ("baseData", "baseType") + + def __init__(self, baseData=None, baseType=None) -> None: + self.baseData = baseData + self.baseType = baseType + + +class DataPoint: + __slots__ = ( + "ns", + "name", + "kind", + "value", + "count", + "min", + "max", + "std_dev", + ) + + def __init__( + self, ns="", name="", kind=None, @@ -63,17 +31,33 @@ class DataPoint(BaseObject): count=None, min=None, max=None, - stdDev=None, + std_dev=None, + ) -> None: + self.ns = ns + self.name = name + self.kind = kind + self.value = value + self.count = count + self.min = min + self.max = max + self.std_dev = std_dev + + +class Envelope: + __slots__ = ( + "ver", + "name", + "time", + "sampleRate", + "seq", + "iKey", + "flags", + "tags", + "data", ) - def __init__(self, *args, **kwargs): - super().__init__(*args, **kwargs) - self.name = self.name - self.value = self.value - - -class Envelope(BaseObject): - _default = BaseObject( + def __init__( + self, ver=1, name="", time="", @@ -83,65 +67,109 @@ class Envelope(BaseObject): flags=None, tags=None, data=None, + ) -> None: + self.ver = ver + self.name = name + self.time = time + self.sampleRate = sampleRate + self.seq = seq + self.iKey = iKey + self.flags = flags + self.tags = tags + self.data = data + + +class Event: + __slots__ = ("ver", "name", "properties", "measurements") + + def __init__(self, ver=2, name="", properties=None, measurements=None): + self.ver = ver + self.name = name + self.properties = properties + self.measurements = measurements + + +class ExceptionData: + __slots__ = ( + "ver", + "exceptions", + "severityLevel", + "problemId", + "properties", + "measurements", ) - def __init__(self, *args, **kwargs): - super().__init__(*args, **kwargs) - self.name = self.name - self.time = self.time - - -class Event(BaseObject): - _default = BaseObject(ver=2, name="", properties=None, measurements=None) - - def __init__(self, *args, **kwargs): - super().__init__(*args, **kwargs) - self.ver = self.ver - self.name = self.name - - -class ExceptionData(BaseObject): - _default = BaseObject( + def __init__( + self, ver=2, - exceptions=[], + exceptions=None, severityLevel=None, problemId=None, properties=None, measurements=None, + ) -> None: + if exceptions is None: + exceptions = [] + self.ver = ver + self.exceptions = exceptions + self.severityLevel = severityLevel + self.problemId = problemId + self.properties = properties + self.measurements = measurements + + +class Message: + __slots__ = ( + "ver", + "message", + "severityLevel", + "properties", + "measurements", ) - def __init__(self, *args, **kwargs): - super().__init__(*args, **kwargs) - self.ver = self.ver - self.exceptions = self.exceptions - - -class Message(BaseObject): - _default = BaseObject( + def __init__( + self, ver=2, message="", severityLevel=None, properties=None, measurements=None, + ) -> None: + self.ver = ver + self.message = message + self.severityLevel = severityLevel + self.properties = properties + self.measurements = measurements + + +class MetricData: + __slots__ = ("ver", "metrics", "properties") + + def __init__(self, ver=2, metrics=None, properties=None) -> None: + if metrics is None: + metrics = [] + self.ver = ver + self.metrics = metrics + self.properties = properties + + +class RemoteDependency: + __slots__ = ( + "ver", + "name", + "id", + "resultCode", + "duration", + "success", + "data", + "type", + "target", + "properties", + "measurements", ) - def __init__(self, *args, **kwargs): - super().__init__(*args, **kwargs) - self.ver = self.ver - self.message = self.message - - -class MetricData(BaseObject): - _default = BaseObject(ver=2, metrics=[], properties=None) - - def __init__(self, *args, **kwargs): - super().__init__(*args, **kwargs) - self.ver = self.ver - self.metrics = self.metrics - - -class RemoteDependency(BaseObject): - _default = BaseObject( + def __init__( + self, ver=2, name="", id="", @@ -153,18 +181,36 @@ class RemoteDependency(BaseObject): target=None, properties=None, measurements=None, + ) -> None: + self.ver = ver + self.name = name + self.id = id + self.resultCode = resultCode + self.duration = duration + self.success = success + self.data = data + self.type = type + self.target = target + self.properties = properties + self.measurements = measurements + + +class Request: + __slots__ = ( + "ver", + "id", + "duration", + "responseCode", + "success", + "source", + "name", + "url", + "properties", + "measurements", ) - def __init__(self, *args, **kwargs): - super().__init__(*args, **kwargs) - self.ver = self.ver - self.name = self.name - self.resultCode = self.resultCode # noqa pylint: disable=invalid-name - self.duration = self.duration - - -class Request(BaseObject): - _default = BaseObject( + def __init__( + self, ver=2, id="", duration="", @@ -175,14 +221,15 @@ class Request(BaseObject): url=None, properties=None, measurements=None, - ) + ) -> None: + self.ver = ver + self.id = id + self.duration = duration + self.responseCode = responseCode + self.success = success + self.source = source + self.name = name + self.url = url + self.properties = properties + self.measurements = measurements - def __init__(self, *args, **kwargs): - super().__init__(*args, **kwargs) - self.ver = self.ver - self.id = self.id # noqa pylint: disable=invalid-name - self.duration = self.duration - self.responseCode = ( # noqa pylint: disable=invalid-name - self.responseCode - ) - self.success = self.success diff --git a/azure_monitor/src/azure_monitor/trace.py b/azure_monitor/src/azure_monitor/trace.py index 14da4cd..4394c21 100644 --- a/azure_monitor/src/azure_monitor/trace.py +++ b/azure_monitor/src/azure_monitor/trace.py @@ -16,8 +16,8 @@ class AzureMonitorSpanExporter(SpanExporter): - def __init__(self, **options): - self.options = utils.Options(**options) + def __init__(self, options: utils.Options): + self.options = options if not self.options.instrumentation_key: raise ValueError("The instrumentation_key is not provided.") @@ -88,7 +88,7 @@ def span_to_envelope(self, span): # noqa pylint: disable=too-many-branches ), duration=utils.ns_to_duration(span.end_time - span.start_time), responseCode=str(span.status.value), - success=False, # Modify based off attributes or Status + success=False, # Modify based off attributes or Status properties={}, ) envelope.data = protocol.Data( @@ -116,7 +116,7 @@ def span_to_envelope(self, span): # noqa pylint: disable=too-many-branches ), resultCode=str(span.status.value), duration=utils.ns_to_duration(span.end_time - span.start_time), - success=False, # Modify based off attributes or Status + success=False, # Modify based off attributes or Status properties={}, ) envelope.data = protocol.Data( @@ -124,7 +124,7 @@ def span_to_envelope(self, span): # noqa pylint: disable=too-many-branches ) if span.kind in (SpanKind.CLIENT, SpanKind.PRODUCER): if "component" in span.attributes and \ - span.attributes["component"] == "http": + span.attributes["component"] == "http": data.type = "HTTP" if "http.url" in span.attributes: url = span.attributes["http.url"] @@ -165,6 +165,5 @@ def span_to_envelope(self, span): # noqa pylint: disable=too-many-branches } ) data.properties["_MS.links"] = json.dumps(links) - print(data.properties["_MS.links"]) # TODO: tracestate, tags return envelope diff --git a/azure_monitor/src/azure_monitor/utils.py b/azure_monitor/src/azure_monitor/utils.py index 33a596d..6d06033 100644 --- a/azure_monitor/src/azure_monitor/utils.py +++ b/azure_monitor/src/azure_monitor/utils.py @@ -5,7 +5,6 @@ import platform import sys -from azure_monitor.protocol import BaseObject from azure_monitor.version import __version__ as ext_version from opentelemetry.sdk.version import __version__ as opentelemetry_version @@ -21,6 +20,7 @@ ), } + def ns_to_duration(nanoseconds): value = (nanoseconds + 500000) // 1000000 # duration in milliseconds value, microseconds = divmod(value, 1000) @@ -32,9 +32,15 @@ def ns_to_duration(nanoseconds): ) -class Options(BaseObject): - _default = BaseObject( +class Options: + __slots__ = ("endpoint", "instrumentation_key", "timeout") + + def __init__( + self, endpoint="https://dc.services.visualstudio.com/v2/track", instrumentation_key=os.getenv("APPINSIGHTS_INSTRUMENTATIONKEY", None), timeout=10.0, # networking timeout in seconds - ) + ) -> None: + self.endpoint = endpoint + self.instrumentation_key = instrumentation_key + self.timeout = timeout diff --git a/azure_monitor/tests/test_protocol.py b/azure_monitor/tests/test_protocol.py index 927cb4d..21c800e 100644 --- a/azure_monitor/tests/test_protocol.py +++ b/azure_monitor/tests/test_protocol.py @@ -5,18 +5,19 @@ from azure_monitor import protocol + class TestProtocol(unittest.TestCase): - def test_object(self): - data = protocol.BaseObject() - self.assertEqual(repr(data), '{}') - data.foo = 1 - self.assertEqual(data.foo, 1) - self.assertEqual(data['foo'], 1) - data['bar'] = 2 - self.assertEqual(data.bar, 2) - self.assertEqual(data['bar'], 2) - self.assertRaises(KeyError, lambda: data['baz']) - self.assertRaises(AttributeError, lambda: data.baz) + # def test_object(self): + # data = protocol.BaseObject() + # self.assertEqual(repr(data), '{}') + # data.foo = 1 + # self.assertEqual(data.foo, 1) + # self.assertEqual(data['foo'], 1) + # data['bar'] = 2 + # self.assertEqual(data.bar, 2) + # self.assertEqual(data['bar'], 2) + # self.assertRaises(KeyError, lambda: data['baz']) + # self.assertRaises(AttributeError, lambda: data.baz) def test_data(self): data = protocol.Data() diff --git a/azure_monitor/tests/trace/test_trace.py b/azure_monitor/tests/trace/test_trace.py index ab76f6f..581e530 100644 --- a/azure_monitor/tests/trace/test_trace.py +++ b/azure_monitor/tests/trace/test_trace.py @@ -1,31 +1,31 @@ # Copyright (c) Microsoft Corporation. All rights reserved. # Licensed under the MIT License. -import mock -import time import unittest from azure_monitor.trace import AzureMonitorSpanExporter +from azure_monitor.utils import Options +from azure_monitor.protocol import Data + class TestAzureExporter(unittest.TestCase): def test_ctor(self): - from azure_monitor.utils import Options - instrumentation_key = Options._default.instrumentation_key - Options._default.instrumentation_key = None - self.assertRaises(ValueError, lambda: AzureMonitorSpanExporter()) - Options._default.instrumentation_key = instrumentation_key + options = Options() + instrumentation_key = options.instrumentation_key + options.instrumentation_key = None + self.assertRaises(ValueError, lambda: AzureMonitorSpanExporter(options=options)) + options.instrumentation_key = instrumentation_key def test_span_to_envelope(self): from opentelemetry.trace import Link, SpanContext, SpanKind from opentelemetry.trace.status import StatusCanonicalCode from opentelemetry.sdk.trace import Span - exporter = AzureMonitorSpanExporter( - instrumentation_key='12345678-1234-5678-abcd-12345678abcd' - ) + options = Options(instrumentation_key="12345678-1234-5678-abcd-12345678abcd") + exporter = AzureMonitorSpanExporter(options) parent_span = Span( - name='test', + name="test", context=SpanContext( trace_id=36873507687745823477771305566750195431, span_id=12030755672171557338, @@ -37,7 +37,7 @@ def test_span_to_envelope(self): # SpanKind.CLIENT HTTP span = Span( - name='test', + name="test", context=SpanContext( trace_id=36873507687745823477771305566750195431, span_id=12030755672171557337, @@ -47,62 +47,42 @@ def test_span_to_envelope(self): trace_config=None, resource=None, attributes={ - 'component': 'http', - 'http.method': 'GET', - 'http.url': 'https://www.wikipedia.org/wiki/Rabbit', - 'http.status_code': 200, + "component": "http", + "http.method": "GET", + "http.url": "https://www.wikipedia.org/wiki/Rabbit", + "http.status_code": 200, }, events=None, - links=None, - kind=SpanKind.CLIENT + links=[], + kind=SpanKind.CLIENT, ) span.status = StatusCanonicalCode.OK span.start_time = start_time span.end_time = end_time envelope = exporter.span_to_envelope(span) + self.assertEqual(envelope.iKey, "12345678-1234-5678-abcd-12345678abcd") self.assertEqual( - envelope.iKey, - '12345678-1234-5678-abcd-12345678abcd') - self.assertEqual( - envelope.name, - 'Microsoft.ApplicationInsights.RemoteDependency') - self.assertEqual( - envelope.tags['ai.operation.parentId'], - 'a6f5d48acb4d31da') - self.assertEqual( - envelope.tags['ai.operation.id'], - '1bbd944a73a05d89eab5d3740a213ee7') - self.assertEqual( - envelope.time, - '2019-12-04T21:18:36.027613Z') - self.assertEqual( - envelope.data.baseData.name, - 'GET//wiki/Rabbit') - self.assertEqual( - envelope.data.baseData.data, - 'https://www.wikipedia.org/wiki/Rabbit') - self.assertEqual( - envelope.data.baseData.target, - 'www.wikipedia.org') - self.assertEqual( - envelope.data.baseData.id, - 'a6f5d48acb4d31d9') - self.assertEqual( - envelope.data.baseData.resultCode, - '200') - self.assertEqual( - envelope.data.baseData.duration, - '0.00:00:01.001') + envelope.name, "Microsoft.ApplicationInsights.RemoteDependency" + ) + self.assertEqual(envelope.tags["ai.operation.parentId"], "a6f5d48acb4d31da") self.assertEqual( - envelope.data.baseData.type, - 'HTTP') + envelope.tags["ai.operation.id"], "1bbd944a73a05d89eab5d3740a213ee7" + ) + self.assertEqual(envelope.time, "2019-12-04T21:18:36.027613Z") + self.assertEqual(envelope.data.baseData.name, "GET//wiki/Rabbit") self.assertEqual( - envelope.data.baseType, - 'RemoteDependencyData') + envelope.data.baseData.data, "https://www.wikipedia.org/wiki/Rabbit" + ) + self.assertEqual(envelope.data.baseData.target, "www.wikipedia.org") + self.assertEqual(envelope.data.baseData.id, "a6f5d48acb4d31d9") + self.assertEqual(envelope.data.baseData.resultCode, "200") + self.assertEqual(envelope.data.baseData.duration, "0.00:00:01.001") + self.assertEqual(envelope.data.baseData.type, "HTTP") + self.assertEqual(envelope.data.baseType, "RemoteDependencyData") # SpanKind.CLIENT unknown type span = Span( - name='test', + name="test", context=SpanContext( trace_id=36873507687745823477771305566750195431, span_id=12030755672171557337, @@ -113,47 +93,31 @@ def test_span_to_envelope(self): resource=None, attributes={}, events=None, - links=None, - kind=SpanKind.CLIENT + links=[], + kind=SpanKind.CLIENT, ) span.status = StatusCanonicalCode.OK span.start_time = start_time span.end_time = end_time envelope = exporter.span_to_envelope(span) + self.assertEqual(envelope.iKey, "12345678-1234-5678-abcd-12345678abcd") self.assertEqual( - envelope.iKey, - '12345678-1234-5678-abcd-12345678abcd') - self.assertEqual( - envelope.name, - 'Microsoft.ApplicationInsights.RemoteDependency') - self.assertEqual( - envelope.tags['ai.operation.parentId'], - 'a6f5d48acb4d31da') - self.assertEqual( - envelope.tags['ai.operation.id'], - '1bbd944a73a05d89eab5d3740a213ee7') - self.assertEqual( - envelope.time, - '2019-12-04T21:18:36.027613Z') - self.assertEqual( - envelope.data.baseData.name, - 'test') - self.assertEqual( - envelope.data.baseData.id, - 'a6f5d48acb4d31d9') - self.assertEqual( - envelope.data.baseData.duration, - '0.00:00:01.001') - self.assertEqual( - envelope.data.baseData.type, - None) + envelope.name, "Microsoft.ApplicationInsights.RemoteDependency" + ) + self.assertEqual(envelope.tags["ai.operation.parentId"], "a6f5d48acb4d31da") self.assertEqual( - envelope.data.baseType, - 'RemoteDependencyData') + envelope.tags["ai.operation.id"], "1bbd944a73a05d89eab5d3740a213ee7" + ) + self.assertEqual(envelope.time, "2019-12-04T21:18:36.027613Z") + self.assertEqual(envelope.data.baseData.name, "test") + self.assertEqual(envelope.data.baseData.id, "a6f5d48acb4d31d9") + self.assertEqual(envelope.data.baseData.duration, "0.00:00:01.001") + self.assertEqual(envelope.data.baseData.type, None) + self.assertEqual(envelope.data.baseType, "RemoteDependencyData") # SpanKind.CLIENT missing method span = Span( - name='test', + name="test", context=SpanContext( trace_id=36873507687745823477771305566750195431, span_id=12030755672171557337, @@ -163,61 +127,41 @@ def test_span_to_envelope(self): trace_config=None, resource=None, attributes={ - 'component': 'http', - 'http.url': 'https://www.wikipedia.org/wiki/Rabbit', - 'http.status_code': 200, + "component": "http", + "http.url": "https://www.wikipedia.org/wiki/Rabbit", + "http.status_code": 200, }, events=None, - links=None, - kind=SpanKind.CLIENT + links=[], + kind=SpanKind.CLIENT, ) span.status = StatusCanonicalCode.OK span.start_time = start_time span.end_time = end_time envelope = exporter.span_to_envelope(span) + self.assertEqual(envelope.iKey, "12345678-1234-5678-abcd-12345678abcd") self.assertEqual( - envelope.iKey, - '12345678-1234-5678-abcd-12345678abcd') - self.assertEqual( - envelope.name, - 'Microsoft.ApplicationInsights.RemoteDependency') - self.assertEqual( - envelope.tags['ai.operation.parentId'], - 'a6f5d48acb4d31da') - self.assertEqual( - envelope.tags['ai.operation.id'], - '1bbd944a73a05d89eab5d3740a213ee7') - self.assertEqual( - envelope.time, - '2019-12-04T21:18:36.027613Z') - self.assertEqual( - envelope.data.baseData.name, - 'test') - self.assertEqual( - envelope.data.baseData.data, - 'https://www.wikipedia.org/wiki/Rabbit') - self.assertEqual( - envelope.data.baseData.target, - 'www.wikipedia.org') - self.assertEqual( - envelope.data.baseData.id, - 'a6f5d48acb4d31d9') - self.assertEqual( - envelope.data.baseData.resultCode, - '200') - self.assertEqual( - envelope.data.baseData.duration, - '0.00:00:01.001') + envelope.name, "Microsoft.ApplicationInsights.RemoteDependency" + ) + self.assertEqual(envelope.tags["ai.operation.parentId"], "a6f5d48acb4d31da") self.assertEqual( - envelope.data.baseData.type, - 'HTTP') + envelope.tags["ai.operation.id"], "1bbd944a73a05d89eab5d3740a213ee7" + ) + self.assertEqual(envelope.time, "2019-12-04T21:18:36.027613Z") + self.assertEqual(envelope.data.baseData.name, "test") self.assertEqual( - envelope.data.baseType, - 'RemoteDependencyData') + envelope.data.baseData.data, "https://www.wikipedia.org/wiki/Rabbit" + ) + self.assertEqual(envelope.data.baseData.target, "www.wikipedia.org") + self.assertEqual(envelope.data.baseData.id, "a6f5d48acb4d31d9") + self.assertEqual(envelope.data.baseData.resultCode, "200") + self.assertEqual(envelope.data.baseData.duration, "0.00:00:01.001") + self.assertEqual(envelope.data.baseData.type, "HTTP") + self.assertEqual(envelope.data.baseType, "RemoteDependencyData") # SpanKind.SERVER HTTP - 200 request span = Span( - name='test', + name="test", context=SpanContext( trace_id=36873507687745823477771305566750195431, span_id=12030755672171557337, @@ -227,64 +171,42 @@ def test_span_to_envelope(self): trace_config=None, resource=None, attributes={ - 'component': 'http', - 'http.method': 'GET', - 'http.path': '/wiki/Rabbit', - 'http.route': '/wiki/Rabbit', - 'http.url': 'https://www.wikipedia.org/wiki/Rabbit', - 'http.status_code': 200, + "component": "http", + "http.method": "GET", + "http.path": "/wiki/Rabbit", + "http.route": "/wiki/Rabbit", + "http.url": "https://www.wikipedia.org/wiki/Rabbit", + "http.status_code": 200, }, events=None, - links=None, - kind=SpanKind.SERVER + links=[], + kind=SpanKind.SERVER, ) span.status = StatusCanonicalCode.OK span.start_time = start_time span.end_time = end_time envelope = exporter.span_to_envelope(span) + self.assertEqual(envelope.iKey, "12345678-1234-5678-abcd-12345678abcd") + self.assertEqual(envelope.name, "Microsoft.ApplicationInsights.Request") + self.assertEqual(envelope.tags["ai.operation.parentId"], "a6f5d48acb4d31da") self.assertEqual( - envelope.iKey, - '12345678-1234-5678-abcd-12345678abcd') - self.assertEqual( - envelope.name, - 'Microsoft.ApplicationInsights.Request') - self.assertEqual( - envelope.tags['ai.operation.parentId'], - 'a6f5d48acb4d31da') - self.assertEqual( - envelope.tags['ai.operation.id'], - '1bbd944a73a05d89eab5d3740a213ee7') - self.assertEqual( - envelope.tags['ai.operation.name'], - 'GET /wiki/Rabbit') - self.assertEqual( - envelope.time, - '2019-12-04T21:18:36.027613Z') - self.assertEqual( - envelope.data.baseData.id, - 'a6f5d48acb4d31d9') - self.assertEqual( - envelope.data.baseData.duration, - '0.00:00:01.001') - self.assertEqual( - envelope.data.baseData.responseCode, - '200') - self.assertEqual( - envelope.data.baseData.name, - 'GET /wiki/Rabbit') - self.assertEqual( - envelope.data.baseData.success, - True) - self.assertEqual( - envelope.data.baseData.url, - 'https://www.wikipedia.org/wiki/Rabbit') - self.assertEqual( - envelope.data.baseType, - 'RequestData') + envelope.tags["ai.operation.id"], "1bbd944a73a05d89eab5d3740a213ee7" + ) + self.assertEqual(envelope.tags["ai.operation.name"], "GET /wiki/Rabbit") + self.assertEqual(envelope.time, "2019-12-04T21:18:36.027613Z") + self.assertEqual(envelope.data.baseData.id, "a6f5d48acb4d31d9") + self.assertEqual(envelope.data.baseData.duration, "0.00:00:01.001") + self.assertEqual(envelope.data.baseData.responseCode, "200") + self.assertEqual(envelope.data.baseData.name, "GET /wiki/Rabbit") + self.assertEqual(envelope.data.baseData.success, True) + self.assertEqual( + envelope.data.baseData.url, "https://www.wikipedia.org/wiki/Rabbit" + ) + self.assertEqual(envelope.data.baseType, "RequestData") # SpanKind.SERVER HTTP - Failed request span = Span( - name='test', + name="test", context=SpanContext( trace_id=36873507687745823477771305566750195431, span_id=12030755672171557337, @@ -294,64 +216,42 @@ def test_span_to_envelope(self): trace_config=None, resource=None, attributes={ - 'component': 'http', - 'http.method': 'GET', - 'http.path': '/wiki/Rabbit', - 'http.route': '/wiki/Rabbit', - 'http.url': 'https://www.wikipedia.org/wiki/Rabbit', - 'http.status_code': 400, + "component": "http", + "http.method": "GET", + "http.path": "/wiki/Rabbit", + "http.route": "/wiki/Rabbit", + "http.url": "https://www.wikipedia.org/wiki/Rabbit", + "http.status_code": 400, }, events=None, - links=None, - kind=SpanKind.SERVER + links=[], + kind=SpanKind.SERVER, ) span.status = StatusCanonicalCode.OK span.start_time = start_time span.end_time = end_time envelope = exporter.span_to_envelope(span) + self.assertEqual(envelope.iKey, "12345678-1234-5678-abcd-12345678abcd") + self.assertEqual(envelope.name, "Microsoft.ApplicationInsights.Request") + self.assertEqual(envelope.tags["ai.operation.parentId"], "a6f5d48acb4d31da") self.assertEqual( - envelope.iKey, - '12345678-1234-5678-abcd-12345678abcd') - self.assertEqual( - envelope.name, - 'Microsoft.ApplicationInsights.Request') - self.assertEqual( - envelope.tags['ai.operation.parentId'], - 'a6f5d48acb4d31da') - self.assertEqual( - envelope.tags['ai.operation.id'], - '1bbd944a73a05d89eab5d3740a213ee7') - self.assertEqual( - envelope.tags['ai.operation.name'], - 'GET /wiki/Rabbit') - self.assertEqual( - envelope.time, - '2019-12-04T21:18:36.027613Z') - self.assertEqual( - envelope.data.baseData.id, - 'a6f5d48acb4d31d9') - self.assertEqual( - envelope.data.baseData.duration, - '0.00:00:01.001') - self.assertEqual( - envelope.data.baseData.responseCode, - '400') - self.assertEqual( - envelope.data.baseData.name, - 'GET /wiki/Rabbit') - self.assertEqual( - envelope.data.baseData.success, - False) - self.assertEqual( - envelope.data.baseData.url, - 'https://www.wikipedia.org/wiki/Rabbit') - self.assertEqual( - envelope.data.baseType, - 'RequestData') + envelope.tags["ai.operation.id"], "1bbd944a73a05d89eab5d3740a213ee7" + ) + self.assertEqual(envelope.tags["ai.operation.name"], "GET /wiki/Rabbit") + self.assertEqual(envelope.time, "2019-12-04T21:18:36.027613Z") + self.assertEqual(envelope.data.baseData.id, "a6f5d48acb4d31d9") + self.assertEqual(envelope.data.baseData.duration, "0.00:00:01.001") + self.assertEqual(envelope.data.baseData.responseCode, "400") + self.assertEqual(envelope.data.baseData.name, "GET /wiki/Rabbit") + self.assertEqual(envelope.data.baseData.success, False) + self.assertEqual( + envelope.data.baseData.url, "https://www.wikipedia.org/wiki/Rabbit" + ) + self.assertEqual(envelope.data.baseType, "RequestData") # SpanKind.SERVER unknown type span = Span( - name='test', + name="test", context=SpanContext( trace_id=36873507687745823477771305566750195431, span_id=12030755672171557337, @@ -361,49 +261,35 @@ def test_span_to_envelope(self): trace_config=None, resource=None, attributes={ - 'component': 'http', - 'http.method': 'GET', - 'http.path': '/wiki/Rabbit', - 'http.route': '/wiki/Rabbit', - 'http.url': 'https://www.wikipedia.org/wiki/Rabbit', - 'http.status_code': 400, + "component": "http", + "http.method": "GET", + "http.path": "/wiki/Rabbit", + "http.route": "/wiki/Rabbit", + "http.url": "https://www.wikipedia.org/wiki/Rabbit", + "http.status_code": 400, }, events=None, - links=None, - kind=SpanKind.SERVER + links=[], + kind=SpanKind.SERVER, ) span.status = StatusCanonicalCode.OK span.start_time = start_time span.end_time = end_time envelope = exporter.span_to_envelope(span) + self.assertEqual(envelope.iKey, "12345678-1234-5678-abcd-12345678abcd") + self.assertEqual(envelope.name, "Microsoft.ApplicationInsights.Request") + self.assertEqual(envelope.tags["ai.operation.parentId"], "a6f5d48acb4d31da") self.assertEqual( - envelope.iKey, - '12345678-1234-5678-abcd-12345678abcd') - self.assertEqual( - envelope.name, - 'Microsoft.ApplicationInsights.Request') - self.assertEqual( - envelope.tags['ai.operation.parentId'], - 'a6f5d48acb4d31da') - self.assertEqual( - envelope.tags['ai.operation.id'], - '1bbd944a73a05d89eab5d3740a213ee7') - self.assertEqual( - envelope.time, - '2019-12-04T21:18:36.027613Z') - self.assertEqual( - envelope.data.baseData.id, - 'a6f5d48acb4d31d9') - self.assertEqual( - envelope.data.baseData.duration, - '0.00:00:01.001') - self.assertEqual( - envelope.data.baseType, - 'RequestData') + envelope.tags["ai.operation.id"], "1bbd944a73a05d89eab5d3740a213ee7" + ) + self.assertEqual(envelope.time, "2019-12-04T21:18:36.027613Z") + self.assertEqual(envelope.data.baseData.id, "a6f5d48acb4d31d9") + self.assertEqual(envelope.data.baseData.duration, "0.00:00:01.001") + self.assertEqual(envelope.data.baseType, "RequestData") # SpanKind.INTERNAL span = Span( - name='test', + name="test", context=SpanContext( trace_id=36873507687745823477771305566750195431, span_id=12030755672171557337, @@ -412,49 +298,33 @@ def test_span_to_envelope(self): sampler=None, trace_config=None, resource=None, - attributes={'key1': 'value1'}, + attributes={"key1": "value1"}, events=None, - links=None, - kind=SpanKind.INTERNAL + links=[], + kind=SpanKind.INTERNAL, ) span.status = StatusCanonicalCode.OK span.start_time = start_time span.end_time = end_time envelope = exporter.span_to_envelope(span) + self.assertEqual(envelope.iKey, "12345678-1234-5678-abcd-12345678abcd") self.assertEqual( - envelope.iKey, - '12345678-1234-5678-abcd-12345678abcd') - self.assertEqual( - envelope.name, - 'Microsoft.ApplicationInsights.RemoteDependency') - self.assertRaises( - KeyError, - lambda: envelope.tags['ai.operation.parentId']) - self.assertEqual( - envelope.tags['ai.operation.id'], - '1bbd944a73a05d89eab5d3740a213ee7') - self.assertEqual( - envelope.time, - '2019-12-04T21:18:36.027613Z') - self.assertEqual( - envelope.data.baseData.name, - 'test') - self.assertEqual( - envelope.data.baseData.duration, - '0.00:00:01.001') - self.assertEqual( - envelope.data.baseData.id, - 'a6f5d48acb4d31d9') - self.assertEqual( - envelope.data.baseData.type, - 'InProc') + envelope.name, "Microsoft.ApplicationInsights.RemoteDependency" + ) + self.assertRaises(KeyError, lambda: envelope.tags["ai.operation.parentId"]) self.assertEqual( - envelope.data.baseType, - 'RemoteDependencyData') + envelope.tags["ai.operation.id"], "1bbd944a73a05d89eab5d3740a213ee7" + ) + self.assertEqual(envelope.time, "2019-12-04T21:18:36.027613Z") + self.assertEqual(envelope.data.baseData.name, "test") + self.assertEqual(envelope.data.baseData.duration, "0.00:00:01.001") + self.assertEqual(envelope.data.baseData.id, "a6f5d48acb4d31d9") + self.assertEqual(envelope.data.baseData.type, "InProc") + self.assertEqual(envelope.data.baseType, "RemoteDependencyData") # Attributes span = Span( - name='test', + name="test", context=SpanContext( trace_id=36873507687745823477771305566750195431, span_id=12030755672171557337, @@ -464,34 +334,36 @@ def test_span_to_envelope(self): trace_config=None, resource=None, attributes={ - 'component': 'http', - 'http.method': 'GET', - 'http.url': 'https://www.wikipedia.org/wiki/Rabbit', - 'http.status_code': 200, - 'test': 'asd' + "component": "http", + "http.method": "GET", + "http.url": "https://www.wikipedia.org/wiki/Rabbit", + "http.status_code": 200, + "test": "asd", }, events=None, - links=None, - kind=SpanKind.CLIENT + links=[], + kind=SpanKind.CLIENT, ) span.status = StatusCanonicalCode.OK span.start_time = start_time span.end_time = end_time envelope = exporter.span_to_envelope(span) - self.assertEqual( - len(envelope.data.baseData.properties), 2) - self.assertEqual( - envelope.data.baseData.properties['component'], 'http') - self.assertEqual(envelope.data.baseData.properties['test'], 'asd') + self.assertEqual(len(envelope.data.baseData.properties), 2) + self.assertEqual(envelope.data.baseData.properties["component"], "http") + self.assertEqual(envelope.data.baseData.properties["test"], "asd") # Links links = [] - links.append(Link(context=SpanContext( - trace_id=36873507687745823477771305566750195432, - span_id=12030755672171557338, - ))) + links.append( + Link( + context=SpanContext( + trace_id=36873507687745823477771305566750195432, + span_id=12030755672171557338, + ) + ) + ) span = Span( - name='test', + name="test", context=SpanContext( trace_id=36873507687745823477771305566750195431, span_id=12030755672171557337, @@ -501,29 +373,29 @@ def test_span_to_envelope(self): trace_config=None, resource=None, attributes={ - 'component': 'http', - 'http.method': 'GET', - 'http.url': 'https://www.wikipedia.org/wiki/Rabbit', - 'http.status_code': 200, + "component": "http", + "http.method": "GET", + "http.url": "https://www.wikipedia.org/wiki/Rabbit", + "http.status_code": 200, }, events=None, links=links, - kind=SpanKind.CLIENT + kind=SpanKind.CLIENT, ) span.status = StatusCanonicalCode.OK span.start_time = start_time span.end_time = end_time envelope = exporter.span_to_envelope(span) - self.assertEqual( - len(envelope.data.baseData.properties), 2) - links_json = '[{"operation_Id": ' + \ - '"1bbd944a73a05d89eab5d3740a213ee8", "id": "a6f5d48acb4d31da"}]' - self.assertEqual(envelope.data.baseData.properties['_MS.links'], links_json) + self.assertEqual(len(envelope.data.baseData.properties), 2) + links_json = ( + '[{"operation_Id": ' + + '"1bbd944a73a05d89eab5d3740a213ee8", "id": "a6f5d48acb4d31da"}]' + ) + self.assertEqual(envelope.data.baseData.properties["_MS.links"], links_json) - # Status span = Span( - name='test', + name="test", context=SpanContext( trace_id=36873507687745823477771305566750195431, span_id=12030755672171557337, @@ -533,25 +405,24 @@ def test_span_to_envelope(self): trace_config=None, resource=None, attributes={ - 'component': 'http', - 'http.method': 'GET', - 'http.url': 'https://www.wikipedia.org/wiki/Rabbit', - 'http.status_code': 500, + "component": "http", + "http.method": "GET", + "http.url": "https://www.wikipedia.org/wiki/Rabbit", + "http.status_code": 500, }, events=None, - links=None, - kind=SpanKind.SERVER + links=[], + kind=SpanKind.SERVER, ) span.status = StatusCanonicalCode.OK span.start_time = start_time span.end_time = end_time envelope = exporter.span_to_envelope(span) - self.assertEqual( - envelope.data.baseData.responseCode, '500') + self.assertEqual(envelope.data.baseData.responseCode, "500") self.assertFalse(envelope.data.baseData.success) span = Span( - name='test', + name="test", context=SpanContext( trace_id=36873507687745823477771305566750195431, span_id=12030755672171557337, @@ -561,25 +432,24 @@ def test_span_to_envelope(self): trace_config=None, resource=None, attributes={ - 'component': 'http', - 'http.method': 'GET', - 'http.url': 'https://www.wikipedia.org/wiki/Rabbit', - 'http.status_code': 500, + "component": "http", + "http.method": "GET", + "http.url": "https://www.wikipedia.org/wiki/Rabbit", + "http.status_code": 500, }, events=None, - links=None, - kind=SpanKind.CLIENT + links=[], + kind=SpanKind.CLIENT, ) span.status = StatusCanonicalCode.OK span.start_time = start_time span.end_time = end_time envelope = exporter.span_to_envelope(span) - self.assertEqual( - envelope.data.baseData.resultCode, '500') + self.assertEqual(envelope.data.baseData.resultCode, "500") self.assertFalse(envelope.data.baseData.success) span = Span( - name='test', + name="test", context=SpanContext( trace_id=36873507687745823477771305566750195431, span_id=12030755672171557337, @@ -589,25 +459,24 @@ def test_span_to_envelope(self): trace_config=None, resource=None, attributes={ - 'component': 'http', - 'http.method': 'GET', - 'http.url': 'https://www.wikipedia.org/wiki/Rabbit', + "component": "http", + "http.method": "GET", + "http.url": "https://www.wikipedia.org/wiki/Rabbit", }, events=None, - links=None, - kind=SpanKind.SERVER + links=[], + kind=SpanKind.SERVER, ) span.status = StatusCanonicalCode.OK span.start_time = start_time span.end_time = end_time span.status = StatusCanonicalCode.OK envelope = exporter.span_to_envelope(span) - self.assertEqual( - envelope.data.baseData.responseCode, '0') + self.assertEqual(envelope.data.baseData.responseCode, "0") self.assertTrue(envelope.data.baseData.success) span = Span( - name='test', + name="test", context=SpanContext( trace_id=36873507687745823477771305566750195431, span_id=12030755672171557337, @@ -617,25 +486,24 @@ def test_span_to_envelope(self): trace_config=None, resource=None, attributes={ - 'component': 'http', - 'http.method': 'GET', - 'http.url': 'https://www.wikipedia.org/wiki/Rabbit', + "component": "http", + "http.method": "GET", + "http.url": "https://www.wikipedia.org/wiki/Rabbit", }, events=None, - links=None, - kind=SpanKind.CLIENT + links=[], + kind=SpanKind.CLIENT, ) span.status = StatusCanonicalCode.OK span.start_time = start_time span.end_time = end_time span.status = StatusCanonicalCode.OK envelope = exporter.span_to_envelope(span) - self.assertEqual( - envelope.data.baseData.resultCode, '0') + self.assertEqual(envelope.data.baseData.resultCode, "0") self.assertTrue(envelope.data.baseData.success) span = Span( - name='test', + name="test", context=SpanContext( trace_id=36873507687745823477771305566750195431, span_id=12030755672171557337, @@ -645,24 +513,23 @@ def test_span_to_envelope(self): trace_config=None, resource=None, attributes={ - 'component': 'http', - 'http.method': 'GET', - 'http.url': 'https://www.wikipedia.org/wiki/Rabbit', + "component": "http", + "http.method": "GET", + "http.url": "https://www.wikipedia.org/wiki/Rabbit", }, events=None, - links=None, - kind=SpanKind.SERVER + links=[], + kind=SpanKind.SERVER, ) span.start_time = start_time span.end_time = end_time span.status = StatusCanonicalCode.UNKNOWN envelope = exporter.span_to_envelope(span) - self.assertEqual( - envelope.data.baseData.responseCode, '2') + self.assertEqual(envelope.data.baseData.responseCode, "2") self.assertFalse(envelope.data.baseData.success) span = Span( - name='test', + name="test", context=SpanContext( trace_id=36873507687745823477771305566750195431, span_id=12030755672171557337, @@ -672,18 +539,17 @@ def test_span_to_envelope(self): trace_config=None, resource=None, attributes={ - 'component': 'http', - 'http.method': 'GET', - 'http.url': 'https://www.wikipedia.org/wiki/Rabbit', + "component": "http", + "http.method": "GET", + "http.url": "https://www.wikipedia.org/wiki/Rabbit", }, events=None, - links=None, - kind=SpanKind.CLIENT + links=[], + kind=SpanKind.CLIENT, ) span.start_time = start_time span.end_time = end_time span.status = StatusCanonicalCode.UNKNOWN envelope = exporter.span_to_envelope(span) - self.assertEqual( - envelope.data.baseData.resultCode, '2') + self.assertEqual(envelope.data.baseData.resultCode, "2") self.assertFalse(envelope.data.baseData.success) From b1d05e68aaacca86e58cef1887f4c37d670ae54d Mon Sep 17 00:00:00 2001 From: Leighton Date: Thu, 16 Jan 2020 14:06:29 -0800 Subject: [PATCH 018/109] Add property attributes for SERVER --- azure_monitor/src/azure_monitor/trace.py | 6 ++ azure_monitor/tests/trace/test_trace.py | 96 ++++++++++++++++++++++++ 2 files changed, 102 insertions(+) diff --git a/azure_monitor/src/azure_monitor/trace.py b/azure_monitor/src/azure_monitor/trace.py index 14da4cd..714ccb5 100644 --- a/azure_monitor/src/azure_monitor/trace.py +++ b/azure_monitor/src/azure_monitor/trace.py @@ -99,8 +99,13 @@ def span_to_envelope(self, span): # noqa pylint: disable=too-many-branches if "http.route" in span.attributes: data.name = data.name + " " + span.attributes["http.route"] envelope.tags["ai.operation.name"] = data.name + data.properties["request.name"] = data.name + elif 'http.path' in span.attributes: + data.properties['request.name'] = data.name + \ + ' ' + span.attributes['http.path'] if "http.url" in span.attributes: data.url = span.attributes["http.url"] + data.properties['request.url'] = span.attributes['http.url'] if "http.status_code" in span.attributes: status_code = span.attributes["http.status_code"] data.responseCode = str(status_code) @@ -146,6 +151,7 @@ def span_to_envelope(self, span): # noqa pylint: disable=too-many-branches data.success = True else: # SpanKind.INTERNAL data.type = "InProc" + data.success = True for key in span.attributes: # This removes redundant data from ApplicationInsights if key.startswith('http.'): diff --git a/azure_monitor/tests/trace/test_trace.py b/azure_monitor/tests/trace/test_trace.py index ab76f6f..75c06f9 100644 --- a/azure_monitor/tests/trace/test_trace.py +++ b/azure_monitor/tests/trace/test_trace.py @@ -448,6 +448,9 @@ def test_span_to_envelope(self): self.assertEqual( envelope.data.baseData.type, 'InProc') + self.assertEqual( + envelope.data.baseData.success, + True) self.assertEqual( envelope.data.baseType, 'RemoteDependencyData') @@ -687,3 +690,96 @@ def test_span_to_envelope(self): self.assertEqual( envelope.data.baseData.resultCode, '2') self.assertFalse(envelope.data.baseData.success) + + # Server route attribute + span = Span( + name='test', + context=SpanContext( + trace_id=36873507687745823477771305566750195431, + span_id=12030755672171557337, + ), + parent=parent_span, + sampler=None, + trace_config=None, + resource=None, + attributes={ + 'component': 'HTTP', + 'http.method': 'GET', + 'http.route': '/wiki/Rabbit', + 'http.path': '/wiki/Rabbitz', + 'http.url': 'https://www.wikipedia.org/wiki/Rabbit', + 'http.status_code': 400, + }, + events=None, + links=None, + kind=SpanKind.SERVER + ) + span.start_time = start_time + span.end_time = end_time + span.status = StatusCanonicalCode.OK + envelope = exporter.span_to_envelope(span) + self.assertEqual( + envelope.data.baseData.properties['request.name'], 'GET /wiki/Rabbit') + self.assertEqual( + envelope.data.baseData.properties['request.url'], 'https://www.wikipedia.org/wiki/Rabbit') + + # Server route attribute missing + span = Span( + name='test', + context=SpanContext( + trace_id=36873507687745823477771305566750195431, + span_id=12030755672171557337, + ), + parent=parent_span, + sampler=None, + trace_config=None, + resource=None, + attributes={ + 'component': 'HTTP', + 'http.method': 'GET', + 'http.path': '/wiki/Rabbitz', + 'http.url': 'https://www.wikipedia.org/wiki/Rabbit', + 'http.status_code': 400, + }, + events=None, + links=None, + kind=SpanKind.SERVER + ) + span.start_time = start_time + span.end_time = end_time + span.status = StatusCanonicalCode.OK + envelope = exporter.span_to_envelope(span) + self.assertEqual( + envelope.data.baseData.properties['request.name'], 'GET /wiki/Rabbitz') + self.assertEqual( + envelope.data.baseData.properties['request.url'], 'https://www.wikipedia.org/wiki/Rabbit') + + # Server route and path attribute missing + span = Span( + name='test', + context=SpanContext( + trace_id=36873507687745823477771305566750195431, + span_id=12030755672171557337, + ), + parent=parent_span, + sampler=None, + trace_config=None, + resource=None, + attributes={ + 'component': 'HTTP', + 'http.method': 'GET', + 'http.url': 'https://www.wikipedia.org/wiki/Rabbit', + 'http.status_code': 400, + }, + events=None, + links=None, + kind=SpanKind.SERVER + ) + span.start_time = start_time + span.end_time = end_time + span.status = StatusCanonicalCode.OK + envelope = exporter.span_to_envelope(span) + self.assertIsNone( + envelope.data.baseData.properties.get('request.name')) + self.assertEqual( + envelope.data.baseData.properties['request.url'], 'https://www.wikipedia.org/wiki/Rabbit') From 1c8fa815eda491e64b799ffa7593537f9583713c Mon Sep 17 00:00:00 2001 From: Victor Lima Date: Fri, 17 Jan 2020 15:55:38 -0300 Subject: [PATCH 019/109] Fixed variable names to me snake case --- azure_monitor/src/azure_monitor/protocol.py | 143 ++++++++++++++++---- azure_monitor/src/azure_monitor/trace.py | 16 +-- azure_monitor/tests/test_protocol.py | 4 +- azure_monitor/tests/trace/test_trace.py | 139 ++++++++++--------- 4 files changed, 197 insertions(+), 105 deletions(-) diff --git a/azure_monitor/src/azure_monitor/protocol.py b/azure_monitor/src/azure_monitor/protocol.py index 54e1c48..b347b52 100644 --- a/azure_monitor/src/azure_monitor/protocol.py +++ b/azure_monitor/src/azure_monitor/protocol.py @@ -3,11 +3,17 @@ class Data: - __slots__ = ("baseData", "baseType") + __slots__ = ("base_data", "base_type") - def __init__(self, baseData=None, baseType=None) -> None: - self.baseData = baseData - self.baseType = baseType + def __init__(self, base_data=None, base_type=None) -> None: + self.base_data = base_data + self.base_type = base_type + + def to_dict(self): + return { + "baseData": self.base_data.to_dict(), + "baseType": self.base_type, + } class DataPoint: @@ -42,15 +48,27 @@ def __init__( self.max = max self.std_dev = std_dev + def to_dict(self): + return { + "ns": self.ns, + "name": self.name, + "kind": self.kind, + "value": self.value, + "count": self.count, + "min": self.min, + "max": self.max, + "stdDev": self.std_dev, + } + class Envelope: __slots__ = ( "ver", "name", "time", - "sampleRate", + "sample_rate", "seq", - "iKey", + "ikey", "flags", "tags", "data", @@ -61,9 +79,9 @@ def __init__( ver=1, name="", time="", - sampleRate=None, + sample_rate=None, seq=None, - iKey=None, + ikey=None, flags=None, tags=None, data=None, @@ -71,13 +89,26 @@ def __init__( self.ver = ver self.name = name self.time = time - self.sampleRate = sampleRate + self.sample_rate = sample_rate self.seq = seq - self.iKey = iKey + self.ikey = ikey self.flags = flags self.tags = tags self.data = data + def to_dict(self): + return { + "ver": self.ver, + "name": self.name, + "time": self.time, + "sampleRate": self.sample_rate, + "seq": self.seq, + "iKey": self.ikey, + "flags": self.flags, + "tags": self.tags, + "data": self.data.to_dict(), + } + class Event: __slots__ = ("ver", "name", "properties", "measurements") @@ -88,13 +119,21 @@ def __init__(self, ver=2, name="", properties=None, measurements=None): self.properties = properties self.measurements = measurements + def to_dict(self): + return { + "ver": self.ver, + "name": self.name, + "properties": self.properties, + "measurements": self.measurements, + } + class ExceptionData: __slots__ = ( "ver", "exceptions", - "severityLevel", - "problemId", + "severity_level", + "problem_id", "properties", "measurements", ) @@ -103,8 +142,8 @@ def __init__( self, ver=2, exceptions=None, - severityLevel=None, - problemId=None, + severity_level=None, + problem_id=None, properties=None, measurements=None, ) -> None: @@ -112,17 +151,27 @@ def __init__( exceptions = [] self.ver = ver self.exceptions = exceptions - self.severityLevel = severityLevel - self.problemId = problemId + self.severity_level = severity_level + self.problem_id = problem_id self.properties = properties self.measurements = measurements + def to_dict(self): + return { + "ver": self.ver, + "exceptions": self.exceptions, + "severityLevel": self.severity_level, + "problemId": self.problem_id, + "properties": self.properties, + "measurements": self.measurements, + } + class Message: __slots__ = ( "ver", "message", - "severityLevel", + "severity_level", "properties", "measurements", ) @@ -131,16 +180,25 @@ def __init__( self, ver=2, message="", - severityLevel=None, + severity_level=None, properties=None, measurements=None, ) -> None: self.ver = ver self.message = message - self.severityLevel = severityLevel + self.severity_level = severity_level self.properties = properties self.measurements = measurements + def to_dict(self): + return { + "ver": self.ver, + "message": self.message, + "severityLevel": self.severity_level, + "properties": self.properties, + "measurements": self.measurements, + } + class MetricData: __slots__ = ("ver", "metrics", "properties") @@ -152,13 +210,20 @@ def __init__(self, ver=2, metrics=None, properties=None) -> None: self.metrics = metrics self.properties = properties + def to_dict(self): + return { + "ver": self.ver, + "metrics": self.metrics, + "properties": self.properties, + } + class RemoteDependency: __slots__ = ( "ver", "name", "id", - "resultCode", + "result_code", "duration", "success", "data", @@ -173,7 +238,7 @@ def __init__( ver=2, name="", id="", - resultCode="", + result_code="", duration="", success=True, data=None, @@ -185,7 +250,7 @@ def __init__( self.ver = ver self.name = name self.id = id - self.resultCode = resultCode + self.result_code = result_code self.duration = duration self.success = success self.data = data @@ -194,13 +259,28 @@ def __init__( self.properties = properties self.measurements = measurements + def to_dict(self): + return { + "ver": self.ver, + "name": self.name, + "id": self.id, + "resultCode": self.result_code, + "duration": self.duration, + "success": self.success, + "data": self.data, + "type": self.type, + "target": self.target, + "properties": self.properties, + "measurements": self.measurements, + } + class Request: __slots__ = ( "ver", "id", "duration", - "responseCode", + "response_code", "success", "source", "name", @@ -214,7 +294,7 @@ def __init__( ver=2, id="", duration="", - responseCode="", + response_code="", success=True, source=None, name=None, @@ -225,7 +305,7 @@ def __init__( self.ver = ver self.id = id self.duration = duration - self.responseCode = responseCode + self.response_code = response_code self.success = success self.source = source self.name = name @@ -233,3 +313,16 @@ def __init__( self.properties = properties self.measurements = measurements + def to_dict(self): + return { + "ver": self.ver, + "id": self.id, + "duration": self.duration, + "responseCode": self.response_code, + "success": self.success, + "source": self.source, + "name": self.name, + "url": self.url, + "properties": self.properties, + "measurements": self.measurements, + } diff --git a/azure_monitor/src/azure_monitor/trace.py b/azure_monitor/src/azure_monitor/trace.py index 4394c21..e348e39 100644 --- a/azure_monitor/src/azure_monitor/trace.py +++ b/azure_monitor/src/azure_monitor/trace.py @@ -16,7 +16,7 @@ class AzureMonitorSpanExporter(SpanExporter): - def __init__(self, options: utils.Options): + def __init__(self, options: "utils.Options"): self.options = options if not self.options.instrumentation_key: raise ValueError("The instrumentation_key is not provided.") @@ -66,7 +66,7 @@ def export(self, spans): def span_to_envelope(self, span): # noqa pylint: disable=too-many-branches envelope = protocol.Envelope( - iKey=self.options.instrumentation_key, + ikey=self.options.instrumentation_key, tags=dict(utils.azure_monitor_context), time=ns_to_iso_str(span.start_time), ) @@ -87,12 +87,12 @@ def span_to_envelope(self, span): # noqa pylint: disable=too-many-branches span.context.span_id ), duration=utils.ns_to_duration(span.end_time - span.start_time), - responseCode=str(span.status.value), + response_code=str(span.status.value), success=False, # Modify based off attributes or Status properties={}, ) envelope.data = protocol.Data( - baseData=data, baseType="RequestData" + base_data=data, base_type="RequestData" ) if "http.method" in span.attributes: data.name = span.attributes["http.method"] @@ -103,7 +103,7 @@ def span_to_envelope(self, span): # noqa pylint: disable=too-many-branches data.url = span.attributes["http.url"] if "http.status_code" in span.attributes: status_code = span.attributes["http.status_code"] - data.responseCode = str(status_code) + data.response_code = str(status_code) data.success = 200 <= status_code < 400 elif span.status == StatusCanonicalCode.OK: data.success = True @@ -114,13 +114,13 @@ def span_to_envelope(self, span): # noqa pylint: disable=too-many-branches id="{:016x}".format( span.context.span_id ), - resultCode=str(span.status.value), + result_code=str(span.status.value), duration=utils.ns_to_duration(span.end_time - span.start_time), success=False, # Modify based off attributes or Status properties={}, ) envelope.data = protocol.Data( - baseData=data, baseType="RemoteDependencyData" + base_data=data, base_type="RemoteDependencyData" ) if span.kind in (SpanKind.CLIENT, SpanKind.PRODUCER): if "component" in span.attributes and \ @@ -140,7 +140,7 @@ def span_to_envelope(self, span): # noqa pylint: disable=too-many-branches + "/" + parse_url.path if "http.status_code" in span.attributes: status_code = span.attributes["http.status_code"] - data.resultCode = str(status_code) + data.result_code = str(status_code) data.success = 200 <= status_code < 400 elif span.status == StatusCanonicalCode.OK: data.success = True diff --git a/azure_monitor/tests/test_protocol.py b/azure_monitor/tests/test_protocol.py index 21c800e..0844688 100644 --- a/azure_monitor/tests/test_protocol.py +++ b/azure_monitor/tests/test_protocol.py @@ -21,8 +21,8 @@ class TestProtocol(unittest.TestCase): def test_data(self): data = protocol.Data() - self.assertIsNone(data.baseData) - self.assertIsNone(data.baseType) + self.assertIsNone(data.base_data) + self.assertIsNone(data.base_type) def test_envelope(self): data = protocol.Envelope() diff --git a/azure_monitor/tests/trace/test_trace.py b/azure_monitor/tests/trace/test_trace.py index 581e530..15c5f87 100644 --- a/azure_monitor/tests/trace/test_trace.py +++ b/azure_monitor/tests/trace/test_trace.py @@ -5,7 +5,6 @@ from azure_monitor.trace import AzureMonitorSpanExporter from azure_monitor.utils import Options -from azure_monitor.protocol import Data class TestAzureExporter(unittest.TestCase): @@ -60,7 +59,7 @@ def test_span_to_envelope(self): span.start_time = start_time span.end_time = end_time envelope = exporter.span_to_envelope(span) - self.assertEqual(envelope.iKey, "12345678-1234-5678-abcd-12345678abcd") + self.assertEqual(envelope.ikey, "12345678-1234-5678-abcd-12345678abcd") self.assertEqual( envelope.name, "Microsoft.ApplicationInsights.RemoteDependency" ) @@ -69,16 +68,16 @@ def test_span_to_envelope(self): envelope.tags["ai.operation.id"], "1bbd944a73a05d89eab5d3740a213ee7" ) self.assertEqual(envelope.time, "2019-12-04T21:18:36.027613Z") - self.assertEqual(envelope.data.baseData.name, "GET//wiki/Rabbit") + self.assertEqual(envelope.data.base_data.name, "GET//wiki/Rabbit") self.assertEqual( - envelope.data.baseData.data, "https://www.wikipedia.org/wiki/Rabbit" - ) - self.assertEqual(envelope.data.baseData.target, "www.wikipedia.org") - self.assertEqual(envelope.data.baseData.id, "a6f5d48acb4d31d9") - self.assertEqual(envelope.data.baseData.resultCode, "200") - self.assertEqual(envelope.data.baseData.duration, "0.00:00:01.001") - self.assertEqual(envelope.data.baseData.type, "HTTP") - self.assertEqual(envelope.data.baseType, "RemoteDependencyData") + envelope.data.base_data.data, "https://www.wikipedia.org/wiki/Rabbit" + ) + self.assertEqual(envelope.data.base_data.target, "www.wikipedia.org") + self.assertEqual(envelope.data.base_data.id, "a6f5d48acb4d31d9") + self.assertEqual(envelope.data.base_data.result_code, "200") + self.assertEqual(envelope.data.base_data.duration, "0.00:00:01.001") + self.assertEqual(envelope.data.base_data.type, "HTTP") + self.assertEqual(envelope.data.base_type, "RemoteDependencyData") # SpanKind.CLIENT unknown type span = Span( @@ -100,7 +99,7 @@ def test_span_to_envelope(self): span.start_time = start_time span.end_time = end_time envelope = exporter.span_to_envelope(span) - self.assertEqual(envelope.iKey, "12345678-1234-5678-abcd-12345678abcd") + self.assertEqual(envelope.ikey, "12345678-1234-5678-abcd-12345678abcd") self.assertEqual( envelope.name, "Microsoft.ApplicationInsights.RemoteDependency" ) @@ -109,11 +108,11 @@ def test_span_to_envelope(self): envelope.tags["ai.operation.id"], "1bbd944a73a05d89eab5d3740a213ee7" ) self.assertEqual(envelope.time, "2019-12-04T21:18:36.027613Z") - self.assertEqual(envelope.data.baseData.name, "test") - self.assertEqual(envelope.data.baseData.id, "a6f5d48acb4d31d9") - self.assertEqual(envelope.data.baseData.duration, "0.00:00:01.001") - self.assertEqual(envelope.data.baseData.type, None) - self.assertEqual(envelope.data.baseType, "RemoteDependencyData") + self.assertEqual(envelope.data.base_data.name, "test") + self.assertEqual(envelope.data.base_data.id, "a6f5d48acb4d31d9") + self.assertEqual(envelope.data.base_data.duration, "0.00:00:01.001") + self.assertEqual(envelope.data.base_data.type, None) + self.assertEqual(envelope.data.base_type, "RemoteDependencyData") # SpanKind.CLIENT missing method span = Span( @@ -139,7 +138,7 @@ def test_span_to_envelope(self): span.start_time = start_time span.end_time = end_time envelope = exporter.span_to_envelope(span) - self.assertEqual(envelope.iKey, "12345678-1234-5678-abcd-12345678abcd") + self.assertEqual(envelope.ikey, "12345678-1234-5678-abcd-12345678abcd") self.assertEqual( envelope.name, "Microsoft.ApplicationInsights.RemoteDependency" ) @@ -148,16 +147,16 @@ def test_span_to_envelope(self): envelope.tags["ai.operation.id"], "1bbd944a73a05d89eab5d3740a213ee7" ) self.assertEqual(envelope.time, "2019-12-04T21:18:36.027613Z") - self.assertEqual(envelope.data.baseData.name, "test") + self.assertEqual(envelope.data.base_data.name, "test") self.assertEqual( - envelope.data.baseData.data, "https://www.wikipedia.org/wiki/Rabbit" - ) - self.assertEqual(envelope.data.baseData.target, "www.wikipedia.org") - self.assertEqual(envelope.data.baseData.id, "a6f5d48acb4d31d9") - self.assertEqual(envelope.data.baseData.resultCode, "200") - self.assertEqual(envelope.data.baseData.duration, "0.00:00:01.001") - self.assertEqual(envelope.data.baseData.type, "HTTP") - self.assertEqual(envelope.data.baseType, "RemoteDependencyData") + envelope.data.base_data.data, "https://www.wikipedia.org/wiki/Rabbit" + ) + self.assertEqual(envelope.data.base_data.target, "www.wikipedia.org") + self.assertEqual(envelope.data.base_data.id, "a6f5d48acb4d31d9") + self.assertEqual(envelope.data.base_data.result_code, "200") + self.assertEqual(envelope.data.base_data.duration, "0.00:00:01.001") + self.assertEqual(envelope.data.base_data.type, "HTTP") + self.assertEqual(envelope.data.base_type, "RemoteDependencyData") # SpanKind.SERVER HTTP - 200 request span = Span( @@ -186,7 +185,7 @@ def test_span_to_envelope(self): span.start_time = start_time span.end_time = end_time envelope = exporter.span_to_envelope(span) - self.assertEqual(envelope.iKey, "12345678-1234-5678-abcd-12345678abcd") + self.assertEqual(envelope.ikey, "12345678-1234-5678-abcd-12345678abcd") self.assertEqual(envelope.name, "Microsoft.ApplicationInsights.Request") self.assertEqual(envelope.tags["ai.operation.parentId"], "a6f5d48acb4d31da") self.assertEqual( @@ -194,15 +193,15 @@ def test_span_to_envelope(self): ) self.assertEqual(envelope.tags["ai.operation.name"], "GET /wiki/Rabbit") self.assertEqual(envelope.time, "2019-12-04T21:18:36.027613Z") - self.assertEqual(envelope.data.baseData.id, "a6f5d48acb4d31d9") - self.assertEqual(envelope.data.baseData.duration, "0.00:00:01.001") - self.assertEqual(envelope.data.baseData.responseCode, "200") - self.assertEqual(envelope.data.baseData.name, "GET /wiki/Rabbit") - self.assertEqual(envelope.data.baseData.success, True) + self.assertEqual(envelope.data.base_data.id, "a6f5d48acb4d31d9") + self.assertEqual(envelope.data.base_data.duration, "0.00:00:01.001") + self.assertEqual(envelope.data.base_data.response_code, "200") + self.assertEqual(envelope.data.base_data.name, "GET /wiki/Rabbit") + self.assertEqual(envelope.data.base_data.success, True) self.assertEqual( - envelope.data.baseData.url, "https://www.wikipedia.org/wiki/Rabbit" + envelope.data.base_data.url, "https://www.wikipedia.org/wiki/Rabbit" ) - self.assertEqual(envelope.data.baseType, "RequestData") + self.assertEqual(envelope.data.base_type, "RequestData") # SpanKind.SERVER HTTP - Failed request span = Span( @@ -231,7 +230,7 @@ def test_span_to_envelope(self): span.start_time = start_time span.end_time = end_time envelope = exporter.span_to_envelope(span) - self.assertEqual(envelope.iKey, "12345678-1234-5678-abcd-12345678abcd") + self.assertEqual(envelope.ikey, "12345678-1234-5678-abcd-12345678abcd") self.assertEqual(envelope.name, "Microsoft.ApplicationInsights.Request") self.assertEqual(envelope.tags["ai.operation.parentId"], "a6f5d48acb4d31da") self.assertEqual( @@ -239,15 +238,15 @@ def test_span_to_envelope(self): ) self.assertEqual(envelope.tags["ai.operation.name"], "GET /wiki/Rabbit") self.assertEqual(envelope.time, "2019-12-04T21:18:36.027613Z") - self.assertEqual(envelope.data.baseData.id, "a6f5d48acb4d31d9") - self.assertEqual(envelope.data.baseData.duration, "0.00:00:01.001") - self.assertEqual(envelope.data.baseData.responseCode, "400") - self.assertEqual(envelope.data.baseData.name, "GET /wiki/Rabbit") - self.assertEqual(envelope.data.baseData.success, False) + self.assertEqual(envelope.data.base_data.id, "a6f5d48acb4d31d9") + self.assertEqual(envelope.data.base_data.duration, "0.00:00:01.001") + self.assertEqual(envelope.data.base_data.response_code, "400") + self.assertEqual(envelope.data.base_data.name, "GET /wiki/Rabbit") + self.assertEqual(envelope.data.base_data.success, False) self.assertEqual( - envelope.data.baseData.url, "https://www.wikipedia.org/wiki/Rabbit" + envelope.data.base_data.url, "https://www.wikipedia.org/wiki/Rabbit" ) - self.assertEqual(envelope.data.baseType, "RequestData") + self.assertEqual(envelope.data.base_type, "RequestData") # SpanKind.SERVER unknown type span = Span( @@ -276,16 +275,16 @@ def test_span_to_envelope(self): span.start_time = start_time span.end_time = end_time envelope = exporter.span_to_envelope(span) - self.assertEqual(envelope.iKey, "12345678-1234-5678-abcd-12345678abcd") + self.assertEqual(envelope.ikey, "12345678-1234-5678-abcd-12345678abcd") self.assertEqual(envelope.name, "Microsoft.ApplicationInsights.Request") self.assertEqual(envelope.tags["ai.operation.parentId"], "a6f5d48acb4d31da") self.assertEqual( envelope.tags["ai.operation.id"], "1bbd944a73a05d89eab5d3740a213ee7" ) self.assertEqual(envelope.time, "2019-12-04T21:18:36.027613Z") - self.assertEqual(envelope.data.baseData.id, "a6f5d48acb4d31d9") - self.assertEqual(envelope.data.baseData.duration, "0.00:00:01.001") - self.assertEqual(envelope.data.baseType, "RequestData") + self.assertEqual(envelope.data.base_data.id, "a6f5d48acb4d31d9") + self.assertEqual(envelope.data.base_data.duration, "0.00:00:01.001") + self.assertEqual(envelope.data.base_type, "RequestData") # SpanKind.INTERNAL span = Span( @@ -307,7 +306,7 @@ def test_span_to_envelope(self): span.start_time = start_time span.end_time = end_time envelope = exporter.span_to_envelope(span) - self.assertEqual(envelope.iKey, "12345678-1234-5678-abcd-12345678abcd") + self.assertEqual(envelope.ikey, "12345678-1234-5678-abcd-12345678abcd") self.assertEqual( envelope.name, "Microsoft.ApplicationInsights.RemoteDependency" ) @@ -316,11 +315,11 @@ def test_span_to_envelope(self): envelope.tags["ai.operation.id"], "1bbd944a73a05d89eab5d3740a213ee7" ) self.assertEqual(envelope.time, "2019-12-04T21:18:36.027613Z") - self.assertEqual(envelope.data.baseData.name, "test") - self.assertEqual(envelope.data.baseData.duration, "0.00:00:01.001") - self.assertEqual(envelope.data.baseData.id, "a6f5d48acb4d31d9") - self.assertEqual(envelope.data.baseData.type, "InProc") - self.assertEqual(envelope.data.baseType, "RemoteDependencyData") + self.assertEqual(envelope.data.base_data.name, "test") + self.assertEqual(envelope.data.base_data.duration, "0.00:00:01.001") + self.assertEqual(envelope.data.base_data.id, "a6f5d48acb4d31d9") + self.assertEqual(envelope.data.base_data.type, "InProc") + self.assertEqual(envelope.data.base_type, "RemoteDependencyData") # Attributes span = Span( @@ -348,9 +347,9 @@ def test_span_to_envelope(self): span.start_time = start_time span.end_time = end_time envelope = exporter.span_to_envelope(span) - self.assertEqual(len(envelope.data.baseData.properties), 2) - self.assertEqual(envelope.data.baseData.properties["component"], "http") - self.assertEqual(envelope.data.baseData.properties["test"], "asd") + self.assertEqual(len(envelope.data.base_data.properties), 2) + self.assertEqual(envelope.data.base_data.properties["component"], "http") + self.assertEqual(envelope.data.base_data.properties["test"], "asd") # Links links = [] @@ -386,12 +385,12 @@ def test_span_to_envelope(self): span.start_time = start_time span.end_time = end_time envelope = exporter.span_to_envelope(span) - self.assertEqual(len(envelope.data.baseData.properties), 2) + self.assertEqual(len(envelope.data.base_data.properties), 2) links_json = ( '[{"operation_Id": ' + '"1bbd944a73a05d89eab5d3740a213ee8", "id": "a6f5d48acb4d31da"}]' ) - self.assertEqual(envelope.data.baseData.properties["_MS.links"], links_json) + self.assertEqual(envelope.data.base_data.properties["_MS.links"], links_json) # Status span = Span( @@ -418,8 +417,8 @@ def test_span_to_envelope(self): span.start_time = start_time span.end_time = end_time envelope = exporter.span_to_envelope(span) - self.assertEqual(envelope.data.baseData.responseCode, "500") - self.assertFalse(envelope.data.baseData.success) + self.assertEqual(envelope.data.base_data.response_code, "500") + self.assertFalse(envelope.data.base_data.success) span = Span( name="test", @@ -445,8 +444,8 @@ def test_span_to_envelope(self): span.start_time = start_time span.end_time = end_time envelope = exporter.span_to_envelope(span) - self.assertEqual(envelope.data.baseData.resultCode, "500") - self.assertFalse(envelope.data.baseData.success) + self.assertEqual(envelope.data.base_data.result_code, "500") + self.assertFalse(envelope.data.base_data.success) span = Span( name="test", @@ -472,8 +471,8 @@ def test_span_to_envelope(self): span.end_time = end_time span.status = StatusCanonicalCode.OK envelope = exporter.span_to_envelope(span) - self.assertEqual(envelope.data.baseData.responseCode, "0") - self.assertTrue(envelope.data.baseData.success) + self.assertEqual(envelope.data.base_data.response_code, "0") + self.assertTrue(envelope.data.base_data.success) span = Span( name="test", @@ -499,8 +498,8 @@ def test_span_to_envelope(self): span.end_time = end_time span.status = StatusCanonicalCode.OK envelope = exporter.span_to_envelope(span) - self.assertEqual(envelope.data.baseData.resultCode, "0") - self.assertTrue(envelope.data.baseData.success) + self.assertEqual(envelope.data.base_data.result_code, "0") + self.assertTrue(envelope.data.base_data.success) span = Span( name="test", @@ -525,8 +524,8 @@ def test_span_to_envelope(self): span.end_time = end_time span.status = StatusCanonicalCode.UNKNOWN envelope = exporter.span_to_envelope(span) - self.assertEqual(envelope.data.baseData.responseCode, "2") - self.assertFalse(envelope.data.baseData.success) + self.assertEqual(envelope.data.base_data.response_code, "2") + self.assertFalse(envelope.data.base_data.success) span = Span( name="test", @@ -551,5 +550,5 @@ def test_span_to_envelope(self): span.end_time = end_time span.status = StatusCanonicalCode.UNKNOWN envelope = exporter.span_to_envelope(span) - self.assertEqual(envelope.data.baseData.resultCode, "2") - self.assertFalse(envelope.data.baseData.success) + self.assertEqual(envelope.data.base_data.result_code, "2") + self.assertFalse(envelope.data.base_data.success) From 4e1b50e309ffb8a7fcda2aab7bb38e0f05be63b6 Mon Sep 17 00:00:00 2001 From: Leighton Date: Fri, 31 Jan 2020 11:37:26 -0800 Subject: [PATCH 020/109] CI up until lint --- .coveragerc | 12 + .flake8 | 19 + .isort.cfg | 16 + .pylintrc | 489 ++++++++++++++ azure_monitor/examples/client.py | 7 +- azure_monitor/examples/request.py | 8 +- azure_monitor/examples/server.py | 13 +- azure_monitor/examples/trace.py | 5 +- azure_monitor/setup.py | 4 +- azure_monitor/src/azure_monitor/protocol.py | 13 +- azure_monitor/src/azure_monitor/trace.py | 72 +-- azure_monitor/src/azure_monitor/utils.py | 4 +- azure_monitor/tests/__init__.py | 2 +- azure_monitor/tests/test_protocol.py | 19 +- azure_monitor/tests/test_utils.py | 12 +- azure_monitor/tests/trace/__init__.py | 2 +- azure_monitor/tests/trace/test_trace.py | 681 ++++++++++---------- dev-requirements.txt | 10 + scripts/pylint.sh | 24 + tox.ini | 55 ++ 20 files changed, 1037 insertions(+), 430 deletions(-) create mode 100644 .coveragerc create mode 100644 .flake8 create mode 100644 .isort.cfg create mode 100644 .pylintrc create mode 100644 dev-requirements.txt create mode 100644 scripts/pylint.sh create mode 100644 tox.ini diff --git a/.coveragerc b/.coveragerc new file mode 100644 index 0000000..eb9ca5a --- /dev/null +++ b/.coveragerc @@ -0,0 +1,12 @@ +[run] +branch = True +omit = + azure_monitor/setup.py + azure_monitor/tests/* + +[report] +fail_under = 100 +show_missing = True +omit = + azure_monitor/setup.py + azure_monitor/tests/* diff --git a/.flake8 b/.flake8 new file mode 100644 index 0000000..6d1c2bc --- /dev/null +++ b/.flake8 @@ -0,0 +1,19 @@ +[flake8] +ignore = + E501 # line too long, defer to black + F401 # unused import, defer to pylint + W503 # allow line breaks after binary ops, not after + E203 # allow whitespace before ':' (https://github.com/psf/black#slices) +exclude = + .bzr + .git + .hg + .svn + .tox + CVS + .venv*/ + venv*/ + target + __pycache__ + ext/opentelemetry-ext-jaeger/src/opentelemetry/ext/jaeger/gen/ + ext/opentelemetry-ext-jaeger/build/* diff --git a/.isort.cfg b/.isort.cfg new file mode 100644 index 0000000..bb84caa --- /dev/null +++ b/.isort.cfg @@ -0,0 +1,16 @@ +[settings] +include_trailing_comma=True +force_grid_wrap=0 +use_parentheses=True +line_length=79 + +; 3 stands for Vertical Hanging Indent, e.g. +; from third_party import ( +; lib1, +; lib2, +; lib3, +; ) +; docs: https://github.com/timothycrosley/isort#multi-line-output-modes +multi_line_output=3 +skip=target + diff --git a/.pylintrc b/.pylintrc new file mode 100644 index 0000000..1aa1e10 --- /dev/null +++ b/.pylintrc @@ -0,0 +1,489 @@ +[MASTER] + +# A comma-separated list of package or module names from where C extensions may +# be loaded. Extensions are loading into the active Python interpreter and may +# run arbitrary code. +extension-pkg-whitelist= + +# Add files or directories to the blacklist. They should be base names, not +# paths. +ignore=CVS,gen + +# Add files or directories matching the regex patterns to the blacklist. The +# regex matches against base names, not paths. +ignore-patterns= + +# Python code to execute, usually for sys.path manipulation such as +# pygtk.require(). +#init-hook= + +# Use multiple processes to speed up Pylint. Specifying 0 will auto-detect the +# number of processors available to use. +jobs=0 + +# Control the amount of potential inferred values when inferring a single +# object. This can help the performance when dealing with large functions or +# complex, nested conditions. +limit-inference-results=100 + +# List of plugins (as comma separated values of python modules names) to load, +# usually to register additional checkers. +load-plugins= + +# Pickle collected data for later comparisons. +persistent=yes + +# Specify a configuration file. +#rcfile= + +# When enabled, pylint would attempt to guess common misconfiguration and emit +# user-friendly hints instead of false-positive error messages. +suggestion-mode=yes + +# Allow loading of arbitrary C extensions. Extensions are imported into the +# active Python interpreter and may run arbitrary code. +unsafe-load-any-extension=no + + +[MESSAGES CONTROL] + +# Only show warnings with the listed confidence levels. Leave empty to show +# all. Valid levels: HIGH, INFERENCE, INFERENCE_FAILURE, UNDEFINED. +confidence= + +# Disable the message, report, category or checker with the given id(s). You +# can either give multiple identifiers separated by comma (,) or put this +# option multiple times (only on the command line, not in the configuration +# file where it should appear only once). You can also use "--disable=all" to +# disable everything first and then reenable specific checks. For example, if +# you want to run only the similarities checker, you can use "--disable=all +# --enable=similarities". If you want to run only the classes checker, but have +# no Warning level messages displayed, use "--disable=all --enable=classes +# --disable=W". +disable=missing-docstring, + fixme, # Warns about FIXME, TODO, etc. comments. + too-few-public-methods, # Might be good to re-enable this later. + too-many-instance-attributes, + too-many-arguments, + ungrouped-imports, # Leave this up to isort + wrong-import-order, # Leave this up to isort + bad-continuation, # Leave this up to black + line-too-long, # Leave this up to black + exec-used + +# Enable the message, report, category or checker with the given id(s). You can +# either give multiple identifier separated by comma (,) or put this option +# multiple time (only on the command line, not in the configuration file where +# it should appear only once). See also the "--disable" option for examples. +# enable=c-extension-no-member + + +[REPORTS] + +# Python expression which should return a note less than 10 (10 is the highest +# note). You have access to the variables errors warning, statement which +# respectively contain the number of errors / warnings messages and the total +# number of statements analyzed. This is used by the global evaluation report +# (RP0004). +#evaluation=10.0 - ((float(5 * error + warning + refactor + convention) / statement) * 10) + +# Template used to display messages. This is a python new-style format string +# used to format the message information. See doc for all details. +#msg-template= + +# Set the output format. Available formats are text, parseable, colorized, json +# and msvs (visual studio). You can also give a reporter class, e.g. +# mypackage.mymodule.MyReporterClass. +#output-format=text + +# Tells whether to display a full report or only the messages. +#reports=no + +# Activate the evaluation score. +score=yes + + +[REFACTORING] + +# Maximum number of nested blocks for function / method body +max-nested-blocks=5 + +# Complete name of functions that never returns. When checking for +# inconsistent-return-statements if a never returning function is called then +# it will be considered as an explicit return statement and no message will be +# printed. +never-returning-functions=sys.exit + + +[LOGGING] + +# Format style used to check logging format string. `old` means using % +# formatting, while `new` is for `{}` formatting. +logging-format-style=old + +# Logging modules to check that the string format arguments are in logging +# function parameter format. +logging-modules=logging + + +[SPELLING] + +# Limits count of emitted suggestions for spelling mistakes. +max-spelling-suggestions=4 + +# Spelling dictionary name. Available dictionaries: none. To make it working +# install python-enchant package.. +spelling-dict= + +# List of comma separated words that should not be checked. +spelling-ignore-words= + +# A path to a file that contains private dictionary; one word per line. +spelling-private-dict-file= + +# Tells whether to store unknown words to indicated private dictionary in +# --spelling-private-dict-file option instead of raising a message. +spelling-store-unknown-words=no + + +[MISCELLANEOUS] + +# List of note tags to take in consideration, separated by a comma. +notes=FIXME, + XXX, + TODO + + +[TYPECHECK] + +# List of decorators that produce context managers, such as +# contextlib.contextmanager. Add to this list to register other decorators that +# produce valid context managers. +contextmanager-decorators=contextlib.contextmanager + +# List of members which are set dynamically and missed by pylint inference +# system, and so shouldn't trigger E1101 when accessed. Python regular +# expressions are accepted. +generated-members= + +# Tells whether missing members accessed in mixin class should be ignored. A +# mixin class is detected if its name ends with "mixin" (case insensitive). +#ignore-mixin-members=yes + +# Tells whether to warn about missing members when the owner of the attribute +# is inferred to be None. +#ignore-none=yes + +# This flag controls whether pylint should warn about no-member and similar +# checks whenever an opaque object is returned when inferring. The inference +# can return multiple potential results while evaluating a Python object, but +# some branches might not be evaluated, which results in partial inference. In +# that case, it might be useful to still emit no-member and other checks for +# the rest of the inferred objects. +#ignore-on-opaque-inference=yes + +# List of class names for which member attributes should not be checked (useful +# for classes with dynamically set attributes). This supports the use of +# qualified names. +ignored-classes=optparse.Values,thread._local,_thread._local + +# List of module names for which member attributes should not be checked +# (useful for modules/projects where namespaces are manipulated during runtime +# and thus existing member attributes cannot be deduced by static analysis. It +# supports qualified module names, as well as Unix pattern matching. +ignored-modules= + +# Show a hint with possible names when a member name was not found. The aspect +# of finding the hint is based on edit distance. +missing-member-hint=yes + +# The minimum edit distance a name should have in order to be considered a +# similar match for a missing member name. +missing-member-hint-distance=1 + +# The total number of similar names that should be taken in consideration when +# showing a hint for a missing member. +missing-member-max-choices=1 + + +[VARIABLES] + +# List of additional names supposed to be defined in builtins. Remember that +# you should avoid defining new builtins when possible. +additional-builtins= + +# Tells whether unused global variables should be treated as a violation. +allow-global-unused-variables=yes + +# List of strings which can identify a callback function by name. A callback +# name must start or end with one of those strings. +callbacks=cb_, + _cb + +# A regular expression matching the name of dummy variables (i.e. expected to +# not be used). +dummy-variables-rgx=_+$|(_[a-zA-Z0-9_]*[a-zA-Z0-9]+?$)|dummy|^ignored_|^unused_ + +# Argument names that match this expression will be ignored. Default to name +# with leading underscore. +ignored-argument-names=_.*|^ignored_|^unused_ + +# Tells whether we should check for unused import in __init__ files. +init-import=no + +# List of qualified module names which can have objects that can redefine +# builtins. +redefining-builtins-modules=six.moves,past.builtins,future.builtins,builtins,io + + +[FORMAT] + +# Expected format of line ending, e.g. empty (any line ending), LF or CRLF. +expected-line-ending-format=LF + +# Regexp for a line that is allowed to be longer than the limit. +ignore-long-lines=^\s*(# )??$ + +# Number of spaces of indent required inside a hanging or continued line. +indent-after-paren=4 + +# String used as indentation unit. This is usually " " (4 spaces) or "\t" (1 +# tab). +indent-string=' ' + +# Maximum number of characters on a single line. +max-line-length=79 + +# Maximum number of lines in a module. +max-module-lines=1000 + +# List of optional constructs for which whitespace checking is disabled. `dict- +# separator` is used to allow tabulation in dicts, etc.: {1 : 1,\n222: 2}. +# `trailing-comma` allows a space between comma and closing bracket: (a, ). +# `empty-line` allows space-only lines. +no-space-check=trailing-comma, + dict-separator + +# Allow the body of a class to be on the same line as the declaration if body +# contains single statement. +single-line-class-stmt=no + +# Allow the body of an if to be on the same line as the test if there is no +# else. +single-line-if-stmt=no + + +[SIMILARITIES] + +# Ignore comments when computing similarities. +ignore-comments=yes + +# Ignore docstrings when computing similarities. +ignore-docstrings=yes + +# Ignore imports when computing similarities. +ignore-imports=no + +# Minimum lines number of a similarity. +min-similarity-lines=4 + + +[BASIC] + +# Naming style matching correct argument names. +argument-naming-style=snake_case + +# Regular expression matching correct argument names. Overrides argument- +# naming-style. +#argument-rgx= + +# Naming style matching correct attribute names. +attr-naming-style=snake_case + +# Regular expression matching correct attribute names. Overrides attr-naming- +# style. +#attr-rgx= + +# Bad variable names which should always be refused, separated by a comma. +bad-names=foo, + bar, + baz, + toto, + tutu, + tata + +# Naming style matching correct class attribute names. +class-attribute-naming-style=any + +# Regular expression matching correct class attribute names. Overrides class- +# attribute-naming-style. +#class-attribute-rgx= + +# Naming style matching correct class names. +class-naming-style=PascalCase + +# Regular expression matching correct class names. Overrides class-naming- +# style. +#class-rgx= + +# Naming style matching correct constant names. +const-naming-style=any + +# Regular expression matching correct constant names. Overrides const-naming- +# style. +#const-rgx= + +# Minimum line length for functions/classes that require docstrings, shorter +# ones are exempt. +docstring-min-length=-1 + +# Naming style matching correct function names. +function-naming-style=snake_case + +# Regular expression matching correct function names. Overrides function- +# naming-style. +#function-rgx= + +# Good variable names which should always be accepted, separated by a comma. +good-names=_, + log, + logger + +# Include a hint for the correct naming format with invalid-name. +include-naming-hint=yes + +# Naming style matching correct inline iteration names. +inlinevar-naming-style=any + +# Regular expression matching correct inline iteration names. Overrides +# inlinevar-naming-style. +#inlinevar-rgx= + +# Naming style matching correct method names. +method-naming-style=snake_case + +# Regular expression matching correct method names. Overrides method-naming- +# style. +#method-rgx= + +# Naming style matching correct module names. +module-naming-style=snake_case + +# Regular expression matching correct module names. Overrides module-naming- +# style. +#module-rgx= + +# Colon-delimited sets of names that determine each other's naming style when +# the name regexes allow several styles. +name-group= + +# Regular expression which should only match function or class names that do +# not require a docstring. +no-docstring-rgx=^_ + +# List of decorators that produce properties, such as abc.abstractproperty. Add +# to this list to register other decorators that produce valid properties. +# These decorators are taken in consideration only for invalid-name. +property-classes=abc.abstractproperty + +# Naming style matching correct variable names. +variable-naming-style=snake_case + +# Regular expression matching correct variable names. Overrides variable- +# naming-style. +variable-rgx=(([a-z_][a-z0-9_]{1,})|(_[a-z0-9_]*)|(__[a-z][a-z0-9_]+__))$ + + +[IMPORTS] + +# Allow wildcard imports from modules that define __all__. +allow-wildcard-with-all=no + +# Analyse import fallback blocks. This can be used to support both Python 2 and +# 3 compatible code, which means that the block might have code that exists +# only in one or another interpreter, leading to false positives when analysed. +analyse-fallback-blocks=yes + +# Deprecated modules which should not be used, separated by a comma. +deprecated-modules=optparse,tkinter.tix + +# Create a graph of external dependencies in the given file (report RP0402 must +# not be disabled). +ext-import-graph= + +# Create a graph of every (i.e. internal and external) dependencies in the +# given file (report RP0402 must not be disabled). +import-graph= + +# Create a graph of internal dependencies in the given file (report RP0402 must +# not be disabled). +int-import-graph= + +# Force import order to recognize a module as part of the standard +# compatibility libraries. +known-standard-library=six + +# Force import order to recognize a module as part of a third party library. +known-third-party=enchant + + +[CLASSES] + +# List of method names used to declare (i.e. assign) instance attributes. +defining-attr-methods=__init__, + __new__, + setUp + +# List of member names, which should be excluded from the protected access +# warning. +exclude-protected=_asdict, + _fields, + _replace, + _source, + _make + +# List of valid names for the first argument in a class method. +valid-classmethod-first-arg=cls + +# List of valid names for the first argument in a metaclass class method. +valid-metaclass-classmethod-first-arg=cls + + +[DESIGN] + +# Maximum number of arguments for function / method. +max-args=5 + +# Maximum number of attributes for a class (see R0902). +max-attributes=7 + +# Maximum number of boolean expressions in an if statement. +max-bool-expr=5 + +# Maximum number of branch for function / method body. +max-branches=12 + +# Maximum number of locals for function / method body. +max-locals=15 + +# Maximum number of parents for a class (see R0901). +max-parents=7 + +# Maximum number of public methods for a class (see R0904). +max-public-methods=20 + +# Maximum number of return / yield for function / method body. +max-returns=6 + +# Maximum number of statements in function / method body. +max-statements=50 + +# Minimum number of public methods for a class (see R0903). +min-public-methods=2 + + +[EXCEPTIONS] + +# Exceptions that will emit a warning when being caught. Defaults to +# "Exception". +overgeneral-exceptions=Exception diff --git a/azure_monitor/examples/client.py b/azure_monitor/examples/client.py index 6607615..6cc7eae 100644 --- a/azure_monitor/examples/client.py +++ b/azure_monitor/examples/client.py @@ -1,13 +1,16 @@ # Copyright (c) Microsoft Corporation. All rights reserved. # Licensed under the MIT License. +# pylint: disable=import-error +# pylint: disable=no-member +# pylint: disable=no-name-in-module import requests - -from azure_monitor import AzureMonitorSpanExporter from opentelemetry import trace from opentelemetry.ext import http_requests from opentelemetry.sdk.trace import TracerSource from opentelemetry.sdk.trace.export import BatchExportSpanProcessor +from azure_monitor import AzureMonitorSpanExporter + trace.set_preferred_tracer_source_implementation(lambda T: TracerSource()) tracer = trace.tracer_source().get_tracer(__name__) http_requests.enable(trace.tracer_source()) diff --git a/azure_monitor/examples/request.py b/azure_monitor/examples/request.py index 975ac86..12d871e 100644 --- a/azure_monitor/examples/request.py +++ b/azure_monitor/examples/request.py @@ -1,13 +1,16 @@ # Copyright (c) Microsoft Corporation. All rights reserved. # Licensed under the MIT License. +# pylint: disable=import-error +# pylint: disable=no-member +# pylint: disable=no-name-in-module import requests - -from azure_monitor import AzureMonitorSpanExporter from opentelemetry import trace from opentelemetry.ext import http_requests from opentelemetry.sdk.trace import TracerSource from opentelemetry.sdk.trace.export import SimpleExportSpanProcessor +from azure_monitor import AzureMonitorSpanExporter + trace.set_preferred_tracer_source_implementation(lambda T: TracerSource()) http_requests.enable(trace.tracer_source()) @@ -19,4 +22,3 @@ with tracer.start_as_current_span("parent"): response = requests.get("", timeout=5) - diff --git a/azure_monitor/examples/server.py b/azure_monitor/examples/server.py index bdeb4f9..b7a3049 100644 --- a/azure_monitor/examples/server.py +++ b/azure_monitor/examples/server.py @@ -1,21 +1,26 @@ # Copyright (c) Microsoft Corporation. All rights reserved. # Licensed under the MIT License. +# pylint: disable=import-error +# pylint: disable=no-member +# pylint: disable=no-name-in-module import flask import requests - -from azure_monitor import AzureMonitorSpanExporter from opentelemetry import trace from opentelemetry.ext import http_requests from opentelemetry.ext.wsgi import OpenTelemetryMiddleware from opentelemetry.sdk.trace import TracerSource from opentelemetry.sdk.trace.export import BatchExportSpanProcessor +from azure_monitor import AzureMonitorSpanExporter + # The preferred tracer implementation must be set, as the opentelemetry-api # defines the interface with a no-op implementation. trace.set_preferred_tracer_source_implementation(lambda T: TracerSource()) tracer = trace.tracer_source().get_tracer(__name__) -exporter = AzureMonitorSpanExporter(instrumentation_key="") +exporter = AzureMonitorSpanExporter( + instrumentation_key="" +) # SpanExporter receives the spans and send them to the target location. span_processor = BatchExportSpanProcessor(exporter) @@ -37,5 +42,5 @@ def hello(): if __name__ == "__main__": - app.run(host='localhost', port=8080, threaded=True) + app.run(host="localhost", port=8080, threaded=True) span_processor.shutdown() diff --git a/azure_monitor/examples/trace.py b/azure_monitor/examples/trace.py index 031fc50..ea77507 100644 --- a/azure_monitor/examples/trace.py +++ b/azure_monitor/examples/trace.py @@ -1,10 +1,13 @@ # Copyright (c) Microsoft Corporation. All rights reserved. # Licensed under the MIT License. -from azure_monitor import AzureMonitorSpanExporter +# pylint: disable=import-error +# pylint: disable=no-member from opentelemetry import trace from opentelemetry.sdk.trace import TracerSource from opentelemetry.sdk.trace.export import SimpleExportSpanProcessor +from azure_monitor import AzureMonitorSpanExporter + trace.set_preferred_tracer_source_implementation(lambda T: TracerSource()) span_processor = SimpleExportSpanProcessor( AzureMonitorSpanExporter(instrumentation_key="") diff --git a/azure_monitor/setup.py b/azure_monitor/setup.py index e38bdd3..ae8edff 100644 --- a/azure_monitor/setup.py +++ b/azure_monitor/setup.py @@ -5,9 +5,7 @@ import setuptools BASE_DIR = os.path.dirname(__file__) -VERSION_FILENAME = os.path.join( - BASE_DIR, "src", "azure_monitor", "version.py" -) +VERSION_FILENAME = os.path.join(BASE_DIR, "src", "azure_monitor", "version.py") PACKAGE_INFO = {} with open(VERSION_FILENAME) as f: exec(f.read(), PACKAGE_INFO) diff --git a/azure_monitor/src/azure_monitor/protocol.py b/azure_monitor/src/azure_monitor/protocol.py index 0de20a9..cda6a95 100644 --- a/azure_monitor/src/azure_monitor/protocol.py +++ b/azure_monitor/src/azure_monitor/protocol.py @@ -1,5 +1,7 @@ # Copyright (c) Microsoft Corporation. All rights reserved. # Licensed under the MIT License. + + class BaseObject(dict): def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) @@ -13,10 +15,7 @@ def __repr__(self): for item in current.items(): if item[0] not in tmp: tmp[item[0]] = item[1] - if ( - current._default # noqa pylint: disable=protected-access - == current - ): + if current._default == current: # noqa pylint: disable=protected-access break current = current._default # noqa pylint: disable=protected-access return repr(tmp) @@ -122,7 +121,7 @@ class Message(BaseObject): message="", severityLevel=None, properties=None, - measurements=None, + measurements=None ) def __init__(self, *args, **kwargs): @@ -182,7 +181,5 @@ def __init__(self, *args, **kwargs): self.ver = self.ver self.id = self.id # noqa pylint: disable=invalid-name self.duration = self.duration - self.responseCode = ( # noqa pylint: disable=invalid-name - self.responseCode - ) + self.responseCode = self.responseCode # noqa pylint: disable=invalid-name self.success = self.success diff --git a/azure_monitor/src/azure_monitor/trace.py b/azure_monitor/src/azure_monitor/trace.py index 714ccb5..40b9822 100644 --- a/azure_monitor/src/azure_monitor/trace.py +++ b/azure_monitor/src/azure_monitor/trace.py @@ -5,13 +5,13 @@ from urllib.parse import urlparse import requests - -from azure_monitor import protocol, utils from opentelemetry.sdk.trace.export import SpanExporter, SpanExportResult from opentelemetry.sdk.util import ns_to_iso_str from opentelemetry.trace import Span, SpanKind from opentelemetry.trace.status import StatusCanonicalCode +from azure_monitor import protocol, utils + logger = logging.getLogger(__name__) @@ -65,47 +65,46 @@ def export(self, spans): return SpanExportResult.FAILED_NOT_RETRYABLE def span_to_envelope(self, span): # noqa pylint: disable=too-many-branches + # pylint: disable=too-many-statements envelope = protocol.Envelope( iKey=self.options.instrumentation_key, tags=dict(utils.azure_monitor_context), time=ns_to_iso_str(span.start_time), ) - envelope.tags["ai.operation.id"] = "{:032x}".format( - span.context.trace_id - ) + envelope.tags["ai.operation.id"] = \ + "{:032x}".format(span.context.trace_id) parent = span.parent if isinstance(parent, Span): parent = parent.context if parent: - envelope.tags[ - "ai.operation.parentId" - ] = "{:016x}".format(parent.span_id) + envelope.tags["ai.operation.parentId"] = \ + "{:016x}".format(parent.span_id) if span.kind in (SpanKind.CONSUMER, SpanKind.SERVER): envelope.name = "Microsoft.ApplicationInsights.Request" data = protocol.Request( - id="{:016x}".format( - span.context.span_id - ), + id="{:016x}".format(span.context.span_id), duration=utils.ns_to_duration(span.end_time - span.start_time), responseCode=str(span.status.value), - success=False, # Modify based off attributes or Status + success=False, # Modify based off attributes or Status properties={}, ) envelope.data = protocol.Data( - baseData=data, baseType="RequestData" + baseData=data, + baseType="RequestData" ) if "http.method" in span.attributes: data.name = span.attributes["http.method"] - if "http.route" in span.attributes: - data.name = data.name + " " + span.attributes["http.route"] - envelope.tags["ai.operation.name"] = data.name - data.properties["request.name"] = data.name - elif 'http.path' in span.attributes: - data.properties['request.name'] = data.name + \ - ' ' + span.attributes['http.path'] + if "http.route" in span.attributes: + data.name = data.name + " " + span.attributes["http.route"] + envelope.tags["ai.operation.name"] = data.name + data.properties["request.name"] = data.name + elif "http.path" in span.attributes: + data.properties["request.name"] = ( + data.name + " " + span.attributes["http.path"] + ) if "http.url" in span.attributes: data.url = span.attributes["http.url"] - data.properties['request.url'] = span.attributes['http.url'] + data.properties["request.url"] = span.attributes["http.url"] if "http.status_code" in span.attributes: status_code = span.attributes["http.status_code"] data.responseCode = str(status_code) @@ -116,20 +115,20 @@ def span_to_envelope(self, span): # noqa pylint: disable=too-many-branches envelope.name = "Microsoft.ApplicationInsights.RemoteDependency" data = protocol.RemoteDependency( name=span.name, - id="{:016x}".format( - span.context.span_id - ), + id="{:016x}".format(span.context.span_id), resultCode=str(span.status.value), duration=utils.ns_to_duration(span.end_time - span.start_time), - success=False, # Modify based off attributes or Status + success=False, # Modify based off attributes or Status properties={}, ) envelope.data = protocol.Data( baseData=data, baseType="RemoteDependencyData" ) if span.kind in (SpanKind.CLIENT, SpanKind.PRODUCER): - if "component" in span.attributes and \ - span.attributes["component"] == "http": + if ( + "component" in span.attributes + and span.attributes["component"] == "http" + ): data.type = "HTTP" if "http.url" in span.attributes: url = span.attributes["http.url"] @@ -141,8 +140,10 @@ def span_to_envelope(self, span): # noqa pylint: disable=too-many-branches data.target = parse_url.netloc if "http.method" in span.attributes: # name is METHOD/path - data.name = span.attributes["http.method"] \ - + "/" + parse_url.path + data.name = ( + span.attributes["http.method"] + "/" + + parse_url.path + ) if "http.status_code" in span.attributes: status_code = span.attributes["http.status_code"] data.resultCode = str(status_code) @@ -154,23 +155,20 @@ def span_to_envelope(self, span): # noqa pylint: disable=too-many-branches data.success = True for key in span.attributes: # This removes redundant data from ApplicationInsights - if key.startswith('http.'): + if key.startswith("http."): continue data.properties[key] = span.attributes[key] if span.links: links = [] for link in span.links: + operation_id = "{:032x}".format(link.context.trace_id) + span_id = "{:016x}".format(link.context.span_id) links.append( { - "operation_Id": "{:032x}".format( - link.context.trace_id - ), - "id": "{:016x}".format( - link.context.span_id - ), + "operation_Id": operation_id, + "id": span_id, } ) data.properties["_MS.links"] = json.dumps(links) - print(data.properties["_MS.links"]) # TODO: tracestate, tags return envelope diff --git a/azure_monitor/src/azure_monitor/utils.py b/azure_monitor/src/azure_monitor/utils.py index 33a596d..0546510 100644 --- a/azure_monitor/src/azure_monitor/utils.py +++ b/azure_monitor/src/azure_monitor/utils.py @@ -5,9 +5,10 @@ import platform import sys +from opentelemetry.sdk.version import __version__ as opentelemetry_version + from azure_monitor.protocol import BaseObject from azure_monitor.version import __version__ as ext_version -from opentelemetry.sdk.version import __version__ as opentelemetry_version azure_monitor_context = { "ai.cloud.role": os.path.basename(sys.argv[0]) or "Python Application", @@ -21,6 +22,7 @@ ), } + def ns_to_duration(nanoseconds): value = (nanoseconds + 500000) // 1000000 # duration in milliseconds value, microseconds = divmod(value, 1000) diff --git a/azure_monitor/tests/__init__.py b/azure_monitor/tests/__init__.py index 6fcf0de..5b7f7a9 100644 --- a/azure_monitor/tests/__init__.py +++ b/azure_monitor/tests/__init__.py @@ -1,2 +1,2 @@ # Copyright (c) Microsoft Corporation. All rights reserved. -# Licensed under the MIT License. \ No newline at end of file +# Licensed under the MIT License. diff --git a/azure_monitor/tests/test_protocol.py b/azure_monitor/tests/test_protocol.py index 927cb4d..695f296 100644 --- a/azure_monitor/tests/test_protocol.py +++ b/azure_monitor/tests/test_protocol.py @@ -5,17 +5,18 @@ from azure_monitor import protocol + class TestProtocol(unittest.TestCase): def test_object(self): data = protocol.BaseObject() - self.assertEqual(repr(data), '{}') + self.assertEqual(repr(data), "{}") data.foo = 1 self.assertEqual(data.foo, 1) - self.assertEqual(data['foo'], 1) - data['bar'] = 2 + self.assertEqual(data["foo"], 1) + data["bar"] = 2 self.assertEqual(data.bar, 2) - self.assertEqual(data['bar'], 2) - self.assertRaises(KeyError, lambda: data['baz']) + self.assertEqual(data["bar"], 2) + self.assertRaises(KeyError, lambda: data["baz"]) self.assertRaises(AttributeError, lambda: data.baz) def test_data(self): @@ -23,6 +24,10 @@ def test_data(self): self.assertIsNone(data.baseData) self.assertIsNone(data.baseType) + def test_data_point(self): + data = protocol.DataPoint() + self.assertEqual(data.ns, "") + def test_envelope(self): data = protocol.Envelope() self.assertEqual(data.ver, 1) @@ -39,6 +44,10 @@ def test_message(self): data = protocol.Message() self.assertEqual(data.ver, 2) + def test_metric_data(self): + data = protocol.MetricData() + self.assertEqual(data.ver, 2) + def test_remote_dependency(self): data = protocol.RemoteDependency() self.assertEqual(data.ver, 2) diff --git a/azure_monitor/tests/test_utils.py b/azure_monitor/tests/test_utils.py index 6832c4f..e3385c2 100644 --- a/azure_monitor/tests/test_utils.py +++ b/azure_monitor/tests/test_utils.py @@ -9,9 +9,9 @@ class TestUtils(unittest.TestCase): def test_nanoseconds_to_duration(self): ns_to_duration = utils.ns_to_duration - self.assertEqual(ns_to_duration(0), '0.00:00:00.000') - self.assertEqual(ns_to_duration(1000000), '0.00:00:00.001') - self.assertEqual(ns_to_duration(1000000000), '0.00:00:01.000') - self.assertEqual(ns_to_duration(60 * 1000000000), '0.00:01:00.000') - self.assertEqual(ns_to_duration(3600 * 1000000000), '0.01:00:00.000') - self.assertEqual(ns_to_duration(86400 * 1000000000), '1.00:00:00.000') + self.assertEqual(ns_to_duration(0), "0.00:00:00.000") + self.assertEqual(ns_to_duration(1000000), "0.00:00:00.001") + self.assertEqual(ns_to_duration(1000000000), "0.00:00:01.000") + self.assertEqual(ns_to_duration(60 * 1000000000), "0.00:01:00.000") + self.assertEqual(ns_to_duration(3600 * 1000000000), "0.01:00:00.000") + self.assertEqual(ns_to_duration(86400 * 1000000000), "1.00:00:00.000") diff --git a/azure_monitor/tests/trace/__init__.py b/azure_monitor/tests/trace/__init__.py index 6fcf0de..5b7f7a9 100644 --- a/azure_monitor/tests/trace/__init__.py +++ b/azure_monitor/tests/trace/__init__.py @@ -1,2 +1,2 @@ # Copyright (c) Microsoft Corporation. All rights reserved. -# Licensed under the MIT License. \ No newline at end of file +# Licensed under the MIT License. diff --git a/azure_monitor/tests/trace/test_trace.py b/azure_monitor/tests/trace/test_trace.py index 75c06f9..984cd98 100644 --- a/azure_monitor/tests/trace/test_trace.py +++ b/azure_monitor/tests/trace/test_trace.py @@ -1,31 +1,32 @@ # Copyright (c) Microsoft Corporation. All rights reserved. # Licensed under the MIT License. -import mock -import time +import json import unittest from azure_monitor.trace import AzureMonitorSpanExporter +from azure_monitor.utils import Options + +from opentelemetry.trace import Link, SpanContext, SpanKind +from opentelemetry.trace.status import StatusCanonicalCode +from opentelemetry.sdk.trace import Span class TestAzureExporter(unittest.TestCase): def test_ctor(self): - from azure_monitor.utils import Options + # pylint: disable=W0212 instrumentation_key = Options._default.instrumentation_key Options._default.instrumentation_key = None - self.assertRaises(ValueError, lambda: AzureMonitorSpanExporter()) + self.assertRaises(ValueError, AzureMonitorSpanExporter()) Options._default.instrumentation_key = instrumentation_key def test_span_to_envelope(self): - from opentelemetry.trace import Link, SpanContext, SpanKind - from opentelemetry.trace.status import StatusCanonicalCode - from opentelemetry.sdk.trace import Span - + # pylint: disable=R0915 exporter = AzureMonitorSpanExporter( - instrumentation_key='12345678-1234-5678-abcd-12345678abcd' + instrumentation_key="12345678-1234-5678-abcd-12345678abcd" ) parent_span = Span( - name='test', + name="test", context=SpanContext( trace_id=36873507687745823477771305566750195431, span_id=12030755672171557338, @@ -37,7 +38,7 @@ def test_span_to_envelope(self): # SpanKind.CLIENT HTTP span = Span( - name='test', + name="test", context=SpanContext( trace_id=36873507687745823477771305566750195431, span_id=12030755672171557337, @@ -47,62 +48,46 @@ def test_span_to_envelope(self): trace_config=None, resource=None, attributes={ - 'component': 'http', - 'http.method': 'GET', - 'http.url': 'https://www.wikipedia.org/wiki/Rabbit', - 'http.status_code': 200, + "component": "http", + "http.method": "GET", + "http.url": "https://www.wikipedia.org/wiki/Rabbit", + "http.status_code": 200, }, events=None, links=None, - kind=SpanKind.CLIENT + kind=SpanKind.CLIENT, ) span.status = StatusCanonicalCode.OK span.start_time = start_time span.end_time = end_time envelope = exporter.span_to_envelope(span) + self.assertEqual(envelope.iKey, "12345678-1234-5678-abcd-12345678abcd") self.assertEqual( - envelope.iKey, - '12345678-1234-5678-abcd-12345678abcd') - self.assertEqual( - envelope.name, - 'Microsoft.ApplicationInsights.RemoteDependency') - self.assertEqual( - envelope.tags['ai.operation.parentId'], - 'a6f5d48acb4d31da') - self.assertEqual( - envelope.tags['ai.operation.id'], - '1bbd944a73a05d89eab5d3740a213ee7') + envelope.name, "Microsoft.ApplicationInsights.RemoteDependency" + ) self.assertEqual( - envelope.time, - '2019-12-04T21:18:36.027613Z') + envelope.tags["ai.operation.parentId"], + "a6f5d48acb4d31da") self.assertEqual( - envelope.data.baseData.name, - 'GET//wiki/Rabbit') + envelope.tags["ai.operation.id"], + "1bbd944a73a05d89eab5d3740a213ee7" + ) + self.assertEqual(envelope.time, "2019-12-04T21:18:36.027613Z") + self.assertEqual(envelope.data.baseData.name, "GET//wiki/Rabbit") self.assertEqual( envelope.data.baseData.data, - 'https://www.wikipedia.org/wiki/Rabbit') - self.assertEqual( - envelope.data.baseData.target, - 'www.wikipedia.org') - self.assertEqual( - envelope.data.baseData.id, - 'a6f5d48acb4d31d9') - self.assertEqual( - envelope.data.baseData.resultCode, - '200') - self.assertEqual( - envelope.data.baseData.duration, - '0.00:00:01.001') - self.assertEqual( - envelope.data.baseData.type, - 'HTTP') - self.assertEqual( - envelope.data.baseType, - 'RemoteDependencyData') + "https://www.wikipedia.org/wiki/Rabbit" + ) + self.assertEqual(envelope.data.baseData.target, "www.wikipedia.org") + self.assertEqual(envelope.data.baseData.id, "a6f5d48acb4d31d9") + self.assertEqual(envelope.data.baseData.resultCode, "200") + self.assertEqual(envelope.data.baseData.duration, "0.00:00:01.001") + self.assertEqual(envelope.data.baseData.type, "HTTP") + self.assertEqual(envelope.data.baseType, "RemoteDependencyData") # SpanKind.CLIENT unknown type span = Span( - name='test', + name="test", context=SpanContext( trace_id=36873507687745823477771305566750195431, span_id=12030755672171557337, @@ -114,46 +99,33 @@ def test_span_to_envelope(self): attributes={}, events=None, links=None, - kind=SpanKind.CLIENT + kind=SpanKind.CLIENT, ) span.status = StatusCanonicalCode.OK span.start_time = start_time span.end_time = end_time envelope = exporter.span_to_envelope(span) + self.assertEqual(envelope.iKey, "12345678-1234-5678-abcd-12345678abcd") self.assertEqual( - envelope.iKey, - '12345678-1234-5678-abcd-12345678abcd') - self.assertEqual( - envelope.name, - 'Microsoft.ApplicationInsights.RemoteDependency') - self.assertEqual( - envelope.tags['ai.operation.parentId'], - 'a6f5d48acb4d31da') - self.assertEqual( - envelope.tags['ai.operation.id'], - '1bbd944a73a05d89eab5d3740a213ee7') - self.assertEqual( - envelope.time, - '2019-12-04T21:18:36.027613Z') - self.assertEqual( - envelope.data.baseData.name, - 'test') - self.assertEqual( - envelope.data.baseData.id, - 'a6f5d48acb4d31d9') - self.assertEqual( - envelope.data.baseData.duration, - '0.00:00:01.001') + envelope.name, "Microsoft.ApplicationInsights.RemoteDependency" + ) self.assertEqual( - envelope.data.baseData.type, - None) + envelope.tags["ai.operation.parentId"], + "a6f5d48acb4d31da") self.assertEqual( - envelope.data.baseType, - 'RemoteDependencyData') + envelope.tags["ai.operation.id"], + "1bbd944a73a05d89eab5d3740a213ee7" + ) + self.assertEqual(envelope.time, "2019-12-04T21:18:36.027613Z") + self.assertEqual(envelope.data.baseData.name, "test") + self.assertEqual(envelope.data.baseData.id, "a6f5d48acb4d31d9") + self.assertEqual(envelope.data.baseData.duration, "0.00:00:01.001") + self.assertEqual(envelope.data.baseData.type, None) + self.assertEqual(envelope.data.baseType, "RemoteDependencyData") # SpanKind.CLIENT missing method span = Span( - name='test', + name="test", context=SpanContext( trace_id=36873507687745823477771305566750195431, span_id=12030755672171557337, @@ -163,61 +135,45 @@ def test_span_to_envelope(self): trace_config=None, resource=None, attributes={ - 'component': 'http', - 'http.url': 'https://www.wikipedia.org/wiki/Rabbit', - 'http.status_code': 200, + "component": "http", + "http.url": "https://www.wikipedia.org/wiki/Rabbit", + "http.status_code": 200, }, events=None, links=None, - kind=SpanKind.CLIENT + kind=SpanKind.CLIENT, ) span.status = StatusCanonicalCode.OK span.start_time = start_time span.end_time = end_time envelope = exporter.span_to_envelope(span) + self.assertEqual(envelope.iKey, "12345678-1234-5678-abcd-12345678abcd") self.assertEqual( - envelope.iKey, - '12345678-1234-5678-abcd-12345678abcd') - self.assertEqual( - envelope.name, - 'Microsoft.ApplicationInsights.RemoteDependency') - self.assertEqual( - envelope.tags['ai.operation.parentId'], - 'a6f5d48acb4d31da') - self.assertEqual( - envelope.tags['ai.operation.id'], - '1bbd944a73a05d89eab5d3740a213ee7') + envelope.name, "Microsoft.ApplicationInsights.RemoteDependency" + ) self.assertEqual( - envelope.time, - '2019-12-04T21:18:36.027613Z') + envelope.tags["ai.operation.parentId"], + "a6f5d48acb4d31da") self.assertEqual( - envelope.data.baseData.name, - 'test') + envelope.tags["ai.operation.id"], + "1bbd944a73a05d89eab5d3740a213ee7" + ) + self.assertEqual(envelope.time, "2019-12-04T21:18:36.027613Z") + self.assertEqual(envelope.data.baseData.name, "test") self.assertEqual( envelope.data.baseData.data, - 'https://www.wikipedia.org/wiki/Rabbit') - self.assertEqual( - envelope.data.baseData.target, - 'www.wikipedia.org') - self.assertEqual( - envelope.data.baseData.id, - 'a6f5d48acb4d31d9') - self.assertEqual( - envelope.data.baseData.resultCode, - '200') - self.assertEqual( - envelope.data.baseData.duration, - '0.00:00:01.001') - self.assertEqual( - envelope.data.baseData.type, - 'HTTP') - self.assertEqual( - envelope.data.baseType, - 'RemoteDependencyData') + "https://www.wikipedia.org/wiki/Rabbit" + ) + self.assertEqual(envelope.data.baseData.target, "www.wikipedia.org") + self.assertEqual(envelope.data.baseData.id, "a6f5d48acb4d31d9") + self.assertEqual(envelope.data.baseData.resultCode, "200") + self.assertEqual(envelope.data.baseData.duration, "0.00:00:01.001") + self.assertEqual(envelope.data.baseData.type, "HTTP") + self.assertEqual(envelope.data.baseType, "RemoteDependencyData") # SpanKind.SERVER HTTP - 200 request span = Span( - name='test', + name="test", context=SpanContext( trace_id=36873507687745823477771305566750195431, span_id=12030755672171557337, @@ -227,64 +183,49 @@ def test_span_to_envelope(self): trace_config=None, resource=None, attributes={ - 'component': 'http', - 'http.method': 'GET', - 'http.path': '/wiki/Rabbit', - 'http.route': '/wiki/Rabbit', - 'http.url': 'https://www.wikipedia.org/wiki/Rabbit', - 'http.status_code': 200, + "component": "http", + "http.method": "GET", + "http.path": "/wiki/Rabbit", + "http.route": "/wiki/Rabbit", + "http.url": "https://www.wikipedia.org/wiki/Rabbit", + "http.status_code": 200, }, events=None, links=None, - kind=SpanKind.SERVER + kind=SpanKind.SERVER, ) span.status = StatusCanonicalCode.OK span.start_time = start_time span.end_time = end_time envelope = exporter.span_to_envelope(span) - self.assertEqual( - envelope.iKey, - '12345678-1234-5678-abcd-12345678abcd') + self.assertEqual(envelope.iKey, "12345678-1234-5678-abcd-12345678abcd") self.assertEqual( envelope.name, - 'Microsoft.ApplicationInsights.Request') - self.assertEqual( - envelope.tags['ai.operation.parentId'], - 'a6f5d48acb4d31da') + "Microsoft.ApplicationInsights.Request") self.assertEqual( - envelope.tags['ai.operation.id'], - '1bbd944a73a05d89eab5d3740a213ee7') + envelope.tags["ai.operation.parentId"], + "a6f5d48acb4d31da") self.assertEqual( - envelope.tags['ai.operation.name'], - 'GET /wiki/Rabbit') - self.assertEqual( - envelope.time, - '2019-12-04T21:18:36.027613Z') - self.assertEqual( - envelope.data.baseData.id, - 'a6f5d48acb4d31d9') - self.assertEqual( - envelope.data.baseData.duration, - '0.00:00:01.001') - self.assertEqual( - envelope.data.baseData.responseCode, - '200') - self.assertEqual( - envelope.data.baseData.name, - 'GET /wiki/Rabbit') - self.assertEqual( - envelope.data.baseData.success, - True) + envelope.tags["ai.operation.id"], + "1bbd944a73a05d89eab5d3740a213ee7" + ) self.assertEqual( - envelope.data.baseData.url, - 'https://www.wikipedia.org/wiki/Rabbit') + envelope.tags["ai.operation.name"], + "GET /wiki/Rabbit") + self.assertEqual(envelope.time, "2019-12-04T21:18:36.027613Z") + self.assertEqual(envelope.data.baseData.id, "a6f5d48acb4d31d9") + self.assertEqual(envelope.data.baseData.duration, "0.00:00:01.001") + self.assertEqual(envelope.data.baseData.responseCode, "200") + self.assertEqual(envelope.data.baseData.name, "GET /wiki/Rabbit") + self.assertEqual(envelope.data.baseData.success, True) self.assertEqual( - envelope.data.baseType, - 'RequestData') + envelope.data.baseData.url, "https://www.wikipedia.org/wiki/Rabbit" + ) + self.assertEqual(envelope.data.baseType, "RequestData") # SpanKind.SERVER HTTP - Failed request span = Span( - name='test', + name="test", context=SpanContext( trace_id=36873507687745823477771305566750195431, span_id=12030755672171557337, @@ -294,64 +235,49 @@ def test_span_to_envelope(self): trace_config=None, resource=None, attributes={ - 'component': 'http', - 'http.method': 'GET', - 'http.path': '/wiki/Rabbit', - 'http.route': '/wiki/Rabbit', - 'http.url': 'https://www.wikipedia.org/wiki/Rabbit', - 'http.status_code': 400, + "component": "http", + "http.method": "GET", + "http.path": "/wiki/Rabbit", + "http.route": "/wiki/Rabbit", + "http.url": "https://www.wikipedia.org/wiki/Rabbit", + "http.status_code": 400, }, events=None, links=None, - kind=SpanKind.SERVER + kind=SpanKind.SERVER, ) span.status = StatusCanonicalCode.OK span.start_time = start_time span.end_time = end_time envelope = exporter.span_to_envelope(span) - self.assertEqual( - envelope.iKey, - '12345678-1234-5678-abcd-12345678abcd') + self.assertEqual(envelope.iKey, "12345678-1234-5678-abcd-12345678abcd") self.assertEqual( envelope.name, - 'Microsoft.ApplicationInsights.Request') - self.assertEqual( - envelope.tags['ai.operation.parentId'], - 'a6f5d48acb4d31da') - self.assertEqual( - envelope.tags['ai.operation.id'], - '1bbd944a73a05d89eab5d3740a213ee7') - self.assertEqual( - envelope.tags['ai.operation.name'], - 'GET /wiki/Rabbit') - self.assertEqual( - envelope.time, - '2019-12-04T21:18:36.027613Z') - self.assertEqual( - envelope.data.baseData.id, - 'a6f5d48acb4d31d9') - self.assertEqual( - envelope.data.baseData.duration, - '0.00:00:01.001') - self.assertEqual( - envelope.data.baseData.responseCode, - '400') + "Microsoft.ApplicationInsights.Request") self.assertEqual( - envelope.data.baseData.name, - 'GET /wiki/Rabbit') + envelope.tags["ai.operation.parentId"], + "a6f5d48acb4d31da") self.assertEqual( - envelope.data.baseData.success, - False) + envelope.tags["ai.operation.id"], + "1bbd944a73a05d89eab5d3740a213ee7" + ) self.assertEqual( - envelope.data.baseData.url, - 'https://www.wikipedia.org/wiki/Rabbit') + envelope.tags["ai.operation.name"], + "GET /wiki/Rabbit") + self.assertEqual(envelope.time, "2019-12-04T21:18:36.027613Z") + self.assertEqual(envelope.data.baseData.id, "a6f5d48acb4d31d9") + self.assertEqual(envelope.data.baseData.duration, "0.00:00:01.001") + self.assertEqual(envelope.data.baseData.responseCode, "400") + self.assertEqual(envelope.data.baseData.name, "GET /wiki/Rabbit") + self.assertEqual(envelope.data.baseData.success, False) self.assertEqual( - envelope.data.baseType, - 'RequestData') + envelope.data.baseData.url, "https://www.wikipedia.org/wiki/Rabbit" + ) + self.assertEqual(envelope.data.baseType, "RequestData") # SpanKind.SERVER unknown type span = Span( - name='test', + name="test", context=SpanContext( trace_id=36873507687745823477771305566750195431, span_id=12030755672171557337, @@ -361,49 +287,40 @@ def test_span_to_envelope(self): trace_config=None, resource=None, attributes={ - 'component': 'http', - 'http.method': 'GET', - 'http.path': '/wiki/Rabbit', - 'http.route': '/wiki/Rabbit', - 'http.url': 'https://www.wikipedia.org/wiki/Rabbit', - 'http.status_code': 400, + "component": "http", + "http.method": "GET", + "http.path": "/wiki/Rabbit", + "http.route": "/wiki/Rabbit", + "http.url": "https://www.wikipedia.org/wiki/Rabbit", + "http.status_code": 400, }, events=None, links=None, - kind=SpanKind.SERVER + kind=SpanKind.SERVER, ) span.status = StatusCanonicalCode.OK span.start_time = start_time span.end_time = end_time envelope = exporter.span_to_envelope(span) - self.assertEqual( - envelope.iKey, - '12345678-1234-5678-abcd-12345678abcd') + self.assertEqual(envelope.iKey, "12345678-1234-5678-abcd-12345678abcd") self.assertEqual( envelope.name, - 'Microsoft.ApplicationInsights.Request') - self.assertEqual( - envelope.tags['ai.operation.parentId'], - 'a6f5d48acb4d31da') + "Microsoft.ApplicationInsights.Request") self.assertEqual( - envelope.tags['ai.operation.id'], - '1bbd944a73a05d89eab5d3740a213ee7') + envelope.tags["ai.operation.parentId"], + "a6f5d48acb4d31da") self.assertEqual( - envelope.time, - '2019-12-04T21:18:36.027613Z') - self.assertEqual( - envelope.data.baseData.id, - 'a6f5d48acb4d31d9') - self.assertEqual( - envelope.data.baseData.duration, - '0.00:00:01.001') - self.assertEqual( - envelope.data.baseType, - 'RequestData') + envelope.tags["ai.operation.id"], + "1bbd944a73a05d89eab5d3740a213ee7" + ) + self.assertEqual(envelope.time, "2019-12-04T21:18:36.027613Z") + self.assertEqual(envelope.data.baseData.id, "a6f5d48acb4d31d9") + self.assertEqual(envelope.data.baseData.duration, "0.00:00:01.001") + self.assertEqual(envelope.data.baseType, "RequestData") # SpanKind.INTERNAL span = Span( - name='test', + name="test", context=SpanContext( trace_id=36873507687745823477771305566750195431, span_id=12030755672171557337, @@ -412,52 +329,37 @@ def test_span_to_envelope(self): sampler=None, trace_config=None, resource=None, - attributes={'key1': 'value1'}, + attributes={"key1": "value1"}, events=None, links=None, - kind=SpanKind.INTERNAL + kind=SpanKind.INTERNAL, ) span.status = StatusCanonicalCode.OK span.start_time = start_time span.end_time = end_time envelope = exporter.span_to_envelope(span) + self.assertEqual(envelope.iKey, "12345678-1234-5678-abcd-12345678abcd") self.assertEqual( - envelope.iKey, - '12345678-1234-5678-abcd-12345678abcd') - self.assertEqual( - envelope.name, - 'Microsoft.ApplicationInsights.RemoteDependency') + envelope.name, "Microsoft.ApplicationInsights.RemoteDependency" + ) self.assertRaises( KeyError, - lambda: envelope.tags['ai.operation.parentId']) - self.assertEqual( - envelope.tags['ai.operation.id'], - '1bbd944a73a05d89eab5d3740a213ee7') - self.assertEqual( - envelope.time, - '2019-12-04T21:18:36.027613Z') - self.assertEqual( - envelope.data.baseData.name, - 'test') - self.assertEqual( - envelope.data.baseData.duration, - '0.00:00:01.001') - self.assertEqual( - envelope.data.baseData.id, - 'a6f5d48acb4d31d9') + lambda: envelope.tags["ai.operation.parentId"]) self.assertEqual( - envelope.data.baseData.type, - 'InProc') - self.assertEqual( - envelope.data.baseData.success, - True) - self.assertEqual( - envelope.data.baseType, - 'RemoteDependencyData') + envelope.tags["ai.operation.id"], + "1bbd944a73a05d89eab5d3740a213ee7" + ) + self.assertEqual(envelope.time, "2019-12-04T21:18:36.027613Z") + self.assertEqual(envelope.data.baseData.name, "test") + self.assertEqual(envelope.data.baseData.duration, "0.00:00:01.001") + self.assertEqual(envelope.data.baseData.id, "a6f5d48acb4d31d9") + self.assertEqual(envelope.data.baseData.type, "InProc") + self.assertEqual(envelope.data.baseData.success, True) + self.assertEqual(envelope.data.baseType, "RemoteDependencyData") # Attributes span = Span( - name='test', + name="test", context=SpanContext( trace_id=36873507687745823477771305566750195431, span_id=12030755672171557337, @@ -467,34 +369,38 @@ def test_span_to_envelope(self): trace_config=None, resource=None, attributes={ - 'component': 'http', - 'http.method': 'GET', - 'http.url': 'https://www.wikipedia.org/wiki/Rabbit', - 'http.status_code': 200, - 'test': 'asd' + "component": "http", + "http.method": "GET", + "http.url": "https://www.wikipedia.org/wiki/Rabbit", + "http.status_code": 200, + "test": "asd", }, events=None, links=None, - kind=SpanKind.CLIENT + kind=SpanKind.CLIENT, ) span.status = StatusCanonicalCode.OK span.start_time = start_time span.end_time = end_time envelope = exporter.span_to_envelope(span) + self.assertEqual(len(envelope.data.baseData.properties), 2) self.assertEqual( - len(envelope.data.baseData.properties), 2) - self.assertEqual( - envelope.data.baseData.properties['component'], 'http') - self.assertEqual(envelope.data.baseData.properties['test'], 'asd') + envelope.data.baseData.properties["component"], + "http") + self.assertEqual(envelope.data.baseData.properties["test"], "asd") # Links links = [] - links.append(Link(context=SpanContext( - trace_id=36873507687745823477771305566750195432, - span_id=12030755672171557338, - ))) + links.append( + Link( + context=SpanContext( + trace_id=36873507687745823477771305566750195432, + span_id=12030755672171557338, + ) + ) + ) span = Span( - name='test', + name="test", context=SpanContext( trace_id=36873507687745823477771305566750195431, span_id=12030755672171557337, @@ -504,29 +410,27 @@ def test_span_to_envelope(self): trace_config=None, resource=None, attributes={ - 'component': 'http', - 'http.method': 'GET', - 'http.url': 'https://www.wikipedia.org/wiki/Rabbit', - 'http.status_code': 200, + "component": "http", + "http.method": "GET", + "http.url": "https://www.wikipedia.org/wiki/Rabbit", + "http.status_code": 200, }, events=None, links=links, - kind=SpanKind.CLIENT + kind=SpanKind.CLIENT, ) span.status = StatusCanonicalCode.OK span.start_time = start_time span.end_time = end_time envelope = exporter.span_to_envelope(span) - self.assertEqual( - len(envelope.data.baseData.properties), 2) - links_json = '[{"operation_Id": ' + \ - '"1bbd944a73a05d89eab5d3740a213ee8", "id": "a6f5d48acb4d31da"}]' - self.assertEqual(envelope.data.baseData.properties['_MS.links'], links_json) + self.assertEqual(len(envelope.data.baseData.properties), 2) + json_dict = json.loads( + envelope.data.baseData.properties["_MS.links"])[0] + self.assertEqual(json_dict["id"], "a6f5d48acb4d31da") - # Status span = Span( - name='test', + name="test", context=SpanContext( trace_id=36873507687745823477771305566750195431, span_id=12030755672171557337, @@ -536,25 +440,24 @@ def test_span_to_envelope(self): trace_config=None, resource=None, attributes={ - 'component': 'http', - 'http.method': 'GET', - 'http.url': 'https://www.wikipedia.org/wiki/Rabbit', - 'http.status_code': 500, + "component": "http", + "http.method": "GET", + "http.url": "https://www.wikipedia.org/wiki/Rabbit", + "http.status_code": 500, }, events=None, links=None, - kind=SpanKind.SERVER + kind=SpanKind.SERVER, ) span.status = StatusCanonicalCode.OK span.start_time = start_time span.end_time = end_time envelope = exporter.span_to_envelope(span) - self.assertEqual( - envelope.data.baseData.responseCode, '500') + self.assertEqual(envelope.data.baseData.responseCode, "500") self.assertFalse(envelope.data.baseData.success) span = Span( - name='test', + name="test", context=SpanContext( trace_id=36873507687745823477771305566750195431, span_id=12030755672171557337, @@ -564,25 +467,24 @@ def test_span_to_envelope(self): trace_config=None, resource=None, attributes={ - 'component': 'http', - 'http.method': 'GET', - 'http.url': 'https://www.wikipedia.org/wiki/Rabbit', - 'http.status_code': 500, + "component": "http", + "http.method": "GET", + "http.url": "https://www.wikipedia.org/wiki/Rabbit", + "http.status_code": 500, }, events=None, links=None, - kind=SpanKind.CLIENT + kind=SpanKind.CLIENT, ) span.status = StatusCanonicalCode.OK span.start_time = start_time span.end_time = end_time envelope = exporter.span_to_envelope(span) - self.assertEqual( - envelope.data.baseData.resultCode, '500') + self.assertEqual(envelope.data.baseData.resultCode, "500") self.assertFalse(envelope.data.baseData.success) span = Span( - name='test', + name="test", context=SpanContext( trace_id=36873507687745823477771305566750195431, span_id=12030755672171557337, @@ -592,25 +494,24 @@ def test_span_to_envelope(self): trace_config=None, resource=None, attributes={ - 'component': 'http', - 'http.method': 'GET', - 'http.url': 'https://www.wikipedia.org/wiki/Rabbit', + "component": "http", + "http.method": "GET", + "http.url": "https://www.wikipedia.org/wiki/Rabbit", }, events=None, links=None, - kind=SpanKind.SERVER + kind=SpanKind.SERVER, ) span.status = StatusCanonicalCode.OK span.start_time = start_time span.end_time = end_time span.status = StatusCanonicalCode.OK envelope = exporter.span_to_envelope(span) - self.assertEqual( - envelope.data.baseData.responseCode, '0') + self.assertEqual(envelope.data.baseData.responseCode, "0") self.assertTrue(envelope.data.baseData.success) span = Span( - name='test', + name="test", context=SpanContext( trace_id=36873507687745823477771305566750195431, span_id=12030755672171557337, @@ -620,25 +521,24 @@ def test_span_to_envelope(self): trace_config=None, resource=None, attributes={ - 'component': 'http', - 'http.method': 'GET', - 'http.url': 'https://www.wikipedia.org/wiki/Rabbit', + "component": "http", + "http.method": "GET", + "http.url": "https://www.wikipedia.org/wiki/Rabbit", }, events=None, links=None, - kind=SpanKind.CLIENT + kind=SpanKind.CLIENT, ) span.status = StatusCanonicalCode.OK span.start_time = start_time span.end_time = end_time span.status = StatusCanonicalCode.OK envelope = exporter.span_to_envelope(span) - self.assertEqual( - envelope.data.baseData.resultCode, '0') + self.assertEqual(envelope.data.baseData.resultCode, "0") self.assertTrue(envelope.data.baseData.success) span = Span( - name='test', + name="test", context=SpanContext( trace_id=36873507687745823477771305566750195431, span_id=12030755672171557337, @@ -648,24 +548,23 @@ def test_span_to_envelope(self): trace_config=None, resource=None, attributes={ - 'component': 'http', - 'http.method': 'GET', - 'http.url': 'https://www.wikipedia.org/wiki/Rabbit', + "component": "http", + "http.method": "GET", + "http.url": "https://www.wikipedia.org/wiki/Rabbit", }, events=None, links=None, - kind=SpanKind.SERVER + kind=SpanKind.SERVER, ) span.start_time = start_time span.end_time = end_time span.status = StatusCanonicalCode.UNKNOWN envelope = exporter.span_to_envelope(span) - self.assertEqual( - envelope.data.baseData.responseCode, '2') + self.assertEqual(envelope.data.baseData.responseCode, "2") self.assertFalse(envelope.data.baseData.success) span = Span( - name='test', + name="test", context=SpanContext( trace_id=36873507687745823477771305566750195431, span_id=12030755672171557337, @@ -675,25 +574,24 @@ def test_span_to_envelope(self): trace_config=None, resource=None, attributes={ - 'component': 'http', - 'http.method': 'GET', - 'http.url': 'https://www.wikipedia.org/wiki/Rabbit', + "component": "http", + "http.method": "GET", + "http.url": "https://www.wikipedia.org/wiki/Rabbit", }, events=None, links=None, - kind=SpanKind.CLIENT + kind=SpanKind.CLIENT, ) span.start_time = start_time span.end_time = end_time span.status = StatusCanonicalCode.UNKNOWN envelope = exporter.span_to_envelope(span) - self.assertEqual( - envelope.data.baseData.resultCode, '2') + self.assertEqual(envelope.data.baseData.resultCode, "2") self.assertFalse(envelope.data.baseData.success) # Server route attribute span = Span( - name='test', + name="test", context=SpanContext( trace_id=36873507687745823477771305566750195431, span_id=12030755672171557337, @@ -703,29 +601,60 @@ def test_span_to_envelope(self): trace_config=None, resource=None, attributes={ - 'component': 'HTTP', - 'http.method': 'GET', - 'http.route': '/wiki/Rabbit', - 'http.path': '/wiki/Rabbitz', - 'http.url': 'https://www.wikipedia.org/wiki/Rabbit', - 'http.status_code': 400, + "component": "HTTP", + "http.method": "GET", + "http.route": "/wiki/Rabbit", + "http.path": "/wiki/Rabbitz", + "http.url": "https://www.wikipedia.org/wiki/Rabbit", + "http.status_code": 400, }, events=None, links=None, - kind=SpanKind.SERVER + kind=SpanKind.SERVER, ) span.start_time = start_time span.end_time = end_time span.status = StatusCanonicalCode.OK envelope = exporter.span_to_envelope(span) self.assertEqual( - envelope.data.baseData.properties['request.name'], 'GET /wiki/Rabbit') + envelope.data.baseData.properties["request.name"], + "GET /wiki/Rabbit" + ) self.assertEqual( - envelope.data.baseData.properties['request.url'], 'https://www.wikipedia.org/wiki/Rabbit') + envelope.data.baseData.properties["request.url"], + "https://www.wikipedia.org/wiki/Rabbit", + ) + + # Server method attribute missing + span = Span( + name="test", + context=SpanContext( + trace_id=36873507687745823477771305566750195431, + span_id=12030755672171557337, + ), + parent=parent_span, + sampler=None, + trace_config=None, + resource=None, + attributes={ + "component": "HTTP", + "http.path": "/wiki/Rabbitz", + "http.url": "https://www.wikipedia.org/wiki/Rabbit", + "http.status_code": 400, + }, + events=None, + links=None, + kind=SpanKind.SERVER, + ) + span.start_time = start_time + span.end_time = end_time + span.status = StatusCanonicalCode.OK + envelope = exporter.span_to_envelope(span) + self.assertIsNone(envelope.data.baseData.name) # Server route attribute missing span = Span( - name='test', + name="test", context=SpanContext( trace_id=36873507687745823477771305566750195431, span_id=12030755672171557337, @@ -735,28 +664,33 @@ def test_span_to_envelope(self): trace_config=None, resource=None, attributes={ - 'component': 'HTTP', - 'http.method': 'GET', - 'http.path': '/wiki/Rabbitz', - 'http.url': 'https://www.wikipedia.org/wiki/Rabbit', - 'http.status_code': 400, + "component": "HTTP", + "http.method": "GET", + "http.path": "/wiki/Rabbitz", + "http.url": "https://www.wikipedia.org/wiki/Rabbit", + "http.status_code": 400, }, events=None, links=None, - kind=SpanKind.SERVER + kind=SpanKind.SERVER, ) span.start_time = start_time span.end_time = end_time span.status = StatusCanonicalCode.OK envelope = exporter.span_to_envelope(span) + self.assertEqual(envelope.data.baseData.name, "GET") self.assertEqual( - envelope.data.baseData.properties['request.name'], 'GET /wiki/Rabbitz') + envelope.data.baseData.properties["request.name"], + "GET /wiki/Rabbitz" + ) self.assertEqual( - envelope.data.baseData.properties['request.url'], 'https://www.wikipedia.org/wiki/Rabbit') + envelope.data.baseData.properties["request.url"], + "https://www.wikipedia.org/wiki/Rabbit", + ) # Server route and path attribute missing span = Span( - name='test', + name="test", context=SpanContext( trace_id=36873507687745823477771305566750195431, span_id=12030755672171557337, @@ -766,20 +700,51 @@ def test_span_to_envelope(self): trace_config=None, resource=None, attributes={ - 'component': 'HTTP', - 'http.method': 'GET', - 'http.url': 'https://www.wikipedia.org/wiki/Rabbit', - 'http.status_code': 400, + "component": "HTTP", + "http.method": "GET", + "http.url": "https://www.wikipedia.org/wiki/Rabbit", + "http.status_code": 400, }, events=None, links=None, - kind=SpanKind.SERVER + kind=SpanKind.SERVER, ) span.start_time = start_time span.end_time = end_time span.status = StatusCanonicalCode.OK envelope = exporter.span_to_envelope(span) self.assertIsNone( - envelope.data.baseData.properties.get('request.name')) + envelope.data.baseData.properties.get("request.name")) self.assertEqual( - envelope.data.baseData.properties['request.url'], 'https://www.wikipedia.org/wiki/Rabbit') + envelope.data.baseData.properties["request.url"], + "https://www.wikipedia.org/wiki/Rabbit", + ) + + # Server http.url missing + span = Span( + name="test", + context=SpanContext( + trace_id=36873507687745823477771305566750195431, + span_id=12030755672171557337, + ), + parent=parent_span, + sampler=None, + trace_config=None, + resource=None, + attributes={ + "component": "HTTP", + "http.method": "GET", + "http.route": "/wiki/Rabbit", + "http.path": "/wiki/Rabbitz", + "http.status_code": 400, + }, + events=None, + links=None, + kind=SpanKind.SERVER, + ) + span.start_time = start_time + span.end_time = end_time + span.status = StatusCanonicalCode.OK + envelope = exporter.span_to_envelope(span) + self.assertIsNone(envelope.data.baseData.url) + self.assertIsNone(envelope.data.baseData.properties.get("request.url")) diff --git a/dev-requirements.txt b/dev-requirements.txt new file mode 100644 index 0000000..6ce5479 --- /dev/null +++ b/dev-requirements.txt @@ -0,0 +1,10 @@ +pylint~=2.3 +flake8~=3.7 +isort~=4.3 +black>=19.3b0,==19.* +mypy==0.740 +sphinx~=2.1 +sphinx-rtd-theme~=0.4 +sphinx-autodoc-typehints~=1.10.2 +pytest!=5.2.3 +pytest-cov>=2.8 diff --git a/scripts/pylint.sh b/scripts/pylint.sh new file mode 100644 index 0000000..d625d56 --- /dev/null +++ b/scripts/pylint.sh @@ -0,0 +1,24 @@ +# Copyright 2019, OpenCensus Authors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +set -ev + +# Run pylint on directories +function pylint_dir { + python -m pip install --upgrade pylint + pylint $(find azure_monitor -type f -name "*.py") + return $? +} + +pylint_dir diff --git a/tox.ini b/tox.ini new file mode 100644 index 0000000..6bffb9a --- /dev/null +++ b/tox.ini @@ -0,0 +1,55 @@ +[tox] +skipsdist = True +skip_missing_interpreters = True +envlist = + ; py3{4,5,6,7,8}-test-{azure_monitor} + ; py3{4}-coverage + + lint + ; py37-{mypy} + ; docs + +[travis] +python = + 3.7: py37, lint, docs + +[testenv] +deps = + -c dev-requirements.txt + test: pytest + coverage: pytest + coverage: pytest-cov + mypy: mypy + +changedir = + test-azure_monitor: azure_monitor/tests + +commands_pre = + python -m pip install -U pip + test-azure_monitor: pip install -e {toxinidir}/azure_monitor + coverage: pip install -e {toxinidir}/azure_monitor + +commands = + test: pytest {posargs} + coverage: coverage erase + coverage: pytest --ignore-glob=*/setup.py --cov azure_monitor --cov-append --cov-report term-missing + coverage: coverage report + +[testenv:lint] +basepython: python3.7 +recreate = True +deps = + -c dev-requirements.txt + pylint + flake8 + isort + black + +commands_pre = + pip install -e {toxinidir}/azure_monitor + +commands = +; black . --diff --check +; isort --diff --check-only --recursive . +; flake8 + bash ./scripts/pylint.sh \ No newline at end of file From 3fbf875f9f10aa604cab1acbe679dc88b4e85b45 Mon Sep 17 00:00:00 2001 From: Leighton Date: Fri, 31 Jan 2020 11:53:09 -0800 Subject: [PATCH 021/109] lint --- .isort.cfg | 2 +- azure_monitor/examples/server.py | 2 +- azure_monitor/src/azure_monitor/protocol.py | 10 ++- azure_monitor/src/azure_monitor/trace.py | 25 +++--- azure_monitor/tests/trace/test_trace.py | 89 +++++++++++---------- pyproject.toml | 2 + tox.ini | 8 +- 7 files changed, 72 insertions(+), 66 deletions(-) create mode 100644 pyproject.toml diff --git a/.isort.cfg b/.isort.cfg index bb84caa..837d197 100644 --- a/.isort.cfg +++ b/.isort.cfg @@ -13,4 +13,4 @@ line_length=79 ; docs: https://github.com/timothycrosley/isort#multi-line-output-modes multi_line_output=3 skip=target - +known_third_party=opentelemetry diff --git a/azure_monitor/examples/server.py b/azure_monitor/examples/server.py index b7a3049..1a5b507 100644 --- a/azure_monitor/examples/server.py +++ b/azure_monitor/examples/server.py @@ -3,7 +3,6 @@ # pylint: disable=import-error # pylint: disable=no-member # pylint: disable=no-name-in-module -import flask import requests from opentelemetry import trace from opentelemetry.ext import http_requests @@ -11,6 +10,7 @@ from opentelemetry.sdk.trace import TracerSource from opentelemetry.sdk.trace.export import BatchExportSpanProcessor +import flask from azure_monitor import AzureMonitorSpanExporter # The preferred tracer implementation must be set, as the opentelemetry-api diff --git a/azure_monitor/src/azure_monitor/protocol.py b/azure_monitor/src/azure_monitor/protocol.py index cda6a95..6e61cb0 100644 --- a/azure_monitor/src/azure_monitor/protocol.py +++ b/azure_monitor/src/azure_monitor/protocol.py @@ -15,7 +15,9 @@ def __repr__(self): for item in current.items(): if item[0] not in tmp: tmp[item[0]] = item[1] - if current._default == current: # noqa pylint: disable=protected-access + if ( + current._default == current + ): # noqa pylint: disable=protected-access break current = current._default # noqa pylint: disable=protected-access return repr(tmp) @@ -121,7 +123,7 @@ class Message(BaseObject): message="", severityLevel=None, properties=None, - measurements=None + measurements=None, ) def __init__(self, *args, **kwargs): @@ -181,5 +183,7 @@ def __init__(self, *args, **kwargs): self.ver = self.ver self.id = self.id # noqa pylint: disable=invalid-name self.duration = self.duration - self.responseCode = self.responseCode # noqa pylint: disable=invalid-name + self.responseCode = ( + self.responseCode + ) # noqa pylint: disable=invalid-name self.success = self.success diff --git a/azure_monitor/src/azure_monitor/trace.py b/azure_monitor/src/azure_monitor/trace.py index 40b9822..9257270 100644 --- a/azure_monitor/src/azure_monitor/trace.py +++ b/azure_monitor/src/azure_monitor/trace.py @@ -71,14 +71,16 @@ def span_to_envelope(self, span): # noqa pylint: disable=too-many-branches tags=dict(utils.azure_monitor_context), time=ns_to_iso_str(span.start_time), ) - envelope.tags["ai.operation.id"] = \ - "{:032x}".format(span.context.trace_id) + envelope.tags["ai.operation.id"] = "{:032x}".format( + span.context.trace_id + ) parent = span.parent if isinstance(parent, Span): parent = parent.context if parent: - envelope.tags["ai.operation.parentId"] = \ - "{:016x}".format(parent.span_id) + envelope.tags["ai.operation.parentId"] = "{:016x}".format( + parent.span_id + ) if span.kind in (SpanKind.CONSUMER, SpanKind.SERVER): envelope.name = "Microsoft.ApplicationInsights.Request" data = protocol.Request( @@ -89,8 +91,7 @@ def span_to_envelope(self, span): # noqa pylint: disable=too-many-branches properties={}, ) envelope.data = protocol.Data( - baseData=data, - baseType="RequestData" + baseData=data, baseType="RequestData" ) if "http.method" in span.attributes: data.name = span.attributes["http.method"] @@ -141,8 +142,9 @@ def span_to_envelope(self, span): # noqa pylint: disable=too-many-branches if "http.method" in span.attributes: # name is METHOD/path data.name = ( - span.attributes["http.method"] + "/" + - parse_url.path + span.attributes["http.method"] + + "/" + + parse_url.path ) if "http.status_code" in span.attributes: status_code = span.attributes["http.status_code"] @@ -163,12 +165,7 @@ def span_to_envelope(self, span): # noqa pylint: disable=too-many-branches for link in span.links: operation_id = "{:032x}".format(link.context.trace_id) span_id = "{:016x}".format(link.context.span_id) - links.append( - { - "operation_Id": operation_id, - "id": span_id, - } - ) + links.append({"operation_Id": operation_id, "id": span_id}) data.properties["_MS.links"] = json.dumps(links) # TODO: tracestate, tags return envelope diff --git a/azure_monitor/tests/trace/test_trace.py b/azure_monitor/tests/trace/test_trace.py index 984cd98..6e72a95 100644 --- a/azure_monitor/tests/trace/test_trace.py +++ b/azure_monitor/tests/trace/test_trace.py @@ -4,12 +4,13 @@ import json import unittest +from opentelemetry.sdk.trace import Span +from opentelemetry.trace import Link, SpanContext, SpanKind +from opentelemetry.trace.status import StatusCanonicalCode + from azure_monitor.trace import AzureMonitorSpanExporter from azure_monitor.utils import Options -from opentelemetry.trace import Link, SpanContext, SpanKind -from opentelemetry.trace.status import StatusCanonicalCode -from opentelemetry.sdk.trace import Span class TestAzureExporter(unittest.TestCase): def test_ctor(self): @@ -66,17 +67,17 @@ def test_span_to_envelope(self): envelope.name, "Microsoft.ApplicationInsights.RemoteDependency" ) self.assertEqual( - envelope.tags["ai.operation.parentId"], - "a6f5d48acb4d31da") + envelope.tags["ai.operation.parentId"], "a6f5d48acb4d31da" + ) self.assertEqual( envelope.tags["ai.operation.id"], - "1bbd944a73a05d89eab5d3740a213ee7" + "1bbd944a73a05d89eab5d3740a213ee7", ) self.assertEqual(envelope.time, "2019-12-04T21:18:36.027613Z") self.assertEqual(envelope.data.baseData.name, "GET//wiki/Rabbit") self.assertEqual( envelope.data.baseData.data, - "https://www.wikipedia.org/wiki/Rabbit" + "https://www.wikipedia.org/wiki/Rabbit", ) self.assertEqual(envelope.data.baseData.target, "www.wikipedia.org") self.assertEqual(envelope.data.baseData.id, "a6f5d48acb4d31d9") @@ -110,11 +111,11 @@ def test_span_to_envelope(self): envelope.name, "Microsoft.ApplicationInsights.RemoteDependency" ) self.assertEqual( - envelope.tags["ai.operation.parentId"], - "a6f5d48acb4d31da") + envelope.tags["ai.operation.parentId"], "a6f5d48acb4d31da" + ) self.assertEqual( envelope.tags["ai.operation.id"], - "1bbd944a73a05d89eab5d3740a213ee7" + "1bbd944a73a05d89eab5d3740a213ee7", ) self.assertEqual(envelope.time, "2019-12-04T21:18:36.027613Z") self.assertEqual(envelope.data.baseData.name, "test") @@ -152,17 +153,17 @@ def test_span_to_envelope(self): envelope.name, "Microsoft.ApplicationInsights.RemoteDependency" ) self.assertEqual( - envelope.tags["ai.operation.parentId"], - "a6f5d48acb4d31da") + envelope.tags["ai.operation.parentId"], "a6f5d48acb4d31da" + ) self.assertEqual( envelope.tags["ai.operation.id"], - "1bbd944a73a05d89eab5d3740a213ee7" + "1bbd944a73a05d89eab5d3740a213ee7", ) self.assertEqual(envelope.time, "2019-12-04T21:18:36.027613Z") self.assertEqual(envelope.data.baseData.name, "test") self.assertEqual( envelope.data.baseData.data, - "https://www.wikipedia.org/wiki/Rabbit" + "https://www.wikipedia.org/wiki/Rabbit", ) self.assertEqual(envelope.data.baseData.target, "www.wikipedia.org") self.assertEqual(envelope.data.baseData.id, "a6f5d48acb4d31d9") @@ -200,18 +201,18 @@ def test_span_to_envelope(self): envelope = exporter.span_to_envelope(span) self.assertEqual(envelope.iKey, "12345678-1234-5678-abcd-12345678abcd") self.assertEqual( - envelope.name, - "Microsoft.ApplicationInsights.Request") + envelope.name, "Microsoft.ApplicationInsights.Request" + ) self.assertEqual( - envelope.tags["ai.operation.parentId"], - "a6f5d48acb4d31da") + envelope.tags["ai.operation.parentId"], "a6f5d48acb4d31da" + ) self.assertEqual( envelope.tags["ai.operation.id"], - "1bbd944a73a05d89eab5d3740a213ee7" + "1bbd944a73a05d89eab5d3740a213ee7", ) self.assertEqual( - envelope.tags["ai.operation.name"], - "GET /wiki/Rabbit") + envelope.tags["ai.operation.name"], "GET /wiki/Rabbit" + ) self.assertEqual(envelope.time, "2019-12-04T21:18:36.027613Z") self.assertEqual(envelope.data.baseData.id, "a6f5d48acb4d31d9") self.assertEqual(envelope.data.baseData.duration, "0.00:00:01.001") @@ -252,18 +253,18 @@ def test_span_to_envelope(self): envelope = exporter.span_to_envelope(span) self.assertEqual(envelope.iKey, "12345678-1234-5678-abcd-12345678abcd") self.assertEqual( - envelope.name, - "Microsoft.ApplicationInsights.Request") + envelope.name, "Microsoft.ApplicationInsights.Request" + ) self.assertEqual( - envelope.tags["ai.operation.parentId"], - "a6f5d48acb4d31da") + envelope.tags["ai.operation.parentId"], "a6f5d48acb4d31da" + ) self.assertEqual( envelope.tags["ai.operation.id"], - "1bbd944a73a05d89eab5d3740a213ee7" + "1bbd944a73a05d89eab5d3740a213ee7", ) self.assertEqual( - envelope.tags["ai.operation.name"], - "GET /wiki/Rabbit") + envelope.tags["ai.operation.name"], "GET /wiki/Rabbit" + ) self.assertEqual(envelope.time, "2019-12-04T21:18:36.027613Z") self.assertEqual(envelope.data.baseData.id, "a6f5d48acb4d31d9") self.assertEqual(envelope.data.baseData.duration, "0.00:00:01.001") @@ -304,14 +305,14 @@ def test_span_to_envelope(self): envelope = exporter.span_to_envelope(span) self.assertEqual(envelope.iKey, "12345678-1234-5678-abcd-12345678abcd") self.assertEqual( - envelope.name, - "Microsoft.ApplicationInsights.Request") + envelope.name, "Microsoft.ApplicationInsights.Request" + ) self.assertEqual( - envelope.tags["ai.operation.parentId"], - "a6f5d48acb4d31da") + envelope.tags["ai.operation.parentId"], "a6f5d48acb4d31da" + ) self.assertEqual( envelope.tags["ai.operation.id"], - "1bbd944a73a05d89eab5d3740a213ee7" + "1bbd944a73a05d89eab5d3740a213ee7", ) self.assertEqual(envelope.time, "2019-12-04T21:18:36.027613Z") self.assertEqual(envelope.data.baseData.id, "a6f5d48acb4d31d9") @@ -343,11 +344,11 @@ def test_span_to_envelope(self): envelope.name, "Microsoft.ApplicationInsights.RemoteDependency" ) self.assertRaises( - KeyError, - lambda: envelope.tags["ai.operation.parentId"]) + KeyError, lambda: envelope.tags["ai.operation.parentId"] + ) self.assertEqual( envelope.tags["ai.operation.id"], - "1bbd944a73a05d89eab5d3740a213ee7" + "1bbd944a73a05d89eab5d3740a213ee7", ) self.assertEqual(envelope.time, "2019-12-04T21:18:36.027613Z") self.assertEqual(envelope.data.baseData.name, "test") @@ -385,8 +386,8 @@ def test_span_to_envelope(self): envelope = exporter.span_to_envelope(span) self.assertEqual(len(envelope.data.baseData.properties), 2) self.assertEqual( - envelope.data.baseData.properties["component"], - "http") + envelope.data.baseData.properties["component"], "http" + ) self.assertEqual(envelope.data.baseData.properties["test"], "asd") # Links @@ -424,8 +425,9 @@ def test_span_to_envelope(self): span.end_time = end_time envelope = exporter.span_to_envelope(span) self.assertEqual(len(envelope.data.baseData.properties), 2) - json_dict = json.loads( - envelope.data.baseData.properties["_MS.links"])[0] + json_dict = json.loads(envelope.data.baseData.properties["_MS.links"])[ + 0 + ] self.assertEqual(json_dict["id"], "a6f5d48acb4d31da") # Status @@ -618,7 +620,7 @@ def test_span_to_envelope(self): envelope = exporter.span_to_envelope(span) self.assertEqual( envelope.data.baseData.properties["request.name"], - "GET /wiki/Rabbit" + "GET /wiki/Rabbit", ) self.assertEqual( envelope.data.baseData.properties["request.url"], @@ -681,7 +683,7 @@ def test_span_to_envelope(self): self.assertEqual(envelope.data.baseData.name, "GET") self.assertEqual( envelope.data.baseData.properties["request.name"], - "GET /wiki/Rabbitz" + "GET /wiki/Rabbitz", ) self.assertEqual( envelope.data.baseData.properties["request.url"], @@ -714,7 +716,8 @@ def test_span_to_envelope(self): span.status = StatusCanonicalCode.OK envelope = exporter.span_to_envelope(span) self.assertIsNone( - envelope.data.baseData.properties.get("request.name")) + envelope.data.baseData.properties.get("request.name") + ) self.assertEqual( envelope.data.baseData.properties["request.url"], "https://www.wikipedia.org/wiki/Rabbit", diff --git a/pyproject.toml b/pyproject.toml new file mode 100644 index 0000000..a8f43fe --- /dev/null +++ b/pyproject.toml @@ -0,0 +1,2 @@ +[tool.black] +line-length = 79 diff --git a/tox.ini b/tox.ini index 6bffb9a..71924e2 100644 --- a/tox.ini +++ b/tox.ini @@ -49,7 +49,7 @@ commands_pre = pip install -e {toxinidir}/azure_monitor commands = -; black . --diff --check -; isort --diff --check-only --recursive . -; flake8 - bash ./scripts/pylint.sh \ No newline at end of file + black . --diff --check + isort --diff --check-only --recursive . + flake8 +; bash ./scripts/pylint.sh \ No newline at end of file From 7cae035c55c64b86dc4fb2d79d4dae1016ceac3d Mon Sep 17 00:00:00 2001 From: Leighton Date: Fri, 31 Jan 2020 12:00:30 -0800 Subject: [PATCH 022/109] Add travis --- .travis.yml | 20 ++++++++++++++++++++ tox.ini | 8 +++----- 2 files changed, 23 insertions(+), 5 deletions(-) create mode 100644 .travis.yml diff --git a/.travis.yml b/.travis.yml new file mode 100644 index 0000000..2cfc918 --- /dev/null +++ b/.travis.yml @@ -0,0 +1,20 @@ +dist: xenial + +language: python + +python: + - '3.4' + - '3.5' + - '3.6' + - '3.7' + - '3.8' + +install: + - pip install tox-travis + +script: + - tox + +after_success: + - pip install codecov + - codecov -v diff --git a/tox.ini b/tox.ini index 71924e2..49190bc 100644 --- a/tox.ini +++ b/tox.ini @@ -2,12 +2,10 @@ skipsdist = True skip_missing_interpreters = True envlist = - ; py3{4,5,6,7,8}-test-{azure_monitor} - ; py3{4}-coverage + py3{4,5,6,7,8}-test-{azure_monitor} + py3{4}-coverage lint - ; py37-{mypy} - ; docs [travis] python = @@ -52,4 +50,4 @@ commands = black . --diff --check isort --diff --check-only --recursive . flake8 -; bash ./scripts/pylint.sh \ No newline at end of file + bash ./scripts/pylint.sh From eea7876ae00faf577d423dea372e633d35bd0205 Mon Sep 17 00:00:00 2001 From: Leighton Date: Fri, 31 Jan 2020 12:31:19 -0800 Subject: [PATCH 023/109] Update --- .flake8 | 3 +-- .pylintrc | 2 +- scripts/pylint.sh | 15 ++------------- tox.ini | 2 +- 4 files changed, 5 insertions(+), 17 deletions(-) diff --git a/.flake8 b/.flake8 index 6d1c2bc..c7299ee 100644 --- a/.flake8 +++ b/.flake8 @@ -15,5 +15,4 @@ exclude = venv*/ target __pycache__ - ext/opentelemetry-ext-jaeger/src/opentelemetry/ext/jaeger/gen/ - ext/opentelemetry-ext-jaeger/build/* + diff --git a/.pylintrc b/.pylintrc index 1aa1e10..4e25a1e 100644 --- a/.pylintrc +++ b/.pylintrc @@ -239,7 +239,7 @@ redefining-builtins-modules=six.moves,past.builtins,future.builtins,builtins,io [FORMAT] # Expected format of line ending, e.g. empty (any line ending), LF or CRLF. -expected-line-ending-format=LF +expected-line-ending-format= # Regexp for a line that is allowed to be longer than the limit. ignore-long-lines=^\s*(# )??$ diff --git a/scripts/pylint.sh b/scripts/pylint.sh index d625d56..f757fac 100644 --- a/scripts/pylint.sh +++ b/scripts/pylint.sh @@ -1,16 +1,5 @@ -# Copyright 2019, OpenCensus Authors -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. set -ev diff --git a/tox.ini b/tox.ini index 49190bc..2e7d0ae 100644 --- a/tox.ini +++ b/tox.ini @@ -24,7 +24,7 @@ changedir = commands_pre = python -m pip install -U pip - test-azure_monitor: pip install -e {toxinidir}/azure_monitor + test-azure_monitor: pip install {toxinidir}/azure_monitor coverage: pip install -e {toxinidir}/azure_monitor commands = From d1240ce7daa8f4a2a1ae2107fcf3ea6d133ae04f Mon Sep 17 00:00:00 2001 From: Leighton Date: Fri, 31 Jan 2020 12:34:45 -0800 Subject: [PATCH 024/109] remove editable --- tox.ini | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tox.ini b/tox.ini index 2e7d0ae..1124a7d 100644 --- a/tox.ini +++ b/tox.ini @@ -25,7 +25,7 @@ changedir = commands_pre = python -m pip install -U pip test-azure_monitor: pip install {toxinidir}/azure_monitor - coverage: pip install -e {toxinidir}/azure_monitor + coverage: pip install {toxinidir}/azure_monitor commands = test: pytest {posargs} @@ -44,7 +44,7 @@ deps = black commands_pre = - pip install -e {toxinidir}/azure_monitor + pip install {toxinidir}/azure_monitor commands = black . --diff --check From bfedaa27c8a0202d85ba7a0492a3ac3e4dd86bfc Mon Sep 17 00:00:00 2001 From: Leighton Date: Fri, 31 Jan 2020 12:39:09 -0800 Subject: [PATCH 025/109] setuptools --- tox.ini | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tox.ini b/tox.ini index 1124a7d..ecd8159 100644 --- a/tox.ini +++ b/tox.ini @@ -23,7 +23,7 @@ changedir = test-azure_monitor: azure_monitor/tests commands_pre = - python -m pip install -U pip + python -m pip install -U pip setuptools test-azure_monitor: pip install {toxinidir}/azure_monitor coverage: pip install {toxinidir}/azure_monitor From 37729d97da7d152841fe59e115c606f6fa196b32 Mon Sep 17 00:00:00 2001 From: Leighton Date: Fri, 31 Jan 2020 12:52:26 -0800 Subject: [PATCH 026/109] fix test --- azure_monitor/tests/trace/test_trace.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/azure_monitor/tests/trace/test_trace.py b/azure_monitor/tests/trace/test_trace.py index 6e72a95..96eb45b 100644 --- a/azure_monitor/tests/trace/test_trace.py +++ b/azure_monitor/tests/trace/test_trace.py @@ -17,7 +17,8 @@ def test_ctor(self): # pylint: disable=W0212 instrumentation_key = Options._default.instrumentation_key Options._default.instrumentation_key = None - self.assertRaises(ValueError, AzureMonitorSpanExporter()) + with self.assertRaises(ValueError): + AzureMonitorSpanExporter() Options._default.instrumentation_key = instrumentation_key def test_span_to_envelope(self): From d7ad83a138470a75a1cea7d42f303d77c619e00d Mon Sep 17 00:00:00 2001 From: Leighton Date: Fri, 31 Jan 2020 13:45:04 -0800 Subject: [PATCH 027/109] editable --- azure_monitor/src/azure_monitor/protocol.py | 4 ++-- azure_monitor/src/azure_monitor/trace.py | 2 +- azure_monitor/tests/trace/test_trace.py | 2 +- tox.ini | 6 +++--- 4 files changed, 7 insertions(+), 7 deletions(-) diff --git a/azure_monitor/src/azure_monitor/protocol.py b/azure_monitor/src/azure_monitor/protocol.py index 6e61cb0..fd9e32a 100644 --- a/azure_monitor/src/azure_monitor/protocol.py +++ b/azure_monitor/src/azure_monitor/protocol.py @@ -184,6 +184,6 @@ def __init__(self, *args, **kwargs): self.id = self.id # noqa pylint: disable=invalid-name self.duration = self.duration self.responseCode = ( - self.responseCode - ) # noqa pylint: disable=invalid-name + self.responseCode # noqa pylint: disable=invalid-name + ) self.success = self.success diff --git a/azure_monitor/src/azure_monitor/trace.py b/azure_monitor/src/azure_monitor/trace.py index 9257270..68eb537 100644 --- a/azure_monitor/src/azure_monitor/trace.py +++ b/azure_monitor/src/azure_monitor/trace.py @@ -14,7 +14,7 @@ logger = logging.getLogger(__name__) - +# pylint: disable=import-error class AzureMonitorSpanExporter(SpanExporter): def __init__(self, **options): self.options = utils.Options(**options) diff --git a/azure_monitor/tests/trace/test_trace.py b/azure_monitor/tests/trace/test_trace.py index 96eb45b..b89868c 100644 --- a/azure_monitor/tests/trace/test_trace.py +++ b/azure_monitor/tests/trace/test_trace.py @@ -11,7 +11,7 @@ from azure_monitor.trace import AzureMonitorSpanExporter from azure_monitor.utils import Options - +# pylint: disable=import-error class TestAzureExporter(unittest.TestCase): def test_ctor(self): # pylint: disable=W0212 diff --git a/tox.ini b/tox.ini index ecd8159..69d5fb5 100644 --- a/tox.ini +++ b/tox.ini @@ -3,13 +3,13 @@ skipsdist = True skip_missing_interpreters = True envlist = py3{4,5,6,7,8}-test-{azure_monitor} - py3{4}-coverage + py3{4,5,6,7,8}-coverage lint [travis] python = - 3.7: py37, lint, docs + 3.7: py37, lint [testenv] deps = @@ -25,7 +25,7 @@ changedir = commands_pre = python -m pip install -U pip setuptools test-azure_monitor: pip install {toxinidir}/azure_monitor - coverage: pip install {toxinidir}/azure_monitor + coverage: pip install -e {toxinidir}/azure_monitor commands = test: pytest {posargs} From 76871ff5ad8eeba5d09fda3ac70818c6612ac72d Mon Sep 17 00:00:00 2001 From: Leighton Date: Fri, 31 Jan 2020 13:47:44 -0800 Subject: [PATCH 028/109] coverage --- .coveragerc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.coveragerc b/.coveragerc index eb9ca5a..f41a787 100644 --- a/.coveragerc +++ b/.coveragerc @@ -5,7 +5,7 @@ omit = azure_monitor/tests/* [report] -fail_under = 100 +fail_under = 80 show_missing = True omit = azure_monitor/setup.py From 43780d7c89fadf9ad4a2d43fa247a4ba5e498766 Mon Sep 17 00:00:00 2001 From: Leighton Date: Fri, 31 Jan 2020 13:57:43 -0800 Subject: [PATCH 029/109] isort --- azure_monitor/src/azure_monitor/trace.py | 1 + azure_monitor/tests/trace/test_trace.py | 1 + 2 files changed, 2 insertions(+) diff --git a/azure_monitor/src/azure_monitor/trace.py b/azure_monitor/src/azure_monitor/trace.py index 68eb537..c0388c3 100644 --- a/azure_monitor/src/azure_monitor/trace.py +++ b/azure_monitor/src/azure_monitor/trace.py @@ -14,6 +14,7 @@ logger = logging.getLogger(__name__) + # pylint: disable=import-error class AzureMonitorSpanExporter(SpanExporter): def __init__(self, **options): diff --git a/azure_monitor/tests/trace/test_trace.py b/azure_monitor/tests/trace/test_trace.py index b89868c..c994564 100644 --- a/azure_monitor/tests/trace/test_trace.py +++ b/azure_monitor/tests/trace/test_trace.py @@ -11,6 +11,7 @@ from azure_monitor.trace import AzureMonitorSpanExporter from azure_monitor.utils import Options + # pylint: disable=import-error class TestAzureExporter(unittest.TestCase): def test_ctor(self): From ae7e68aab744ce1cc7e1ca7414893855673af2bd Mon Sep 17 00:00:00 2001 From: Leighton Date: Fri, 31 Jan 2020 14:01:27 -0800 Subject: [PATCH 030/109] imports --- azure_monitor/src/azure_monitor/trace.py | 2 +- azure_monitor/src/azure_monitor/utils.py | 1 + azure_monitor/tests/trace/test_trace.py | 1 + 3 files changed, 3 insertions(+), 1 deletion(-) diff --git a/azure_monitor/src/azure_monitor/trace.py b/azure_monitor/src/azure_monitor/trace.py index c0388c3..f72320e 100644 --- a/azure_monitor/src/azure_monitor/trace.py +++ b/azure_monitor/src/azure_monitor/trace.py @@ -4,6 +4,7 @@ import logging from urllib.parse import urlparse +# pylint: disable=import-error import requests from opentelemetry.sdk.trace.export import SpanExporter, SpanExportResult from opentelemetry.sdk.util import ns_to_iso_str @@ -15,7 +16,6 @@ logger = logging.getLogger(__name__) -# pylint: disable=import-error class AzureMonitorSpanExporter(SpanExporter): def __init__(self, **options): self.options = utils.Options(**options) diff --git a/azure_monitor/src/azure_monitor/utils.py b/azure_monitor/src/azure_monitor/utils.py index 0546510..e50628e 100644 --- a/azure_monitor/src/azure_monitor/utils.py +++ b/azure_monitor/src/azure_monitor/utils.py @@ -5,6 +5,7 @@ import platform import sys +# pylint: disable=import-error from opentelemetry.sdk.version import __version__ as opentelemetry_version from azure_monitor.protocol import BaseObject diff --git a/azure_monitor/tests/trace/test_trace.py b/azure_monitor/tests/trace/test_trace.py index c994564..9131da1 100644 --- a/azure_monitor/tests/trace/test_trace.py +++ b/azure_monitor/tests/trace/test_trace.py @@ -4,6 +4,7 @@ import json import unittest +# pylint: disable=import-error from opentelemetry.sdk.trace import Span from opentelemetry.trace import Link, SpanContext, SpanKind from opentelemetry.trace.status import StatusCanonicalCode From 0f0976991a8348e11685d3659a3706a65bab5571 Mon Sep 17 00:00:00 2001 From: Leighton Date: Fri, 31 Jan 2020 14:02:18 -0800 Subject: [PATCH 031/109] name --- azure_monitor/src/azure_monitor/protocol.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/azure_monitor/src/azure_monitor/protocol.py b/azure_monitor/src/azure_monitor/protocol.py index fd9e32a..7315c67 100644 --- a/azure_monitor/src/azure_monitor/protocol.py +++ b/azure_monitor/src/azure_monitor/protocol.py @@ -183,7 +183,7 @@ def __init__(self, *args, **kwargs): self.ver = self.ver self.id = self.id # noqa pylint: disable=invalid-name self.duration = self.duration - self.responseCode = ( + self.responseCode = ( # noqa pylint: disable=invalid-name self.responseCode # noqa pylint: disable=invalid-name ) self.success = self.success From 2dcaf36d5f94d392f7b2b5a5ff26dd3787b9bba3 Mon Sep 17 00:00:00 2001 From: Leighton Date: Fri, 31 Jan 2020 14:02:35 -0800 Subject: [PATCH 032/109] name --- azure_monitor/src/azure_monitor/protocol.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/azure_monitor/src/azure_monitor/protocol.py b/azure_monitor/src/azure_monitor/protocol.py index 7315c67..25f72b8 100644 --- a/azure_monitor/src/azure_monitor/protocol.py +++ b/azure_monitor/src/azure_monitor/protocol.py @@ -183,7 +183,7 @@ def __init__(self, *args, **kwargs): self.ver = self.ver self.id = self.id # noqa pylint: disable=invalid-name self.duration = self.duration - self.responseCode = ( # noqa pylint: disable=invalid-name + self.responseCode = ( # noqa pylint: disable=invalid-name self.responseCode # noqa pylint: disable=invalid-name ) self.success = self.success From 93b5b975001ce609224d4f0298a7f4c577ee561f Mon Sep 17 00:00:00 2001 From: Hector Hernandez Guzman Date: Wed, 19 Feb 2020 16:22:50 -0800 Subject: [PATCH 033/109] Adding readme for examples Creating traces folder Fixing issue with Status new property in API --- azure_monitor/examples/traces/README.md | 58 +++++++++++++++++++ azure_monitor/examples/{ => traces}/client.py | 0 .../examples/{ => traces}/request.py | 2 +- azure_monitor/examples/{ => traces}/server.py | 0 azure_monitor/examples/{ => traces}/trace.py | 4 +- azure_monitor/src/azure_monitor/trace.py | 4 +- 6 files changed, 64 insertions(+), 4 deletions(-) create mode 100644 azure_monitor/examples/traces/README.md rename azure_monitor/examples/{ => traces}/client.py (100%) rename azure_monitor/examples/{ => traces}/request.py (92%) rename azure_monitor/examples/{ => traces}/server.py (100%) rename azure_monitor/examples/{ => traces}/trace.py (87%) diff --git a/azure_monitor/examples/traces/README.md b/azure_monitor/examples/traces/README.md new file mode 100644 index 0000000..d388724 --- /dev/null +++ b/azure_monitor/examples/traces/README.md @@ -0,0 +1,58 @@ + + +# Overview + +* Azure Monitor examples require opentelemetry packages version 0.4a0 + +## Installation + +```sh +$ pip install opentelemetry-azure-monitor-exporter +``` + +## Run the Applications + +### Trace + +* Update the code in trace.py to use your INSTRUMENTATION_KEY + +* Run the sample + +```sh +$ # from this directory +$ python trace.py +``` + +### Request + +* Update the code in request.py to use your INSTRUMENTATION_KEY + +* Run the sample + +```sh +$ pip install opentelemetry-ext-http-requests +$ # from this directory +$ python request.py +``` + +### Server + +* Update the code in server.py to use your INSTRUMENTATION_KEY + +* Run the sample + +```sh +$ pip install opentelemetry-ext-http-requests +$ pip install opentelemetry-ext-wsgi +$ # from this directory +$ python server.py +``` + +* Open http://localhost:8080/ + + +## Explore the data + +After running the applications, data would be available in [Azure]( +https://docs.microsoft.com/en-us/azure/azure-monitor/app/app-insights-overview#where-do-i-see-my-telemetry) + diff --git a/azure_monitor/examples/client.py b/azure_monitor/examples/traces/client.py similarity index 100% rename from azure_monitor/examples/client.py rename to azure_monitor/examples/traces/client.py diff --git a/azure_monitor/examples/request.py b/azure_monitor/examples/traces/request.py similarity index 92% rename from azure_monitor/examples/request.py rename to azure_monitor/examples/traces/request.py index 12d871e..aadd55f 100644 --- a/azure_monitor/examples/request.py +++ b/azure_monitor/examples/traces/request.py @@ -21,4 +21,4 @@ tracer = trace.tracer_source().get_tracer(__name__) with tracer.start_as_current_span("parent"): - response = requests.get("", timeout=5) + response = requests.get("https://azure.microsoft.com/", timeout=5) diff --git a/azure_monitor/examples/server.py b/azure_monitor/examples/traces/server.py similarity index 100% rename from azure_monitor/examples/server.py rename to azure_monitor/examples/traces/server.py diff --git a/azure_monitor/examples/trace.py b/azure_monitor/examples/traces/trace.py similarity index 87% rename from azure_monitor/examples/trace.py rename to azure_monitor/examples/traces/trace.py index ea77507..8f94033 100644 --- a/azure_monitor/examples/trace.py +++ b/azure_monitor/examples/traces/trace.py @@ -10,7 +10,9 @@ trace.set_preferred_tracer_source_implementation(lambda T: TracerSource()) span_processor = SimpleExportSpanProcessor( - AzureMonitorSpanExporter(instrumentation_key="") + AzureMonitorSpanExporter( + instrumentation_key="" + ) ) trace.tracer_source().add_span_processor(span_processor) tracer = trace.tracer_source().get_tracer(__name__) diff --git a/azure_monitor/src/azure_monitor/trace.py b/azure_monitor/src/azure_monitor/trace.py index f72320e..61d6e22 100644 --- a/azure_monitor/src/azure_monitor/trace.py +++ b/azure_monitor/src/azure_monitor/trace.py @@ -87,7 +87,7 @@ def span_to_envelope(self, span): # noqa pylint: disable=too-many-branches data = protocol.Request( id="{:016x}".format(span.context.span_id), duration=utils.ns_to_duration(span.end_time - span.start_time), - responseCode=str(span.status.value), + responseCode=str(span.status.canonical_code), success=False, # Modify based off attributes or Status properties={}, ) @@ -118,7 +118,7 @@ def span_to_envelope(self, span): # noqa pylint: disable=too-many-branches data = protocol.RemoteDependency( name=span.name, id="{:016x}".format(span.context.span_id), - resultCode=str(span.status.value), + resultCode=str(span.status.canonical_code), duration=utils.ns_to_duration(span.end_time - span.start_time), success=False, # Modify based off attributes or Status properties={}, From f6904dd22a00eabf95fa594be06412b6b222e87a Mon Sep 17 00:00:00 2001 From: Hector Hernandez Guzman Date: Wed, 19 Feb 2020 16:54:40 -0800 Subject: [PATCH 034/109] Test --- tox.ini | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tox.ini b/tox.ini index 69d5fb5..eccd753 100644 --- a/tox.ini +++ b/tox.ini @@ -31,7 +31,7 @@ commands = test: pytest {posargs} coverage: coverage erase coverage: pytest --ignore-glob=*/setup.py --cov azure_monitor --cov-append --cov-report term-missing - coverage: coverage report + coverage: coverage report [testenv:lint] basepython: python3.7 From 0a9e4c6d7fb77ce4397e0cb7019cd6069412e399 Mon Sep 17 00:00:00 2001 From: Hector Hernandez Guzman Date: Fri, 21 Feb 2020 14:05:09 -0800 Subject: [PATCH 035/109] Updating OpenTelemetry dependency version --- azure_monitor/examples/traces/README.md | 5 ----- azure_monitor/setup.cfg | 4 ++-- 2 files changed, 2 insertions(+), 7 deletions(-) diff --git a/azure_monitor/examples/traces/README.md b/azure_monitor/examples/traces/README.md index d388724..8ae4ec8 100644 --- a/azure_monitor/examples/traces/README.md +++ b/azure_monitor/examples/traces/README.md @@ -1,9 +1,4 @@ - -# Overview - -* Azure Monitor examples require opentelemetry packages version 0.4a0 - ## Installation ```sh diff --git a/azure_monitor/setup.cfg b/azure_monitor/setup.cfg index d6b9775..c52347e 100644 --- a/azure_monitor/setup.cfg +++ b/azure_monitor/setup.cfg @@ -27,8 +27,8 @@ package_dir= =src packages=find_namespace: install_requires = - opentelemetry-api - opentelemetry-sdk + opentelemetry-api >= 0.4a0 + opentelemetry-sdk >= 0.4a0 requests ~= 2.0 [options.packages.find] From 1c25861690bb7a60b7c46ce3781759a46650d79c Mon Sep 17 00:00:00 2001 From: Hector Hernandez Guzman Date: Mon, 24 Feb 2020 15:58:10 -0800 Subject: [PATCH 036/109] Adding connection string support Adding instrumentation key validation --- azure_monitor/src/azure_monitor/trace.py | 2 - azure_monitor/src/azure_monitor/utils.py | 102 +++++++++- azure_monitor/tests/test_utils.py | 247 +++++++++++++++++++++++ azure_monitor/tests/trace/test_trace.py | 4 + 4 files changed, 351 insertions(+), 4 deletions(-) diff --git a/azure_monitor/src/azure_monitor/trace.py b/azure_monitor/src/azure_monitor/trace.py index f72320e..b4bbc54 100644 --- a/azure_monitor/src/azure_monitor/trace.py +++ b/azure_monitor/src/azure_monitor/trace.py @@ -19,8 +19,6 @@ class AzureMonitorSpanExporter(SpanExporter): def __init__(self, **options): self.options = utils.Options(**options) - if not self.options.instrumentation_key: - raise ValueError("The instrumentation_key is not provided.") def export(self, spans): envelopes = tuple(map(self.span_to_envelope, spans)) diff --git a/azure_monitor/src/azure_monitor/utils.py b/azure_monitor/src/azure_monitor/utils.py index e50628e..3ddfc46 100644 --- a/azure_monitor/src/azure_monitor/utils.py +++ b/azure_monitor/src/azure_monitor/utils.py @@ -3,6 +3,7 @@ import locale import os import platform +import re import sys # pylint: disable=import-error @@ -11,6 +12,7 @@ from azure_monitor.protocol import BaseObject from azure_monitor.version import __version__ as ext_version + azure_monitor_context = { "ai.cloud.role": os.path.basename(sys.argv[0]) or "Python Application", "ai.cloud.roleInstance": platform.node(), @@ -35,9 +37,105 @@ def ns_to_duration(nanoseconds): ) +INGESTION_ENDPOINT = "ingestionendpoint" +INSTRUMENTATION_KEY = "instrumentationkey" + +# Validate UUID format +# Specs taken from https://tools.ietf.org/html/rfc4122 +uuid_regex_pattern = re.compile( + "^[0-9a-f]{8}-" + "[0-9a-f]{4}-" + "[1-5][0-9a-f]{3}-" + "[89ab][0-9a-f]{3}-" + "[0-9a-f]{12}$" +) + + class Options(BaseObject): + def __init__(self, *args, **kwargs): + super(Options, self).__init__(*args, **kwargs) + self._initialize() + self._validate_instrumentation_key() + _default = BaseObject( + connection_string=None, + instrumentation_key=None, endpoint="https://dc.services.visualstudio.com/v2/track", - instrumentation_key=os.getenv("APPINSIGHTS_INSTRUMENTATIONKEY", None), - timeout=10.0, # networking timeout in seconds + timeout=10.0, ) + + def _initialize(self): + code_cs = self._parse_connection_string(self.connection_string) + code_ikey = self.instrumentation_key + env_cs = self._parse_connection_string( + os.getenv("APPLICATIONINSIGHTS_CONNECTION_STRING") + ) + env_ikey = os.getenv("APPINSIGHTS_INSTRUMENTATIONKEY") + + # The priority of which value takes on the instrumentation key is: + # 1. Key from explicitly passed in connection string + # 2. Key from explicitly passed in instrumentation key + # 3. Key from connection string in environment variable + # 4. Key from instrumentation key in environment variable + self.instrumentation_key = ( + code_cs.get(INSTRUMENTATION_KEY) + or code_ikey + or env_cs.get(INSTRUMENTATION_KEY) + or env_ikey + ) + # The priority of the ingestion endpoint is as follows: + # 1. The endpoint explicitly passed in connection string + # 2. The endpoint from the connection string in environment variable + # 3. The default breeze endpoint + endpoint = ( + code_cs.get(INGESTION_ENDPOINT) + or env_cs.get(INGESTION_ENDPOINT) + or "https://dc.services.visualstudio.com" + ) + self.endpoint = endpoint + "/v2/track" + + def _validate_instrumentation_key(self): + """Validates the instrumentation key used for Azure Monitor. + An instrumentation key cannot be null or empty. An instrumentation key + is valid for Azure Monitor only if it is a valid UUID. + :param instrumentation_key: The instrumentation key to validate + """ + if not self.instrumentation_key: + raise ValueError("Instrumentation key cannot be none or empty.") + match = uuid_regex_pattern.match(self.instrumentation_key) + if not match: + raise ValueError("Invalid instrumentation key.") + + def _parse_connection_string(self, connection_string): + if connection_string is None: + return {} + try: + pairs = connection_string.split(";") + result = dict(s.split("=") for s in pairs) + # Convert keys to lower-case due to case type-insensitive checking + result = {key.lower(): value for key, value in result.items()} + except Exception: + raise ValueError("Invalid connection string") + # Validate authorization + auth = result.get("authorization") + if auth is not None and auth.lower() != "ikey": + raise ValueError("Invalid authorization mechanism") + # Construct the ingestion endpoint if not passed in explicitly + if result.get(INGESTION_ENDPOINT) is None: + endpoint_suffix = "" + location_prefix = "" + suffix = result.get("endpointsuffix") + if suffix is not None: + endpoint_suffix = suffix + # Get regional information if provided + prefix = result.get("location") + if prefix is not None: + location_prefix = prefix + "." + endpoint = ( + "https://" + location_prefix + "dc." + endpoint_suffix + ) + result[INGESTION_ENDPOINT] = endpoint + else: + # Default to None if cannot construct + result[INGESTION_ENDPOINT] = None + return result diff --git a/azure_monitor/tests/test_utils.py b/azure_monitor/tests/test_utils.py index e3385c2..ff171a3 100644 --- a/azure_monitor/tests/test_utils.py +++ b/azure_monitor/tests/test_utils.py @@ -1,12 +1,19 @@ # Copyright (c) Microsoft Corporation. All rights reserved. # Licensed under the MIT License. +import os import unittest from azure_monitor import utils class TestUtils(unittest.TestCase): + def setUp(self): + os.environ.clear() + self._valid_instrumentation_key = ( + "1234abcd-5678-4efa-8abc-1234567890ab" + ) + def test_nanoseconds_to_duration(self): ns_to_duration = utils.ns_to_duration self.assertEqual(ns_to_duration(0), "0.00:00:00.000") @@ -15,3 +22,243 @@ def test_nanoseconds_to_duration(self): self.assertEqual(ns_to_duration(60 * 1000000000), "0.00:01:00.000") self.assertEqual(ns_to_duration(3600 * 1000000000), "0.01:00:00.000") self.assertEqual(ns_to_duration(86400 * 1000000000), "1.00:00:00.000") + + def test_validate_instrumentation_key(self): + options = utils.Options( + instrumentation_key=self._valid_instrumentation_key + ) + self.assertEqual( + options.instrumentation_key, self._valid_instrumentation_key + ) + + def test_invalid_key_none(self): + self.assertRaises( + ValueError, lambda: utils.Options(instrumentation_key=None) + ) + + def test_invalid_key_empty(self): + self.assertRaises( + ValueError, lambda: utils.Options(instrumentation_key="") + ) + + def test_invalid_key_prefix(self): + self.assertRaises( + ValueError, + lambda: utils.Options( + instrumentation_key="test1234abcd-5678-4efa-8abc-1234567890ab" + ), + ) + + def test_invalid_key_suffix(self): + self.assertRaises( + ValueError, + lambda: utils.Options( + instrumentation_key="1234abcd-5678-4efa-8abc-1234567890abtest" + ), + ) + + def test_invalid_key_length(self): + self.assertRaises( + ValueError, + lambda: utils.Options( + instrumentation_key="1234abcd-5678-4efa-8abc-12234567890ab" + ), + ) + + def test_invalid_key_dashes(self): + self.assertRaises( + ValueError, + lambda: utils.Options( + instrumentation_key="1234abcda5678-4efa-8abc-1234567890ab" + ), + ) + + def test_invalid_key_section1_length(self): + self.assertRaises( + ValueError, + lambda: utils.Options( + instrumentation_key="1234abcda-678-4efa-8abc-1234567890ab" + ), + ) + + def test_invalid_key_section2_length(self): + self.assertRaises( + ValueError, + lambda: utils.Options( + instrumentation_key="1234abcd-678-a4efa-8abc-1234567890ab" + ), + ) + + def test_invalid_key_section3_length(self): + self.assertRaises( + ValueError, + lambda: utils.Options( + instrumentation_key="1234abcd-6789-4ef-8cabc-1234567890ab" + ), + ) + + def test_invalid_key_section4_length(self): + self.assertRaises( + ValueError, + lambda: utils.Options( + instrumentation_key="1234abcd-678-4efa-8bc-11234567890ab" + ), + ) + + def test_invalid_key_section5_length(self): + self.assertRaises( + ValueError, + lambda: utils.Options( + instrumentation_key="234abcd-678-4efa-8abc-11234567890ab" + ), + ) + + def test_invalid_key_section1_hex(self): + self.assertRaises( + ValueError, + lambda: utils.Options( + instrumentation_key="x234abcd-5678-4efa-8abc-1234567890ab" + ), + ) + + def test_invalid_key_section2_hex(self): + self.assertRaises( + ValueError, + lambda: utils.Options( + instrumentation_key="1234abcd-x678-4efa-8abc-1234567890ab" + ), + ) + + def test_invalid_key_section3_hex(self): + self.assertRaises( + ValueError, + lambda: utils.Options( + instrumentation_key="1234abcd-5678-4xfa-8abc-1234567890ab" + ), + ) + + def test_invalid_key_section4_hex(self): + self.assertRaises( + ValueError, + lambda: utils.Options( + instrumentation_key="1234abcd-5678-4xfa-8abc-1234567890ab" + ), + ) + + def test_invalid_key_section5_hex(self): + self.assertRaises( + ValueError, + lambda: utils.Options( + instrumentation_key="1234abcd-5678-4xfa-8abc-1234567890ab" + ), + ) + + def test_invalid_key_version(self): + self.assertRaises( + ValueError, + lambda: utils.Options( + instrumentation_key="1234abcd-5678-6efa-8abc-1234567890ab" + ), + ) + + def test_invalid_key_variant(self): + self.assertRaises( + ValueError, + lambda: utils.Options( + instrumentation_key="1234abcd-5678-4efa-2abc-1234567890ab" + ), + ) + + def test_process_options_ikey_code_cs(self): + os.environ[ + "APPLICATIONINSIGHTS_CONNECTION_STRING" + ] = "Authorization=ikey;InstrumentationKey=789" + os.environ["APPINSIGHTS_INSTRUMENTATIONKEY"] = "101112" + options = utils.Options( + connection_string="Authorization=ikey;InstrumentationKey=" + + self._valid_instrumentation_key, + instrumentation_key="456", + ) + self.assertEqual( + options.instrumentation_key, self._valid_instrumentation_key + ) + + def test_process_options_ikey_code_ikey(self): + os.environ[ + "APPLICATIONINSIGHTS_CONNECTION_STRING" + ] = "Authorization=ikey;InstrumentationKey=789" + os.environ["APPINSIGHTS_INSTRUMENTATIONKEY"] = "101112" + options = utils.Options( + connection_string=None, + instrumentation_key=self._valid_instrumentation_key, + ) + self.assertEqual( + options.instrumentation_key, self._valid_instrumentation_key + ) + + def test_process_options_ikey_env_cs(self): + os.environ["APPLICATIONINSIGHTS_CONNECTION_STRING"] = ( + "Authorization=ikey;InstrumentationKey=" + + self._valid_instrumentation_key + ) + os.environ["APPINSIGHTS_INSTRUMENTATIONKEY"] = "101112" + options = utils.Options( + connection_string=None, instrumentation_key=None + ) + self.assertEqual( + options.instrumentation_key, self._valid_instrumentation_key + ) + + def test_process_options_ikey_env_ikey(self): + os.environ[ + "APPINSIGHTS_INSTRUMENTATIONKEY" + ] = self._valid_instrumentation_key + options = utils.Options( + connection_string=None, instrumentation_key=None + ) + self.assertEqual( + options.instrumentation_key, self._valid_instrumentation_key + ) + + def test_process_options_endpoint_code_cs(self): + os.environ[ + "APPLICATIONINSIGHTS_CONNECTION_STRING" + ] = "Authorization=ikey;IngestionEndpoint=456;InstrumentationKey=" + options = utils.Options( + connection_string="Authorization=ikey;IngestionEndpoint=123", + instrumentation_key=self._valid_instrumentation_key, + ) + self.assertEqual(options.endpoint, "123/v2/track") + + def test_process_options_endpoint_env_cs(self): + os.environ[ + "APPLICATIONINSIGHTS_CONNECTION_STRING" + ] = "Authorization=ikey;IngestionEndpoint=456" + options = utils.Options( + connection_string=None, + instrumentation_key=self._valid_instrumentation_key, + ) + self.assertEqual(options.endpoint, "456/v2/track") + + def test_process_options_endpoint_default(self): + options = utils.Options( + connection_string=None, + instrumentation_key=self._valid_instrumentation_key, + ) + self.assertEqual( + options.endpoint, "https://dc.services.visualstudio.com/v2/track" + ) + + def test_parse_connection_string_invalid(self): + self.assertRaises( + ValueError, lambda: utils.Options(connection_string="asd") + ) + + def test_parse_connection_string_invalid_auth(self): + self.assertRaises( + ValueError, + lambda: utils.Options( + connection_string="Authorization=asd", + instrumentation_key=self._valid_instrumentation_key, + ), + ) diff --git a/azure_monitor/tests/trace/test_trace.py b/azure_monitor/tests/trace/test_trace.py index 9131da1..29e14bb 100644 --- a/azure_monitor/tests/trace/test_trace.py +++ b/azure_monitor/tests/trace/test_trace.py @@ -2,6 +2,7 @@ # Licensed under the MIT License. import json +import os import unittest # pylint: disable=import-error @@ -15,6 +16,9 @@ # pylint: disable=import-error class TestAzureExporter(unittest.TestCase): + def setUp(self): + os.environ.clear() + def test_ctor(self): # pylint: disable=W0212 instrumentation_key = Options._default.instrumentation_key From 8f808f0f29b94139d7e08c361acf933d12755c1d Mon Sep 17 00:00:00 2001 From: Victor Lima Date: Mon, 24 Feb 2020 20:59:00 -0300 Subject: [PATCH 037/109] Converting envelope keys to match Microsoft Application Insights API request body --- azure_monitor/src/azure_monitor/trace.py | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/azure_monitor/src/azure_monitor/trace.py b/azure_monitor/src/azure_monitor/trace.py index 9b4f7a7..fdeeb20 100644 --- a/azure_monitor/src/azure_monitor/trace.py +++ b/azure_monitor/src/azure_monitor/trace.py @@ -23,7 +23,7 @@ def __init__(self, options: "utils.Options"): raise ValueError("The instrumentation_key is not provided.") def export(self, spans): - envelopes = tuple(map(self.span_to_envelope, spans)) + envelopes = self.span_to_envelopes(spans) try: response = requests.post( @@ -65,6 +65,13 @@ def export(self, spans): return SpanExportResult.FAILED_NOT_RETRYABLE + def span_to_envelopes(self, spans): + envelopes = [] + for span in spans: + envelopes.append(self.span_to_envelope(span).to_dict()) + + return envelopes + def span_to_envelope(self, span): # noqa pylint: disable=too-many-branches # pylint: disable=too-many-statements envelope = protocol.Envelope( From 447713200726b74474f9f1040464989bfd1d5a53 Mon Sep 17 00:00:00 2001 From: Victor Lima Date: Mon, 24 Feb 2020 21:01:37 -0300 Subject: [PATCH 038/109] Changed method name in trace --- azure_monitor/src/azure_monitor/trace.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/azure_monitor/src/azure_monitor/trace.py b/azure_monitor/src/azure_monitor/trace.py index fdeeb20..fb1a590 100644 --- a/azure_monitor/src/azure_monitor/trace.py +++ b/azure_monitor/src/azure_monitor/trace.py @@ -23,7 +23,7 @@ def __init__(self, options: "utils.Options"): raise ValueError("The instrumentation_key is not provided.") def export(self, spans): - envelopes = self.span_to_envelopes(spans) + envelopes = self.spans_to_envelopes(spans) try: response = requests.post( @@ -65,7 +65,7 @@ def export(self, spans): return SpanExportResult.FAILED_NOT_RETRYABLE - def span_to_envelopes(self, spans): + def spans_to_envelopes(self, spans): envelopes = [] for span in spans: envelopes.append(self.span_to_envelope(span).to_dict()) From 3090b98af39ef140d4058c6c575c1d216d0c9aea Mon Sep 17 00:00:00 2001 From: Hector Hernandez Guzman Date: Tue, 25 Feb 2020 11:52:09 -0800 Subject: [PATCH 039/109] Adding telemetry processors --- azure_monitor/src/azure_monitor/exporter.py | 45 ++++++++++++ azure_monitor/src/azure_monitor/trace.py | 81 +++++++++++---------- azure_monitor/tests/test_base_exporter.py | 81 +++++++++++++++++++++ 3 files changed, 170 insertions(+), 37 deletions(-) create mode 100644 azure_monitor/src/azure_monitor/exporter.py create mode 100644 azure_monitor/tests/test_base_exporter.py diff --git a/azure_monitor/src/azure_monitor/exporter.py b/azure_monitor/src/azure_monitor/exporter.py new file mode 100644 index 0000000..2c77ef2 --- /dev/null +++ b/azure_monitor/src/azure_monitor/exporter.py @@ -0,0 +1,45 @@ +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. + +import logging + +logger = logging.getLogger(__name__) + +class BaseExporter(object): + def __init__(self): + self._telemetry_processors = [] + + def add_telemetry_processor(self, processor): + """Adds telemetry processor to the collection. Telemetry processors + will be called one by one before telemetry item is pushed for sending + and in the order they were added. + :param processor: The processor to add. + """ + self._telemetry_processors.append(processor) + + def clear_telemetry_processors(self): + """Removes all telemetry processors""" + self._telemetry_processors = [] + + def apply_telemetry_processors(self, envelopes): + """Applies all telemetry processors in the order they were added. + This function will return the list of envelopes to be exported after + each processor has been run sequentially. Individual processors can + throw exceptions and fail, but the applying of all telemetry processors + will proceed (not fast fail). Processors also return True if envelope + should be included for exporting, False otherwise. + :param envelopes: The envelopes to apply each processor to. + """ + filtered_envelopes = [] + for envelope in envelopes: + accepted = True + for processor in self._telemetry_processors: + try: + if processor(envelope) is False: + accepted = False + break + except Exception as ex: + logger.warning("Telemetry processor failed with: %s.", ex) + if accepted: + filtered_envelopes.append(envelope) + return filtered_envelopes diff --git a/azure_monitor/src/azure_monitor/trace.py b/azure_monitor/src/azure_monitor/trace.py index f72320e..c570e27 100644 --- a/azure_monitor/src/azure_monitor/trace.py +++ b/azure_monitor/src/azure_monitor/trace.py @@ -12,58 +12,65 @@ from opentelemetry.trace.status import StatusCanonicalCode from azure_monitor import protocol, utils +from azure_monitor.exporter import BaseExporter logger = logging.getLogger(__name__) -class AzureMonitorSpanExporter(SpanExporter): +class AzureMonitorSpanExporter(SpanExporter, BaseExporter): def __init__(self, **options): self.options = utils.Options(**options) + super(AzureMonitorSpanExporter, self).__init__() if not self.options.instrumentation_key: raise ValueError("The instrumentation_key is not provided.") def export(self, spans): - envelopes = tuple(map(self.span_to_envelope, spans)) - - try: - response = requests.post( - url=self.options.endpoint, - data=json.dumps(envelopes), - headers={ - "Accept": "application/json", - "Content-Type": "application/json; charset=utf-8", - }, - timeout=self.options.timeout, - ) - except requests.RequestException as ex: - logger.warning("Transient client side error %s.", ex) - return SpanExportResult.FAILED_RETRYABLE + envelopes = map(self.span_to_envelope, spans) + envelopes_to_export = tuple(self.apply_telemetry_processors(envelopes)) + if len(envelopes_to_export) > 1: + try: + response = requests.post( + url=self.options.endpoint, + data=json.dumps(envelopes_to_export), + headers={ + "Accept": "application/json", + "Content-Type": "application/json; charset=utf-8", + }, + timeout=self.options.timeout, + ) + except requests.RequestException as ex: + logger.warning("Transient client side error %s.", ex) + return SpanExportResult.FAILED_RETRYABLE - text = "N/A" - data = None # noqa pylint: disable=unused-variable - try: - text = response.text - except Exception as ex: # noqa pylint: disable=broad-except - logger.warning("Error while reading response body %s.", ex) - else: + text = "N/A" + data = None # noqa pylint: disable=unused-variable try: - data = json.loads(text) # noqa pylint: disable=unused-variable - except Exception: # noqa pylint: disable=broad-except - pass + text = response.text + except Exception as ex: # noqa pylint: disable=broad-except + logger.warning("Error while reading response body %s.", ex) + else: + try: + data = json.loads( + text + ) # noqa pylint: disable=unused-variable + except Exception: # noqa pylint: disable=broad-except + pass - if response.status_code == 200: - logger.info("Transmission succeeded: %s.", text) - return SpanExportResult.SUCCESS + if response.status_code == 200: + logger.info("Transmission succeeded: %s.", text) + return SpanExportResult.SUCCESS - if response.status_code in ( - 206, # Partial Content - 429, # Too Many Requests - 500, # Internal Server Error - 503, # Service Unavailable - ): - return SpanExportResult.FAILED_RETRYABLE + if response.status_code in ( + 206, # Partial Content + 429, # Too Many Requests + 500, # Internal Server Error + 503, # Service Unavailable + ): + return SpanExportResult.FAILED_RETRYABLE - return SpanExportResult.FAILED_NOT_RETRYABLE + return SpanExportResult.FAILED_NOT_RETRYABLE + # No spans to export + return SpanExportResult.SUCCESS def span_to_envelope(self, span): # noqa pylint: disable=too-many-branches # pylint: disable=too-many-statements diff --git a/azure_monitor/tests/test_base_exporter.py b/azure_monitor/tests/test_base_exporter.py new file mode 100644 index 0000000..cb880b7 --- /dev/null +++ b/azure_monitor/tests/test_base_exporter.py @@ -0,0 +1,81 @@ +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. + +import unittest + +from azure_monitor.exporter import BaseExporter +from azure_monitor.protocol import Envelope + +# pylint: disable=W0212 +class TestBaseExporter(unittest.TestCase): + def test_telemetry_processor_add(self): + base = BaseExporter() + base.add_telemetry_processor(lambda: True) + self.assertEqual(len(base._telemetry_processors), 1) + + def test_telemetry_processor_clear(self): + base = BaseExporter() + base.add_telemetry_processor(lambda: True) + self.assertEqual(len(base._telemetry_processors), 1) + base.clear_telemetry_processors() + self.assertEqual(len(base._telemetry_processors), 0) + + def test_telemetry_processor_apply(self): + base = BaseExporter() + + def callback_function(envelope): + envelope.baseType += "_world" + + base.add_telemetry_processor(callback_function) + envelope = Envelope() + envelope.baseType = "type1" + base.apply_telemetry_processors([envelope]) + self.assertEqual(envelope.baseType, "type1_world") + + def test_telemetry_processor_apply_multiple(self): + base = BaseExporter() + base._telemetry_processors = [] + + def callback_function(envelope): + envelope.baseType += "_world" + + def callback_function2(envelope): + envelope.baseType += "_world2" + + base.add_telemetry_processor(callback_function) + base.add_telemetry_processor(callback_function2) + envelope = Envelope() + envelope.baseType = "type1" + base.apply_telemetry_processors([envelope]) + self.assertEqual(envelope.baseType, "type1_world_world2") + + def test_telemetry_processor_apply_exception(self): + base = BaseExporter() + + def callback_function(envelope): + raise ValueError() + + def callback_function2(envelope): + envelope.baseType += "_world2" + + base.add_telemetry_processor(callback_function) + base.add_telemetry_processor(callback_function2) + envelope = Envelope() + envelope.baseType = "type1" + base.apply_telemetry_processors([envelope]) + self.assertEqual(envelope.baseType, "type1_world2") + + def test_telemetry_processor_apply_not_accepted(self): + base = BaseExporter() + + def callback_function(envelope): + return envelope.baseType == "type2" + + base.add_telemetry_processor(callback_function) + envelope = Envelope() + envelope.baseType = "type1" + envelope2 = Envelope() + envelope2.baseType = "type2" + envelopes = base.apply_telemetry_processors([envelope, envelope2]) + self.assertEqual(len(envelopes), 1) + self.assertEqual(envelopes[0].baseType, "type2") From 3c754a1ddfaef117fe124e0cf2e7dc4dfe42f55d Mon Sep 17 00:00:00 2001 From: Hector Hernandez Guzman Date: Wed, 26 Feb 2020 11:36:25 -0800 Subject: [PATCH 040/109] Addressing comments --- azure_monitor/src/azure_monitor/exporter.py | 16 +++++++++++----- azure_monitor/src/azure_monitor/trace.py | 7 +++---- 2 files changed, 14 insertions(+), 9 deletions(-) diff --git a/azure_monitor/src/azure_monitor/exporter.py b/azure_monitor/src/azure_monitor/exporter.py index 2c77ef2..9a2af59 100644 --- a/azure_monitor/src/azure_monitor/exporter.py +++ b/azure_monitor/src/azure_monitor/exporter.py @@ -3,16 +3,21 @@ import logging +from azure_monitor import utils + logger = logging.getLogger(__name__) -class BaseExporter(object): - def __init__(self): + +class BaseExporter: + def __init__(self, **options): self._telemetry_processors = [] + self.options = utils.Options(**options) def add_telemetry_processor(self, processor): - """Adds telemetry processor to the collection. Telemetry processors - will be called one by one before telemetry item is pushed for sending - and in the order they were added. + """Adds telemetry processor to the collection. + + Telemetry processors will be called one by one before telemetry + item is pushed for sending and in the order they were added. :param processor: The processor to add. """ self._telemetry_processors.append(processor) @@ -23,6 +28,7 @@ def clear_telemetry_processors(self): def apply_telemetry_processors(self, envelopes): """Applies all telemetry processors in the order they were added. + This function will return the list of envelopes to be exported after each processor has been run sequentially. Individual processors can throw exceptions and fail, but the applying of all telemetry processors diff --git a/azure_monitor/src/azure_monitor/trace.py b/azure_monitor/src/azure_monitor/trace.py index c570e27..ff3cf9d 100644 --- a/azure_monitor/src/azure_monitor/trace.py +++ b/azure_monitor/src/azure_monitor/trace.py @@ -17,17 +17,16 @@ logger = logging.getLogger(__name__) -class AzureMonitorSpanExporter(SpanExporter, BaseExporter): +class AzureMonitorSpanExporter(BaseExporter, SpanExporter): def __init__(self, **options): - self.options = utils.Options(**options) - super(AzureMonitorSpanExporter, self).__init__() + super(AzureMonitorSpanExporter, self).__init__(**options) if not self.options.instrumentation_key: raise ValueError("The instrumentation_key is not provided.") def export(self, spans): envelopes = map(self.span_to_envelope, spans) envelopes_to_export = tuple(self.apply_telemetry_processors(envelopes)) - if len(envelopes_to_export) > 1: + if envelopes_to_export: try: response = requests.post( url=self.options.endpoint, From de8894269182c3c9678b2fb5388c3330c53a5bf7 Mon Sep 17 00:00:00 2001 From: Hector Hernandez Guzman Date: Wed, 26 Feb 2020 11:39:58 -0800 Subject: [PATCH 041/109] Addressing comments --- azure_monitor/examples/traces/README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/azure_monitor/examples/traces/README.md b/azure_monitor/examples/traces/README.md index 8ae4ec8..df64170 100644 --- a/azure_monitor/examples/traces/README.md +++ b/azure_monitor/examples/traces/README.md @@ -9,7 +9,7 @@ $ pip install opentelemetry-azure-monitor-exporter ### Trace -* Update the code in trace.py to use your INSTRUMENTATION_KEY +* Update the code in trace.py to use your `INSTRUMENTATION_KEY` * Run the sample @@ -20,7 +20,7 @@ $ python trace.py ### Request -* Update the code in request.py to use your INSTRUMENTATION_KEY +* Update the code in request.py to use your `INSTRUMENTATION_KEY` * Run the sample @@ -32,7 +32,7 @@ $ python request.py ### Server -* Update the code in server.py to use your INSTRUMENTATION_KEY +* Update the code in server.py to use your `INSTRUMENTATION_KEY` * Run the sample From 3f1e09cbac0b311efd9ad5b9ce0e931b2458ccf6 Mon Sep 17 00:00:00 2001 From: Victor Lima Date: Wed, 26 Feb 2020 18:26:28 -0300 Subject: [PATCH 042/109] Fixed trace canonical code reference --- azure_monitor/src/azure_monitor/trace.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/azure_monitor/src/azure_monitor/trace.py b/azure_monitor/src/azure_monitor/trace.py index 211c5ab..d15c817 100644 --- a/azure_monitor/src/azure_monitor/trace.py +++ b/azure_monitor/src/azure_monitor/trace.py @@ -121,7 +121,7 @@ def span_to_envelope(self, span): # noqa pylint: disable=too-many-branches status_code = span.attributes["http.status_code"] data.response_code = str(status_code) data.success = 200 <= status_code < 400 - elif span.status == StatusCanonicalCode.OK: + elif span.status.canonical_code == StatusCanonicalCode.OK: data.success = True else: envelope.name = "Microsoft.ApplicationInsights.RemoteDependency" @@ -161,7 +161,7 @@ def span_to_envelope(self, span): # noqa pylint: disable=too-many-branches status_code = span.attributes["http.status_code"] data.result_code = str(status_code) data.success = 200 <= status_code < 400 - elif span.status == StatusCanonicalCode.OK: + elif span.status.canonical_code == StatusCanonicalCode.OK: data.success = True else: # SpanKind.INTERNAL data.type = "InProc" From 38311747eb6908ccdbc1b621566cff163d8eb741 Mon Sep 17 00:00:00 2001 From: Victor Lima Date: Wed, 26 Feb 2020 18:33:34 -0300 Subject: [PATCH 043/109] Made Options a subclass of BaseObject --- azure_monitor/src/azure_monitor/utils.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/azure_monitor/src/azure_monitor/utils.py b/azure_monitor/src/azure_monitor/utils.py index 854951e..db0e2b5 100644 --- a/azure_monitor/src/azure_monitor/utils.py +++ b/azure_monitor/src/azure_monitor/utils.py @@ -8,6 +8,7 @@ from opentelemetry.sdk.version import __version__ as opentelemetry_version from azure_monitor.version import __version__ as ext_version +from .protocol import BaseObject azure_monitor_context = { "ai.cloud.role": os.path.basename(sys.argv[0]) or "Python Application", @@ -33,7 +34,7 @@ def ns_to_duration(nanoseconds): ) -class Options: +class Options(BaseObject): __slots__ = ("endpoint", "instrumentation_key", "timeout") def __init__( From 05a0ec69b684b91668e52db2bf533bedd147070d Mon Sep 17 00:00:00 2001 From: Victor Lima Date: Wed, 26 Feb 2020 18:58:21 -0300 Subject: [PATCH 044/109] Removed unnecessary status checks --- azure_monitor/src/azure_monitor/trace.py | 4 ---- 1 file changed, 4 deletions(-) diff --git a/azure_monitor/src/azure_monitor/trace.py b/azure_monitor/src/azure_monitor/trace.py index d15c817..c028f2f 100644 --- a/azure_monitor/src/azure_monitor/trace.py +++ b/azure_monitor/src/azure_monitor/trace.py @@ -121,8 +121,6 @@ def span_to_envelope(self, span): # noqa pylint: disable=too-many-branches status_code = span.attributes["http.status_code"] data.response_code = str(status_code) data.success = 200 <= status_code < 400 - elif span.status.canonical_code == StatusCanonicalCode.OK: - data.success = True else: envelope.name = "Microsoft.ApplicationInsights.RemoteDependency" data = protocol.RemoteDependency( @@ -161,8 +159,6 @@ def span_to_envelope(self, span): # noqa pylint: disable=too-many-branches status_code = span.attributes["http.status_code"] data.result_code = str(status_code) data.success = 200 <= status_code < 400 - elif span.status.canonical_code == StatusCanonicalCode.OK: - data.success = True else: # SpanKind.INTERNAL data.type = "InProc" data.success = True From 30d041775e723a3572647e296be9eb6cc492b2db Mon Sep 17 00:00:00 2001 From: Victor Lima Date: Wed, 26 Feb 2020 20:01:31 -0300 Subject: [PATCH 045/109] Removed base_type from envelope and updated tests --- azure_monitor/src/azure_monitor/protocol.py | 7 +-- azure_monitor/tests/test_base_exporter.py | 55 +++++++++++++-------- 2 files changed, 37 insertions(+), 25 deletions(-) diff --git a/azure_monitor/src/azure_monitor/protocol.py b/azure_monitor/src/azure_monitor/protocol.py index ea4f1c4..998f308 100644 --- a/azure_monitor/src/azure_monitor/protocol.py +++ b/azure_monitor/src/azure_monitor/protocol.py @@ -87,8 +87,7 @@ class Envelope(BaseObject): "ikey", "flags", "tags", - "data", - "base_type" + "data" ) def __init__( @@ -101,8 +100,7 @@ def __init__( ikey=None, flags=None, tags=None, - data=None, - base_type=None + data=None ) -> None: self.ver = ver self.name = name @@ -113,7 +111,6 @@ def __init__( self.flags = flags self.tags = tags self.data = data - self.base_type = base_type def to_dict(self): return { diff --git a/azure_monitor/tests/test_base_exporter.py b/azure_monitor/tests/test_base_exporter.py index d221e9f..d1deaec 100644 --- a/azure_monitor/tests/test_base_exporter.py +++ b/azure_monitor/tests/test_base_exporter.py @@ -4,7 +4,7 @@ import unittest from azure_monitor.exporter import BaseExporter -from azure_monitor.protocol import Envelope +from azure_monitor.protocol import Data, Envelope # pylint: disable=W0212 class TestBaseExporter(unittest.TestCase): @@ -24,30 +24,36 @@ def test_telemetry_processor_apply(self): base = BaseExporter() def callback_function(envelope): - envelope.base_type += "_world" + envelope.data.base_type += "_world" base.add_telemetry_processor(callback_function) - envelope = Envelope() - envelope.base_type = "type1" + envelope = Envelope( + data=Data( + base_type="type1" + ) + ) base.apply_telemetry_processors([envelope]) - self.assertEqual(envelope.base_type, "type1_world") + self.assertEqual(envelope.data.base_type, "type1_world") def test_telemetry_processor_apply_multiple(self): base = BaseExporter() base._telemetry_processors = [] def callback_function(envelope): - envelope.base_type += "_world" + envelope.data.base_type += "_world" def callback_function2(envelope): - envelope.base_type += "_world2" + envelope.data.base_type += "_world2" base.add_telemetry_processor(callback_function) base.add_telemetry_processor(callback_function2) - envelope = Envelope() - envelope.base_type = "type1" + envelope = Envelope( + data=Data( + base_type="type1" + ) + ) base.apply_telemetry_processors([envelope]) - self.assertEqual(envelope.base_type, "type1_world_world2") + self.assertEqual(envelope.data.base_type, "type1_world_world2") def test_telemetry_processor_apply_exception(self): base = BaseExporter() @@ -56,26 +62,35 @@ def callback_function(envelope): raise ValueError() def callback_function2(envelope): - envelope.base_type += "_world2" + envelope.data.base_type += "_world2" base.add_telemetry_processor(callback_function) base.add_telemetry_processor(callback_function2) - envelope = Envelope() - envelope.base_type = "type1" + envelope = Envelope( + data=Data( + base_type="type1" + ) + ) base.apply_telemetry_processors([envelope]) - self.assertEqual(envelope.base_type, "type1_world2") + self.assertEqual(envelope.data.base_type, "type1_world2") def test_telemetry_processor_apply_not_accepted(self): base = BaseExporter() def callback_function(envelope): - return envelope.base_type == "type2" + return envelope.data.base_type == "type2" base.add_telemetry_processor(callback_function) - envelope = Envelope() - envelope.base_type = "type1" - envelope2 = Envelope() - envelope2.base_type = "type2" + envelope = Envelope( + data=Data( + base_type="type1" + ) + ) + envelope2 = Envelope( + data=Data( + base_type="type2" + ) + ) envelopes = base.apply_telemetry_processors([envelope, envelope2]) self.assertEqual(len(envelopes), 1) - self.assertEqual(envelopes[0].base_type, "type2") + self.assertEqual(envelopes[0].data.base_type, "type2") From f00ff22836bd8f86f59d06fae84f4336e9423758 Mon Sep 17 00:00:00 2001 From: Hector Hernandez Guzman Date: Wed, 26 Feb 2020 15:13:30 -0800 Subject: [PATCH 046/109] WIP --- .gitignore | 3 + azure_monitor/examples/traces/trace.py | 4 +- azure_monitor/src/azure_monitor/utils.py | 4 +- azure_monitor/tests/trace/test_trace.py | 122 +++++++++++------------ 4 files changed, 66 insertions(+), 67 deletions(-) diff --git a/.gitignore b/.gitignore index 894a44c..a127971 100644 --- a/.gitignore +++ b/.gitignore @@ -102,3 +102,6 @@ venv.bak/ # mypy .mypy_cache/ + +# vscode +.vscode/ diff --git a/azure_monitor/examples/traces/trace.py b/azure_monitor/examples/traces/trace.py index 8f94033..ea77507 100644 --- a/azure_monitor/examples/traces/trace.py +++ b/azure_monitor/examples/traces/trace.py @@ -10,9 +10,7 @@ trace.set_preferred_tracer_source_implementation(lambda T: TracerSource()) span_processor = SimpleExportSpanProcessor( - AzureMonitorSpanExporter( - instrumentation_key="" - ) + AzureMonitorSpanExporter(instrumentation_key="") ) trace.tracer_source().add_span_processor(span_processor) tracer = trace.tracer_source().get_tracer(__name__) diff --git a/azure_monitor/src/azure_monitor/utils.py b/azure_monitor/src/azure_monitor/utils.py index 3ddfc46..8a6bcac 100644 --- a/azure_monitor/src/azure_monitor/utils.py +++ b/azure_monitor/src/azure_monitor/utils.py @@ -131,8 +131,8 @@ def _parse_connection_string(self, connection_string): prefix = result.get("location") if prefix is not None: location_prefix = prefix + "." - endpoint = ( - "https://" + location_prefix + "dc." + endpoint_suffix + endpoint = "https://{0}dc.{1}".format( + location_prefix, endpoint_suffix ) result[INGESTION_ENDPOINT] = endpoint else: diff --git a/azure_monitor/tests/trace/test_trace.py b/azure_monitor/tests/trace/test_trace.py index 29e14bb..84928d7 100644 --- a/azure_monitor/tests/trace/test_trace.py +++ b/azure_monitor/tests/trace/test_trace.py @@ -65,9 +65,9 @@ def test_span_to_envelope(self): links=None, kind=SpanKind.CLIENT, ) - span.status = StatusCanonicalCode.OK - span.start_time = start_time - span.end_time = end_time + span.set_status(StatusCanonicalCode.OK) + span.start(start_time=start_time) + span.end(end_time=end_time) envelope = exporter.span_to_envelope(span) self.assertEqual(envelope.iKey, "12345678-1234-5678-abcd-12345678abcd") self.assertEqual( @@ -109,9 +109,9 @@ def test_span_to_envelope(self): links=None, kind=SpanKind.CLIENT, ) - span.status = StatusCanonicalCode.OK - span.start_time = start_time - span.end_time = end_time + span.set_status(StatusCanonicalCode.OK) + span.start(start_time=start_time) + span.end(end_time=end_time) envelope = exporter.span_to_envelope(span) self.assertEqual(envelope.iKey, "12345678-1234-5678-abcd-12345678abcd") self.assertEqual( @@ -151,9 +151,9 @@ def test_span_to_envelope(self): links=None, kind=SpanKind.CLIENT, ) - span.status = StatusCanonicalCode.OK - span.start_time = start_time - span.end_time = end_time + span.set_status(StatusCanonicalCode.OK) + span.start(start_time=start_time) + span.end(end_time=end_time) envelope = exporter.span_to_envelope(span) self.assertEqual(envelope.iKey, "12345678-1234-5678-abcd-12345678abcd") self.assertEqual( @@ -202,9 +202,9 @@ def test_span_to_envelope(self): links=None, kind=SpanKind.SERVER, ) - span.status = StatusCanonicalCode.OK - span.start_time = start_time - span.end_time = end_time + span.set_status(StatusCanonicalCode.OK) + span.start(start_time=start_time) + span.end(end_time=end_time) envelope = exporter.span_to_envelope(span) self.assertEqual(envelope.iKey, "12345678-1234-5678-abcd-12345678abcd") self.assertEqual( @@ -254,9 +254,9 @@ def test_span_to_envelope(self): links=None, kind=SpanKind.SERVER, ) - span.status = StatusCanonicalCode.OK - span.start_time = start_time - span.end_time = end_time + span.set_status(StatusCanonicalCode.OK) + span.start(start_time=start_time) + span.end(end_time=end_time) envelope = exporter.span_to_envelope(span) self.assertEqual(envelope.iKey, "12345678-1234-5678-abcd-12345678abcd") self.assertEqual( @@ -306,9 +306,9 @@ def test_span_to_envelope(self): links=None, kind=SpanKind.SERVER, ) - span.status = StatusCanonicalCode.OK - span.start_time = start_time - span.end_time = end_time + span.set_status(StatusCanonicalCode.OK) + span.start(start_time=start_time) + span.end(end_time=end_time) envelope = exporter.span_to_envelope(span) self.assertEqual(envelope.iKey, "12345678-1234-5678-abcd-12345678abcd") self.assertEqual( @@ -342,9 +342,9 @@ def test_span_to_envelope(self): links=None, kind=SpanKind.INTERNAL, ) - span.status = StatusCanonicalCode.OK - span.start_time = start_time - span.end_time = end_time + span.set_status(StatusCanonicalCode.OK) + span.start(start_time=start_time) + span.end(end_time=end_time) envelope = exporter.span_to_envelope(span) self.assertEqual(envelope.iKey, "12345678-1234-5678-abcd-12345678abcd") self.assertEqual( @@ -387,9 +387,9 @@ def test_span_to_envelope(self): links=None, kind=SpanKind.CLIENT, ) - span.status = StatusCanonicalCode.OK - span.start_time = start_time - span.end_time = end_time + span.set_status(StatusCanonicalCode.OK) + span.start(start_time=start_time) + span.end(end_time=end_time) envelope = exporter.span_to_envelope(span) self.assertEqual(len(envelope.data.baseData.properties), 2) self.assertEqual( @@ -427,9 +427,9 @@ def test_span_to_envelope(self): links=links, kind=SpanKind.CLIENT, ) - span.status = StatusCanonicalCode.OK - span.start_time = start_time - span.end_time = end_time + span.set_status(StatusCanonicalCode.OK) + span.start(start_time=start_time) + span.end(end_time=end_time) envelope = exporter.span_to_envelope(span) self.assertEqual(len(envelope.data.baseData.properties), 2) json_dict = json.loads(envelope.data.baseData.properties["_MS.links"])[ @@ -458,9 +458,9 @@ def test_span_to_envelope(self): links=None, kind=SpanKind.SERVER, ) - span.status = StatusCanonicalCode.OK - span.start_time = start_time - span.end_time = end_time + span.set_status(StatusCanonicalCode.OK) + span.start(start_time=start_time) + span.end(end_time=end_time) envelope = exporter.span_to_envelope(span) self.assertEqual(envelope.data.baseData.responseCode, "500") self.assertFalse(envelope.data.baseData.success) @@ -485,9 +485,9 @@ def test_span_to_envelope(self): links=None, kind=SpanKind.CLIENT, ) - span.status = StatusCanonicalCode.OK - span.start_time = start_time - span.end_time = end_time + span.set_status(StatusCanonicalCode.OK) + span.start(start_time=start_time) + span.end(end_time=end_time) envelope = exporter.span_to_envelope(span) self.assertEqual(envelope.data.baseData.resultCode, "500") self.assertFalse(envelope.data.baseData.success) @@ -511,10 +511,9 @@ def test_span_to_envelope(self): links=None, kind=SpanKind.SERVER, ) - span.status = StatusCanonicalCode.OK - span.start_time = start_time - span.end_time = end_time - span.status = StatusCanonicalCode.OK + span.set_status(StatusCanonicalCode.OK) + span.start(start_time=start_time) + span.end(end_time=end_time) envelope = exporter.span_to_envelope(span) self.assertEqual(envelope.data.baseData.responseCode, "0") self.assertTrue(envelope.data.baseData.success) @@ -538,10 +537,9 @@ def test_span_to_envelope(self): links=None, kind=SpanKind.CLIENT, ) - span.status = StatusCanonicalCode.OK - span.start_time = start_time - span.end_time = end_time - span.status = StatusCanonicalCode.OK + span.set_status(StatusCanonicalCode.OK) + span.start(start_time=start_time) + span.end(end_time=end_time) envelope = exporter.span_to_envelope(span) self.assertEqual(envelope.data.baseData.resultCode, "0") self.assertTrue(envelope.data.baseData.success) @@ -565,9 +563,9 @@ def test_span_to_envelope(self): links=None, kind=SpanKind.SERVER, ) - span.start_time = start_time - span.end_time = end_time - span.status = StatusCanonicalCode.UNKNOWN + span.set_status(StatusCanonicalCode.UNKNOWN) + span.start(start_time=start_time) + span.end(end_time=end_time) envelope = exporter.span_to_envelope(span) self.assertEqual(envelope.data.baseData.responseCode, "2") self.assertFalse(envelope.data.baseData.success) @@ -591,9 +589,9 @@ def test_span_to_envelope(self): links=None, kind=SpanKind.CLIENT, ) - span.start_time = start_time - span.end_time = end_time - span.status = StatusCanonicalCode.UNKNOWN + span.set_status(StatusCanonicalCode.UNKNOWN) + span.start(start_time=start_time) + span.end(end_time=end_time) envelope = exporter.span_to_envelope(span) self.assertEqual(envelope.data.baseData.resultCode, "2") self.assertFalse(envelope.data.baseData.success) @@ -621,9 +619,9 @@ def test_span_to_envelope(self): links=None, kind=SpanKind.SERVER, ) - span.start_time = start_time - span.end_time = end_time - span.status = StatusCanonicalCode.OK + span.set_status(StatusCanonicalCode.OK) + span.start(start_time=start_time) + span.end(end_time=end_time) envelope = exporter.span_to_envelope(span) self.assertEqual( envelope.data.baseData.properties["request.name"], @@ -655,9 +653,9 @@ def test_span_to_envelope(self): links=None, kind=SpanKind.SERVER, ) - span.start_time = start_time - span.end_time = end_time - span.status = StatusCanonicalCode.OK + span.set_status(StatusCanonicalCode.OK) + span.start(start_time=start_time) + span.end(end_time=end_time) envelope = exporter.span_to_envelope(span) self.assertIsNone(envelope.data.baseData.name) @@ -683,9 +681,9 @@ def test_span_to_envelope(self): links=None, kind=SpanKind.SERVER, ) - span.start_time = start_time - span.end_time = end_time - span.status = StatusCanonicalCode.OK + span.set_status(StatusCanonicalCode.OK) + span.start(start_time=start_time) + span.end(end_time=end_time) envelope = exporter.span_to_envelope(span) self.assertEqual(envelope.data.baseData.name, "GET") self.assertEqual( @@ -718,9 +716,9 @@ def test_span_to_envelope(self): links=None, kind=SpanKind.SERVER, ) - span.start_time = start_time - span.end_time = end_time - span.status = StatusCanonicalCode.OK + span.set_status(StatusCanonicalCode.OK) + span.start(start_time=start_time) + span.end(end_time=end_time) envelope = exporter.span_to_envelope(span) self.assertIsNone( envelope.data.baseData.properties.get("request.name") @@ -752,9 +750,9 @@ def test_span_to_envelope(self): links=None, kind=SpanKind.SERVER, ) - span.start_time = start_time - span.end_time = end_time - span.status = StatusCanonicalCode.OK + span.set_status(StatusCanonicalCode.OK) + span.start(start_time=start_time) + span.end(end_time=end_time) envelope = exporter.span_to_envelope(span) self.assertIsNone(envelope.data.baseData.url) self.assertIsNone(envelope.data.baseData.properties.get("request.url")) From 279ebda31ad47f9f798e50fe1aee59a2e2ca5b1a Mon Sep 17 00:00:00 2001 From: Hector Hernandez Guzman Date: Wed, 26 Feb 2020 16:04:37 -0800 Subject: [PATCH 047/109] Fixing lint and tox errors --- azure_monitor/src/azure_monitor/exporter.py | 4 +- azure_monitor/src/azure_monitor/protocol.py | 4 +- azure_monitor/src/azure_monitor/trace.py | 6 +- azure_monitor/src/azure_monitor/utils.py | 19 ++-- azure_monitor/tests/test_base_exporter.py | 35 ++----- azure_monitor/tests/test_protocol.py | 2 +- azure_monitor/tests/trace/test_trace.py | 105 +++++++++++++------- 7 files changed, 100 insertions(+), 75 deletions(-) diff --git a/azure_monitor/src/azure_monitor/exporter.py b/azure_monitor/src/azure_monitor/exporter.py index 9a2af59..3fe5b3f 100644 --- a/azure_monitor/src/azure_monitor/exporter.py +++ b/azure_monitor/src/azure_monitor/exporter.py @@ -14,8 +14,8 @@ def __init__(self, **options): self.options = utils.Options(**options) def add_telemetry_processor(self, processor): - """Adds telemetry processor to the collection. - + """Adds telemetry processor to the collection. + Telemetry processors will be called one by one before telemetry item is pushed for sending and in the order they were added. :param processor: The processor to add. diff --git a/azure_monitor/src/azure_monitor/protocol.py b/azure_monitor/src/azure_monitor/protocol.py index 998f308..46c0834 100644 --- a/azure_monitor/src/azure_monitor/protocol.py +++ b/azure_monitor/src/azure_monitor/protocol.py @@ -87,7 +87,7 @@ class Envelope(BaseObject): "ikey", "flags", "tags", - "data" + "data", ) def __init__( @@ -100,7 +100,7 @@ def __init__( ikey=None, flags=None, tags=None, - data=None + data=None, ) -> None: self.ver = ver self.name = name diff --git a/azure_monitor/src/azure_monitor/trace.py b/azure_monitor/src/azure_monitor/trace.py index 62e36a6..5dc630e 100644 --- a/azure_monitor/src/azure_monitor/trace.py +++ b/azure_monitor/src/azure_monitor/trace.py @@ -96,7 +96,8 @@ def span_to_envelope(self, span): # noqa pylint: disable=too-many-branches id="{:016x}".format(span.context.span_id), duration=utils.ns_to_duration(span.end_time - span.start_time), response_code=str(span.status.canonical_code.value), - success=span.status.canonical_code == StatusCanonicalCode.OK, # Modify based off attributes or Status + success=span.status.canonical_code + == StatusCanonicalCode.OK, # Modify based off attributes or Status properties={}, ) envelope.data = protocol.Data( @@ -126,7 +127,8 @@ def span_to_envelope(self, span): # noqa pylint: disable=too-many-branches id="{:016x}".format(span.context.span_id), result_code=str(span.status.canonical_code.value), duration=utils.ns_to_duration(span.end_time - span.start_time), - success=span.status.canonical_code == StatusCanonicalCode.OK, # Modify based off attributes or Status + success=span.status.canonical_code + == StatusCanonicalCode.OK, # Modify based off attributes or Status properties={}, ) envelope.data = protocol.Data( diff --git a/azure_monitor/src/azure_monitor/utils.py b/azure_monitor/src/azure_monitor/utils.py index 676d053..2672596 100644 --- a/azure_monitor/src/azure_monitor/utils.py +++ b/azure_monitor/src/azure_monitor/utils.py @@ -6,11 +6,10 @@ import re import sys - from opentelemetry.sdk.version import __version__ as opentelemetry_version -from azure_monitor.version import __version__ as ext_version -from .protocol import BaseObject +from azure_monitor.protocol import BaseObject +from azure_monitor.version import __version__ as ext_version azure_monitor_context = { "ai.cloud.role": os.path.basename(sys.argv[0]) or "Python Application", @@ -51,8 +50,13 @@ def ns_to_duration(nanoseconds): class Options(BaseObject): - - __slots__ = ("connection_string","endpoint", "instrumentation_key", "timeout") + + __slots__ = ( + "connection_string", + "endpoint", + "instrumentation_key", + "timeout", + ) def __init__( self, @@ -67,7 +71,7 @@ def __init__( self.timeout = timeout self._initialize() self._validate_instrumentation_key() - + def _initialize(self): code_cs = self._parse_connection_string(self.connection_string) code_ikey = self.instrumentation_key @@ -97,7 +101,7 @@ def _initialize(self): or "https://dc.services.visualstudio.com" ) self.endpoint = endpoint + "/v2/track" - + def _validate_instrumentation_key(self): """Validates the instrumentation key used for Azure Monitor. An instrumentation key cannot be null or empty. An instrumentation key @@ -143,4 +147,3 @@ def _parse_connection_string(self, connection_string): # Default to None if cannot construct result[INGESTION_ENDPOINT] = None return result - diff --git a/azure_monitor/tests/test_base_exporter.py b/azure_monitor/tests/test_base_exporter.py index 354a5c2..617b777 100644 --- a/azure_monitor/tests/test_base_exporter.py +++ b/azure_monitor/tests/test_base_exporter.py @@ -7,11 +7,14 @@ from azure_monitor.exporter import BaseExporter from azure_monitor.protocol import Data, Envelope + # pylint: disable=W0212 class TestBaseExporter(unittest.TestCase): @classmethod def setUpClass(cls): - os.environ["APPINSIGHTS_INSTRUMENTATIONKEY"] = "1234abcd-5678-4efa-8abc-1234567890ab" + os.environ[ + "APPINSIGHTS_INSTRUMENTATIONKEY" + ] = "1234abcd-5678-4efa-8abc-1234567890ab" def test_telemetry_processor_add(self): base = BaseExporter() @@ -32,11 +35,7 @@ def callback_function(envelope): envelope.data.base_type += "_world" base.add_telemetry_processor(callback_function) - envelope = Envelope( - data=Data( - base_type="type1" - ) - ) + envelope = Envelope(data=Data(base_type="type1")) base.apply_telemetry_processors([envelope]) self.assertEqual(envelope.data.base_type, "type1_world") @@ -52,11 +51,7 @@ def callback_function2(envelope): base.add_telemetry_processor(callback_function) base.add_telemetry_processor(callback_function2) - envelope = Envelope( - data=Data( - base_type="type1" - ) - ) + envelope = Envelope(data=Data(base_type="type1")) base.apply_telemetry_processors([envelope]) self.assertEqual(envelope.data.base_type, "type1_world_world2") @@ -71,11 +66,7 @@ def callback_function2(envelope): base.add_telemetry_processor(callback_function) base.add_telemetry_processor(callback_function2) - envelope = Envelope( - data=Data( - base_type="type1" - ) - ) + envelope = Envelope(data=Data(base_type="type1")) base.apply_telemetry_processors([envelope]) self.assertEqual(envelope.data.base_type, "type1_world2") @@ -86,16 +77,8 @@ def callback_function(envelope): return envelope.data.base_type == "type2" base.add_telemetry_processor(callback_function) - envelope = Envelope( - data=Data( - base_type="type1" - ) - ) - envelope2 = Envelope( - data=Data( - base_type="type2" - ) - ) + envelope = Envelope(data=Data(base_type="type1")) + envelope2 = Envelope(data=Data(base_type="type2")) envelopes = base.apply_telemetry_processors([envelope, envelope2]) self.assertEqual(len(envelopes), 1) self.assertEqual(envelopes[0].data.base_type, "type2") diff --git a/azure_monitor/tests/test_protocol.py b/azure_monitor/tests/test_protocol.py index 25a5206..0248b12 100644 --- a/azure_monitor/tests/test_protocol.py +++ b/azure_monitor/tests/test_protocol.py @@ -9,7 +9,7 @@ class TestProtocol(unittest.TestCase): def test_object(self): data = protocol.BaseObject() - self.assertEqual(repr(data), '{}') + self.assertEqual(repr(data), "{}") def test_data(self): data = protocol.Data() diff --git a/azure_monitor/tests/trace/test_trace.py b/azure_monitor/tests/trace/test_trace.py index 307758e..17730f4 100644 --- a/azure_monitor/tests/trace/test_trace.py +++ b/azure_monitor/tests/trace/test_trace.py @@ -24,11 +24,9 @@ def test_ctor(self): self.assertRaises(ValueError, lambda: AzureMonitorSpanExporter()) def test_span_to_envelope(self): - from opentelemetry.trace import Link, SpanContext, SpanKind - from opentelemetry.trace.status import StatusCanonicalCode - from opentelemetry.sdk.trace import Span - - options = {"instrumentation_key": "12345678-1234-5678-abcd-12345678abcd"} + options = { + "instrumentation_key": "12345678-1234-5678-abcd-12345678abcd" + } exporter = AzureMonitorSpanExporter(**options) parent_span = Span( @@ -39,7 +37,7 @@ def test_span_to_envelope(self): ), ) - start_time = 1575494316027612800 + start_time = 1575494316027613500 end_time = start_time + 1001000000 # SpanKind.CLIENT HTTP @@ -71,14 +69,18 @@ def test_span_to_envelope(self): self.assertEqual( envelope.name, "Microsoft.ApplicationInsights.RemoteDependency" ) - self.assertEqual(envelope.tags["ai.operation.parentId"], "a6f5d48acb4d31da") self.assertEqual( - envelope.tags["ai.operation.id"], "1bbd944a73a05d89eab5d3740a213ee7" + envelope.tags["ai.operation.parentId"], "a6f5d48acb4d31da" + ) + self.assertEqual( + envelope.tags["ai.operation.id"], + "1bbd944a73a05d89eab5d3740a213ee7", ) self.assertEqual(envelope.time, "2019-12-04T21:18:36.027613Z") self.assertEqual(envelope.data.base_data.name, "GET//wiki/Rabbit") self.assertEqual( - envelope.data.base_data.data, "https://www.wikipedia.org/wiki/Rabbit" + envelope.data.base_data.data, + "https://www.wikipedia.org/wiki/Rabbit", ) self.assertEqual(envelope.data.base_data.target, "www.wikipedia.org") self.assertEqual(envelope.data.base_data.id, "a6f5d48acb4d31d9") @@ -111,9 +113,12 @@ def test_span_to_envelope(self): self.assertEqual( envelope.name, "Microsoft.ApplicationInsights.RemoteDependency" ) - self.assertEqual(envelope.tags["ai.operation.parentId"], "a6f5d48acb4d31da") self.assertEqual( - envelope.tags["ai.operation.id"], "1bbd944a73a05d89eab5d3740a213ee7" + envelope.tags["ai.operation.parentId"], "a6f5d48acb4d31da" + ) + self.assertEqual( + envelope.tags["ai.operation.id"], + "1bbd944a73a05d89eab5d3740a213ee7", ) self.assertEqual(envelope.time, "2019-12-04T21:18:36.027613Z") self.assertEqual(envelope.data.base_data.name, "test") @@ -150,14 +155,18 @@ def test_span_to_envelope(self): self.assertEqual( envelope.name, "Microsoft.ApplicationInsights.RemoteDependency" ) - self.assertEqual(envelope.tags["ai.operation.parentId"], "a6f5d48acb4d31da") self.assertEqual( - envelope.tags["ai.operation.id"], "1bbd944a73a05d89eab5d3740a213ee7" + envelope.tags["ai.operation.parentId"], "a6f5d48acb4d31da" + ) + self.assertEqual( + envelope.tags["ai.operation.id"], + "1bbd944a73a05d89eab5d3740a213ee7", ) self.assertEqual(envelope.time, "2019-12-04T21:18:36.027613Z") self.assertEqual(envelope.data.base_data.name, "test") self.assertEqual( - envelope.data.base_data.data, "https://www.wikipedia.org/wiki/Rabbit" + envelope.data.base_data.data, + "https://www.wikipedia.org/wiki/Rabbit", ) self.assertEqual(envelope.data.base_data.target, "www.wikipedia.org") self.assertEqual(envelope.data.base_data.id, "a6f5d48acb4d31d9") @@ -194,12 +203,19 @@ def test_span_to_envelope(self): span.end(end_time=end_time) envelope = exporter.span_to_envelope(span) self.assertEqual(envelope.ikey, "12345678-1234-5678-abcd-12345678abcd") - self.assertEqual(envelope.name, "Microsoft.ApplicationInsights.Request") - self.assertEqual(envelope.tags["ai.operation.parentId"], "a6f5d48acb4d31da") self.assertEqual( - envelope.tags["ai.operation.id"], "1bbd944a73a05d89eab5d3740a213ee7" + envelope.name, "Microsoft.ApplicationInsights.Request" + ) + self.assertEqual( + envelope.tags["ai.operation.parentId"], "a6f5d48acb4d31da" + ) + self.assertEqual( + envelope.tags["ai.operation.id"], + "1bbd944a73a05d89eab5d3740a213ee7", + ) + self.assertEqual( + envelope.tags["ai.operation.name"], "GET /wiki/Rabbit" ) - self.assertEqual(envelope.tags["ai.operation.name"], "GET /wiki/Rabbit") self.assertEqual(envelope.time, "2019-12-04T21:18:36.027613Z") self.assertEqual(envelope.data.base_data.id, "a6f5d48acb4d31d9") self.assertEqual(envelope.data.base_data.duration, "0.00:00:01.001") @@ -207,7 +223,8 @@ def test_span_to_envelope(self): self.assertEqual(envelope.data.base_data.name, "GET /wiki/Rabbit") self.assertEqual(envelope.data.base_data.success, True) self.assertEqual( - envelope.data.base_data.url, "https://www.wikipedia.org/wiki/Rabbit" + envelope.data.base_data.url, + "https://www.wikipedia.org/wiki/Rabbit", ) self.assertEqual(envelope.data.base_type, "RequestData") @@ -239,12 +256,19 @@ def test_span_to_envelope(self): span.end(end_time=end_time) envelope = exporter.span_to_envelope(span) self.assertEqual(envelope.ikey, "12345678-1234-5678-abcd-12345678abcd") - self.assertEqual(envelope.name, "Microsoft.ApplicationInsights.Request") - self.assertEqual(envelope.tags["ai.operation.parentId"], "a6f5d48acb4d31da") self.assertEqual( - envelope.tags["ai.operation.id"], "1bbd944a73a05d89eab5d3740a213ee7" + envelope.name, "Microsoft.ApplicationInsights.Request" + ) + self.assertEqual( + envelope.tags["ai.operation.parentId"], "a6f5d48acb4d31da" + ) + self.assertEqual( + envelope.tags["ai.operation.id"], + "1bbd944a73a05d89eab5d3740a213ee7", + ) + self.assertEqual( + envelope.tags["ai.operation.name"], "GET /wiki/Rabbit" ) - self.assertEqual(envelope.tags["ai.operation.name"], "GET /wiki/Rabbit") self.assertEqual(envelope.time, "2019-12-04T21:18:36.027613Z") self.assertEqual(envelope.data.base_data.id, "a6f5d48acb4d31d9") self.assertEqual(envelope.data.base_data.duration, "0.00:00:01.001") @@ -252,7 +276,8 @@ def test_span_to_envelope(self): self.assertEqual(envelope.data.base_data.name, "GET /wiki/Rabbit") self.assertEqual(envelope.data.base_data.success, False) self.assertEqual( - envelope.data.base_data.url, "https://www.wikipedia.org/wiki/Rabbit" + envelope.data.base_data.url, + "https://www.wikipedia.org/wiki/Rabbit", ) self.assertEqual(envelope.data.base_type, "RequestData") @@ -284,10 +309,15 @@ def test_span_to_envelope(self): span.end(end_time=end_time) envelope = exporter.span_to_envelope(span) self.assertEqual(envelope.ikey, "12345678-1234-5678-abcd-12345678abcd") - self.assertEqual(envelope.name, "Microsoft.ApplicationInsights.Request") - self.assertEqual(envelope.tags["ai.operation.parentId"], "a6f5d48acb4d31da") self.assertEqual( - envelope.tags["ai.operation.id"], "1bbd944a73a05d89eab5d3740a213ee7" + envelope.name, "Microsoft.ApplicationInsights.Request" + ) + self.assertEqual( + envelope.tags["ai.operation.parentId"], "a6f5d48acb4d31da" + ) + self.assertEqual( + envelope.tags["ai.operation.id"], + "1bbd944a73a05d89eab5d3740a213ee7", ) self.assertEqual(envelope.time, "2019-12-04T21:18:36.027613Z") self.assertEqual(envelope.data.base_data.id, "a6f5d48acb4d31d9") @@ -318,9 +348,12 @@ def test_span_to_envelope(self): self.assertEqual( envelope.name, "Microsoft.ApplicationInsights.RemoteDependency" ) - self.assertRaises(KeyError, lambda: envelope.tags["ai.operation.parentId"]) + self.assertRaises( + KeyError, lambda: envelope.tags["ai.operation.parentId"] + ) self.assertEqual( - envelope.tags["ai.operation.id"], "1bbd944a73a05d89eab5d3740a213ee7" + envelope.tags["ai.operation.id"], + "1bbd944a73a05d89eab5d3740a213ee7", ) self.assertEqual(envelope.time, "2019-12-04T21:18:36.027613Z") self.assertEqual(envelope.data.base_data.name, "test") @@ -356,7 +389,9 @@ def test_span_to_envelope(self): span.end(end_time=end_time) envelope = exporter.span_to_envelope(span) self.assertEqual(len(envelope.data.base_data.properties), 2) - self.assertEqual(envelope.data.base_data.properties["component"], "http") + self.assertEqual( + envelope.data.base_data.properties["component"], "http" + ) self.assertEqual(envelope.data.base_data.properties["test"], "asd") # Links @@ -394,9 +429,9 @@ def test_span_to_envelope(self): span.end(end_time=end_time) envelope = exporter.span_to_envelope(span) self.assertEqual(len(envelope.data.base_data.properties), 2) - json_dict = json.loads(envelope.data.base_data.properties["_MS.links"])[ - 0 - ] + json_dict = json.loads( + envelope.data.base_data.properties["_MS.links"] + )[0] self.assertEqual(json_dict["id"], "a6f5d48acb4d31da") # Status @@ -717,4 +752,6 @@ def test_span_to_envelope(self): span.status = Status(canonical_code=StatusCanonicalCode.OK) envelope = exporter.span_to_envelope(span) self.assertIsNone(envelope.data.base_data.url) - self.assertIsNone(envelope.data.base_data.properties.get("request.url")) + self.assertIsNone( + envelope.data.base_data.properties.get("request.url") + ) From e362067546f7707cebfa1b237731b06420e65552 Mon Sep 17 00:00:00 2001 From: Hector Hernandez Guzman Date: Thu, 27 Feb 2020 16:34:19 -0800 Subject: [PATCH 048/109] Adding localStorage support Adding data response when 206 responseCode --- azure_monitor/src/azure_monitor/exporter.py | 107 +++++++- azure_monitor/src/azure_monitor/protocol.py | 8 +- azure_monitor/src/azure_monitor/storage.py | 178 +++++++++++++ azure_monitor/src/azure_monitor/trace.py | 63 ++--- azure_monitor/src/azure_monitor/utils.py | 78 ++++++ azure_monitor/tests/test_storage.py | 135 ++++++++++ azure_monitor/tests/test_utils.py | 19 ++ azure_monitor/tests/trace/test_trace.py | 274 +++++++++++++++++++- 8 files changed, 805 insertions(+), 57 deletions(-) create mode 100644 azure_monitor/src/azure_monitor/storage.py create mode 100644 azure_monitor/tests/test_storage.py diff --git a/azure_monitor/src/azure_monitor/exporter.py b/azure_monitor/src/azure_monitor/exporter.py index 3fe5b3f..dc80f40 100644 --- a/azure_monitor/src/azure_monitor/exporter.py +++ b/azure_monitor/src/azure_monitor/exporter.py @@ -1,9 +1,14 @@ # Copyright (c) Microsoft Corporation. All rights reserved. # Licensed under the MIT License. - +import json import logging +# pylint: disable=import-error +import requests + + from azure_monitor import utils +from azure_monitor.storage import LocalFileStorage logger = logging.getLogger(__name__) @@ -12,6 +17,12 @@ class BaseExporter: def __init__(self, **options): self._telemetry_processors = [] self.options = utils.Options(**options) + self.storage = LocalFileStorage( + path=self.options.storage_path, + max_size=self.options.storage_max_size, + maintenance_period=self.options.storage_maintenance_period, + retention_period=self.options.storage_retention_period, + ) def add_telemetry_processor(self, processor): """Adds telemetry processor to the collection. @@ -49,3 +60,97 @@ def apply_telemetry_processors(self, envelopes): if accepted: filtered_envelopes.append(envelope) return filtered_envelopes + + def _transmit_from_storage(self): + for blob in self.storage.gets(): + # give a few more seconds for blob lease operation + # to reduce the chance of race (for perf consideration) + if blob.lease(self.options.timeout + 5): + envelopes = blob.get() # TODO: handle error + result = self._transmit(envelopes) + if result == utils.ExportResult.FAILED_RETRYABLE: + blob.lease(1) + else: + blob.delete(silent=True) + + def _transmit(self, envelopes_to_export): + """ + Transmit the data envelopes to the ingestion service. + + Returns an ExportResult, this function should never + throw an exception. + """ + if envelopes_to_export: + try: + response = requests.post( + url=self.options.endpoint, + data=json.dumps(envelopes_to_export), + headers={ + "Accept": "application/json", + "Content-Type": "application/json; charset=utf-8", + }, + timeout=self.options.timeout, + ) + except Exception as ex: + logger.warning("Transient client side error %s.", ex) + return utils.ExportResult.FAILED_RETRYABLE + + text = "N/A" + data = None + try: + text = response.text + except Exception as ex: # noqa pylint: disable=broad-except + logger.warning("Error while reading response body %s.", ex) + else: + try: + data = json.loads(text) + except Exception: # noqa pylint: disable=broad-except + pass + + if response.status_code == 200: + logger.info("Transmission succeeded: %s.", text) + return utils.ExportResult.SUCCESS + if response.status_code == 206: # Partial Content + # TODO: store the unsent data + if data: + try: + resend_envelopes = [] + for error in data["errors"]: + if error["statusCode"] in ( + 429, # Too Many Requests + 500, # Internal Server Error + 503, # Service Unavailable + ): + resend_envelopes.append( + envelopes_to_export[error["index"]] + ) + else: + logger.error( + "Data drop %s: %s %s.", + error["statusCode"], + error["message"], + envelopes_to_export[error["index"]], + ) + if resend_envelopes: + self.storage.put(resend_envelopes) + except Exception as ex: + logger.error( + "Error while processing %s: %s %s.", + response.status_code, + text, + ex, + ) + return utils.ExportResult.FAILED_NOT_RETRYABLE + # cannot parse response body, fallback to retry + + if response.status_code in ( + 206, # Partial Content + 429, # Too Many Requests + 500, # Internal Server Error + 503, # Service Unavailable + ): + return utils.ExportResult.FAILED_RETRYABLE + + return utils.ExportResult.FAILED_NOT_RETRYABLE + # No spans to export + return utils.ExportResult.SUCCESS diff --git a/azure_monitor/src/azure_monitor/protocol.py b/azure_monitor/src/azure_monitor/protocol.py index 46c0834..5981f6c 100644 --- a/azure_monitor/src/azure_monitor/protocol.py +++ b/azure_monitor/src/azure_monitor/protocol.py @@ -27,8 +27,8 @@ def __init__(self, base_data=None, base_type=None) -> None: def to_dict(self): return { - "baseData": self.base_data.to_dict(), - "baseType": self.base_type, + "baseData": self.base_data.to_dict() if self.base_data else None, + "baseType": self.base_type.to_dict() if self.base_type else None, } @@ -122,8 +122,8 @@ def to_dict(self): "iKey": self.ikey, "flags": self.flags, "tags": self.tags, - "data": self.data.to_dict(), - "baseType": self.base_type, + "data": self.data.to_dict() if self.data else None, + "baseType": self.data.base_type if self.data else None, } diff --git a/azure_monitor/src/azure_monitor/storage.py b/azure_monitor/src/azure_monitor/storage.py new file mode 100644 index 0000000..10145f4 --- /dev/null +++ b/azure_monitor/src/azure_monitor/storage.py @@ -0,0 +1,178 @@ +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. + +import datetime +import json +import os +import random + +from azure_monitor.utils import PeriodicTask + + +def _fmt(timestamp): + return timestamp.strftime("%Y-%m-%dT%H%M%S.%f") + + +def _now(): + return datetime.datetime.utcnow() + + +def _seconds(seconds): + return datetime.timedelta(seconds=seconds) + + +class LocalFileBlob(object): + def __init__(self, fullpath): + self.fullpath = fullpath + + def delete(self, silent=False): + try: + os.remove(self.fullpath) + except Exception: + if not silent: + raise + + def get(self, silent=False): + try: + with open(self.fullpath, "r") as file: + return tuple( + json.loads(line.strip()) for line in file.readlines() + ) + except Exception: + if not silent: + raise + + def put(self, data, lease_period=0, silent=False): + try: + fullpath = self.fullpath + ".tmp" + with open(fullpath, "w") as file: + for item in data: + file.write(json.dumps(item)) + # The official Python doc: Do not use os.linesep as a line + # terminator when writing files opened in text mode (the + # default); use a single '\n' instead, on all platforms. + file.write("\n") + if lease_period: + timestamp = _now() + _seconds(lease_period) + self.fullpath += "@{}.lock".format(_fmt(timestamp)) + os.rename(fullpath, self.fullpath) + return self + except Exception: + if not silent: + raise + + def lease(self, period): + timestamp = _now() + _seconds(period) + fullpath = self.fullpath + if fullpath.endswith(".lock"): + fullpath = fullpath[: fullpath.rindex("@")] + fullpath += "@{}.lock".format(_fmt(timestamp)) + try: + os.rename(self.fullpath, fullpath) + except Exception: + return None + self.fullpath = fullpath + return self + + +class LocalFileStorage(object): + def __init__( + self, + path, + max_size=100 * 1024 * 1024, # 100MB + maintenance_period=60, # 1 minute + retention_period=7 * 24 * 60 * 60, # 7 days + write_timeout=60, # 1 minute + ): + self.path = os.path.abspath(path) + self.max_size = max_size + self.maintenance_period = maintenance_period + self.retention_period = retention_period + self.write_timeout = write_timeout + self._maintenance_routine(silent=False) + self._maintenance_task = PeriodicTask( + interval=self.maintenance_period, + function=self._maintenance_routine, + kwargs={"silent": True}, + ) + self._maintenance_task.daemon = True + self._maintenance_task.start() + + def close(self): + self._maintenance_task.cancel() + self._maintenance_task.join() + + def __enter__(self): + return self + + def __exit__(self, type, value, traceback): + self.close() + + def _maintenance_routine(self, silent=False): + try: + if not os.path.isdir(self.path): + os.makedirs(self.path) + except Exception: + if not silent: + raise + try: + for blob in self.gets(): + pass + except Exception: + if not silent: + raise + + def gets(self): + now = _now() + lease_deadline = _fmt(now) + retention_deadline = _fmt(now - _seconds(self.retention_period)) + timeout_deadline = _fmt(now - _seconds(self.write_timeout)) + for name in sorted(os.listdir(self.path)): + path = os.path.join(self.path, name) + if not os.path.isfile(path): + continue # skip if not a file + if path.endswith(".tmp"): + if name < timeout_deadline: + try: + os.remove(path) # TODO: log data loss + except Exception: + pass # keep silent + if path.endswith(".lock"): + if path[path.rindex("@") + 1 : -5] > lease_deadline: + continue # under lease + new_path = path[: path.rindex("@")] + try: + os.rename(path, new_path) + except Exception: + continue # keep silent + path = new_path + if path.endswith(".blob"): + if name < retention_deadline: + try: + os.remove(path) # TODO: log data loss + except Exception: + pass # keep silent + else: + yield LocalFileBlob(path) + + def get(self): + cursor = self.gets() + try: + return next(cursor) + except StopIteration: + pass + return None + + def put(self, data, lease_period=0, silent=False): + blob = LocalFileBlob( + os.path.join( + self.path, + "{}-{}.blob".format( + _fmt(_now()), + "{:08x}".format( + random.getrandbits(32) + ), # thread-safe random + ), + ) + ) + return blob.put(data, lease_period=lease_period, silent=silent) diff --git a/azure_monitor/src/azure_monitor/trace.py b/azure_monitor/src/azure_monitor/trace.py index 5dc630e..e5b5db1 100644 --- a/azure_monitor/src/azure_monitor/trace.py +++ b/azure_monitor/src/azure_monitor/trace.py @@ -4,8 +4,6 @@ import logging from urllib.parse import urlparse -# pylint: disable=import-error -import requests from opentelemetry.sdk.trace.export import SpanExporter, SpanExportResult from opentelemetry.sdk.util import ns_to_iso_str from opentelemetry.trace import Span, SpanKind @@ -21,59 +19,28 @@ class AzureMonitorSpanExporter(BaseExporter, SpanExporter): def __init__(self, **options): super(AzureMonitorSpanExporter, self).__init__(**options) - def export(self, spans): + def export(self, spans) -> SpanExportResult: envelopes = map(self.span_to_envelope, spans) envelopes_to_export = map( lambda x: x.to_dict(), tuple(self.apply_telemetry_processors(envelopes)), ) + try: + result = self._transmit(envelopes_to_export) + if result == SpanExportResult.FAILED_RETRYABLE: + self.storage.put(envelopes, result) + if len(envelopes_to_export) < self.options.max_batch_size: + self._transmit_from_storage() + return utils.get_trace_export_result(result) + except Exception: + logger.exception("Exception occurred while exporting the data.") - if envelopes_to_export: - try: - response = requests.post( - url=self.options.endpoint, - data=json.dumps(envelopes_to_export), - headers={ - "Accept": "application/json", - "Content-Type": "application/json; charset=utf-8", - }, - timeout=self.options.timeout, - ) - except requests.RequestException as ex: - logger.warning("Transient client side error %s.", ex) - return SpanExportResult.FAILED_RETRYABLE + def span_to_envelope( + self, span + ) -> protocol.Envelope: # noqa pylint: disable=too-many-branches - text = "N/A" - data = None # noqa pylint: disable=unused-variable - try: - text = response.text - except Exception as ex: # noqa pylint: disable=broad-except - logger.warning("Error while reading response body %s.", ex) - else: - try: - data = json.loads( - text - ) # noqa pylint: disable=unused-variable - except Exception: # noqa pylint: disable=broad-except - pass - - if response.status_code == 200: - logger.info("Transmission succeeded: %s.", text) - return SpanExportResult.SUCCESS - - if response.status_code in ( - 206, # Partial Content - 429, # Too Many Requests - 500, # Internal Server Error - 503, # Service Unavailable - ): - return SpanExportResult.FAILED_RETRYABLE - - return SpanExportResult.FAILED_NOT_RETRYABLE - # No spans to export - return SpanExportResult.SUCCESS - - def span_to_envelope(self, span): # noqa pylint: disable=too-many-branches + if not span: + return None # pylint: disable=too-many-statements envelope = protocol.Envelope( ikey=self.options.instrumentation_key, diff --git a/azure_monitor/src/azure_monitor/utils.py b/azure_monitor/src/azure_monitor/utils.py index 2672596..f584f82 100644 --- a/azure_monitor/src/azure_monitor/utils.py +++ b/azure_monitor/src/azure_monitor/utils.py @@ -5,9 +5,14 @@ import platform import re import sys +import threading +import time +from enum import Enum +from opentelemetry.sdk.trace.export import SpanExportResult from opentelemetry.sdk.version import __version__ as opentelemetry_version + from azure_monitor.protocol import BaseObject from azure_monitor.version import __version__ as ext_version @@ -49,12 +54,34 @@ def ns_to_duration(nanoseconds): ) +class ExportResult(Enum): + SUCCESS = 0 + FAILED_RETRYABLE = 1 + FAILED_NOT_RETRYABLE = 2 + + +def get_trace_export_result(result: ExportResult): + if result == ExportResult.SUCCESS: + return SpanExportResult.SUCCESS + elif result == ExportResult.FAILED_RETRYABLE: + return SpanExportResult.FAILED_RETRYABLE + elif result == ExportResult.FAILED_NOT_RETRYABLE: + return SpanExportResult.FAILED_NOT_RETRYABLE + else: + return None + + class Options(BaseObject): __slots__ = ( "connection_string", "endpoint", "instrumentation_key", + "max_batch_size", + "storage_maintenance_period", + "storage_max_size", + "storage_path", + "storage_retention_period", "timeout", ) @@ -63,11 +90,26 @@ def __init__( connection_string=None, endpoint="https://dc.services.visualstudio.com/v2/track", instrumentation_key=None, + max_batch_size=100, + storage_maintenance_period=60, + storage_max_size=100 * 1024 * 1024, + storage_path=os.path.join( + os.path.expanduser("~"), + ".opentelemetry", + ".azure", + os.path.basename(sys.argv[0]) or ".console", + ), + storage_retention_period=7 * 24 * 60 * 60, timeout=10.0, # networking timeout in seconds ) -> None: self.connection_string = connection_string self.endpoint = endpoint self.instrumentation_key = instrumentation_key + self.max_batch_size = max_batch_size + self.storage_maintenance_period = storage_maintenance_period + self.storage_max_size = storage_max_size + self.storage_path = storage_path + self.storage_retention_period = storage_retention_period self.timeout = timeout self._initialize() self._validate_instrumentation_key() @@ -147,3 +189,39 @@ def _parse_connection_string(self, connection_string): # Default to None if cannot construct result[INGESTION_ENDPOINT] = None return result + + +class PeriodicTask(threading.Thread): + """Thread that periodically calls a given function. + + :type interval: int or float + :param interval: Seconds between calls to the function. + + :type function: function + :param function: The function to call. + + :type args: list + :param args: The args passed in while calling `function`. + + :type kwargs: dict + :param args: The kwargs passed in while calling `function`. + """ + + def __init__(self, interval, function, args=None, kwargs=None): + super(PeriodicTask, self).__init__() + self.interval = interval + self.function = function + self.args = args or [] + self.kwargs = kwargs or {} + self.finished = threading.Event() + + def run(self): + wait_time = self.interval + while not self.finished.wait(wait_time): + start_time = time.time() + self.function(*self.args, **self.kwargs) + elapsed_time = time.time() - start_time + wait_time = max(self.interval - elapsed_time, 0) + + def cancel(self): + self.finished.set() diff --git a/azure_monitor/tests/test_storage.py b/azure_monitor/tests/test_storage.py new file mode 100644 index 0000000..1b30767 --- /dev/null +++ b/azure_monitor/tests/test_storage.py @@ -0,0 +1,135 @@ +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. + +import os +import shutil +import unittest +from unittest import mock + +from azure_monitor.storage import ( + LocalFileBlob, + LocalFileStorage, + _now, + _seconds, +) + +TEST_FOLDER = os.path.abspath(".test") + + +def setUpModule(): + os.makedirs(TEST_FOLDER) + + +def tearDownModule(): + shutil.rmtree(TEST_FOLDER) + + +def throw(exc_type, *args, **kwargs): + def func(*_args, **_kwargs): + raise exc_type(*args, **kwargs) + + return func + + +class TestLocalFileBlob(unittest.TestCase): + def test_delete(self): + blob = LocalFileBlob(os.path.join(TEST_FOLDER, "foobar")) + blob.delete(silent=True) + self.assertRaises(Exception, lambda: blob.delete()) + self.assertRaises(Exception, lambda: blob.delete(silent=False)) + + def test_get(self): + blob = LocalFileBlob(os.path.join(TEST_FOLDER, "foobar")) + self.assertIsNone(blob.get(silent=True)) + self.assertRaises(Exception, lambda: blob.get()) + self.assertRaises(Exception, lambda: blob.get(silent=False)) + + def test_put_error(self): + blob = LocalFileBlob(os.path.join(TEST_FOLDER, "foobar")) + with mock.patch("os.rename", side_effect=throw(Exception)): + self.assertRaises(Exception, lambda: blob.put([1, 2, 3])) + + def test_put_without_lease(self): + blob = LocalFileBlob(os.path.join(TEST_FOLDER, "foobar.blob")) + input = (1, 2, 3) + blob.delete(silent=True) + blob.put(input) + self.assertEqual(blob.get(), input) + + def test_put_with_lease(self): + blob = LocalFileBlob(os.path.join(TEST_FOLDER, "foobar.blob")) + input = (1, 2, 3) + blob.delete(silent=True) + blob.put(input, lease_period=0.01) + blob.lease(0.01) + self.assertEqual(blob.get(), input) + + def test_lease_error(self): + blob = LocalFileBlob(os.path.join(TEST_FOLDER, "foobar.blob")) + blob.delete(silent=True) + self.assertEqual(blob.lease(0.01), None) + + +class TestLocalFileStorage(unittest.TestCase): + def test_get_nothing(self): + with LocalFileStorage(os.path.join(TEST_FOLDER, "test", "a")) as stor: + pass + with LocalFileStorage(os.path.join(TEST_FOLDER, "test")) as stor: + self.assertIsNone(stor.get()) + + def test_get(self): + now = _now() + with LocalFileStorage(os.path.join(TEST_FOLDER, "foo")) as stor: + stor.put((1, 2, 3), lease_period=10) + with mock.patch("azure_monitor.storage._now") as m: + m.return_value = now - _seconds(30 * 24 * 60 * 60) + stor.put((1, 2, 3)) + stor.put((1, 2, 3), lease_period=10) + with mock.patch("os.rename"): + stor.put((1, 2, 3)) + with mock.patch("os.rename"): + stor.put((1, 2, 3)) + with mock.patch("os.remove", side_effect=throw(Exception)): + with mock.patch("os.rename", side_effect=throw(Exception)): + self.assertIsNone(stor.get()) + self.assertIsNone(stor.get()) + + def test_put(self): + input = (1, 2, 3) + with LocalFileStorage(os.path.join(TEST_FOLDER, "bar")) as stor: + stor.put(input) + self.assertEqual(stor.get().get(), input) + with LocalFileStorage(os.path.join(TEST_FOLDER, "bar")) as stor: + self.assertEqual(stor.get().get(), input) + with mock.patch("os.rename", side_effect=throw(Exception)): + self.assertIsNone(stor.put(input, silent=True)) + self.assertRaises(Exception, lambda: stor.put(input)) + + def test_maintanence_routine(self): + with mock.patch("os.makedirs") as m: + m.return_value = None + self.assertRaises( + Exception, + lambda: LocalFileStorage(os.path.join(TEST_FOLDER, "baz")), + ) + with mock.patch("os.makedirs", side_effect=throw(Exception)): + self.assertRaises( + Exception, + lambda: LocalFileStorage(os.path.join(TEST_FOLDER, "baz")), + ) + with mock.patch("os.listdir", side_effect=throw(Exception)): + self.assertRaises( + Exception, + lambda: LocalFileStorage(os.path.join(TEST_FOLDER, "baz")), + ) + with LocalFileStorage(os.path.join(TEST_FOLDER, "baz")) as stor: + with mock.patch("os.listdir", side_effect=throw(Exception)): + stor._maintenance_routine(silent=True) + self.assertRaises( + Exception, lambda: stor._maintenance_routine() + ) + with mock.patch("os.path.isdir", side_effect=throw(Exception)): + stor._maintenance_routine(silent=True) + self.assertRaises( + Exception, lambda: stor._maintenance_routine() + ) diff --git a/azure_monitor/tests/test_utils.py b/azure_monitor/tests/test_utils.py index ff171a3..d236e79 100644 --- a/azure_monitor/tests/test_utils.py +++ b/azure_monitor/tests/test_utils.py @@ -4,6 +4,8 @@ import os import unittest +from opentelemetry.sdk.trace.export import SpanExportResult + from azure_monitor import utils @@ -262,3 +264,20 @@ def test_parse_connection_string_invalid_auth(self): instrumentation_key=self._valid_instrumentation_key, ), ) + + def test_get_trace_export_result(self): + self.assertEqual( + utils.get_trace_export_result(utils.ExportResult.SUCCESS), + SpanExportResult.SUCCESS, + ) + self.assertEqual( + utils.get_trace_export_result( + utils.ExportResult.FAILED_NOT_RETRYABLE + ), + SpanExportResult.FAILED_NOT_RETRYABLE, + ) + self.assertEqual( + utils.get_trace_export_result(utils.ExportResult.FAILED_RETRYABLE), + SpanExportResult.FAILED_RETRYABLE, + ) + self.assertEqual(utils.get_trace_export_result(None), None) diff --git a/azure_monitor/tests/trace/test_trace.py b/azure_monitor/tests/trace/test_trace.py index 17730f4..ef86a4b 100644 --- a/azure_monitor/tests/trace/test_trace.py +++ b/azure_monitor/tests/trace/test_trace.py @@ -4,24 +4,275 @@ import json import os +import shutil import unittest +from unittest import mock # pylint: disable=import-error from opentelemetry.sdk.trace import Span +from opentelemetry.sdk.trace.export import SpanExportResult from opentelemetry.trace import Link, SpanContext, SpanKind from opentelemetry.trace.status import Status, StatusCanonicalCode +from azure_monitor.protocol import Envelope from azure_monitor.trace import AzureMonitorSpanExporter from azure_monitor.utils import Options +TEST_FOLDER = os.path.abspath(".test.exporter") + + +def setUpModule(): + os.makedirs(TEST_FOLDER) + + +def tearDownModule(): + shutil.rmtree(TEST_FOLDER) + + +def throw(exc_type, *args, **kwargs): + def func(*_args, **_kwargs): + raise exc_type(*args, **kwargs) + + return func + # pylint: disable=import-error class TestAzureExporter(unittest.TestCase): - def setUp(self): - os.environ.clear() + @classmethod + def setUpClass(self): + os.environ[ + "APPINSIGHTS_INSTRUMENTATIONKEY" + ] = "1234abcd-5678-4efa-8abc-1234567890ab" + + @mock.patch("requests.post", return_value=mock.Mock()) + def test_export_empty(self, request_mock): + exporter = AzureMonitorSpanExporter( + storage_path=os.path.join(TEST_FOLDER, self.id()) + ) + exporter.export([]) + self.assertEqual(len(os.listdir(exporter.storage.path)), 0) + + @mock.patch("azure_monitor.trace.logger") + def test_export_exception(self, mock_logger): + exporter = AzureMonitorSpanExporter( + storage_path=os.path.join(TEST_FOLDER, self.id()) + ) + exporter.export([None]) + mock_logger.exception.assert_called() + + @mock.patch( + "azure_monitor.trace.AzureMonitorSpanExporter.span_to_envelope" + ) # noqa: E501 + def test_export_failure(self, span_to_envelope_mock): + span_to_envelope_mock.return_value = ["bar"] + exporter = AzureMonitorSpanExporter( + storage_path=os.path.join(TEST_FOLDER, self.id()) + ) + with mock.patch( + "azure_monitor.trace.AzureMonitorSpanExporter._transmit" + ) as transmit: # noqa: E501 + test_span = Span( + name="test", + context=SpanContext( + trace_id=36873507687745823477771305566750195431, + span_id=12030755672171557338, + ), + ) + test_span.start() + test_span.end() + transmit.return_value = SpanExportResult.FAILED_RETRYABLE + exporter.export([test_span]) + self.assertEqual(len(os.listdir(exporter.storage.path)), 1) + self.assertIsNone(exporter.storage.get()) + + @mock.patch( + "azure_monitor.trace.AzureMonitorSpanExporter.span_to_envelope" + ) # noqa: E501 + def test_export_success(self, span_to_envelope_mock): + span_to_envelope_mock.return_value = ["bar"] + exporter = AzureMonitorSpanExporter( + max_batch_size=1, storage_path=os.path.join(TEST_FOLDER, self.id()) + ) + with mock.patch( + "azure_monitor.trace.AzureMonitorSpanExporter._transmit" + ) as transmit: # noqa: E501 + transmit.return_value = 0 + exporter.export([]) + exporter.export(["foo"]) + self.assertEqual(len(os.listdir(exporter.storage.path)), 0) + exporter.export(["foo"]) + self.assertEqual(len(os.listdir(exporter.storage.path)), 0) + + def test_transmission_nothing(self): + exporter = AzureMonitorSpanExporter( + storage_path=os.path.join(TEST_FOLDER, self.id()) + ) + with mock.patch("requests.post") as post: + post.return_value = None + exporter._transmit_from_storage() - def test_ctor(self): - self.assertRaises(ValueError, lambda: AzureMonitorSpanExporter()) + def test_transmission_request_exception(self): + exporter = AzureMonitorSpanExporter( + storage_path=os.path.join(TEST_FOLDER, self.id()) + ) + envelopes_to_export = map(lambda x: x.to_dict(), tuple([Envelope()])) + exporter.storage.put(envelopes_to_export) + with mock.patch("requests.post", throw(Exception)): + exporter._transmit_from_storage() + self.assertIsNone(exporter.storage.get()) + self.assertEqual(len(os.listdir(exporter.storage.path)), 1) + + @mock.patch("requests.post", return_value=mock.Mock()) + def test_transmission_lease_failure(self, requests_mock): + requests_mock.return_value = MockResponse(200, "unknown") + exporter = AzureMonitorSpanExporter( + storage_path=os.path.join(TEST_FOLDER, self.id()) + ) + envelopes_to_export = map(lambda x: x.to_dict(), tuple([Envelope()])) + exporter.storage.put(envelopes_to_export) + with mock.patch( + "azure_monitor.storage.LocalFileBlob.lease" + ) as lease: # noqa: E501 + lease.return_value = False + exporter._transmit_from_storage() + self.assertTrue(exporter.storage.get()) + + def test_transmission_response_exception(self): + exporter = AzureMonitorSpanExporter( + storage_path=os.path.join(TEST_FOLDER, self.id()) + ) + envelopes_to_export = map(lambda x: x.to_dict(), tuple([Envelope()])) + exporter.storage.put(envelopes_to_export) + with mock.patch("requests.post") as post: + post.return_value = MockResponse(200, None) + del post.return_value.text + exporter._transmit_from_storage() + self.assertIsNone(exporter.storage.get()) + self.assertEqual(len(os.listdir(exporter.storage.path)), 0) + + def test_transmission_200(self): + exporter = AzureMonitorSpanExporter( + storage_path=os.path.join(TEST_FOLDER, self.id()) + ) + envelopes_to_export = map(lambda x: x.to_dict(), tuple([Envelope()])) + exporter.storage.put(envelopes_to_export) + with mock.patch("requests.post") as post: + post.return_value = MockResponse(200, "unknown") + exporter._transmit_from_storage() + self.assertIsNone(exporter.storage.get()) + self.assertEqual(len(os.listdir(exporter.storage.path)), 0) + + def test_transmission_206(self): + exporter = AzureMonitorSpanExporter( + storage_path=os.path.join(TEST_FOLDER, self.id()) + ) + envelopes_to_export = map(lambda x: x.to_dict(), tuple([Envelope()])) + exporter.storage.put(envelopes_to_export) + with mock.patch("requests.post") as post: + post.return_value = MockResponse(206, "unknown") + exporter._transmit_from_storage() + self.assertIsNone(exporter.storage.get()) + self.assertEqual(len(os.listdir(exporter.storage.path)), 1) + + def test_transmission_206_500(self): + exporter = AzureMonitorSpanExporter( + storage_path=os.path.join(TEST_FOLDER, self.id()) + ) + test_envelope = Envelope(name="testEnvelope") + envelopes_to_export = map( + lambda x: x.to_dict(), + tuple([Envelope(), Envelope(), test_envelope]), + ) + exporter.storage.put(envelopes_to_export) + with mock.patch("requests.post") as post: + post.return_value = MockResponse( + 206, + json.dumps( + { + "itemsReceived": 5, + "itemsAccepted": 3, + "errors": [ + {"index": 0, "statusCode": 400, "message": ""}, + { + "index": 2, + "statusCode": 500, + "message": "Internal Server Error", + }, + ], + } + ), + ) + exporter._transmit_from_storage() + self.assertEqual(len(os.listdir(exporter.storage.path)), 1) + self.assertEqual( + exporter.storage.get().get()[0]["name"], "testEnvelope" + ) + + def test_transmission_206_nothing_to_retry(self): + exporter = AzureMonitorSpanExporter( + storage_path=os.path.join(TEST_FOLDER, self.id()) + ) + envelopes_to_export = map(lambda x: x.to_dict(), tuple([Envelope()])) + exporter.storage.put(envelopes_to_export) + with mock.patch("requests.post") as post: + post.return_value = MockResponse( + 206, + json.dumps( + { + "itemsReceived": 3, + "itemsAccepted": 2, + "errors": [ + {"index": 0, "statusCode": 400, "message": ""} + ], + } + ), + ) + exporter._transmit_from_storage() + self.assertEqual(len(os.listdir(exporter.storage.path)), 0) + + def test_transmission_206_bogus(self): + exporter = AzureMonitorSpanExporter( + storage_path=os.path.join(TEST_FOLDER, self.id()) + ) + envelopes_to_export = map(lambda x: x.to_dict(), tuple([Envelope()])) + exporter.storage.put(envelopes_to_export) + with mock.patch("requests.post") as post: + post.return_value = MockResponse( + 206, + json.dumps( + { + "itemsReceived": 5, + "itemsAccepted": 3, + "errors": [{"foo": 0, "bar": 1}], + } + ), + ) + exporter._transmit_from_storage() + self.assertIsNone(exporter.storage.get()) + self.assertEqual(len(os.listdir(exporter.storage.path)), 0) + + def test_transmission_400(self): + exporter = AzureMonitorSpanExporter( + storage_path=os.path.join(TEST_FOLDER, self.id()) + ) + envelopes_to_export = map(lambda x: x.to_dict(), tuple([Envelope()])) + exporter.storage.put(envelopes_to_export) + with mock.patch("requests.post") as post: + post.return_value = MockResponse(400, "{}") + exporter._transmit_from_storage() + self.assertEqual(len(os.listdir(exporter.storage.path)), 0) + + def test_transmission_500(self): + exporter = AzureMonitorSpanExporter( + storage_path=os.path.join(TEST_FOLDER, self.id()) + ) + envelopes_to_export = map(lambda x: x.to_dict(), tuple([Envelope()])) + exporter.storage.put(envelopes_to_export) + with mock.patch("requests.post") as post: + post.return_value = MockResponse(500, "{}") + exporter._transmit_from_storage() + self.assertIsNone(exporter.storage.get()) + self.assertEqual(len(os.listdir(exporter.storage.path)), 1) def test_span_to_envelope(self): options = { @@ -755,3 +1006,18 @@ def test_span_to_envelope(self): self.assertIsNone( envelope.data.base_data.properties.get("request.url") ) + + +class MockResponse(object): + def __init__(self, status_code, text): + self.status_code = status_code + self.text = text + + +class MockTransport(object): + def __init__(self, exporter=None): + self.export_called = False + self.exporter = exporter + + def export(self, datas): + self.export_called = True From abc798d3e0304ea4e416e00143f3fb1521096211 Mon Sep 17 00:00:00 2001 From: Hector Hernandez Guzman Date: Fri, 28 Feb 2020 16:16:41 -0800 Subject: [PATCH 049/109] Adding usage instructions --- README.md => CONTRIBUTING.md | 28 ++--- README.rst | 136 +++++++++++++++++++++++++ azure_monitor/README.rst | 19 ---- azure_monitor/examples/traces/trace.py | 31 +++--- 4 files changed, 167 insertions(+), 47 deletions(-) rename README.md => CONTRIBUTING.md (98%) create mode 100644 README.rst delete mode 100644 azure_monitor/README.rst diff --git a/README.md b/CONTRIBUTING.md similarity index 98% rename from README.md rename to CONTRIBUTING.md index b81a84e..8eeee9c 100644 --- a/README.md +++ b/CONTRIBUTING.md @@ -1,14 +1,14 @@ - -# Contributing - -This project welcomes contributions and suggestions. Most contributions require you to agree to a -Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us -the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com. - -When you submit a pull request, a CLA bot will automatically determine whether you need to provide -a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions -provided by the bot. You will only need to do this once across all repos using our CLA. - -This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). -For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or -contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments. + +# Contributing + +This project welcomes contributions and suggestions. Most contributions require you to agree to a +Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us +the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com. + +When you submit a pull request, a CLA bot will automatically determine whether you need to provide +a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions +provided by the bot. You will only need to do this once across all repos using our CLA. + +This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). +For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or +contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments. diff --git a/README.rst b/README.rst new file mode 100644 index 0000000..91b95a8 --- /dev/null +++ b/README.rst @@ -0,0 +1,136 @@ +OpenTelemetry Azure Monitor Exporters +===================================== + +|pypi| + +.. |pypi| image:: https://badge.fury.io/py/opentelemetry-azure-monitor-exporter.svg + :target: https://pypi.org/project/opentelemetry-azure-monitor-exporter/ + +Installation +------------ + +:: + + pip install opentelemetry-azure-monitor-exporter + +Usage +----- + +Trace +~~~~~ + +The **Azure Monitor Trace Exporter** allows you to export `OpenTelemetry`_ traces to `Azure Monitor`_. + +This example shows how to send a span "hello" to Azure Monitor. + +* Create an Azure Monitor resource and get the instrumentation key, more information can be found `here `_. +* Place your instrumentation key in a `connection string` and directly into your code. +* Alternatively, you can specify your `connection string` in an environment variable ``APPLICATIONINSIGHTS_CONNECTION_STRING``. + + .. code:: python + + from azure_monitor import AzureMonitorSpanExporter + from opentelemetry import trace + from opentelemetry.sdk.trace import TracerProvider + from opentelemetry.sdk.trace.export import BatchExportSpanProcessor + + # The preferred tracer implementation must be set, as the opentelemetry-api + # defines the interface with a no-op implementation. + trace.set_preferred_tracer_provider_implementation(lambda T: TracerProvider()) + + # We tell OpenTelemetry who it is that is creating spans. In this case, we have + # no real name (no setup.py), so we make one up. If we had a version, we would + # also specify it here. + tracer = trace.get_tracer(__name__) + + exporter = AzureMonitorSpanExporter( + connection_string='InstrumentationKey=99c42f65-1656-4c41-afde-bd86b709a4a7', + ) + + # SpanExporter receives the spans and send them to the target location. + span_processor = BatchExportSpanProcessor(exporter) + trace.tracer_provider().add_span_processor(span_processor) + + with tracer.start_as_current_span('hello'): + print('Hello World!') + +Integrations +############ + +OpenTelemetry also supports several `integrations `_ which allows to integrate with third party libraries. + +This example shows how to integrate with the `requests `_ library. + +* Create an Azure Monitor resource and get the instrumentation key, more information can be found `here `_. +* Install the `requests integration package using ``pip install opentelemetry-ext-http-requests``. +* Place your instrumentation key in a `connection string` and directly into your code. +* Alternatively, you can specify your `connection string` in an environment variable ``APPLICATIONINSIGHTS_CONNECTION_STRING``. + +.. code:: python + + import requests + + from azure_monitor import AzureMonitorSpanExporter + from opentelemetry import trace + from opentelemetry.ext import http_requests + from opentelemetry.sdk.trace import TracerProvider + from opentelemetry.sdk.trace.export import ( + BatchExportSpanProcessor, + ConsoleSpanExporter, + ) + + # The preferred tracer implementation must be set, as the opentelemetry-api + # defines the interface with a no-op implementation. + trace.set_preferred_tracer_provider_implementation(lambda T: TracerProvider()) + tracer_provider = trace.tracer_provider() + + exporter = AzureMonitorSpanExporter( + connection_string='InstrumentationKey=99c42f65-1656-4c41-afde-bd86b709a4a7', + ) + span_processor = BatchExportSpanProcessor(exporter) + tracer_provider.add_span_processor(span_processor) + + http_requests.enable(tracer_provider) + response = requests.get(url="https://azure.microsoft.com/") + +Modifying Traces +################ + +* You can pass a callback function to the exporter to process telemetry before it is exported. +* Your callback function can return `False` if you do not want this envelope exported. +* Your callback function must accept an [envelope](https://github.com/microsoft/opentelemetry-exporters-python/blob/master/azure_monitor/src/azure_monitor/protocol.py#L80) data type as its parameter. +* You can see the schema for Azure Monitor data types in the envelopes [here](https://github.com/microsoft/opentelemetry-exporters-python/blob/master/azure_monitor/src/azure_monitor/protocol.py). +* The `AzureMonitorSpanExporter` handles `Data` data types. + +.. code:: python + + from azure_monitor import AzureMonitorSpanExporter + from opentelemetry import trace + from opentelemetry.sdk.trace import TracerProvider + from opentelemetry.sdk.trace.export import BatchExportSpanProcessor + + # Callback function to add os_type: linux to span properties + def callback_function(envelope): + envelope.data.baseData.properties['os_type'] = 'linux' + return True + + exporter = AzureMonitorSpanExporter( + connection_string='InstrumentationKey=99c42f65-1656-4c41-afde-bd86b709a4a7' + ) + exporter.add_telemetry_processor(callback_function) + + trace.set_preferred_tracer_provider_implementation(lambda T: TracerProvider()) + tracer = trace.get_tracer(__name__) + span_processor = BatchExportSpanProcessor(exporter) + trace.tracer_provider().add_span_processor(span_processor) + + with tracer.start_as_current_span('hello'): + print('Hello World!') + +References +---------- + +* `Azure Monitor `_ +* `OpenTelemetry Project `_ +* `OpenTelemetry Python Client `_ + diff --git a/azure_monitor/README.rst b/azure_monitor/README.rst deleted file mode 100644 index 3a07f4d..0000000 --- a/azure_monitor/README.rst +++ /dev/null @@ -1,19 +0,0 @@ -OpenTelemetry Azure Monitor Exporters -===================================== - -This library is the home of the Azure Monitor Exporters which is an integration for OpenTelemetry. - -Installation ------------- - -:: - - pip install opentelemetry-azure-monitor-exporter - -References ----------- - -* `Azure Monitor `_ -* `OpenTelemetry Project `_ -* `OpenTelemetry Python Client `_ - diff --git a/azure_monitor/examples/traces/trace.py b/azure_monitor/examples/traces/trace.py index ea77507..acc5dc1 100644 --- a/azure_monitor/examples/traces/trace.py +++ b/azure_monitor/examples/traces/trace.py @@ -1,19 +1,22 @@ -# Copyright (c) Microsoft Corporation. All rights reserved. -# Licensed under the MIT License. -# pylint: disable=import-error -# pylint: disable=no-member +from azure_monitor import AzureMonitorSpanExporter from opentelemetry import trace -from opentelemetry.sdk.trace import TracerSource -from opentelemetry.sdk.trace.export import SimpleExportSpanProcessor +from opentelemetry.sdk.trace import TracerProvider +from opentelemetry.sdk.trace.export import BatchExportSpanProcessor -from azure_monitor import AzureMonitorSpanExporter +# Callback function to add os_type: linux to span properties +def callback_function(envelope): + envelope.data.baseData.properties['os_type'] = 'linux' + return True -trace.set_preferred_tracer_source_implementation(lambda T: TracerSource()) -span_processor = SimpleExportSpanProcessor( - AzureMonitorSpanExporter(instrumentation_key="") +exporter = AzureMonitorSpanExporter( + connection_string='InstrumentationKey=99c42f65-1656-4c41-afde-bd86b709a4a7' ) -trace.tracer_source().add_span_processor(span_processor) -tracer = trace.tracer_source().get_tracer(__name__) +exporter.add_telemetry_processor(callback_function) + +trace.set_preferred_tracer_provider_implementation(lambda T: TracerProvider()) +tracer = trace.get_tracer(__name__) +span_processor = BatchExportSpanProcessor(exporter) +trace.tracer_provider().add_span_processor(span_processor) -with tracer.start_as_current_span("hello") as span: - print("Hello, World!") +with tracer.start_as_current_span('hello'): + print('Hello, World!') \ No newline at end of file From d1ed01ced82740b1c67838d8350b95804ae955ae Mon Sep 17 00:00:00 2001 From: Hector Hernandez Date: Tue, 3 Mar 2020 17:57:40 -0800 Subject: [PATCH 050/109] WIP --- azure_monitor/examples/traces/trace.py | 9 +-- azure_monitor/src/azure_monitor/exporter.py | 3 +- azure_monitor/src/azure_monitor/metrics.py | 71 ++++++++++++++++++ azure_monitor/src/azure_monitor/protocol.py | 4 +- azure_monitor/src/azure_monitor/trace.py | 9 +-- azure_monitor/src/azure_monitor/utils.py | 19 +++-- azure_monitor/tests/metrics/__init__.py | 2 + azure_monitor/tests/metrics/test_metrics.py | 80 +++++++++++++++++++++ azure_monitor/tests/test_base_exporter.py | 26 +++++++ azure_monitor/tests/test_utils.py | 20 ++++++ azure_monitor/tests/trace/test_trace.py | 20 ++++-- 11 files changed, 242 insertions(+), 21 deletions(-) create mode 100644 azure_monitor/src/azure_monitor/metrics.py create mode 100644 azure_monitor/tests/metrics/__init__.py create mode 100644 azure_monitor/tests/metrics/test_metrics.py diff --git a/azure_monitor/examples/traces/trace.py b/azure_monitor/examples/traces/trace.py index acc5dc1..688f5b7 100644 --- a/azure_monitor/examples/traces/trace.py +++ b/azure_monitor/examples/traces/trace.py @@ -5,11 +5,12 @@ # Callback function to add os_type: linux to span properties def callback_function(envelope): - envelope.data.baseData.properties['os_type'] = 'linux' + envelope.data.baseData.properties["os_type"] = "linux" return True + exporter = AzureMonitorSpanExporter( - connection_string='InstrumentationKey=99c42f65-1656-4c41-afde-bd86b709a4a7' + connection_string="InstrumentationKey=99c42f65-1656-4c41-afde-bd86b709a4a7" ) exporter.add_telemetry_processor(callback_function) @@ -18,5 +19,5 @@ def callback_function(envelope): span_processor = BatchExportSpanProcessor(exporter) trace.tracer_provider().add_span_processor(span_processor) -with tracer.start_as_current_span('hello'): - print('Hello, World!') \ No newline at end of file +with tracer.start_as_current_span("hello"): + print("Hello, World!") diff --git a/azure_monitor/src/azure_monitor/exporter.py b/azure_monitor/src/azure_monitor/exporter.py index dc80f40..9edb4c2 100644 --- a/azure_monitor/src/azure_monitor/exporter.py +++ b/azure_monitor/src/azure_monitor/exporter.py @@ -6,7 +6,6 @@ # pylint: disable=import-error import requests - from azure_monitor import utils from azure_monitor.storage import LocalFileStorage @@ -73,7 +72,7 @@ def _transmit_from_storage(self): else: blob.delete(silent=True) - def _transmit(self, envelopes_to_export): + def _transmit(self, envelopes_to_export) -> utils.ExportResult: """ Transmit the data envelopes to the ingestion service. diff --git a/azure_monitor/src/azure_monitor/metrics.py b/azure_monitor/src/azure_monitor/metrics.py new file mode 100644 index 0000000..11530c6 --- /dev/null +++ b/azure_monitor/src/azure_monitor/metrics.py @@ -0,0 +1,71 @@ +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. +import json +import logging +from typing import Sequence +from urllib.parse import urlparse + +from opentelemetry.metrics import Counter, Measure, Metric +from opentelemetry.sdk.metrics.export import ( + MetricRecord, + MetricsExporter, + MetricsExportResult, +) +from opentelemetry.sdk.util import ns_to_iso_str + +from azure_monitor import protocol, utils +from azure_monitor.exporter import BaseExporter + +logger = logging.getLogger(__name__) + + +class AzureMonitorMetricsExporter(BaseExporter, MetricsExporter): + def __init__(self, **options): + super(AzureMonitorMetricsExporter, self).__init__(**options) + + def export(self, metric_records: Sequence[MetricRecord]) -> MetricsExportResult: + envelopes = map(self.metric_to_envelope, metric_records) + envelopes_to_export = map( + lambda x: x.to_dict(), + tuple(self.apply_telemetry_processors(envelopes)), + ) + try: + result = self._transmit(envelopes_to_export) + if result == MetricsExportResult.FAILED_RETRYABLE: + self.storage.put(envelopes, result) + if len(list(envelopes_to_export)) < self.options.max_batch_size: + self._transmit_from_storage() + return utils.get_metrics_export_result(result) + except Exception: + logger.exception("Exception occurred while exporting the data.") + + def metric_to_envelope( + self, metric_record: MetricRecord + ) -> protocol.Envelope: + + if not metric_record: + return None + envelope = protocol.Envelope( + ikey=self.options.instrumentation_key, + tags=dict(utils.azure_monitor_context), + time=ns_to_iso_str( + metric_record.metric.get_handle( + metric_record.label_set + ).last_update_timestamp + ), + ) + envelope.name = "Microsoft.ApplicationInsights.Metric" + + data_point = protocol.DataPoint( + ns=metric_record.metric.name, + name=metric_record.metric.description, + value=metric_record.aggregator.checkpoint, + ) + + properties = {} + for label_tuple in metric_record.label_set.labels: + properties[label_tuple[0]] = label_tuple[1] + + data = protocol.MetricData(metrics=[data_point], properties=properties) + envelope.data = protocol.Data(base_data=data, base_type="MetricData") + return envelope diff --git a/azure_monitor/src/azure_monitor/protocol.py b/azure_monitor/src/azure_monitor/protocol.py index 5981f6c..0e0421e 100644 --- a/azure_monitor/src/azure_monitor/protocol.py +++ b/azure_monitor/src/azure_monitor/protocol.py @@ -27,8 +27,8 @@ def __init__(self, base_data=None, base_type=None) -> None: def to_dict(self): return { - "baseData": self.base_data.to_dict() if self.base_data else None, - "baseType": self.base_type.to_dict() if self.base_type else None, + "baseData": self.base_data, + "baseType": self.base_type, } diff --git a/azure_monitor/src/azure_monitor/trace.py b/azure_monitor/src/azure_monitor/trace.py index e5b5db1..8afb2b6 100644 --- a/azure_monitor/src/azure_monitor/trace.py +++ b/azure_monitor/src/azure_monitor/trace.py @@ -2,6 +2,7 @@ # Licensed under the MIT License. import json import logging +from typing import Sequence from urllib.parse import urlparse from opentelemetry.sdk.trace.export import SpanExporter, SpanExportResult @@ -19,7 +20,7 @@ class AzureMonitorSpanExporter(BaseExporter, SpanExporter): def __init__(self, **options): super(AzureMonitorSpanExporter, self).__init__(**options) - def export(self, spans) -> SpanExportResult: + def export(self, spans: Sequence[Span]) -> SpanExportResult: envelopes = map(self.span_to_envelope, spans) envelopes_to_export = map( lambda x: x.to_dict(), @@ -27,16 +28,16 @@ def export(self, spans) -> SpanExportResult: ) try: result = self._transmit(envelopes_to_export) - if result == SpanExportResult.FAILED_RETRYABLE: + if result == utils.ExportResult.FAILED_RETRYABLE: self.storage.put(envelopes, result) - if len(envelopes_to_export) < self.options.max_batch_size: + if len(list(envelopes_to_export)) < self.options.max_batch_size: self._transmit_from_storage() return utils.get_trace_export_result(result) except Exception: logger.exception("Exception occurred while exporting the data.") def span_to_envelope( - self, span + self, span: Span ) -> protocol.Envelope: # noqa pylint: disable=too-many-branches if not span: diff --git a/azure_monitor/src/azure_monitor/utils.py b/azure_monitor/src/azure_monitor/utils.py index f584f82..2030612 100644 --- a/azure_monitor/src/azure_monitor/utils.py +++ b/azure_monitor/src/azure_monitor/utils.py @@ -9,10 +9,10 @@ import time from enum import Enum +from opentelemetry.sdk.metrics.export import MetricsExportResult from opentelemetry.sdk.trace.export import SpanExportResult from opentelemetry.sdk.version import __version__ as opentelemetry_version - from azure_monitor.protocol import BaseObject from azure_monitor.version import __version__ as ext_version @@ -60,7 +60,7 @@ class ExportResult(Enum): FAILED_NOT_RETRYABLE = 2 -def get_trace_export_result(result: ExportResult): +def get_trace_export_result(result: ExportResult) -> SpanExportResult: if result == ExportResult.SUCCESS: return SpanExportResult.SUCCESS elif result == ExportResult.FAILED_RETRYABLE: @@ -71,6 +71,17 @@ def get_trace_export_result(result: ExportResult): return None +def get_metrics_export_result(result: ExportResult) -> MetricsExportResult: + if result == ExportResult.SUCCESS: + return MetricsExportResult.SUCCESS + elif result == ExportResult.FAILED_RETRYABLE: + return MetricsExportResult.FAILED_RETRYABLE + elif result == ExportResult.FAILED_NOT_RETRYABLE: + return MetricsExportResult.FAILED_NOT_RETRYABLE + else: + return None + + class Options(BaseObject): __slots__ = ( @@ -88,7 +99,6 @@ class Options(BaseObject): def __init__( self, connection_string=None, - endpoint="https://dc.services.visualstudio.com/v2/track", instrumentation_key=None, max_batch_size=100, storage_maintenance_period=60, @@ -101,9 +111,9 @@ def __init__( ), storage_retention_period=7 * 24 * 60 * 60, timeout=10.0, # networking timeout in seconds + **options, ) -> None: self.connection_string = connection_string - self.endpoint = endpoint self.instrumentation_key = instrumentation_key self.max_batch_size = max_batch_size self.storage_maintenance_period = storage_maintenance_period @@ -111,6 +121,7 @@ def __init__( self.storage_path = storage_path self.storage_retention_period = storage_retention_period self.timeout = timeout + self.endpoint = "" self._initialize() self._validate_instrumentation_key() diff --git a/azure_monitor/tests/metrics/__init__.py b/azure_monitor/tests/metrics/__init__.py new file mode 100644 index 0000000..5b7f7a9 --- /dev/null +++ b/azure_monitor/tests/metrics/__init__.py @@ -0,0 +1,2 @@ +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. diff --git a/azure_monitor/tests/metrics/test_metrics.py b/azure_monitor/tests/metrics/test_metrics.py new file mode 100644 index 0000000..93afec0 --- /dev/null +++ b/azure_monitor/tests/metrics/test_metrics.py @@ -0,0 +1,80 @@ +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. + + +import json +import os +import shutil +import unittest +from unittest import mock + +from opentelemetry import metrics +from opentelemetry.sdk.metrics import Counter, Meter +from opentelemetry.sdk.metrics.export import MetricRecord, MetricsExportResult +from opentelemetry.sdk.metrics.export.aggregate import CounterAggregator + +from azure_monitor.protocol import Envelope +from azure_monitor.metrics import AzureMonitorMetricsExporter +from azure_monitor.utils import ExportResult, Options + + +class TestAzureMetricsExporter(unittest.TestCase): + @classmethod + def setUpClass(self): + os.environ[ + "APPINSIGHTS_INSTRUMENTATIONKEY" + ] = "1234abcd-5678-4efa-8abc-1234567890ab" + + metrics.set_preferred_meter_implementation(lambda _: Meter()) + self._meter = metrics.meter() + self._test_metric = self._meter.create_metric( + "testname", + "testdesc", + "unit", + int, + Counter, + ["environment"], + ) + kvp = {"environment": "staging"} + self._test_label_set = self._meter.get_label_set(kvp) + + def test_constructor(self): + """Test the constructor.""" + exporter = AzureMonitorMetricsExporter( + instrumentation_key="4321abcd-5678-4efa-8abc-1234567890ab" + ) + self.assertIsInstance(exporter.options, Options) + self.assertEqual( + exporter.options.instrumentation_key, + "4321abcd-5678-4efa-8abc-1234567890ab", + ) + + def test_export(self,): + record = MetricRecord( + CounterAggregator(), self._test_label_set, self._test_metric + ) + exporter = AzureMonitorMetricsExporter() + with mock.patch( + "azure_monitor.metrics.AzureMonitorMetricsExporter._transmit" + ) as transmit: # noqa: E501 + transmit.return_value = ExportResult.SUCCESS + result = exporter.export([record]) + self.assertEqual(result, MetricsExportResult.SUCCESS) + + # def test_metric_to_envelope(self): + # self.assertEqual(True, False) + + +class MockResponse(object): + def __init__(self, status_code, text): + self.status_code = status_code + self.text = text + + +class MockTransport(object): + def __init__(self, exporter=None): + self.export_called = False + self.exporter = exporter + + def export(self, datas): + self.export_called = True diff --git a/azure_monitor/tests/test_base_exporter.py b/azure_monitor/tests/test_base_exporter.py index 617b777..a3fd661 100644 --- a/azure_monitor/tests/test_base_exporter.py +++ b/azure_monitor/tests/test_base_exporter.py @@ -6,6 +6,7 @@ from azure_monitor.exporter import BaseExporter from azure_monitor.protocol import Data, Envelope +from azure_monitor.utils import Options # pylint: disable=W0212 @@ -16,6 +17,31 @@ def setUpClass(cls): "APPINSIGHTS_INSTRUMENTATIONKEY" ] = "1234abcd-5678-4efa-8abc-1234567890ab" + def test_constructor(self): + """Test the constructor.""" + base = BaseExporter( + instrumentation_key="4321abcd-5678-4efa-8abc-1234567890ab", + max_batch_size=1, + storage_maintenance_period=2, + storage_max_size=3, + storage_path="testStoragePath", + storage_retention_period=4, + timeout=5, + something_else=6, + ) + self.assertIsInstance(base.options, Options) + self.assertEqual( + base.options.instrumentation_key, + "4321abcd-5678-4efa-8abc-1234567890ab", + ) + self.assertEqual(base.options.max_batch_size, 1) + self.assertEqual(base.options.storage_maintenance_period, 2) + self.assertEqual(base.options.storage_max_size, 3) + self.assertEqual(base.options.storage_retention_period, 4) + self.assertEqual(base.options.timeout, 5) + self.assertEqual(base.options.storage_path, "testStoragePath") + self.assertEqual(getattr(base.options, "something_else", None), None) + def test_telemetry_processor_add(self): base = BaseExporter() base.add_telemetry_processor(lambda: True) diff --git a/azure_monitor/tests/test_utils.py b/azure_monitor/tests/test_utils.py index d236e79..a2e2152 100644 --- a/azure_monitor/tests/test_utils.py +++ b/azure_monitor/tests/test_utils.py @@ -4,6 +4,7 @@ import os import unittest +from opentelemetry.sdk.metrics.export import MetricsExportResult from opentelemetry.sdk.trace.export import SpanExportResult from azure_monitor import utils @@ -281,3 +282,22 @@ def test_get_trace_export_result(self): SpanExportResult.FAILED_RETRYABLE, ) self.assertEqual(utils.get_trace_export_result(None), None) + + def test_get_metrics_export_result(self): + self.assertEqual( + utils.get_metrics_export_result(utils.ExportResult.SUCCESS), + MetricsExportResult.SUCCESS, + ) + self.assertEqual( + utils.get_metrics_export_result( + utils.ExportResult.FAILED_NOT_RETRYABLE + ), + MetricsExportResult.FAILED_NOT_RETRYABLE, + ) + self.assertEqual( + utils.get_metrics_export_result( + utils.ExportResult.FAILED_RETRYABLE + ), + MetricsExportResult.FAILED_RETRYABLE, + ) + self.assertEqual(utils.get_metrics_export_result(None), None) diff --git a/azure_monitor/tests/trace/test_trace.py b/azure_monitor/tests/trace/test_trace.py index ef86a4b..9e94944 100644 --- a/azure_monitor/tests/trace/test_trace.py +++ b/azure_monitor/tests/trace/test_trace.py @@ -10,13 +10,12 @@ # pylint: disable=import-error from opentelemetry.sdk.trace import Span -from opentelemetry.sdk.trace.export import SpanExportResult from opentelemetry.trace import Link, SpanContext, SpanKind from opentelemetry.trace.status import Status, StatusCanonicalCode from azure_monitor.protocol import Envelope from azure_monitor.trace import AzureMonitorSpanExporter -from azure_monitor.utils import Options +from azure_monitor.utils import ExportResult,Options TEST_FOLDER = os.path.abspath(".test.exporter") @@ -40,12 +39,23 @@ def func(*_args, **_kwargs): class TestAzureExporter(unittest.TestCase): @classmethod def setUpClass(self): + os.environ.clear() os.environ[ "APPINSIGHTS_INSTRUMENTATIONKEY" ] = "1234abcd-5678-4efa-8abc-1234567890ab" - @mock.patch("requests.post", return_value=mock.Mock()) - def test_export_empty(self, request_mock): + def test_constructor(self): + """Test the constructor.""" + exporter = AzureMonitorSpanExporter( + instrumentation_key="4321abcd-5678-4efa-8abc-1234567890ab" + ) + self.assertIsInstance(exporter.options, Options) + self.assertEqual( + exporter.options.instrumentation_key, + "4321abcd-5678-4efa-8abc-1234567890ab", + ) + + def test_export_empty(self): exporter = AzureMonitorSpanExporter( storage_path=os.path.join(TEST_FOLDER, self.id()) ) @@ -80,7 +90,7 @@ def test_export_failure(self, span_to_envelope_mock): ) test_span.start() test_span.end() - transmit.return_value = SpanExportResult.FAILED_RETRYABLE + transmit.return_value = ExportResult.FAILED_RETRYABLE exporter.export([test_span]) self.assertEqual(len(os.listdir(exporter.storage.path)), 1) self.assertIsNone(exporter.storage.get()) From 838fc8d28c5ae3354303386ce63d0af85cfea492 Mon Sep 17 00:00:00 2001 From: Hector Hernandez Date: Wed, 4 Mar 2020 13:00:49 -0800 Subject: [PATCH 051/109] Adding Metrics exporter Adding typing Adding args docs --- azure_monitor/examples/traces/trace.py | 12 +- azure_monitor/src/azure_monitor/exporter.py | 22 +- azure_monitor/src/azure_monitor/metrics.py | 74 ++++++ azure_monitor/src/azure_monitor/protocol.py | 263 +++++++++++++++----- azure_monitor/src/azure_monitor/trace.py | 9 +- azure_monitor/src/azure_monitor/utils.py | 19 +- azure_monitor/tests/metrics/__init__.py | 2 + azure_monitor/tests/metrics/test_metrics.py | 116 +++++++++ azure_monitor/tests/test_base_exporter.py | 26 ++ azure_monitor/tests/test_utils.py | 20 ++ azure_monitor/tests/trace/test_trace.py | 20 +- 11 files changed, 493 insertions(+), 90 deletions(-) create mode 100644 azure_monitor/src/azure_monitor/metrics.py create mode 100644 azure_monitor/tests/metrics/__init__.py create mode 100644 azure_monitor/tests/metrics/test_metrics.py diff --git a/azure_monitor/examples/traces/trace.py b/azure_monitor/examples/traces/trace.py index acc5dc1..8dc1324 100644 --- a/azure_monitor/examples/traces/trace.py +++ b/azure_monitor/examples/traces/trace.py @@ -1,15 +1,17 @@ -from azure_monitor import AzureMonitorSpanExporter from opentelemetry import trace from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import BatchExportSpanProcessor +from azure_monitor import AzureMonitorSpanExporter + # Callback function to add os_type: linux to span properties def callback_function(envelope): - envelope.data.baseData.properties['os_type'] = 'linux' + envelope.data.baseData.properties["os_type"] = "linux" return True + exporter = AzureMonitorSpanExporter( - connection_string='InstrumentationKey=99c42f65-1656-4c41-afde-bd86b709a4a7' + connection_string="InstrumentationKey=99c42f65-1656-4c41-afde-bd86b709a4a7" ) exporter.add_telemetry_processor(callback_function) @@ -18,5 +20,5 @@ def callback_function(envelope): span_processor = BatchExportSpanProcessor(exporter) trace.tracer_provider().add_span_processor(span_processor) -with tracer.start_as_current_span('hello'): - print('Hello, World!') \ No newline at end of file +with tracer.start_as_current_span("hello"): + print("Hello, World!") diff --git a/azure_monitor/src/azure_monitor/exporter.py b/azure_monitor/src/azure_monitor/exporter.py index dc80f40..1145ecc 100644 --- a/azure_monitor/src/azure_monitor/exporter.py +++ b/azure_monitor/src/azure_monitor/exporter.py @@ -2,12 +2,12 @@ # Licensed under the MIT License. import json import logging +import typing -# pylint: disable=import-error import requests - from azure_monitor import utils +from azure_monitor.protocol import Envelope from azure_monitor.storage import LocalFileStorage logger = logging.getLogger(__name__) @@ -24,7 +24,9 @@ def __init__(self, **options): retention_period=self.options.storage_retention_period, ) - def add_telemetry_processor(self, processor): + def add_telemetry_processor( + self, processor: typing.Callable[..., any] + ) -> None: """Adds telemetry processor to the collection. Telemetry processors will be called one by one before telemetry @@ -33,11 +35,13 @@ def add_telemetry_processor(self, processor): """ self._telemetry_processors.append(processor) - def clear_telemetry_processors(self): + def clear_telemetry_processors(self) -> None: """Removes all telemetry processors""" self._telemetry_processors = [] - def apply_telemetry_processors(self, envelopes): + def apply_telemetry_processors( + self, envelopes: typing.List[Envelope] + ) -> typing.List[Envelope]: """Applies all telemetry processors in the order they were added. This function will return the list of envelopes to be exported after @@ -61,7 +65,7 @@ def apply_telemetry_processors(self, envelopes): filtered_envelopes.append(envelope) return filtered_envelopes - def _transmit_from_storage(self): + def _transmit_from_storage(self) -> None: for blob in self.storage.gets(): # give a few more seconds for blob lease operation # to reduce the chance of race (for perf consideration) @@ -73,14 +77,16 @@ def _transmit_from_storage(self): else: blob.delete(silent=True) - def _transmit(self, envelopes_to_export): + def _transmit( + self, envelopes_to_export: typing.List[Envelope] + ) -> utils.ExportResult: """ Transmit the data envelopes to the ingestion service. Returns an ExportResult, this function should never throw an exception. """ - if envelopes_to_export: + if len(envelopes_to_export) > 0: try: response = requests.post( url=self.options.endpoint, diff --git a/azure_monitor/src/azure_monitor/metrics.py b/azure_monitor/src/azure_monitor/metrics.py new file mode 100644 index 0000000..29d4add --- /dev/null +++ b/azure_monitor/src/azure_monitor/metrics.py @@ -0,0 +1,74 @@ +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. +import json +import logging +from typing import Sequence +from urllib.parse import urlparse + +from opentelemetry.metrics import Counter, Measure, Metric +from opentelemetry.sdk.metrics.export import ( + MetricRecord, + MetricsExporter, + MetricsExportResult, +) +from opentelemetry.sdk.util import ns_to_iso_str + +from azure_monitor import protocol, utils +from azure_monitor.exporter import BaseExporter + +logger = logging.getLogger(__name__) + + +class AzureMonitorMetricsExporter(BaseExporter, MetricsExporter): + def __init__(self, **options): + super(AzureMonitorMetricsExporter, self).__init__(**options) + + def export( + self, metric_records: Sequence[MetricRecord] + ) -> MetricsExportResult: + envelopes = map(self.metric_to_envelope, metric_records) + envelopes_to_export = map( + lambda x: x.to_dict(), + tuple(self.apply_telemetry_processors(envelopes)), + ) + try: + result = self._transmit(envelopes_to_export) + if result == MetricsExportResult.FAILED_RETRYABLE: + self.storage.put(envelopes, result) + if len(list(envelopes_to_export)) < self.options.max_batch_size: + self._transmit_from_storage() + return utils.get_metrics_export_result(result) + except Exception: + logger.exception("Exception occurred while exporting the data.") + + def metric_to_envelope( + self, metric_record: MetricRecord + ) -> protocol.Envelope: + + if not metric_record: + return None + envelope = protocol.Envelope( + ikey=self.options.instrumentation_key, + tags=dict(utils.azure_monitor_context), + time=ns_to_iso_str( + metric_record.metric.get_handle( + metric_record.label_set + ).last_update_timestamp + ), + ) + envelope.name = "Microsoft.ApplicationInsights.Metric" + + data_point = protocol.DataPoint( + ns=metric_record.metric.name, + name=metric_record.metric.description, + value=metric_record.aggregator.checkpoint, + kind=protocol.DataPointType.MEASUREMENT, + ) + + properties = {} + for label_tuple in metric_record.label_set.labels: + properties[label_tuple[0]] = label_tuple[1] + + data = protocol.MetricData(metrics=[data_point], properties=properties) + envelope.data = protocol.Data(base_data=data, base_type="MetricData") + return envelope diff --git a/azure_monitor/src/azure_monitor/protocol.py b/azure_monitor/src/azure_monitor/protocol.py index 5981f6c..9222242 100644 --- a/azure_monitor/src/azure_monitor/protocol.py +++ b/azure_monitor/src/azure_monitor/protocol.py @@ -1,6 +1,9 @@ # Copyright (c) Microsoft Corporation. All rights reserved. # Licensed under the MIT License. +import typing +from enum import Enum + class BaseObject: __slots__ = () @@ -19,6 +22,13 @@ def __repr__(self): class Data(BaseObject): + """Data + + Args: + base_data: Data base data. + base_type: Data base type. + """ + __slots__ = ("base_data", "base_type") def __init__(self, base_data=None, base_type=None) -> None: @@ -26,13 +36,28 @@ def __init__(self, base_data=None, base_type=None) -> None: self.base_type = base_type def to_dict(self): - return { - "baseData": self.base_data.to_dict() if self.base_data else None, - "baseType": self.base_type.to_dict() if self.base_type else None, - } + return {"baseData": self.base_data, "baseType": self.base_type} + + +class DataPointType(Enum): + MEASUREMENT = 0 + AGGREGATION = 1 class DataPoint(BaseObject): + """Metric data single measurement. + + Args: + ns: Namespace of the metric + name: Name of the metric. + kind: Metric type. Single measurement or the aggregated value. + value: Single value for measurement. Sum of individual measurements for the aggregation. + count: Metric weight of the aggregated metric. Should not be set for a measurement. + min: Minimum value of the aggregated metric. Should not be set for a measurement. + max: Maximum value of the aggregated metric. Should not be set for a measurement. + std_dev: Standard deviation of the aggregated metric. Should not be set for a measurement. + """ + __slots__ = ( "ns", "name", @@ -46,14 +71,14 @@ class DataPoint(BaseObject): def __init__( self, - ns="", - name="", - kind=None, - value=0.0, - count=None, - min=None, - max=None, - std_dev=None, + ns: str = "", + name: str = "", + kind: DataPointType = None, + value: float = 0.0, + count: float = None, + min: float = None, + max: float = None, + std_dev: float = None, ) -> None: self.ns = ns self.name = name @@ -78,6 +103,26 @@ def to_dict(self): class Envelope(BaseObject): + """Envelope represents a telemetry item + + Args: + ver: Envelope version. For internal use only. By assigning this the default, + it will not be serialized within the payload unless changed to a value other + than #1. + name: Type name of telemetry data item. + time: Event date time when telemetry item was created. This is the wall clock + time on the client when the event was generated. + There is no guarantee that the client's time is accurate. This field must be + formatted in UTC ISO 8601 format + sample_rate: Sampling rate used in application. This telemetry item represents + 1 / sampleRate actual telemetry items. + seq: Sequence field used to track absolute order of uploaded events. + ikey: The application's instrumentation key. + flags: Key/value collection of flags. + tags: Key/value collection of context properties. + data: Telemetry data item. + """ + __slots__ = ( "ver", "name", @@ -92,15 +137,15 @@ class Envelope(BaseObject): def __init__( self, - ver=1, - name="", - time="", - sample_rate=None, - seq=None, - ikey=None, - flags=None, - tags=None, - data=None, + ver: int = 1, + name: str = "", + time: str = "", + sample_rate: int = None, + seq: str = None, + ikey: str = None, + flags: typing.Dict = None, + tags: typing.Dict = None, + data: Data = None, ) -> None: self.ver = ver self.name = name @@ -128,9 +173,27 @@ def to_dict(self): class Event(BaseObject): + """Instances of Event represent structured event records that can be grouped + and searched by their properties. Event data item also creates a metric of + event count by name. + + Args: + ver: Schema version. + name: Event name. Keep it low cardinality to allow proper grouping and + useful metrics. + properties: Collection of custom properties. + measurements: Collection of custom measurements. + """ + __slots__ = ("ver", "name", "properties", "measurements") - def __init__(self, ver=2, name="", properties=None, measurements=None): + def __init__( + self, + ver: int = 2, + name: str = "", + properties: typing.Dict[str, any] = None, + measurements: any = None, + ): self.ver = ver self.name = name self.properties = properties @@ -146,30 +209,39 @@ def to_dict(self): class ExceptionData(BaseObject): + """An instance of Exception represents a handled or unhandled exception that + occurred during execution of the monitored application. + + Args: + ver: Schema version. + exceptions: Exception chain - list of inner exceptions. + severity_level: Severity level. Mostly used to indicate exception severity + level when it is reported by logging library. + properties: Collection of custom properties. + measurements: Collection of custom measurements. + """ + __slots__ = ( "ver", "exceptions", "severity_level", - "problem_id", "properties", "measurements", ) def __init__( self, - ver=2, - exceptions=None, - severity_level=None, - problem_id=None, - properties=None, - measurements=None, + ver: int = 2, + exceptions: any = None, + severity_level: int = None, + properties: typing.Dict[str, any] = None, + measurements: any = None, ) -> None: if exceptions is None: exceptions = [] self.ver = ver self.exceptions = exceptions self.severity_level = severity_level - self.problem_id = problem_id self.properties = properties self.measurements = measurements @@ -185,27 +257,29 @@ def to_dict(self): class Message(BaseObject): - __slots__ = ( - "ver", - "message", - "severity_level", - "properties", - "measurements", - ) + """Instances of Message represent printf-like trace statements that are + text-searched. The message does not have measurements. + + Args: + ver: Schema version. + message: Trace message. + severity_level: Trace severity level. + properties: Collection of custom properties. + """ + + __slots__ = ("ver", "message", "severity_level", "properties") def __init__( self, - ver=2, - message="", + ver: int = 2, + message: str = "", severity_level=None, - properties=None, - measurements=None, + properties: typing.Dict[str, any] = None, ) -> None: self.ver = ver self.message = message self.severity_level = severity_level self.properties = properties - self.measurements = measurements def to_dict(self): return { @@ -218,9 +292,25 @@ def to_dict(self): class MetricData(BaseObject): + """An instance of the Metric item is a list of measurements (single data points) + and/or aggregations. + + Args: + ver: Data base data. + metrics: List of metrics. Only one metric in the list is currently supported + by Application Insights storage. If multiple data points were sent only the + first one will be used. + properties:Collection of custom properties. + """ + __slots__ = ("ver", "metrics", "properties") - def __init__(self, ver=2, metrics=None, properties=None) -> None: + def __init__( + self, + ver: int = 2, + metrics: typing.List[DataPoint] = None, + properties: typing.Dict[str, any] = None, + ) -> None: if metrics is None: metrics = [] self.ver = ver @@ -236,6 +326,29 @@ def to_dict(self): class RemoteDependency(BaseObject): + """An instance of Remote Dependency represents an interaction of the monitored component + with a remote component/service like SQL or an HTTP endpoint. + + Args: + ver: Schema version. + name: Name of the command initiated with this dependency call. Low cardinality value. + Examples are stored procedure name and URL path template. + id: Identifier of a dependency call instance. Used for correlation with the request + telemetry item corresponding to this dependency call. + result_code: Result code of a dependency call. Examples are SQL error code and HTTP + status code. + duration: Request duration in format: DD.HH:MM:SS.MMMMMM. Must be less than 1000 days. + success: Indication of successfull or unsuccessfull call. + data: Command initiated by this dependency call. Examples are SQL statement and HTTP + URL's with all query parameters. + type: Dependency type name. Very low cardinality value for logical grouping of + dependencies and interpretation of other fields like commandName and resultCode. + Examples are SQL, Azure table, and HTTP. + target: Target site of a dependency call. Examples are server name, host address. + properties: Collection of custom properties. + measurements: Collection of custom measurements. + """ + __slots__ = ( "ver", "name", @@ -252,17 +365,17 @@ class RemoteDependency(BaseObject): def __init__( self, - ver=2, - name="", - id="", - result_code="", - duration="", - success=True, - data=None, - type=None, - target=None, - properties=None, - measurements=None, + ver: int = 2, + name: str = "", + id: str = "", + result_code: str = "", + duration: str = "", + success: bool = True, + data: Data = None, + type: str = None, + target: str = None, + properties: typing.Dict[str, any] = None, + measurements: any = None, ) -> None: self.ver = ver self.name = name @@ -293,6 +406,28 @@ def to_dict(self): class Request(BaseObject): + """An instance of Request represents completion of an external request to the + application to do work and contains a summary of that request execution and the + results. + + Args: + ver: Schema version. + id: Identifier of a request call instance. Used for correlation between request + and other telemetry items. + duration: Request duration in format: DD.HH:MM:SS.MMMMMM. Must be less than 1000 + days. + response_code: Response code from Request + success: Indication of successfull or unsuccessfull call. + source: Source of the request. Examples are the instrumentation key of the caller + or the ip address of the caller. + name: Name of the request. Represents code path taken to process request. Low + cardinality value to allow better grouping of requests. For HTTP requests it + represents the HTTP method and URL path template like 'GET /values/{id}'. + url: Request URL with all query string parameters. + properties: Collection of custom properties. + measurements: Collection of custom measurements. + """ + __slots__ = ( "ver", "id", @@ -308,16 +443,16 @@ class Request(BaseObject): def __init__( self, - ver=2, - id="", - duration="", - response_code="", - success=True, - source=None, - name=None, - url=None, - properties=None, - measurements=None, + ver: int = 2, + id: str = "", + duration: str = "", + response_code: str = "", + success: bool = True, + source: str = None, + name: str = None, + url: str = None, + properties: typing.Dict[str, any] = None, + measurements: any = None, ) -> None: self.ver = ver self.id = id diff --git a/azure_monitor/src/azure_monitor/trace.py b/azure_monitor/src/azure_monitor/trace.py index e5b5db1..8afb2b6 100644 --- a/azure_monitor/src/azure_monitor/trace.py +++ b/azure_monitor/src/azure_monitor/trace.py @@ -2,6 +2,7 @@ # Licensed under the MIT License. import json import logging +from typing import Sequence from urllib.parse import urlparse from opentelemetry.sdk.trace.export import SpanExporter, SpanExportResult @@ -19,7 +20,7 @@ class AzureMonitorSpanExporter(BaseExporter, SpanExporter): def __init__(self, **options): super(AzureMonitorSpanExporter, self).__init__(**options) - def export(self, spans) -> SpanExportResult: + def export(self, spans: Sequence[Span]) -> SpanExportResult: envelopes = map(self.span_to_envelope, spans) envelopes_to_export = map( lambda x: x.to_dict(), @@ -27,16 +28,16 @@ def export(self, spans) -> SpanExportResult: ) try: result = self._transmit(envelopes_to_export) - if result == SpanExportResult.FAILED_RETRYABLE: + if result == utils.ExportResult.FAILED_RETRYABLE: self.storage.put(envelopes, result) - if len(envelopes_to_export) < self.options.max_batch_size: + if len(list(envelopes_to_export)) < self.options.max_batch_size: self._transmit_from_storage() return utils.get_trace_export_result(result) except Exception: logger.exception("Exception occurred while exporting the data.") def span_to_envelope( - self, span + self, span: Span ) -> protocol.Envelope: # noqa pylint: disable=too-many-branches if not span: diff --git a/azure_monitor/src/azure_monitor/utils.py b/azure_monitor/src/azure_monitor/utils.py index f584f82..2030612 100644 --- a/azure_monitor/src/azure_monitor/utils.py +++ b/azure_monitor/src/azure_monitor/utils.py @@ -9,10 +9,10 @@ import time from enum import Enum +from opentelemetry.sdk.metrics.export import MetricsExportResult from opentelemetry.sdk.trace.export import SpanExportResult from opentelemetry.sdk.version import __version__ as opentelemetry_version - from azure_monitor.protocol import BaseObject from azure_monitor.version import __version__ as ext_version @@ -60,7 +60,7 @@ class ExportResult(Enum): FAILED_NOT_RETRYABLE = 2 -def get_trace_export_result(result: ExportResult): +def get_trace_export_result(result: ExportResult) -> SpanExportResult: if result == ExportResult.SUCCESS: return SpanExportResult.SUCCESS elif result == ExportResult.FAILED_RETRYABLE: @@ -71,6 +71,17 @@ def get_trace_export_result(result: ExportResult): return None +def get_metrics_export_result(result: ExportResult) -> MetricsExportResult: + if result == ExportResult.SUCCESS: + return MetricsExportResult.SUCCESS + elif result == ExportResult.FAILED_RETRYABLE: + return MetricsExportResult.FAILED_RETRYABLE + elif result == ExportResult.FAILED_NOT_RETRYABLE: + return MetricsExportResult.FAILED_NOT_RETRYABLE + else: + return None + + class Options(BaseObject): __slots__ = ( @@ -88,7 +99,6 @@ class Options(BaseObject): def __init__( self, connection_string=None, - endpoint="https://dc.services.visualstudio.com/v2/track", instrumentation_key=None, max_batch_size=100, storage_maintenance_period=60, @@ -101,9 +111,9 @@ def __init__( ), storage_retention_period=7 * 24 * 60 * 60, timeout=10.0, # networking timeout in seconds + **options, ) -> None: self.connection_string = connection_string - self.endpoint = endpoint self.instrumentation_key = instrumentation_key self.max_batch_size = max_batch_size self.storage_maintenance_period = storage_maintenance_period @@ -111,6 +121,7 @@ def __init__( self.storage_path = storage_path self.storage_retention_period = storage_retention_period self.timeout = timeout + self.endpoint = "" self._initialize() self._validate_instrumentation_key() diff --git a/azure_monitor/tests/metrics/__init__.py b/azure_monitor/tests/metrics/__init__.py new file mode 100644 index 0000000..5b7f7a9 --- /dev/null +++ b/azure_monitor/tests/metrics/__init__.py @@ -0,0 +1,2 @@ +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. diff --git a/azure_monitor/tests/metrics/test_metrics.py b/azure_monitor/tests/metrics/test_metrics.py new file mode 100644 index 0000000..37f64b5 --- /dev/null +++ b/azure_monitor/tests/metrics/test_metrics.py @@ -0,0 +1,116 @@ +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. + + +import json +import os +import shutil +import unittest +from unittest import mock + +from opentelemetry import metrics +from opentelemetry.sdk.metrics import Counter, Meter +from opentelemetry.sdk.metrics.export import MetricRecord, MetricsExportResult +from opentelemetry.sdk.metrics.export.aggregate import CounterAggregator +from opentelemetry.sdk.util import ns_to_iso_str + +from azure_monitor.metrics import AzureMonitorMetricsExporter +from azure_monitor.protocol import Data, Envelope, MetricData +from azure_monitor.utils import ExportResult, Options + + +class TestAzureMetricsExporter(unittest.TestCase): + @classmethod + def setUpClass(self): + os.environ[ + "APPINSIGHTS_INSTRUMENTATIONKEY" + ] = "1234abcd-5678-4efa-8abc-1234567890ab" + + metrics.set_preferred_meter_implementation(lambda _: Meter()) + self._meter = metrics.meter() + self._test_metric = self._meter.create_metric( + "testname", "testdesc", "unit", int, Counter, ["environment"] + ) + kvp = {"environment": "staging"} + self._test_label_set = self._meter.get_label_set(kvp) + + def test_constructor(self): + """Test the constructor.""" + exporter = AzureMonitorMetricsExporter( + instrumentation_key="4321abcd-5678-4efa-8abc-1234567890ab" + ) + self.assertIsInstance(exporter.options, Options) + self.assertEqual( + exporter.options.instrumentation_key, + "4321abcd-5678-4efa-8abc-1234567890ab", + ) + + def test_export(self,): + record = MetricRecord( + CounterAggregator(), self._test_label_set, self._test_metric + ) + exporter = AzureMonitorMetricsExporter() + with mock.patch( + "azure_monitor.metrics.AzureMonitorMetricsExporter._transmit" + ) as transmit: # noqa: E501 + transmit.return_value = ExportResult.SUCCESS + result = exporter.export([record]) + self.assertEqual(result, MetricsExportResult.SUCCESS) + + def test_metric_to_envelope(self): + aggregator = CounterAggregator() + aggregator.update(123) + aggregator.take_checkpoint() + record = MetricRecord( + aggregator, self._test_label_set, self._test_metric + ) + exporter = AzureMonitorMetricsExporter() + envelope = exporter.metric_to_envelope(record) + self.assertIsInstance(envelope, Envelope) + self.assertEqual(envelope.ver, 1) + self.assertEqual(envelope.name, "Microsoft.ApplicationInsights.Metric") + self.assertEqual( + envelope.time, + ns_to_iso_str( + record.metric.get_handle( + record.label_set + ).last_update_timestamp + ), + ) + self.assertEqual(envelope.sample_rate, None) + self.assertEqual(envelope.seq, None) + self.assertEqual(envelope.ikey, "1234abcd-5678-4efa-8abc-1234567890ab") + self.assertEqual(envelope.flags, None) + + self.assertIsInstance(envelope.data, Data) + self.assertIsInstance(envelope.data.base_data, MetricData) + self.assertEqual(envelope.data.base_data.ver, 2) + self.assertEqual(len(envelope.data.base_data.metrics), 1) + self.assertEqual(envelope.data.base_data.metrics[0].ns, "testname") + self.assertEqual(envelope.data.base_data.metrics[0].name, "testdesc") + self.assertEqual(envelope.data.base_data.metrics[0].value, 123) + self.assertEqual( + envelope.data.base_data.properties["environment"], "staging" + ) + self.assertIsNotNone(envelope.tags["ai.cloud.role"]) + self.assertIsNotNone(envelope.tags["ai.cloud.roleInstance"]) + self.assertIsNotNone(envelope.tags["ai.device.id"]) + self.assertIsNotNone(envelope.tags["ai.device.locale"]) + self.assertIsNotNone(envelope.tags["ai.device.osVersion"]) + self.assertIsNotNone(envelope.tags["ai.device.type"]) + self.assertIsNotNone(envelope.tags["ai.internal.sdkVersion"]) + + +class MockResponse(object): + def __init__(self, status_code, text): + self.status_code = status_code + self.text = text + + +class MockTransport(object): + def __init__(self, exporter=None): + self.export_called = False + self.exporter = exporter + + def export(self, datas): + self.export_called = True diff --git a/azure_monitor/tests/test_base_exporter.py b/azure_monitor/tests/test_base_exporter.py index 617b777..a3fd661 100644 --- a/azure_monitor/tests/test_base_exporter.py +++ b/azure_monitor/tests/test_base_exporter.py @@ -6,6 +6,7 @@ from azure_monitor.exporter import BaseExporter from azure_monitor.protocol import Data, Envelope +from azure_monitor.utils import Options # pylint: disable=W0212 @@ -16,6 +17,31 @@ def setUpClass(cls): "APPINSIGHTS_INSTRUMENTATIONKEY" ] = "1234abcd-5678-4efa-8abc-1234567890ab" + def test_constructor(self): + """Test the constructor.""" + base = BaseExporter( + instrumentation_key="4321abcd-5678-4efa-8abc-1234567890ab", + max_batch_size=1, + storage_maintenance_period=2, + storage_max_size=3, + storage_path="testStoragePath", + storage_retention_period=4, + timeout=5, + something_else=6, + ) + self.assertIsInstance(base.options, Options) + self.assertEqual( + base.options.instrumentation_key, + "4321abcd-5678-4efa-8abc-1234567890ab", + ) + self.assertEqual(base.options.max_batch_size, 1) + self.assertEqual(base.options.storage_maintenance_period, 2) + self.assertEqual(base.options.storage_max_size, 3) + self.assertEqual(base.options.storage_retention_period, 4) + self.assertEqual(base.options.timeout, 5) + self.assertEqual(base.options.storage_path, "testStoragePath") + self.assertEqual(getattr(base.options, "something_else", None), None) + def test_telemetry_processor_add(self): base = BaseExporter() base.add_telemetry_processor(lambda: True) diff --git a/azure_monitor/tests/test_utils.py b/azure_monitor/tests/test_utils.py index d236e79..a2e2152 100644 --- a/azure_monitor/tests/test_utils.py +++ b/azure_monitor/tests/test_utils.py @@ -4,6 +4,7 @@ import os import unittest +from opentelemetry.sdk.metrics.export import MetricsExportResult from opentelemetry.sdk.trace.export import SpanExportResult from azure_monitor import utils @@ -281,3 +282,22 @@ def test_get_trace_export_result(self): SpanExportResult.FAILED_RETRYABLE, ) self.assertEqual(utils.get_trace_export_result(None), None) + + def test_get_metrics_export_result(self): + self.assertEqual( + utils.get_metrics_export_result(utils.ExportResult.SUCCESS), + MetricsExportResult.SUCCESS, + ) + self.assertEqual( + utils.get_metrics_export_result( + utils.ExportResult.FAILED_NOT_RETRYABLE + ), + MetricsExportResult.FAILED_NOT_RETRYABLE, + ) + self.assertEqual( + utils.get_metrics_export_result( + utils.ExportResult.FAILED_RETRYABLE + ), + MetricsExportResult.FAILED_RETRYABLE, + ) + self.assertEqual(utils.get_metrics_export_result(None), None) diff --git a/azure_monitor/tests/trace/test_trace.py b/azure_monitor/tests/trace/test_trace.py index ef86a4b..a0ae717 100644 --- a/azure_monitor/tests/trace/test_trace.py +++ b/azure_monitor/tests/trace/test_trace.py @@ -10,13 +10,12 @@ # pylint: disable=import-error from opentelemetry.sdk.trace import Span -from opentelemetry.sdk.trace.export import SpanExportResult from opentelemetry.trace import Link, SpanContext, SpanKind from opentelemetry.trace.status import Status, StatusCanonicalCode from azure_monitor.protocol import Envelope from azure_monitor.trace import AzureMonitorSpanExporter -from azure_monitor.utils import Options +from azure_monitor.utils import ExportResult, Options TEST_FOLDER = os.path.abspath(".test.exporter") @@ -40,12 +39,23 @@ def func(*_args, **_kwargs): class TestAzureExporter(unittest.TestCase): @classmethod def setUpClass(self): + os.environ.clear() os.environ[ "APPINSIGHTS_INSTRUMENTATIONKEY" ] = "1234abcd-5678-4efa-8abc-1234567890ab" - @mock.patch("requests.post", return_value=mock.Mock()) - def test_export_empty(self, request_mock): + def test_constructor(self): + """Test the constructor.""" + exporter = AzureMonitorSpanExporter( + instrumentation_key="4321abcd-5678-4efa-8abc-1234567890ab" + ) + self.assertIsInstance(exporter.options, Options) + self.assertEqual( + exporter.options.instrumentation_key, + "4321abcd-5678-4efa-8abc-1234567890ab", + ) + + def test_export_empty(self): exporter = AzureMonitorSpanExporter( storage_path=os.path.join(TEST_FOLDER, self.id()) ) @@ -80,7 +90,7 @@ def test_export_failure(self, span_to_envelope_mock): ) test_span.start() test_span.end() - transmit.return_value = SpanExportResult.FAILED_RETRYABLE + transmit.return_value = ExportResult.FAILED_RETRYABLE exporter.export([test_span]) self.assertEqual(len(os.listdir(exporter.storage.path)), 1) self.assertIsNone(exporter.storage.get()) From 82d7a393b71ef43bdeb0a9da1efcd675d485e003 Mon Sep 17 00:00:00 2001 From: Hector Hernandez Date: Thu, 5 Mar 2020 11:12:23 -0800 Subject: [PATCH 052/109] iKey example update --- azure_monitor/examples/traces/trace.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/azure_monitor/examples/traces/trace.py b/azure_monitor/examples/traces/trace.py index 8dc1324..2698c78 100644 --- a/azure_monitor/examples/traces/trace.py +++ b/azure_monitor/examples/traces/trace.py @@ -11,7 +11,7 @@ def callback_function(envelope): exporter = AzureMonitorSpanExporter( - connection_string="InstrumentationKey=99c42f65-1656-4c41-afde-bd86b709a4a7" + connection_string="InstrumentationKey=" ) exporter.add_telemetry_processor(callback_function) From 4cfcc39ca84052f07073e47d87bd0ab98950d654 Mon Sep 17 00:00:00 2001 From: Hector Hernandez Date: Thu, 5 Mar 2020 13:23:40 -0800 Subject: [PATCH 053/109] Addressing comments --- azure_monitor/src/azure_monitor/metrics.py | 3 +- azure_monitor/src/azure_monitor/protocol.py | 101 ++++++++++++++++++-- azure_monitor/src/azure_monitor/trace.py | 3 +- azure_monitor/src/azure_monitor/utils.py | 35 ++++--- 4 files changed, 117 insertions(+), 25 deletions(-) diff --git a/azure_monitor/src/azure_monitor/metrics.py b/azure_monitor/src/azure_monitor/metrics.py index 29d4add..00d2534 100644 --- a/azure_monitor/src/azure_monitor/metrics.py +++ b/azure_monitor/src/azure_monitor/metrics.py @@ -35,7 +35,8 @@ def export( result = self._transmit(envelopes_to_export) if result == MetricsExportResult.FAILED_RETRYABLE: self.storage.put(envelopes, result) - if len(list(envelopes_to_export)) < self.options.max_batch_size: + if result == utils.ExportResult.SUCCESS: + # Try to send any cached events self._transmit_from_storage() return utils.get_metrics_export_result(result) except Exception: diff --git a/azure_monitor/src/azure_monitor/protocol.py b/azure_monitor/src/azure_monitor/protocol.py index 9222242..aaea169 100644 --- a/azure_monitor/src/azure_monitor/protocol.py +++ b/azure_monitor/src/azure_monitor/protocol.py @@ -25,13 +25,14 @@ class Data(BaseObject): """Data Args: - base_data: Data base data. - base_type: Data base type. + base_data: Container for data item (B section). + base_type: Name of item (B section) if any. If telemetry data is + derived straight from this, this should be None. """ __slots__ = ("base_data", "base_type") - def __init__(self, base_data=None, base_type=None) -> None: + def __init__(self, base_data: any = None, base_type: str = None) -> None: self.base_data = base_data self.base_type = base_type @@ -192,7 +193,7 @@ def __init__( ver: int = 2, name: str = "", properties: typing.Dict[str, any] = None, - measurements: any = None, + measurements: typing.Dict[str, int] = None, ): self.ver = ver self.name = name @@ -208,6 +209,64 @@ def to_dict(self): } +class ExceptionDetails(BaseObject): + """Exception details of the exception in a chain. + + Args: + id: In case exception is nested (outer exception contains inner one), + the id and outerId properties are used to represent the nesting. + outer_id: The value of outerId is a reference to an element in + ExceptionDetails that represents the outer exception. + type_name: Exception type name. + message: Exception message. + has_full_stack: Indicates if full exception stack is provided in the exception. + The stack may be trimmed, such as in the case of a StackOverflow exception. + stack: Text describing the stack. Either stack or parsedStack should have a + value. + parsed_stack: List of stack frames. Either stack or parsedStack should have + a value. + """ + + __slots__ = ( + "id", + "outer_id", + "type_name", + "message", + "has_full_stack", + "stack", + "parsed_stack", + ) + + def __init__( + self, + id: int = None, + outer_id: int = None, + type_name: str = None, + message: str = None, + has_full_stack: bool = None, + stack: str = None, + parsed_stack: any = None, + ) -> None: + self.id = id + self.outer_id = outer_id + self.type_name = type_name + self.message = message + self.has_full_stack = has_full_stack + self.stack = stack + self.parsed_stack = parsed_stack + + def to_dict(self): + return { + "id": self.ver, + "outerId": self.outer_id, + "typeName": self.type_name, + "message": self.message, + "hasFullStack ": self.has_full_stack, + "stack": self.stack, + "parsedStack": self.parsed_stack, + } + + class ExceptionData(BaseObject): """An instance of Exception represents a handled or unhandled exception that occurred during execution of the monitored application. @@ -217,6 +276,9 @@ class ExceptionData(BaseObject): exceptions: Exception chain - list of inner exceptions. severity_level: Severity level. Mostly used to indicate exception severity level when it is reported by logging library. + problem_id: Identifier of where the exception was thrown in code. + Used for exceptions grouping. Typically a combination of exception type + and a function from the call stack. properties: Collection of custom properties. measurements: Collection of custom measurements. """ @@ -225,6 +287,7 @@ class ExceptionData(BaseObject): "ver", "exceptions", "severity_level", + "problem_id", "properties", "measurements", ) @@ -232,16 +295,18 @@ class ExceptionData(BaseObject): def __init__( self, ver: int = 2, - exceptions: any = None, + exceptions: typing.List[ExceptionDetails] = None, severity_level: int = None, + problem_id: str = None, properties: typing.Dict[str, any] = None, - measurements: any = None, + measurements: typing.Dict[str, int] = None, ) -> None: if exceptions is None: exceptions = [] self.ver = ver self.exceptions = exceptions self.severity_level = severity_level + self.problem_id = problem_id self.properties = properties self.measurements = measurements @@ -256,6 +321,14 @@ def to_dict(self): } +class DataPointType(Enum): + VERBOSE = 0 + INFORMATION = 1 + WARNING = 2 + ERROR = 3 + CRITICAL = 4 + + class Message(BaseObject): """Instances of Message represent printf-like trace statements that are text-searched. The message does not have measurements. @@ -267,19 +340,27 @@ class Message(BaseObject): properties: Collection of custom properties. """ - __slots__ = ("ver", "message", "severity_level", "properties") + __slots__ = ( + "ver", + "message", + "measurements", + "severity_level", + "properties", + ) def __init__( self, ver: int = 2, message: str = "", - severity_level=None, + severity_level: DataPointType = None, properties: typing.Dict[str, any] = None, + measurements: typing.Dict[str, int] = None, ) -> None: self.ver = ver self.message = message self.severity_level = severity_level self.properties = properties + self.measurements = measurements def to_dict(self): return { @@ -375,7 +456,7 @@ def __init__( type: str = None, target: str = None, properties: typing.Dict[str, any] = None, - measurements: any = None, + measurements: typing.Dict[str, int] = None, ) -> None: self.ver = ver self.name = name @@ -452,7 +533,7 @@ def __init__( name: str = None, url: str = None, properties: typing.Dict[str, any] = None, - measurements: any = None, + measurements: typing.Dict[str, int] = None, ) -> None: self.ver = ver self.id = id diff --git a/azure_monitor/src/azure_monitor/trace.py b/azure_monitor/src/azure_monitor/trace.py index 8afb2b6..23f7be6 100644 --- a/azure_monitor/src/azure_monitor/trace.py +++ b/azure_monitor/src/azure_monitor/trace.py @@ -30,7 +30,8 @@ def export(self, spans: Sequence[Span]) -> SpanExportResult: result = self._transmit(envelopes_to_export) if result == utils.ExportResult.FAILED_RETRYABLE: self.storage.put(envelopes, result) - if len(list(envelopes_to_export)) < self.options.max_batch_size: + if result == utils.ExportResult.SUCCESS: + # Try to send any cached events self._transmit_from_storage() return utils.get_trace_export_result(result) except Exception: diff --git a/azure_monitor/src/azure_monitor/utils.py b/azure_monitor/src/azure_monitor/utils.py index 2030612..b4161f2 100644 --- a/azure_monitor/src/azure_monitor/utils.py +++ b/azure_monitor/src/azure_monitor/utils.py @@ -7,6 +7,7 @@ import sys import threading import time +import typing from enum import Enum from opentelemetry.sdk.metrics.export import MetricsExportResult @@ -83,12 +84,22 @@ def get_metrics_export_result(result: ExportResult) -> MetricsExportResult: class Options(BaseObject): + """Options to configure Azure exporters. + + Args: + connection_string: Azure Connection String. + instrumentation_key: Azure Instrumentation Key. + storage_maintenance_period: Local storage maintenance interval in seconds. + storage_max_size: Local storage maximum size in bytes. + storage_path: Local storage file path. + storage_retention_period: Local storage retention period in seconds + timeout: Request timeout in seconds + """ __slots__ = ( "connection_string", "endpoint", "instrumentation_key", - "max_batch_size", "storage_maintenance_period", "storage_max_size", "storage_path", @@ -98,24 +109,22 @@ class Options(BaseObject): def __init__( self, - connection_string=None, - instrumentation_key=None, - max_batch_size=100, - storage_maintenance_period=60, - storage_max_size=100 * 1024 * 1024, - storage_path=os.path.join( + connection_string: str = None, + instrumentation_key: str = None, + storage_maintenance_period: int = 60, + storage_max_size: int = 100 * 1024 * 1024, + storage_path: str = os.path.join( os.path.expanduser("~"), ".opentelemetry", ".azure", os.path.basename(sys.argv[0]) or ".console", ), - storage_retention_period=7 * 24 * 60 * 60, - timeout=10.0, # networking timeout in seconds + storage_retention_period: int = 7 * 24 * 60 * 60, + timeout: int = 10.0, # networking timeout in seconds **options, ) -> None: self.connection_string = connection_string self.instrumentation_key = instrumentation_key - self.max_batch_size = max_batch_size self.storage_maintenance_period = storage_maintenance_period self.storage_max_size = storage_max_size self.storage_path = storage_path @@ -125,7 +134,7 @@ def __init__( self._initialize() self._validate_instrumentation_key() - def _initialize(self): + def _initialize(self) -> None: code_cs = self._parse_connection_string(self.connection_string) code_ikey = self.instrumentation_key env_cs = self._parse_connection_string( @@ -155,7 +164,7 @@ def _initialize(self): ) self.endpoint = endpoint + "/v2/track" - def _validate_instrumentation_key(self): + def _validate_instrumentation_key(self) -> None: """Validates the instrumentation key used for Azure Monitor. An instrumentation key cannot be null or empty. An instrumentation key is valid for Azure Monitor only if it is a valid UUID. @@ -167,7 +176,7 @@ def _validate_instrumentation_key(self): if not match: raise ValueError("Invalid instrumentation key.") - def _parse_connection_string(self, connection_string): + def _parse_connection_string(self, connection_string) -> typing.Dict: if connection_string is None: return {} try: From 7c4f896ebbe89c8982d02a820ff59794e6dbce8a Mon Sep 17 00:00:00 2001 From: Hector Hernandez Date: Thu, 5 Mar 2020 15:45:45 -0800 Subject: [PATCH 054/109] Addressing comments --- azure_monitor/src/azure_monitor/protocol.py | 4 ++-- azure_monitor/src/azure_monitor/utils.py | 1 - azure_monitor/tests/test_base_exporter.py | 9 +++++---- azure_monitor/tests/trace/test_trace.py | 2 +- 4 files changed, 8 insertions(+), 8 deletions(-) diff --git a/azure_monitor/src/azure_monitor/protocol.py b/azure_monitor/src/azure_monitor/protocol.py index aaea169..a0b0863 100644 --- a/azure_monitor/src/azure_monitor/protocol.py +++ b/azure_monitor/src/azure_monitor/protocol.py @@ -321,7 +321,7 @@ def to_dict(self): } -class DataPointType(Enum): +class SeverityLevel(Enum): VERBOSE = 0 INFORMATION = 1 WARNING = 2 @@ -352,7 +352,7 @@ def __init__( self, ver: int = 2, message: str = "", - severity_level: DataPointType = None, + severity_level: SeverityLevel = None, properties: typing.Dict[str, any] = None, measurements: typing.Dict[str, int] = None, ) -> None: diff --git a/azure_monitor/src/azure_monitor/utils.py b/azure_monitor/src/azure_monitor/utils.py index b4161f2..5b13f63 100644 --- a/azure_monitor/src/azure_monitor/utils.py +++ b/azure_monitor/src/azure_monitor/utils.py @@ -121,7 +121,6 @@ def __init__( ), storage_retention_period: int = 7 * 24 * 60 * 60, timeout: int = 10.0, # networking timeout in seconds - **options, ) -> None: self.connection_string = connection_string self.instrumentation_key = instrumentation_key diff --git a/azure_monitor/tests/test_base_exporter.py b/azure_monitor/tests/test_base_exporter.py index a3fd661..fd74e46 100644 --- a/azure_monitor/tests/test_base_exporter.py +++ b/azure_monitor/tests/test_base_exporter.py @@ -21,26 +21,27 @@ def test_constructor(self): """Test the constructor.""" base = BaseExporter( instrumentation_key="4321abcd-5678-4efa-8abc-1234567890ab", - max_batch_size=1, storage_maintenance_period=2, storage_max_size=3, storage_path="testStoragePath", storage_retention_period=4, timeout=5, - something_else=6, ) self.assertIsInstance(base.options, Options) self.assertEqual( base.options.instrumentation_key, "4321abcd-5678-4efa-8abc-1234567890ab", ) - self.assertEqual(base.options.max_batch_size, 1) self.assertEqual(base.options.storage_maintenance_period, 2) self.assertEqual(base.options.storage_max_size, 3) self.assertEqual(base.options.storage_retention_period, 4) self.assertEqual(base.options.timeout, 5) self.assertEqual(base.options.storage_path, "testStoragePath") - self.assertEqual(getattr(base.options, "something_else", None), None) + + def test_constructor_wrong_options(self): + """Test the constructor with wrong options.""" + with self.assertRaises(TypeError): + base = BaseExporter(something_else=6) def test_telemetry_processor_add(self): base = BaseExporter() diff --git a/azure_monitor/tests/trace/test_trace.py b/azure_monitor/tests/trace/test_trace.py index 07b9839..7a1ceb1 100644 --- a/azure_monitor/tests/trace/test_trace.py +++ b/azure_monitor/tests/trace/test_trace.py @@ -102,7 +102,7 @@ def test_export_failure(self, span_to_envelope_mock): def test_export_success(self, span_to_envelope_mock): span_to_envelope_mock.return_value = ["bar"] exporter = AzureMonitorSpanExporter( - max_batch_size=1, storage_path=os.path.join(TEST_FOLDER, self.id()) + storage_path=os.path.join(TEST_FOLDER, self.id()) ) with mock.patch( "azure_monitor.trace.AzureMonitorSpanExporter._transmit" From 29ca6d849702ffaf131a97373d14d90503e52bd3 Mon Sep 17 00:00:00 2001 From: The Gitter Badger Date: Mon, 9 Mar 2020 19:41:57 +0000 Subject: [PATCH 055/109] Add Gitter badge --- README.rst | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/README.rst b/README.rst index 91b95a8..e87e177 100644 --- a/README.rst +++ b/README.rst @@ -1,6 +1,10 @@ OpenTelemetry Azure Monitor Exporters ===================================== +.. image:: https://badges.gitter.im/Microsoft/azure-monitor-python.svg + :alt: Join the chat at https://gitter.im/Microsoft/azure-monitor-python + :target: https://gitter.im/Microsoft/azure-monitor-python?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge + |pypi| .. |pypi| image:: https://badge.fury.io/py/opentelemetry-azure-monitor-exporter.svg From 9993fa2d92d791f50413fb1e9618770436c5a363 Mon Sep 17 00:00:00 2001 From: Leighton Date: Mon, 9 Mar 2020 22:10:38 -0700 Subject: [PATCH 056/109] readme --- README.md | 168 ++++++++++++++++++++ README.rst | 140 ---------------- azure_monitor/examples/metrics/simple.py | 29 ++++ azure_monitor/examples/traces/client.py | 14 +- azure_monitor/examples/traces/request.py | 14 +- azure_monitor/examples/traces/server.py | 12 +- azure_monitor/src/azure_monitor/__init__.py | 3 +- azure_monitor/src/azure_monitor/exporter.py | 12 +- azure_monitor/src/azure_monitor/metrics.py | 16 +- azure_monitor/src/azure_monitor/protocol.py | 10 +- 10 files changed, 241 insertions(+), 177 deletions(-) create mode 100644 README.md delete mode 100644 README.rst create mode 100644 azure_monitor/examples/metrics/simple.py diff --git a/README.md b/README.md new file mode 100644 index 0000000..eea2dc6 --- /dev/null +++ b/README.md @@ -0,0 +1,168 @@ +# OpenTelemetry Azure Monitor Exporters + +[![Gitter chat](https://img.shields.io/gitter/room/opentelemetry/opentelemetry-python)](https://gitter.im/Microsoft/azure-monitor-python) +[![Build status](https://travis-ci.org/microsoft/opentelemetry-exporters-python.svg?branch=master)](https://travis-ci.org/microsoft/opentelemetry-exporters-python) +[![PyPI version](https://badge.fury.io/py/opentelemetry-azure-monitor-exporter.svg)](https://badge.fury.io/py/opentelemetry-azure-monitor-exporter) + +## Installation + +```sh +pip install opentelemetry-azure-monitor-exporter +``` + +## Usage + +### Trace + +The **Azure Monitor Trace Exporter** allows you to export [OpenTelemetry](https://opentelemetry.io/) traces to [Azure Monitor](https://docs.microsoft.com/azure/azure-monitor/). + +This example shows how to send a span "hello" to Azure Monitor. + +* Create an Azure Monitor resource and get the instrumentation key, more information can be found [here](https://docs.microsoft.com/azure/azure-monitor/app/create-new-resource). +* Place your instrumentation key in a `connection string` and directly into your code. +* Alternatively, you can specify your `connection string` in an environment variable ``APPLICATIONINSIGHTS_CONNECTION_STRING``. + +```python +from azure_monitor import AzureMonitorSpanExporter +from opentelemetry import trace +from opentelemetry.sdk.trace import TracerProvider +from opentelemetry.sdk.trace.export import BatchExportSpanProcessor + +# The preferred tracer implementation must be set, as the opentelemetry-api +# defines the interface with a no-op implementation. +trace.set_preferred_tracer_provider_implementation(lambda T: TracerProvider()) + +# We tell OpenTelemetry who it is that is creating spans. In this case, we have +# no real name (no setup.py), so we make one up. If we had a version, we would +# also specify it here. +tracer = trace.get_tracer(__name__) + +exporter = AzureMonitorSpanExporter( + connection_string='InstrumentationKey=', +) + +# SpanExporter receives the spans and send them to the target location. +span_processor = BatchExportSpanProcessor(exporter) +trace.tracer_provider().add_span_processor(span_processor) + +with tracer.start_as_current_span('hello'): + print('Hello World!') +``` + +#### Integrations + +OpenTelemetry also supports several [integrations](https://github.com/open-telemetry/opentelemetry-python/tree/master/ext) which allows to integrate with third party libraries. + +This example shows how to integrate with the [requests](https://2.python-requests.org/en/master/)_ library. + +* Create an Azure Monitor resource and get the instrumentation key, more information can be found [here](https://docs.microsoft.com/azure/azure-monitor/app/create-new-resource). +* Install the `requests` integration package using ``pip install opentelemetry-ext-http-requests``. +* Place your instrumentation key in a `connection string` and directly into your code. +* Alternatively, you can specify your `connection string` in an environment variable ``APPLICATIONINSIGHTS_CONNECTION_STRING``. + +```python +import requests + +from azure_monitor import AzureMonitorSpanExporter +from opentelemetry import trace +from opentelemetry.ext import http_requests +from opentelemetry.sdk.trace import TracerProvider +from opentelemetry.sdk.trace.export import ( + BatchExportSpanProcessor, + ConsoleSpanExporter, +) + +# The preferred tracer implementation must be set, as the opentelemetry-api +# defines the interface with a no-op implementation. +trace.set_preferred_tracer_provider_implementation(lambda T: TracerProvider()) +tracer_provider = trace.tracer_provider() + +exporter = AzureMonitorSpanExporter( + connection_string='InstrumentationKey=', + ) +span_processor = BatchExportSpanProcessor(exporter) +tracer_provider.add_span_processor(span_processor) + +http_requests.enable(tracer_provider) +response = requests.get(url="https://azure.microsoft.com/") +``` + +#### Modifying Traces + +* You can pass a callback function to the exporter to process telemetry before it is exported. +* Your callback function can return `False` if you do not want this envelope exported. +* Your callback function must accept an [envelope](https://github.com/microsoft/opentelemetry-exporters-python/blob/master/azure_monitor/src/azure_monitor/protocol.py#L80) data type as its parameter. +* You can see the schema for Azure Monitor data types in the envelopes [here](https://github.com/microsoft/opentelemetry-exporters-python/blob/master/azure_monitor/src/azure_monitor/protocol.py). +* The `AzureMonitorSpanExporter` handles `Data` data types. + +```python +from azure_monitor import AzureMonitorSpanExporter +from opentelemetry import trace +from opentelemetry.sdk.trace import TracerProvider +from opentelemetry.sdk.trace.export import BatchExportSpanProcessor + +# Callback function to add os_type: linux to span properties +def callback_function(envelope): + envelope.data.baseData.properties['os_type'] = 'linux' + return True + +exporter = AzureMonitorSpanExporter( + connection_string='InstrumentationKey=' +) +exporter.add_telemetry_processor(callback_function) + +trace.set_preferred_tracer_provider_implementation(lambda T: TracerProvider()) +tracer = trace.get_tracer(__name__) +span_processor = BatchExportSpanProcessor(exporter) +trace.tracer_provider().add_span_processor(span_processor) + +with tracer.start_as_current_span('hello'): + print('Hello World!') +``` + +### Metrics + +The **Azure Monitor Metrics Exporter** allows you to export metrics to [Azure Monitor](https://docs.microsoft.com/azure/azure-monitor/). + +This example shows how to track a counter metric and send it as telemetry every export interval. + +* Create an Azure Monitor resource and get the instrumentation key, more information can be found [here](https://docs.microsoft.com/azure/azure-monitor/app/create-new-resource). +* Place your instrumentation key in a `connection string` and directly into your code. +* Alternatively, you can specify your `connection string` in an environment variable ``APPLICATIONINSIGHTS_CONNECTION_STRING``. + +```python +import time + +from azure_monitor import AzureMonitorMetricsExporter +from opentelemetry import metrics +from opentelemetry.sdk.metrics import Counter, MeterProvider +from opentelemetry.sdk.metrics.export.controller import PushController + +metrics.set_preferred_meter_provider_implementation(lambda _: MeterProvider()) +meter = metrics.get_meter(__name__) +exporter = AzureMonitorMetricsExporter( + connection_string='InstrumentationKey=' +) +controller = PushController(meter, exporter, 5) + +requests_counter = meter.create_metric( + name="requests", + description="number of requests", + unit="1", + value_type=int, + metric_type=Counter, + label_keys=("environment",), +) + +testing_label_set = meter.get_label_set({"environment": "testing"}) + +requests_counter.add(25, testing_label_set) +time.sleep(100) +``` + +# References + +[Azure Monitor](https://docs.microsoft.com/azure/azure-monitor/) +[OpenTelemetry Project](https://opentelemetry.io/) +[OpenTelemetry Python Client](https://github.com/open-telemetry/opentelemetry-python) +[Azure Monitor Python Gitter](https://gitter.im/Microsoft/azure-monitor-python) diff --git a/README.rst b/README.rst deleted file mode 100644 index e87e177..0000000 --- a/README.rst +++ /dev/null @@ -1,140 +0,0 @@ -OpenTelemetry Azure Monitor Exporters -===================================== - -.. image:: https://badges.gitter.im/Microsoft/azure-monitor-python.svg - :alt: Join the chat at https://gitter.im/Microsoft/azure-monitor-python - :target: https://gitter.im/Microsoft/azure-monitor-python?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge - -|pypi| - -.. |pypi| image:: https://badge.fury.io/py/opentelemetry-azure-monitor-exporter.svg - :target: https://pypi.org/project/opentelemetry-azure-monitor-exporter/ - -Installation ------------- - -:: - - pip install opentelemetry-azure-monitor-exporter - -Usage ------ - -Trace -~~~~~ - -The **Azure Monitor Trace Exporter** allows you to export `OpenTelemetry`_ traces to `Azure Monitor`_. - -This example shows how to send a span "hello" to Azure Monitor. - -* Create an Azure Monitor resource and get the instrumentation key, more information can be found `here `_. -* Place your instrumentation key in a `connection string` and directly into your code. -* Alternatively, you can specify your `connection string` in an environment variable ``APPLICATIONINSIGHTS_CONNECTION_STRING``. - - .. code:: python - - from azure_monitor import AzureMonitorSpanExporter - from opentelemetry import trace - from opentelemetry.sdk.trace import TracerProvider - from opentelemetry.sdk.trace.export import BatchExportSpanProcessor - - # The preferred tracer implementation must be set, as the opentelemetry-api - # defines the interface with a no-op implementation. - trace.set_preferred_tracer_provider_implementation(lambda T: TracerProvider()) - - # We tell OpenTelemetry who it is that is creating spans. In this case, we have - # no real name (no setup.py), so we make one up. If we had a version, we would - # also specify it here. - tracer = trace.get_tracer(__name__) - - exporter = AzureMonitorSpanExporter( - connection_string='InstrumentationKey=99c42f65-1656-4c41-afde-bd86b709a4a7', - ) - - # SpanExporter receives the spans and send them to the target location. - span_processor = BatchExportSpanProcessor(exporter) - trace.tracer_provider().add_span_processor(span_processor) - - with tracer.start_as_current_span('hello'): - print('Hello World!') - -Integrations -############ - -OpenTelemetry also supports several `integrations `_ which allows to integrate with third party libraries. - -This example shows how to integrate with the `requests `_ library. - -* Create an Azure Monitor resource and get the instrumentation key, more information can be found `here `_. -* Install the `requests integration package using ``pip install opentelemetry-ext-http-requests``. -* Place your instrumentation key in a `connection string` and directly into your code. -* Alternatively, you can specify your `connection string` in an environment variable ``APPLICATIONINSIGHTS_CONNECTION_STRING``. - -.. code:: python - - import requests - - from azure_monitor import AzureMonitorSpanExporter - from opentelemetry import trace - from opentelemetry.ext import http_requests - from opentelemetry.sdk.trace import TracerProvider - from opentelemetry.sdk.trace.export import ( - BatchExportSpanProcessor, - ConsoleSpanExporter, - ) - - # The preferred tracer implementation must be set, as the opentelemetry-api - # defines the interface with a no-op implementation. - trace.set_preferred_tracer_provider_implementation(lambda T: TracerProvider()) - tracer_provider = trace.tracer_provider() - - exporter = AzureMonitorSpanExporter( - connection_string='InstrumentationKey=99c42f65-1656-4c41-afde-bd86b709a4a7', - ) - span_processor = BatchExportSpanProcessor(exporter) - tracer_provider.add_span_processor(span_processor) - - http_requests.enable(tracer_provider) - response = requests.get(url="https://azure.microsoft.com/") - -Modifying Traces -################ - -* You can pass a callback function to the exporter to process telemetry before it is exported. -* Your callback function can return `False` if you do not want this envelope exported. -* Your callback function must accept an [envelope](https://github.com/microsoft/opentelemetry-exporters-python/blob/master/azure_monitor/src/azure_monitor/protocol.py#L80) data type as its parameter. -* You can see the schema for Azure Monitor data types in the envelopes [here](https://github.com/microsoft/opentelemetry-exporters-python/blob/master/azure_monitor/src/azure_monitor/protocol.py). -* The `AzureMonitorSpanExporter` handles `Data` data types. - -.. code:: python - - from azure_monitor import AzureMonitorSpanExporter - from opentelemetry import trace - from opentelemetry.sdk.trace import TracerProvider - from opentelemetry.sdk.trace.export import BatchExportSpanProcessor - - # Callback function to add os_type: linux to span properties - def callback_function(envelope): - envelope.data.baseData.properties['os_type'] = 'linux' - return True - - exporter = AzureMonitorSpanExporter( - connection_string='InstrumentationKey=99c42f65-1656-4c41-afde-bd86b709a4a7' - ) - exporter.add_telemetry_processor(callback_function) - - trace.set_preferred_tracer_provider_implementation(lambda T: TracerProvider()) - tracer = trace.get_tracer(__name__) - span_processor = BatchExportSpanProcessor(exporter) - trace.tracer_provider().add_span_processor(span_processor) - - with tracer.start_as_current_span('hello'): - print('Hello World!') - -References ----------- - -* `Azure Monitor `_ -* `OpenTelemetry Project `_ -* `OpenTelemetry Python Client `_ - diff --git a/azure_monitor/examples/metrics/simple.py b/azure_monitor/examples/metrics/simple.py new file mode 100644 index 0000000..e8b5bed --- /dev/null +++ b/azure_monitor/examples/metrics/simple.py @@ -0,0 +1,29 @@ +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. +import time + +from azure_monitor import AzureMonitorMetricsExporter +from opentelemetry import metrics +from opentelemetry.sdk.metrics import Counter, MeterProvider +from opentelemetry.sdk.metrics.export.controller import PushController + +metrics.set_preferred_meter_provider_implementation(lambda _: MeterProvider()) +meter = metrics.get_meter(__name__) +exporter = AzureMonitorMetricsExporter( + connection_string="InstrumentationKey=" +) +controller = PushController(meter, exporter, 5) + +requests_counter = meter.create_metric( + name="requests", + description="number of requests", + unit="1", + value_type=int, + metric_type=Counter, + label_keys=("environment",), +) + +testing_label_set = meter.get_label_set({"environment": "testing"}) + +requests_counter.add(25, testing_label_set) +time.sleep(100) diff --git a/azure_monitor/examples/traces/client.py b/azure_monitor/examples/traces/client.py index 6cc7eae..f3cfc1d 100644 --- a/azure_monitor/examples/traces/client.py +++ b/azure_monitor/examples/traces/client.py @@ -6,18 +6,20 @@ import requests from opentelemetry import trace from opentelemetry.ext import http_requests -from opentelemetry.sdk.trace import TracerSource +from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import BatchExportSpanProcessor from azure_monitor import AzureMonitorSpanExporter -trace.set_preferred_tracer_source_implementation(lambda T: TracerSource()) -tracer = trace.tracer_source().get_tracer(__name__) -http_requests.enable(trace.tracer_source()) +trace.set_preferred_tracer_provider_implementation(lambda T: TracerProvider()) +tracer = trace.get_tracer(__name__) +http_requests.enable(trace.tracer_provider()) span_processor = BatchExportSpanProcessor( - AzureMonitorSpanExporter(instrumentation_key="") + AzureMonitorSpanExporter( + connection_string="InstrumentationKey=" + ) ) -trace.tracer_source().add_span_processor(span_processor) +trace.tracer_provider().add_span_processor(span_processor) response = requests.get(url="http://127.0.0.1:8080/") span_processor.shutdown() diff --git a/azure_monitor/examples/traces/request.py b/azure_monitor/examples/traces/request.py index aadd55f..5843ad8 100644 --- a/azure_monitor/examples/traces/request.py +++ b/azure_monitor/examples/traces/request.py @@ -6,19 +6,21 @@ import requests from opentelemetry import trace from opentelemetry.ext import http_requests -from opentelemetry.sdk.trace import TracerSource +from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import SimpleExportSpanProcessor from azure_monitor import AzureMonitorSpanExporter -trace.set_preferred_tracer_source_implementation(lambda T: TracerSource()) +trace.set_preferred_tracer_provider_implementation(lambda T: TracerProvider()) -http_requests.enable(trace.tracer_source()) +http_requests.enable(trace.tracer_provider()) span_processor = SimpleExportSpanProcessor( - AzureMonitorSpanExporter(instrumentation_key="") + AzureMonitorSpanExporter( + connection_string="InstrumentationKey=" + ) ) -trace.tracer_source().add_span_processor(span_processor) -tracer = trace.tracer_source().get_tracer(__name__) +trace.tracer_provider().add_span_processor(span_processor) +tracer = trace.get_tracer(__name__) with tracer.start_as_current_span("parent"): response = requests.get("https://azure.microsoft.com/", timeout=5) diff --git a/azure_monitor/examples/traces/server.py b/azure_monitor/examples/traces/server.py index 1a5b507..8c8528f 100644 --- a/azure_monitor/examples/traces/server.py +++ b/azure_monitor/examples/traces/server.py @@ -7,7 +7,7 @@ from opentelemetry import trace from opentelemetry.ext import http_requests from opentelemetry.ext.wsgi import OpenTelemetryMiddleware -from opentelemetry.sdk.trace import TracerSource +from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import BatchExportSpanProcessor import flask @@ -15,21 +15,21 @@ # The preferred tracer implementation must be set, as the opentelemetry-api # defines the interface with a no-op implementation. -trace.set_preferred_tracer_source_implementation(lambda T: TracerSource()) -tracer = trace.tracer_source().get_tracer(__name__) +trace.set_preferred_tracer_provider_implementation(lambda T: TracerProvider()) +tracer = trace.get_tracer(__name__) exporter = AzureMonitorSpanExporter( - instrumentation_key="" + connection_string="InstrumentationKey=" ) # SpanExporter receives the spans and send them to the target location. span_processor = BatchExportSpanProcessor(exporter) -trace.tracer_source().add_span_processor(span_processor) +trace.tracer_provider().add_span_processor(span_processor) # Integrations are the glue that binds the OpenTelemetry API and the # frameworks and libraries that are used together, automatically creating # Spans and propagating context as appropriate. -http_requests.enable(trace.tracer_source()) +http_requests.enable(trace.tracer_provider()) app = flask.Flask(__name__) app.wsgi_app = OpenTelemetryMiddleware(app.wsgi_app) diff --git a/azure_monitor/src/azure_monitor/__init__.py b/azure_monitor/src/azure_monitor/__init__.py index f3a6f7b..9d1b57f 100644 --- a/azure_monitor/src/azure_monitor/__init__.py +++ b/azure_monitor/src/azure_monitor/__init__.py @@ -1,6 +1,7 @@ # Copyright (c) Microsoft Corporation. All rights reserved. # Licensed under the MIT License. +from azure_monitor.metrics import AzureMonitorMetricsExporter from azure_monitor.trace import AzureMonitorSpanExporter from azure_monitor.version import __version__ # noqa -__all__ = ["AzureMonitorSpanExporter"] +__all__ = ["AzureMonitorMetricsExporter", "AzureMonitorSpanExporter"] diff --git a/azure_monitor/src/azure_monitor/exporter.py b/azure_monitor/src/azure_monitor/exporter.py index 1145ecc..b36a1be 100644 --- a/azure_monitor/src/azure_monitor/exporter.py +++ b/azure_monitor/src/azure_monitor/exporter.py @@ -78,7 +78,7 @@ def _transmit_from_storage(self) -> None: blob.delete(silent=True) def _transmit( - self, envelopes_to_export: typing.List[Envelope] + self, envelopes: typing.List[Envelope] ) -> utils.ExportResult: """ Transmit the data envelopes to the ingestion service. @@ -86,11 +86,11 @@ def _transmit( Returns an ExportResult, this function should never throw an exception. """ - if len(envelopes_to_export) > 0: + if len(envelopes) > 0: try: response = requests.post( url=self.options.endpoint, - data=json.dumps(envelopes_to_export), + data=json.dumps(envelopes), headers={ "Accept": "application/json", "Content-Type": "application/json; charset=utf-8", @@ -98,7 +98,7 @@ def _transmit( timeout=self.options.timeout, ) except Exception as ex: - logger.warning("Transient client side error %s.", ex) + logger.warning("Transient client side error: %s.", ex) return utils.ExportResult.FAILED_RETRYABLE text = "N/A" @@ -128,14 +128,14 @@ def _transmit( 503, # Service Unavailable ): resend_envelopes.append( - envelopes_to_export[error["index"]] + envelopes[error["index"]] ) else: logger.error( "Data drop %s: %s %s.", error["statusCode"], error["message"], - envelopes_to_export[error["index"]], + envelopes[error["index"]], ) if resend_envelopes: self.storage.put(resend_envelopes) diff --git a/azure_monitor/src/azure_monitor/metrics.py b/azure_monitor/src/azure_monitor/metrics.py index 00d2534..0860007 100644 --- a/azure_monitor/src/azure_monitor/metrics.py +++ b/azure_monitor/src/azure_monitor/metrics.py @@ -26,13 +26,13 @@ def __init__(self, **options): def export( self, metric_records: Sequence[MetricRecord] ) -> MetricsExportResult: - envelopes = map(self.metric_to_envelope, metric_records) - envelopes_to_export = map( + envelopes = list(map(self.metric_to_envelope, metric_records)) + envelopes = list(map( lambda x: x.to_dict(), - tuple(self.apply_telemetry_processors(envelopes)), - ) + self.apply_telemetry_processors(envelopes), + )) try: - result = self._transmit(envelopes_to_export) + result = self._transmit(envelopes) if result == MetricsExportResult.FAILED_RETRYABLE: self.storage.put(envelopes, result) if result == utils.ExportResult.SUCCESS: @@ -60,10 +60,10 @@ def metric_to_envelope( envelope.name = "Microsoft.ApplicationInsights.Metric" data_point = protocol.DataPoint( - ns=metric_record.metric.name, - name=metric_record.metric.description, + ns=metric_record.metric.description, + name=metric_record.metric.name, value=metric_record.aggregator.checkpoint, - kind=protocol.DataPointType.MEASUREMENT, + kind=protocol.DataPointType.MEASUREMENT.value, ) properties = {} diff --git a/azure_monitor/src/azure_monitor/protocol.py b/azure_monitor/src/azure_monitor/protocol.py index a0b0863..cd4965f 100644 --- a/azure_monitor/src/azure_monitor/protocol.py +++ b/azure_monitor/src/azure_monitor/protocol.py @@ -37,7 +37,7 @@ def __init__(self, base_data: any = None, base_type: str = None) -> None: self.base_type = base_type def to_dict(self): - return {"baseData": self.base_data, "baseType": self.base_type} + return {"baseData": self.base_data.to_dict(), "baseType": self.base_type} class DataPointType(Enum): @@ -169,7 +169,6 @@ def to_dict(self): "flags": self.flags, "tags": self.tags, "data": self.data.to_dict() if self.data else None, - "baseType": self.data.base_type if self.data else None, } @@ -389,13 +388,16 @@ class MetricData(BaseObject): def __init__( self, ver: int = 2, - metrics: typing.List[DataPoint] = None, + metrics: typing.List[DataPoint] = [], properties: typing.Dict[str, any] = None, ) -> None: if metrics is None: metrics = [] self.ver = ver - self.metrics = metrics + self.metrics = list(map( + lambda x: x.to_dict(), + metrics, + )) self.properties = properties def to_dict(self): From 10928ed5da147c931497f84c8f3d5eba652b858d Mon Sep 17 00:00:00 2001 From: Leighton Date: Mon, 9 Mar 2020 22:41:39 -0700 Subject: [PATCH 057/109] fix tests --- azure_monitor/src/azure_monitor/metrics.py | 1 - azure_monitor/src/azure_monitor/protocol.py | 12 ++++++------ azure_monitor/tests/metrics/test_metrics.py | 15 +++++++-------- 3 files changed, 13 insertions(+), 15 deletions(-) diff --git a/azure_monitor/src/azure_monitor/metrics.py b/azure_monitor/src/azure_monitor/metrics.py index 0860007..d4d1bba 100644 --- a/azure_monitor/src/azure_monitor/metrics.py +++ b/azure_monitor/src/azure_monitor/metrics.py @@ -69,7 +69,6 @@ def metric_to_envelope( properties = {} for label_tuple in metric_record.label_set.labels: properties[label_tuple[0]] = label_tuple[1] - data = protocol.MetricData(metrics=[data_point], properties=properties) envelope.data = protocol.Data(base_data=data, base_type="MetricData") return envelope diff --git a/azure_monitor/src/azure_monitor/protocol.py b/azure_monitor/src/azure_monitor/protocol.py index cd4965f..cba995a 100644 --- a/azure_monitor/src/azure_monitor/protocol.py +++ b/azure_monitor/src/azure_monitor/protocol.py @@ -388,22 +388,22 @@ class MetricData(BaseObject): def __init__( self, ver: int = 2, - metrics: typing.List[DataPoint] = [], + metrics: typing.List[DataPoint] = None, properties: typing.Dict[str, any] = None, ) -> None: if metrics is None: metrics = [] self.ver = ver - self.metrics = list(map( - lambda x: x.to_dict(), - metrics, - )) + self.metrics = metrics self.properties = properties def to_dict(self): return { "ver": self.ver, - "metrics": self.metrics, + "metrics": list(map( + lambda x: x.to_dict(), + self.metrics, + )), "properties": self.properties, } diff --git a/azure_monitor/tests/metrics/test_metrics.py b/azure_monitor/tests/metrics/test_metrics.py index 37f64b5..f83844c 100644 --- a/azure_monitor/tests/metrics/test_metrics.py +++ b/azure_monitor/tests/metrics/test_metrics.py @@ -1,7 +1,5 @@ # Copyright (c) Microsoft Corporation. All rights reserved. # Licensed under the MIT License. - - import json import os import shutil @@ -9,13 +7,13 @@ from unittest import mock from opentelemetry import metrics -from opentelemetry.sdk.metrics import Counter, Meter +from opentelemetry.sdk.metrics import Counter, MeterProvider from opentelemetry.sdk.metrics.export import MetricRecord, MetricsExportResult from opentelemetry.sdk.metrics.export.aggregate import CounterAggregator from opentelemetry.sdk.util import ns_to_iso_str from azure_monitor.metrics import AzureMonitorMetricsExporter -from azure_monitor.protocol import Data, Envelope, MetricData +from azure_monitor.protocol import Data, DataPoint, Envelope, MetricData from azure_monitor.utils import ExportResult, Options @@ -26,8 +24,8 @@ def setUpClass(self): "APPINSIGHTS_INSTRUMENTATIONKEY" ] = "1234abcd-5678-4efa-8abc-1234567890ab" - metrics.set_preferred_meter_implementation(lambda _: Meter()) - self._meter = metrics.meter() + metrics.set_preferred_meter_provider_implementation(lambda _: MeterProvider()) + self._meter = metrics.get_meter(__name__) self._test_metric = self._meter.create_metric( "testname", "testdesc", "unit", int, Counter, ["environment"] ) @@ -86,8 +84,9 @@ def test_metric_to_envelope(self): self.assertIsInstance(envelope.data.base_data, MetricData) self.assertEqual(envelope.data.base_data.ver, 2) self.assertEqual(len(envelope.data.base_data.metrics), 1) - self.assertEqual(envelope.data.base_data.metrics[0].ns, "testname") - self.assertEqual(envelope.data.base_data.metrics[0].name, "testdesc") + self.assertIsInstance(envelope.data.base_data.metrics[0], DataPoint) + self.assertEqual(envelope.data.base_data.metrics[0].ns, "testdesc") + self.assertEqual(envelope.data.base_data.metrics[0].name, "testname") self.assertEqual(envelope.data.base_data.metrics[0].value, 123) self.assertEqual( envelope.data.base_data.properties["environment"], "staging" From 3a021dcb48cfccd78f28f0e40ca93b702b6b4df1 Mon Sep 17 00:00:00 2001 From: Leighton Date: Tue, 10 Mar 2020 10:48:43 -0700 Subject: [PATCH 058/109] build --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index eea2dc6..b6a7296 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,7 @@ # OpenTelemetry Azure Monitor Exporters [![Gitter chat](https://img.shields.io/gitter/room/opentelemetry/opentelemetry-python)](https://gitter.im/Microsoft/azure-monitor-python) -[![Build status](https://travis-ci.org/microsoft/opentelemetry-exporters-python.svg?branch=master)](https://travis-ci.org/microsoft/opentelemetry-exporters-python) + [![PyPI version](https://badge.fury.io/py/opentelemetry-azure-monitor-exporter.svg)](https://badge.fury.io/py/opentelemetry-azure-monitor-exporter) ## Installation From c02e57ac38e716cea39a885c7fbcf388bfe1bbc1 Mon Sep 17 00:00:00 2001 From: Leighton Date: Tue, 10 Mar 2020 10:55:34 -0700 Subject: [PATCH 059/109] comment out tests --- README.md | 2 +- azure_monitor/examples/metrics/simple.py | 48 ++++++------- azure_monitor/examples/traces/client.py | 44 ++++++------ azure_monitor/examples/traces/request.py | 44 ++++++------ azure_monitor/examples/traces/server.py | 92 ++++++++++++------------ azure_monitor/examples/traces/trace.py | 36 +++++----- 6 files changed, 133 insertions(+), 133 deletions(-) diff --git a/README.md b/README.md index b6a7296..eea2dc6 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,7 @@ # OpenTelemetry Azure Monitor Exporters [![Gitter chat](https://img.shields.io/gitter/room/opentelemetry/opentelemetry-python)](https://gitter.im/Microsoft/azure-monitor-python) - +[![Build status](https://travis-ci.org/microsoft/opentelemetry-exporters-python.svg?branch=master)](https://travis-ci.org/microsoft/opentelemetry-exporters-python) [![PyPI version](https://badge.fury.io/py/opentelemetry-azure-monitor-exporter.svg)](https://badge.fury.io/py/opentelemetry-azure-monitor-exporter) ## Installation diff --git a/azure_monitor/examples/metrics/simple.py b/azure_monitor/examples/metrics/simple.py index e8b5bed..14d586a 100644 --- a/azure_monitor/examples/metrics/simple.py +++ b/azure_monitor/examples/metrics/simple.py @@ -1,29 +1,29 @@ -# Copyright (c) Microsoft Corporation. All rights reserved. -# Licensed under the MIT License. -import time +# # Copyright (c) Microsoft Corporation. All rights reserved. +# # Licensed under the MIT License. +# import time -from azure_monitor import AzureMonitorMetricsExporter -from opentelemetry import metrics -from opentelemetry.sdk.metrics import Counter, MeterProvider -from opentelemetry.sdk.metrics.export.controller import PushController +# from azure_monitor import AzureMonitorMetricsExporter +# from opentelemetry import metrics +# from opentelemetry.sdk.metrics import Counter, MeterProvider +# from opentelemetry.sdk.metrics.export.controller import PushController -metrics.set_preferred_meter_provider_implementation(lambda _: MeterProvider()) -meter = metrics.get_meter(__name__) -exporter = AzureMonitorMetricsExporter( - connection_string="InstrumentationKey=" -) -controller = PushController(meter, exporter, 5) +# metrics.set_preferred_meter_provider_implementation(lambda _: MeterProvider()) +# meter = metrics.get_meter(__name__) +# exporter = AzureMonitorMetricsExporter( +# connection_string="InstrumentationKey=" +# ) +# controller = PushController(meter, exporter, 5) -requests_counter = meter.create_metric( - name="requests", - description="number of requests", - unit="1", - value_type=int, - metric_type=Counter, - label_keys=("environment",), -) +# requests_counter = meter.create_metric( +# name="requests", +# description="number of requests", +# unit="1", +# value_type=int, +# metric_type=Counter, +# label_keys=("environment",), +# ) -testing_label_set = meter.get_label_set({"environment": "testing"}) +# testing_label_set = meter.get_label_set({"environment": "testing"}) -requests_counter.add(25, testing_label_set) -time.sleep(100) +# requests_counter.add(25, testing_label_set) +# time.sleep(100) diff --git a/azure_monitor/examples/traces/client.py b/azure_monitor/examples/traces/client.py index f3cfc1d..f51ad10 100644 --- a/azure_monitor/examples/traces/client.py +++ b/azure_monitor/examples/traces/client.py @@ -1,25 +1,25 @@ -# Copyright (c) Microsoft Corporation. All rights reserved. -# Licensed under the MIT License. -# pylint: disable=import-error -# pylint: disable=no-member -# pylint: disable=no-name-in-module -import requests -from opentelemetry import trace -from opentelemetry.ext import http_requests -from opentelemetry.sdk.trace import TracerProvider -from opentelemetry.sdk.trace.export import BatchExportSpanProcessor +# # Copyright (c) Microsoft Corporation. All rights reserved. +# # Licensed under the MIT License. +# # pylint: disable=import-error +# # pylint: disable=no-member +# # pylint: disable=no-name-in-module +# import requests +# from opentelemetry import trace +# from opentelemetry.ext import http_requests +# from opentelemetry.sdk.trace import TracerProvider +# from opentelemetry.sdk.trace.export import BatchExportSpanProcessor -from azure_monitor import AzureMonitorSpanExporter +# from azure_monitor import AzureMonitorSpanExporter -trace.set_preferred_tracer_provider_implementation(lambda T: TracerProvider()) -tracer = trace.get_tracer(__name__) -http_requests.enable(trace.tracer_provider()) -span_processor = BatchExportSpanProcessor( - AzureMonitorSpanExporter( - connection_string="InstrumentationKey=" - ) -) -trace.tracer_provider().add_span_processor(span_processor) +# trace.set_preferred_tracer_provider_implementation(lambda T: TracerProvider()) +# tracer = trace.get_tracer(__name__) +# http_requests.enable(trace.tracer_provider()) +# span_processor = BatchExportSpanProcessor( +# AzureMonitorSpanExporter( +# connection_string="InstrumentationKey=" +# ) +# ) +# trace.tracer_provider().add_span_processor(span_processor) -response = requests.get(url="http://127.0.0.1:8080/") -span_processor.shutdown() +# response = requests.get(url="http://127.0.0.1:8080/") +# span_processor.shutdown() diff --git a/azure_monitor/examples/traces/request.py b/azure_monitor/examples/traces/request.py index 5843ad8..20d68f0 100644 --- a/azure_monitor/examples/traces/request.py +++ b/azure_monitor/examples/traces/request.py @@ -1,26 +1,26 @@ -# Copyright (c) Microsoft Corporation. All rights reserved. -# Licensed under the MIT License. -# pylint: disable=import-error -# pylint: disable=no-member -# pylint: disable=no-name-in-module -import requests -from opentelemetry import trace -from opentelemetry.ext import http_requests -from opentelemetry.sdk.trace import TracerProvider -from opentelemetry.sdk.trace.export import SimpleExportSpanProcessor +# # Copyright (c) Microsoft Corporation. All rights reserved. +# # Licensed under the MIT License. +# # pylint: disable=import-error +# # pylint: disable=no-member +# # pylint: disable=no-name-in-module +# import requests +# from opentelemetry import trace +# from opentelemetry.ext import http_requests +# from opentelemetry.sdk.trace import TracerProvider +# from opentelemetry.sdk.trace.export import SimpleExportSpanProcessor -from azure_monitor import AzureMonitorSpanExporter +# from azure_monitor import AzureMonitorSpanExporter -trace.set_preferred_tracer_provider_implementation(lambda T: TracerProvider()) +# trace.set_preferred_tracer_provider_implementation(lambda T: TracerProvider()) -http_requests.enable(trace.tracer_provider()) -span_processor = SimpleExportSpanProcessor( - AzureMonitorSpanExporter( - connection_string="InstrumentationKey=" - ) -) -trace.tracer_provider().add_span_processor(span_processor) -tracer = trace.get_tracer(__name__) +# http_requests.enable(trace.tracer_provider()) +# span_processor = SimpleExportSpanProcessor( +# AzureMonitorSpanExporter( +# connection_string="InstrumentationKey=" +# ) +# ) +# trace.tracer_provider().add_span_processor(span_processor) +# tracer = trace.get_tracer(__name__) -with tracer.start_as_current_span("parent"): - response = requests.get("https://azure.microsoft.com/", timeout=5) +# with tracer.start_as_current_span("parent"): +# response = requests.get("https://azure.microsoft.com/", timeout=5) diff --git a/azure_monitor/examples/traces/server.py b/azure_monitor/examples/traces/server.py index 8c8528f..62ae190 100644 --- a/azure_monitor/examples/traces/server.py +++ b/azure_monitor/examples/traces/server.py @@ -1,46 +1,46 @@ -# Copyright (c) Microsoft Corporation. All rights reserved. -# Licensed under the MIT License. -# pylint: disable=import-error -# pylint: disable=no-member -# pylint: disable=no-name-in-module -import requests -from opentelemetry import trace -from opentelemetry.ext import http_requests -from opentelemetry.ext.wsgi import OpenTelemetryMiddleware -from opentelemetry.sdk.trace import TracerProvider -from opentelemetry.sdk.trace.export import BatchExportSpanProcessor - -import flask -from azure_monitor import AzureMonitorSpanExporter - -# The preferred tracer implementation must be set, as the opentelemetry-api -# defines the interface with a no-op implementation. -trace.set_preferred_tracer_provider_implementation(lambda T: TracerProvider()) -tracer = trace.get_tracer(__name__) - -exporter = AzureMonitorSpanExporter( - connection_string="InstrumentationKey=" -) - -# SpanExporter receives the spans and send them to the target location. -span_processor = BatchExportSpanProcessor(exporter) -trace.tracer_provider().add_span_processor(span_processor) - -# Integrations are the glue that binds the OpenTelemetry API and the -# frameworks and libraries that are used together, automatically creating -# Spans and propagating context as appropriate. -http_requests.enable(trace.tracer_provider()) -app = flask.Flask(__name__) -app.wsgi_app = OpenTelemetryMiddleware(app.wsgi_app) - - -@app.route("/") -def hello(): - with tracer.start_as_current_span("parent"): - requests.get("https://www.wikipedia.org/wiki/Rabbit") - return "hello" - - -if __name__ == "__main__": - app.run(host="localhost", port=8080, threaded=True) - span_processor.shutdown() +# # Copyright (c) Microsoft Corporation. All rights reserved. +# # Licensed under the MIT License. +# # pylint: disable=import-error +# # pylint: disable=no-member +# # pylint: disable=no-name-in-module +# import requests +# from opentelemetry import trace +# from opentelemetry.ext import http_requests +# from opentelemetry.ext.wsgi import OpenTelemetryMiddleware +# from opentelemetry.sdk.trace import TracerProvider +# from opentelemetry.sdk.trace.export import BatchExportSpanProcessor + +# import flask +# from azure_monitor import AzureMonitorSpanExporter + +# # The preferred tracer implementation must be set, as the opentelemetry-api +# # defines the interface with a no-op implementation. +# trace.set_preferred_tracer_provider_implementation(lambda T: TracerProvider()) +# tracer = trace.get_tracer(__name__) + +# exporter = AzureMonitorSpanExporter( +# connection_string="InstrumentationKey=" +# ) + +# # SpanExporter receives the spans and send them to the target location. +# span_processor = BatchExportSpanProcessor(exporter) +# trace.tracer_provider().add_span_processor(span_processor) + +# # Integrations are the glue that binds the OpenTelemetry API and the +# # frameworks and libraries that are used together, automatically creating +# # Spans and propagating context as appropriate. +# http_requests.enable(trace.tracer_provider()) +# app = flask.Flask(__name__) +# app.wsgi_app = OpenTelemetryMiddleware(app.wsgi_app) + + +# @app.route("/") +# def hello(): +# with tracer.start_as_current_span("parent"): +# requests.get("https://www.wikipedia.org/wiki/Rabbit") +# return "hello" + + +# if __name__ == "__main__": +# app.run(host="localhost", port=8080, threaded=True) +# span_processor.shutdown() diff --git a/azure_monitor/examples/traces/trace.py b/azure_monitor/examples/traces/trace.py index 2698c78..889ca7b 100644 --- a/azure_monitor/examples/traces/trace.py +++ b/azure_monitor/examples/traces/trace.py @@ -1,24 +1,24 @@ -from opentelemetry import trace -from opentelemetry.sdk.trace import TracerProvider -from opentelemetry.sdk.trace.export import BatchExportSpanProcessor +# from opentelemetry import trace +# from opentelemetry.sdk.trace import TracerProvider +# from opentelemetry.sdk.trace.export import BatchExportSpanProcessor -from azure_monitor import AzureMonitorSpanExporter +# from azure_monitor import AzureMonitorSpanExporter -# Callback function to add os_type: linux to span properties -def callback_function(envelope): - envelope.data.baseData.properties["os_type"] = "linux" - return True +# # Callback function to add os_type: linux to span properties +# def callback_function(envelope): +# envelope.data.baseData.properties["os_type"] = "linux" +# return True -exporter = AzureMonitorSpanExporter( - connection_string="InstrumentationKey=" -) -exporter.add_telemetry_processor(callback_function) +# exporter = AzureMonitorSpanExporter( +# connection_string="InstrumentationKey=" +# ) +# exporter.add_telemetry_processor(callback_function) -trace.set_preferred_tracer_provider_implementation(lambda T: TracerProvider()) -tracer = trace.get_tracer(__name__) -span_processor = BatchExportSpanProcessor(exporter) -trace.tracer_provider().add_span_processor(span_processor) +# trace.set_preferred_tracer_provider_implementation(lambda T: TracerProvider()) +# tracer = trace.get_tracer(__name__) +# span_processor = BatchExportSpanProcessor(exporter) +# trace.tracer_provider().add_span_processor(span_processor) -with tracer.start_as_current_span("hello"): - print("Hello, World!") +# with tracer.start_as_current_span("hello"): +# print("Hello, World!") From f30b1c833146a810506c9c3ffbcda5b562c6b30c Mon Sep 17 00:00:00 2001 From: Leighton Date: Tue, 10 Mar 2020 11:01:49 -0700 Subject: [PATCH 060/109] revert tests --- azure_monitor/tests/metrics/test_metrics.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/azure_monitor/tests/metrics/test_metrics.py b/azure_monitor/tests/metrics/test_metrics.py index f83844c..a44b6ea 100644 --- a/azure_monitor/tests/metrics/test_metrics.py +++ b/azure_monitor/tests/metrics/test_metrics.py @@ -7,7 +7,7 @@ from unittest import mock from opentelemetry import metrics -from opentelemetry.sdk.metrics import Counter, MeterProvider +from opentelemetry.sdk.metrics import Counter, Meter from opentelemetry.sdk.metrics.export import MetricRecord, MetricsExportResult from opentelemetry.sdk.metrics.export.aggregate import CounterAggregator from opentelemetry.sdk.util import ns_to_iso_str @@ -24,8 +24,8 @@ def setUpClass(self): "APPINSIGHTS_INSTRUMENTATIONKEY" ] = "1234abcd-5678-4efa-8abc-1234567890ab" - metrics.set_preferred_meter_provider_implementation(lambda _: MeterProvider()) - self._meter = metrics.get_meter(__name__) + metrics.set_preferred_meter_implementation(lambda _: Meter()) + self._meter = metrics.meter() self._test_metric = self._meter.create_metric( "testname", "testdesc", "unit", int, Counter, ["environment"] ) From 5ca52ed4fc8564a2146a9c853464b73aada3dbc7 Mon Sep 17 00:00:00 2001 From: Hector Hernandez Date: Tue, 10 Mar 2020 11:47:58 -0700 Subject: [PATCH 061/109] WIP --- azure_monitor/setup.cfg | 1 + azure_monitor/src/azure_monitor/__init__.py | 11 +- .../azure_monitor/auto_collection/__init__.py | 35 +++ .../http_dependency_metrics.py | 74 +++++ .../auto_collection/http_request_metrics.py | 138 +++++++++ .../auto_collection/performance_metrics.py | 108 +++++++ .../{exporter.py => export/__init__.py} | 54 +++- .../metrics/__init__.py} | 10 +- .../{trace.py => export/trace/__init__.py} | 12 +- azure_monitor/src/azure_monitor/options.py | 149 ++++++++++ azure_monitor/src/azure_monitor/utils.py | 174 ------------ .../tests/auto_collection/__init__.py | 2 + .../test_performance_metrics.py | 56 ++++ azure_monitor/tests/metrics/test_metrics.py | 22 +- azure_monitor/tests/test_base_exporter.py | 6 +- azure_monitor/tests/test_options.py | 255 +++++++++++++++++ azure_monitor/tests/test_utils.py | 267 +----------------- azure_monitor/tests/trace/test_trace.py | 17 +- 18 files changed, 924 insertions(+), 467 deletions(-) create mode 100644 azure_monitor/src/azure_monitor/auto_collection/__init__.py create mode 100644 azure_monitor/src/azure_monitor/auto_collection/http_dependency_metrics.py create mode 100644 azure_monitor/src/azure_monitor/auto_collection/http_request_metrics.py create mode 100644 azure_monitor/src/azure_monitor/auto_collection/performance_metrics.py rename azure_monitor/src/azure_monitor/{exporter.py => export/__init__.py} (79%) rename azure_monitor/src/azure_monitor/{metrics.py => export/metrics/__init__.py} (92%) rename azure_monitor/src/azure_monitor/{trace.py => export/trace/__init__.py} (96%) create mode 100644 azure_monitor/src/azure_monitor/options.py create mode 100644 azure_monitor/tests/auto_collection/__init__.py create mode 100644 azure_monitor/tests/auto_collection/test_performance_metrics.py create mode 100644 azure_monitor/tests/test_options.py diff --git a/azure_monitor/setup.cfg b/azure_monitor/setup.cfg index c52347e..0660283 100644 --- a/azure_monitor/setup.cfg +++ b/azure_monitor/setup.cfg @@ -29,6 +29,7 @@ packages=find_namespace: install_requires = opentelemetry-api >= 0.4a0 opentelemetry-sdk >= 0.4a0 + psutil >= 5.6.3 requests ~= 2.0 [options.packages.find] diff --git a/azure_monitor/src/azure_monitor/__init__.py b/azure_monitor/src/azure_monitor/__init__.py index f3a6f7b..63520fb 100644 --- a/azure_monitor/src/azure_monitor/__init__.py +++ b/azure_monitor/src/azure_monitor/__init__.py @@ -1,6 +1,11 @@ # Copyright (c) Microsoft Corporation. All rights reserved. # Licensed under the MIT License. -from azure_monitor.trace import AzureMonitorSpanExporter -from azure_monitor.version import __version__ # noqa +from azure_monitor.export.metrics import AzureMonitorMetricsExporter +from azure_monitor.export.trace import AzureMonitorSpanExporter +from azure_monitor.options import ExporterOptions -__all__ = ["AzureMonitorSpanExporter"] +__all__ = [ + "AzureMonitorMetricsExporter", + "AzureMonitorSpanExporter", + "ExporterOptions", +] diff --git a/azure_monitor/src/azure_monitor/auto_collection/__init__.py b/azure_monitor/src/azure_monitor/auto_collection/__init__.py new file mode 100644 index 0000000..e90a66b --- /dev/null +++ b/azure_monitor/src/azure_monitor/auto_collection/__init__.py @@ -0,0 +1,35 @@ +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. +# +from opentelemetry.metrics import LabelSet, Meter + +from azure_monitor.auto_collection.performance_metrics import ( + PerformanceMetrics, +) +from azure_monitor.auto_collection.http_dependency_metrics import ( + HttpDependencyMetrics, +) +from azure_monitor.auto_collection.http_request_metrics import ( + HttpRequestMetrics, +) +from azure_monitor.utils import PeriodicTask + +__all__ = ["AutoCollection", "HttpDependencyMetrics", "HttpRequestMetrics", "PerformanceMetrics"] + + +class AutoCollection: + def __init__( + self, meter: Meter, label_set: LabelSet, collection_interval: int = 60 + ): + self._performance_metrics = PerformanceMetrics(meter, label_set) + self._dependency_metrics = HttpDependencyMetrics(meter, label_set) + self._request_metrics = HttpRequestMetrics(meter, label_set) + + self._collect_task = PeriodicTask( + interval=collection_interval, function=self.collect + ) + + def collect(self): + self._performance_metrics.track() + self._dependency_metrics.track() + self._request_metrics.track() diff --git a/azure_monitor/src/azure_monitor/auto_collection/http_dependency_metrics.py b/azure_monitor/src/azure_monitor/auto_collection/http_dependency_metrics.py new file mode 100644 index 0000000..7c7a9c4 --- /dev/null +++ b/azure_monitor/src/azure_monitor/auto_collection/http_dependency_metrics.py @@ -0,0 +1,74 @@ +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. +import logging +import threading +import time + +import requests + +from opentelemetry.metrics import LabelSet, Meter +from opentelemetry.sdk.metrics import Gauge + +logger = logging.getLogger(__name__) + + +def dependency_patch(self) -> None: + result = ORIGINAL_REQUEST(*args, **kwargs) + # Only collect request metric if sent from non-exporter thread + if not execution_context.is_exporter(): + # We don't want multiple threads updating this at once + with _dependency_lock: + count = dependency_map.get("count", 0) + dependency_map["count"] = count + 1 + return result + + +class HttpDependencyMetrics: + def __init__(self, meter: Meter, label_set: LabelSet): + self._meter = meter + self._label_set = label_set + # Patch requests + requests.Session.request = dependency_patch + dependency_rate_metric = self._meter.create_metric( + "\\ApplicationInsights\\Dependency Calls/Sec", + "Outgoing Requests per second", + "rps", + int, + Gauge, + ) + self._dependency_rate_handle = dependency_rate_metric.get_handle( + self._label_set + ) + + def track(self) -> None: + self._track_dependency_rate() + + def _track_dependency_rate(self) -> None: + """ Track Dependency rate + + Calculated by obtaining by getting the number of outgoing requests made + using the requests library within an elapsed time and dividing that + value over the elapsed time. + """ + current_count = dependency_map.get("count", 0) + current_time = time.time() + last_count = dependency_map.get("last_count", 0) + last_time = dependency_map.get("last_time") + last_result = dependency_map.get("last_result", 0) + + try: + # last_time is None the very first time this function is called + if last_time is not None: + elapsed_seconds = current_time - last_time + interval_count = current_count - last_count + result = interval_count / elapsed_seconds + else: + result = 0 + dependency_map["last_time"] = current_time + dependency_map["last_count"] = current_count + dependency_map["last_result"] = result + self._dependency_rate_handle.set(result) + except ZeroDivisionError: + # If elapsed_seconds is 0, exporter call made too close to previous + # Return the previous result if this is the case + self._dependency_rate_handle.set(last_result) diff --git a/azure_monitor/src/azure_monitor/auto_collection/http_request_metrics.py b/azure_monitor/src/azure_monitor/auto_collection/http_request_metrics.py new file mode 100644 index 0000000..0531c75 --- /dev/null +++ b/azure_monitor/src/azure_monitor/auto_collection/http_request_metrics.py @@ -0,0 +1,138 @@ +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. +import logging +import threading +import time + +from http.server import HTTPServer +from opentelemetry.metrics import LabelSet, Meter +from opentelemetry.sdk.metrics import Gauge + +logger = logging.getLogger(__name__) + +ORIGINAL_CONSTRUCTOR = HTTPServer.__init__ + + +def request_patch(func): + def wrapper(self=None): + start_time = time.time() + func(self) + end_time = time.time() + + with _requests_lock: + # Update Count + count = requests_map.get("count", 0) + requests_map["count"] = count + 1 + # Update duration + duration = requests_map.get("duration", 0) + requests_map["duration"] = duration + (end_time - start_time) + + return wrapper + + +def server_patch(*args, **kwargs): + if len(args) >= 3: + handler = args[2] + if handler: + # Patch the handler methods if they exist + if "do_DELETE" in dir(handler): + handler.do_DELETE = request_patch(handler.do_DELETE) + if "do_GET" in dir(handler): + handler.do_GET = request_patch(handler.do_GET) + if "do_HEAD" in dir(handler): + handler.do_HEAD = request_patch(handler.do_HEAD) + if "do_OPTIONS" in dir(handler): + handler.do_OPTIONS = request_patch(handler.do_OPTIONS) + if "do_POST" in dir(handler): + handler.do_POST = request_patch(handler.do_POST) + if "do_PUT" in dir(handler): + handler.do_PUT = request_patch(handler.do_PUT) + result = ORIGINAL_CONSTRUCTOR(*args, **kwargs) + return result + + +class HttpRequestMetrics: + def __init__(self, meter: Meter, label_set: LabelSet): + self._meter = meter + self._label_set = label_set + # Patch the HTTPServer handler to track request information + HTTPServer.__init__ = server_patch + + request_duration_metric = self._meter.create_metric( + "\\ASP.NET Applications(??APP_W3SVC_PROC??)\\Request Execution Time", + "Incoming Requests Average Execution Time", + "milliseconds", + int, + Gauge, + ) + self._request_duration_handle = request_duration_metric.get_handle( + self._label_set + ) + + request_rate_metric = self._meter.create_metric( + "\\ASP.NET Applications(??APP_W3SVC_PROC??)\\Requests/Sec", + "Incoming Requests Average Execution Rate", + "milliseconds", + int, + Gauge, + ) + self._request_rate_handle = request_rate_metric.get_handle( + self._label_set + ) + + def track(self) -> None: + self._track_request_duration() + + def _track_request_duration(self) -> None: + """ Track Request execution time + + Calculated by getting the time it takes to make an incoming request + and dividing over the amount of incoming requests over an elapsed time. + """ + last_average_duration = requests_map.get("last_average_duration", 0) + interval_duration = requests_map.get("duration", 0) - requests_map.get( + "last_duration", 0 + ) + interval_count = requests_map.get("count", 0) - requests_map.get( + "last_count", 0 + ) + try: + result = interval_duration / interval_count + requests_map["last_average_duration"] = result + requests_map["last_duration"] = requests_map.get("duration", 0) + # Convert to milliseconds + self._request_duration_handle(result * 1000.0) + except ZeroDivisionError: + # If interval_count is 0, exporter call made too close to previous + # Return the previous result if this is the case + self._request_duration_handle(last_average_duration * 1000.0) + + def _track_request_rate(self) -> None: + """ Track Request execution rate + + Calculated by obtaining by getting the number of incoming requests + made to an HTTPServer within an elapsed time and dividing that value + over the elapsed time. + """ + current_time = time.time() + last_rate = requests_map.get("last_rate", 0) + last_time = requests_map.get("last_time") + + try: + # last_rate_time is None the first time this function is called + if last_time is not None: + interval_time = current_time - requests_map.get("last_time", 0) + interval_count = requests_map.get( + "count", 0 + ) - requests_map.get("last_count", 0) + result = interval_count / interval_time + else: + result = 0 + requests_map["last_time"] = current_time + requests_map["last_count"] = requests_map.get("count", 0) + requests_map["last_rate"] = result + self._request_rate_handle(result) + except ZeroDivisionError: + # If elapsed_seconds is 0, exporter call made too close to previous + # Return the previous result if this is the case + self._request_rate_handle(last_rate) diff --git a/azure_monitor/src/azure_monitor/auto_collection/performance_metrics.py b/azure_monitor/src/azure_monitor/auto_collection/performance_metrics.py new file mode 100644 index 0000000..a56c6b6 --- /dev/null +++ b/azure_monitor/src/azure_monitor/auto_collection/performance_metrics.py @@ -0,0 +1,108 @@ +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. +import logging + +import psutil + +from opentelemetry.metrics import LabelSet, Meter +from opentelemetry.sdk.metrics import Gauge + +logger = logging.getLogger(__name__) +PROCESS = psutil.Process() + + +class PerformanceMetrics: + def __init__(self, meter: Meter, label_set: LabelSet): + self._meter = meter + self._label_set = label_set + # Create performance metrics + cpu_metric = self._meter.create_metric( + "\\Processor(_Total)\\% Processor Time", + "Processor time as a percentage", + "percentage", + float, + Gauge, + ) + self._cpu_handle = cpu_metric.get_handle(self._label_set) + + memory_metric = self._meter.create_metric( + "\\Memory\\Available Bytes", + "Amount of available memory in bytes", + "byte", + int, + Gauge, + ) + self._memory_handle = memory_metric.get_handle(self._label_set) + + process_cpu_metric = self._meter.create_metric( + "\\Process(??APP_WIN32_PROC??)\\% Processor Time", + "Process CPU usage as a percentage", + "percentage", + float, + Gauge, + ) + self._process_cpu_handle = process_cpu_metric.get_handle( + self._label_set + ) + + process_memory_metric = self._meter.create_metric( + "\\Process(??APP_WIN32_PROC??)\\Private Bytes", + "Amount of memory process has used in bytes", + "byte", + int, + Gauge, + ) + self._process_memory_handle = process_memory_metric.get_handle( + self._label_set + ) + + def track(self) -> None: + self._track_cpu() + self._track_process_cpu() + self._track_memory() + self._track_process_memory() + + def _track_cpu(self) -> None: + """ Track CPU time + + Processor time is defined as a float representing the current system + wide CPU utilization minus idle CPU time as a percentage. Idle CPU + time is defined as the time spent doing nothing. Return values range + from 0.0 to 100.0 inclusive. + """ + cpu_times_percent = psutil.cpu_times_percent() + self._cpu_handle.set(100 - cpu_times_percent.idle) + + def _track_process_cpu(self) -> None: + """ Track Process CPU time + + Returns a derived gauge for the CPU usage for the current process. + Return values range from 0.0 to 100.0 inclusive. + """ + try: + # In the case of a process running on multiple threads on different + # CPU cores, the returned value of cpu_percent() can be > 100.0. We + # normalize the cpu process using the number of logical CPUs + cpu_count = psutil.cpu_count(logical=True) + self._process_cpu_handle.set(PROCESS.cpu_percent() / cpu_count) + except Exception: + logger.exception("Error handling get process cpu usage.") + + def _track_memory(self) -> None: + """ Track Memory + + Available memory is defined as memory that can be given instantly to + processes without the system going into swap. + """ + self._memory_handle.set(psutil.virtual_memory().available) + + def _track_process_memory(self) -> None: + """ Track Memory + + Available memory is defined as memory that can be given instantly to + processes without the system going into swap. + """ + try: + self._process_memory_handle.set(PROCESS.memory_info().rss) + except Exception: + logger.exception("Error handling get process private bytes.") diff --git a/azure_monitor/src/azure_monitor/exporter.py b/azure_monitor/src/azure_monitor/export/__init__.py similarity index 79% rename from azure_monitor/src/azure_monitor/exporter.py rename to azure_monitor/src/azure_monitor/export/__init__.py index 1145ecc..be954da 100644 --- a/azure_monitor/src/azure_monitor/exporter.py +++ b/azure_monitor/src/azure_monitor/export/__init__.py @@ -6,17 +6,29 @@ import requests -from azure_monitor import utils +from enum import Enum + +from opentelemetry.sdk.metrics.export import MetricsExportResult +from opentelemetry.sdk.trace.export import SpanExportResult + +from azure_monitor.options import ExporterOptions from azure_monitor.protocol import Envelope from azure_monitor.storage import LocalFileStorage + logger = logging.getLogger(__name__) +class ExportResult(Enum): + SUCCESS = 0 + FAILED_RETRYABLE = 1 + FAILED_NOT_RETRYABLE = 2 + + class BaseExporter: def __init__(self, **options): self._telemetry_processors = [] - self.options = utils.Options(**options) + self.options = ExporterOptions(**options) self.storage = LocalFileStorage( path=self.options.storage_path, max_size=self.options.storage_max_size, @@ -72,14 +84,14 @@ def _transmit_from_storage(self) -> None: if blob.lease(self.options.timeout + 5): envelopes = blob.get() # TODO: handle error result = self._transmit(envelopes) - if result == utils.ExportResult.FAILED_RETRYABLE: + if result == ExportResult.FAILED_RETRYABLE: blob.lease(1) else: blob.delete(silent=True) def _transmit( self, envelopes_to_export: typing.List[Envelope] - ) -> utils.ExportResult: + ) -> ExportResult: """ Transmit the data envelopes to the ingestion service. @@ -99,7 +111,7 @@ def _transmit( ) except Exception as ex: logger.warning("Transient client side error %s.", ex) - return utils.ExportResult.FAILED_RETRYABLE + return ExportResult.FAILED_RETRYABLE text = "N/A" data = None @@ -115,7 +127,7 @@ def _transmit( if response.status_code == 200: logger.info("Transmission succeeded: %s.", text) - return utils.ExportResult.SUCCESS + return ExportResult.SUCCESS if response.status_code == 206: # Partial Content # TODO: store the unsent data if data: @@ -146,7 +158,7 @@ def _transmit( text, ex, ) - return utils.ExportResult.FAILED_NOT_RETRYABLE + return ExportResult.FAILED_NOT_RETRYABLE # cannot parse response body, fallback to retry if response.status_code in ( @@ -155,8 +167,30 @@ def _transmit( 500, # Internal Server Error 503, # Service Unavailable ): - return utils.ExportResult.FAILED_RETRYABLE + return ExportResult.FAILED_RETRYABLE - return utils.ExportResult.FAILED_NOT_RETRYABLE + return ExportResult.FAILED_NOT_RETRYABLE # No spans to export - return utils.ExportResult.SUCCESS + return ExportResult.SUCCESS + + +def get_trace_export_result(result: ExportResult) -> SpanExportResult: + if result == ExportResult.SUCCESS: + return SpanExportResult.SUCCESS + elif result == ExportResult.FAILED_RETRYABLE: + return SpanExportResult.FAILED_RETRYABLE + elif result == ExportResult.FAILED_NOT_RETRYABLE: + return SpanExportResult.FAILED_NOT_RETRYABLE + else: + return None + + +def get_metrics_export_result(result: ExportResult) -> MetricsExportResult: + if result == ExportResult.SUCCESS: + return MetricsExportResult.SUCCESS + elif result == ExportResult.FAILED_RETRYABLE: + return MetricsExportResult.FAILED_RETRYABLE + elif result == ExportResult.FAILED_NOT_RETRYABLE: + return MetricsExportResult.FAILED_NOT_RETRYABLE + else: + return None diff --git a/azure_monitor/src/azure_monitor/metrics.py b/azure_monitor/src/azure_monitor/export/metrics/__init__.py similarity index 92% rename from azure_monitor/src/azure_monitor/metrics.py rename to azure_monitor/src/azure_monitor/export/metrics/__init__.py index 00d2534..c296c66 100644 --- a/azure_monitor/src/azure_monitor/metrics.py +++ b/azure_monitor/src/azure_monitor/export/metrics/__init__.py @@ -14,7 +14,11 @@ from opentelemetry.sdk.util import ns_to_iso_str from azure_monitor import protocol, utils -from azure_monitor.exporter import BaseExporter +from azure_monitor.export import ( + BaseExporter, + ExportResult, + get_metrics_export_result, +) logger = logging.getLogger(__name__) @@ -35,10 +39,10 @@ def export( result = self._transmit(envelopes_to_export) if result == MetricsExportResult.FAILED_RETRYABLE: self.storage.put(envelopes, result) - if result == utils.ExportResult.SUCCESS: + if result == ExportResult.SUCCESS: # Try to send any cached events self._transmit_from_storage() - return utils.get_metrics_export_result(result) + return get_metrics_export_result(result) except Exception: logger.exception("Exception occurred while exporting the data.") diff --git a/azure_monitor/src/azure_monitor/trace.py b/azure_monitor/src/azure_monitor/export/trace/__init__.py similarity index 96% rename from azure_monitor/src/azure_monitor/trace.py rename to azure_monitor/src/azure_monitor/export/trace/__init__.py index 23f7be6..a40aae2 100644 --- a/azure_monitor/src/azure_monitor/trace.py +++ b/azure_monitor/src/azure_monitor/export/trace/__init__.py @@ -11,7 +11,11 @@ from opentelemetry.trace.status import StatusCanonicalCode from azure_monitor import protocol, utils -from azure_monitor.exporter import BaseExporter +from azure_monitor.export import ( + BaseExporter, + ExportResult, + get_trace_export_result, +) logger = logging.getLogger(__name__) @@ -28,12 +32,12 @@ def export(self, spans: Sequence[Span]) -> SpanExportResult: ) try: result = self._transmit(envelopes_to_export) - if result == utils.ExportResult.FAILED_RETRYABLE: + if result == ExportResult.FAILED_RETRYABLE: self.storage.put(envelopes, result) - if result == utils.ExportResult.SUCCESS: + if result == ExportResult.SUCCESS: # Try to send any cached events self._transmit_from_storage() - return utils.get_trace_export_result(result) + return get_trace_export_result(result) except Exception: logger.exception("Exception occurred while exporting the data.") diff --git a/azure_monitor/src/azure_monitor/options.py b/azure_monitor/src/azure_monitor/options.py new file mode 100644 index 0000000..a6896c5 --- /dev/null +++ b/azure_monitor/src/azure_monitor/options.py @@ -0,0 +1,149 @@ +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. +import os +import sys +import re +import typing + +from azure_monitor.protocol import BaseObject + + +INGESTION_ENDPOINT = "ingestionendpoint" +INSTRUMENTATION_KEY = "instrumentationkey" + +# Validate UUID format +# Specs taken from https://tools.ietf.org/html/rfc4122 +uuid_regex_pattern = re.compile( + "^[0-9a-f]{8}-" + "[0-9a-f]{4}-" + "[1-5][0-9a-f]{3}-" + "[89ab][0-9a-f]{3}-" + "[0-9a-f]{12}$" +) + + +class ExporterOptions(BaseObject): + """Options to configure Azure exporters. + + Args: + connection_string: Azure Connection String. + instrumentation_key: Azure Instrumentation Key. + storage_maintenance_period: Local storage maintenance interval in seconds. + storage_max_size: Local storage maximum size in bytes. + storage_path: Local storage file path. + storage_retention_period: Local storage retention period in seconds + timeout: Request timeout in seconds + """ + + __slots__ = ( + "connection_string", + "endpoint", + "instrumentation_key", + "storage_maintenance_period", + "storage_max_size", + "storage_path", + "storage_retention_period", + "timeout", + ) + + def __init__( + self, + connection_string: str = None, + instrumentation_key: str = None, + storage_maintenance_period: int = 60, + storage_max_size: int = 100 * 1024 * 1024, + storage_path: str = os.path.join( + os.path.expanduser("~"), + ".opentelemetry", + ".azure", + os.path.basename(sys.argv[0]) or ".console", + ), + storage_retention_period: int = 7 * 24 * 60 * 60, + timeout: int = 10.0, # networking timeout in seconds + ) -> None: + self.connection_string = connection_string + self.instrumentation_key = instrumentation_key + self.storage_maintenance_period = storage_maintenance_period + self.storage_max_size = storage_max_size + self.storage_path = storage_path + self.storage_retention_period = storage_retention_period + self.timeout = timeout + self.endpoint = "" + self._initialize() + self._validate_instrumentation_key() + + def _initialize(self) -> None: + code_cs = self._parse_connection_string(self.connection_string) + code_ikey = self.instrumentation_key + env_cs = self._parse_connection_string( + os.getenv("APPLICATIONINSIGHTS_CONNECTION_STRING") + ) + env_ikey = os.getenv("APPINSIGHTS_INSTRUMENTATIONKEY") + + # The priority of which value takes on the instrumentation key is: + # 1. Key from explicitly passed in connection string + # 2. Key from explicitly passed in instrumentation key + # 3. Key from connection string in environment variable + # 4. Key from instrumentation key in environment variable + self.instrumentation_key = ( + code_cs.get(INSTRUMENTATION_KEY) + or code_ikey + or env_cs.get(INSTRUMENTATION_KEY) + or env_ikey + ) + # The priority of the ingestion endpoint is as follows: + # 1. The endpoint explicitly passed in connection string + # 2. The endpoint from the connection string in environment variable + # 3. The default breeze endpoint + endpoint = ( + code_cs.get(INGESTION_ENDPOINT) + or env_cs.get(INGESTION_ENDPOINT) + or "https://dc.services.visualstudio.com" + ) + self.endpoint = endpoint + "/v2/track" + + def _validate_instrumentation_key(self) -> None: + """Validates the instrumentation key used for Azure Monitor. + An instrumentation key cannot be null or empty. An instrumentation key + is valid for Azure Monitor only if it is a valid UUID. + :param instrumentation_key: The instrumentation key to validate + """ + if not self.instrumentation_key: + raise ValueError("Instrumentation key cannot be none or empty.") + match = uuid_regex_pattern.match(self.instrumentation_key) + if not match: + raise ValueError("Invalid instrumentation key.") + + def _parse_connection_string(self, connection_string) -> typing.Dict: + if connection_string is None: + return {} + try: + pairs = connection_string.split(";") + result = dict(s.split("=") for s in pairs) + # Convert keys to lower-case due to case type-insensitive checking + result = {key.lower(): value for key, value in result.items()} + except Exception: + raise ValueError("Invalid connection string") + # Validate authorization + auth = result.get("authorization") + if auth is not None and auth.lower() != "ikey": + raise ValueError("Invalid authorization mechanism") + # Construct the ingestion endpoint if not passed in explicitly + if result.get(INGESTION_ENDPOINT) is None: + endpoint_suffix = "" + location_prefix = "" + suffix = result.get("endpointsuffix") + if suffix is not None: + endpoint_suffix = suffix + # Get regional information if provided + prefix = result.get("location") + if prefix is not None: + location_prefix = prefix + "." + endpoint = "https://{0}dc.{1}".format( + location_prefix, endpoint_suffix + ) + result[INGESTION_ENDPOINT] = endpoint + else: + # Default to None if cannot construct + result[INGESTION_ENDPOINT] = None + return result diff --git a/azure_monitor/src/azure_monitor/utils.py b/azure_monitor/src/azure_monitor/utils.py index 5b13f63..0ad5da3 100644 --- a/azure_monitor/src/azure_monitor/utils.py +++ b/azure_monitor/src/azure_monitor/utils.py @@ -3,15 +3,10 @@ import locale import os import platform -import re import sys import threading import time -import typing -from enum import Enum -from opentelemetry.sdk.metrics.export import MetricsExportResult -from opentelemetry.sdk.trace.export import SpanExportResult from opentelemetry.sdk.version import __version__ as opentelemetry_version from azure_monitor.protocol import BaseObject @@ -41,175 +36,6 @@ def ns_to_duration(nanoseconds): ) -INGESTION_ENDPOINT = "ingestionendpoint" -INSTRUMENTATION_KEY = "instrumentationkey" - -# Validate UUID format -# Specs taken from https://tools.ietf.org/html/rfc4122 -uuid_regex_pattern = re.compile( - "^[0-9a-f]{8}-" - "[0-9a-f]{4}-" - "[1-5][0-9a-f]{3}-" - "[89ab][0-9a-f]{3}-" - "[0-9a-f]{12}$" -) - - -class ExportResult(Enum): - SUCCESS = 0 - FAILED_RETRYABLE = 1 - FAILED_NOT_RETRYABLE = 2 - - -def get_trace_export_result(result: ExportResult) -> SpanExportResult: - if result == ExportResult.SUCCESS: - return SpanExportResult.SUCCESS - elif result == ExportResult.FAILED_RETRYABLE: - return SpanExportResult.FAILED_RETRYABLE - elif result == ExportResult.FAILED_NOT_RETRYABLE: - return SpanExportResult.FAILED_NOT_RETRYABLE - else: - return None - - -def get_metrics_export_result(result: ExportResult) -> MetricsExportResult: - if result == ExportResult.SUCCESS: - return MetricsExportResult.SUCCESS - elif result == ExportResult.FAILED_RETRYABLE: - return MetricsExportResult.FAILED_RETRYABLE - elif result == ExportResult.FAILED_NOT_RETRYABLE: - return MetricsExportResult.FAILED_NOT_RETRYABLE - else: - return None - - -class Options(BaseObject): - """Options to configure Azure exporters. - - Args: - connection_string: Azure Connection String. - instrumentation_key: Azure Instrumentation Key. - storage_maintenance_period: Local storage maintenance interval in seconds. - storage_max_size: Local storage maximum size in bytes. - storage_path: Local storage file path. - storage_retention_period: Local storage retention period in seconds - timeout: Request timeout in seconds - """ - - __slots__ = ( - "connection_string", - "endpoint", - "instrumentation_key", - "storage_maintenance_period", - "storage_max_size", - "storage_path", - "storage_retention_period", - "timeout", - ) - - def __init__( - self, - connection_string: str = None, - instrumentation_key: str = None, - storage_maintenance_period: int = 60, - storage_max_size: int = 100 * 1024 * 1024, - storage_path: str = os.path.join( - os.path.expanduser("~"), - ".opentelemetry", - ".azure", - os.path.basename(sys.argv[0]) or ".console", - ), - storage_retention_period: int = 7 * 24 * 60 * 60, - timeout: int = 10.0, # networking timeout in seconds - ) -> None: - self.connection_string = connection_string - self.instrumentation_key = instrumentation_key - self.storage_maintenance_period = storage_maintenance_period - self.storage_max_size = storage_max_size - self.storage_path = storage_path - self.storage_retention_period = storage_retention_period - self.timeout = timeout - self.endpoint = "" - self._initialize() - self._validate_instrumentation_key() - - def _initialize(self) -> None: - code_cs = self._parse_connection_string(self.connection_string) - code_ikey = self.instrumentation_key - env_cs = self._parse_connection_string( - os.getenv("APPLICATIONINSIGHTS_CONNECTION_STRING") - ) - env_ikey = os.getenv("APPINSIGHTS_INSTRUMENTATIONKEY") - - # The priority of which value takes on the instrumentation key is: - # 1. Key from explicitly passed in connection string - # 2. Key from explicitly passed in instrumentation key - # 3. Key from connection string in environment variable - # 4. Key from instrumentation key in environment variable - self.instrumentation_key = ( - code_cs.get(INSTRUMENTATION_KEY) - or code_ikey - or env_cs.get(INSTRUMENTATION_KEY) - or env_ikey - ) - # The priority of the ingestion endpoint is as follows: - # 1. The endpoint explicitly passed in connection string - # 2. The endpoint from the connection string in environment variable - # 3. The default breeze endpoint - endpoint = ( - code_cs.get(INGESTION_ENDPOINT) - or env_cs.get(INGESTION_ENDPOINT) - or "https://dc.services.visualstudio.com" - ) - self.endpoint = endpoint + "/v2/track" - - def _validate_instrumentation_key(self) -> None: - """Validates the instrumentation key used for Azure Monitor. - An instrumentation key cannot be null or empty. An instrumentation key - is valid for Azure Monitor only if it is a valid UUID. - :param instrumentation_key: The instrumentation key to validate - """ - if not self.instrumentation_key: - raise ValueError("Instrumentation key cannot be none or empty.") - match = uuid_regex_pattern.match(self.instrumentation_key) - if not match: - raise ValueError("Invalid instrumentation key.") - - def _parse_connection_string(self, connection_string) -> typing.Dict: - if connection_string is None: - return {} - try: - pairs = connection_string.split(";") - result = dict(s.split("=") for s in pairs) - # Convert keys to lower-case due to case type-insensitive checking - result = {key.lower(): value for key, value in result.items()} - except Exception: - raise ValueError("Invalid connection string") - # Validate authorization - auth = result.get("authorization") - if auth is not None and auth.lower() != "ikey": - raise ValueError("Invalid authorization mechanism") - # Construct the ingestion endpoint if not passed in explicitly - if result.get(INGESTION_ENDPOINT) is None: - endpoint_suffix = "" - location_prefix = "" - suffix = result.get("endpointsuffix") - if suffix is not None: - endpoint_suffix = suffix - # Get regional information if provided - prefix = result.get("location") - if prefix is not None: - location_prefix = prefix + "." - endpoint = "https://{0}dc.{1}".format( - location_prefix, endpoint_suffix - ) - result[INGESTION_ENDPOINT] = endpoint - else: - # Default to None if cannot construct - result[INGESTION_ENDPOINT] = None - return result - - class PeriodicTask(threading.Thread): """Thread that periodically calls a given function. diff --git a/azure_monitor/tests/auto_collection/__init__.py b/azure_monitor/tests/auto_collection/__init__.py new file mode 100644 index 0000000..5b7f7a9 --- /dev/null +++ b/azure_monitor/tests/auto_collection/__init__.py @@ -0,0 +1,2 @@ +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. diff --git a/azure_monitor/tests/auto_collection/test_performance_metrics.py b/azure_monitor/tests/auto_collection/test_performance_metrics.py new file mode 100644 index 0000000..2670714 --- /dev/null +++ b/azure_monitor/tests/auto_collection/test_performance_metrics.py @@ -0,0 +1,56 @@ +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. + + +import json +import os +import shutil +import unittest +from unittest import mock + +from opentelemetry import metrics +from opentelemetry.sdk.metrics import Gauge, Meter +from opentelemetry.sdk.metrics.export import MetricRecord, MetricsExportResult +from opentelemetry.sdk.metrics.export.aggregate import CounterAggregator +from opentelemetry.sdk.util import ns_to_iso_str + +from azure_monitor.auto_collection import PerformanceMetrics + + +class TestPerformanceMetrics(unittest.TestCase): + @classmethod + def setUpClass(cls): + cls._meter_defaults = (metrics._METER, metrics._METER_FACTORY) + metrics.set_preferred_meter_implementation(lambda _: Meter()) + cls._meter = metrics.meter() + kvp = {"environment": "staging"} + cls._test_label_set = cls._meter.get_label_set(kvp) + + @classmethod + def tearDownClass(cls): + metrics._METER, metrics._METER_FACTORY = cls._meter_defaults + + def test_constructor(self): + """Test the constructor.""" + auto_collector = PerformanceMetrics( + meter=self._meter, label_set=self._test_label_set + ) + self.assertEqual(auto_collector._meter, self._meter) + self.assertEqual(auto_collector._label_set, self._test_label_set) + + result_metric = list(self._meter.metrics)[0] + self.assertIsInstance(result_metric, Gauge) + self.assertEqual( + result_metric.name, "\Processor(_Total)\% Processor Time" + ) + self.assertEqual( + result_metric.description, "Processor time as a percentage" + ) + self.assertEqual(result_metric.unit, "percentage") + self.assertEqual(result_metric.value_type, float) + self.assertEqual( + result_metric.get_handle( + self._test_label_set + ).aggregator.checkpoint, + 0, + ) diff --git a/azure_monitor/tests/metrics/test_metrics.py b/azure_monitor/tests/metrics/test_metrics.py index 37f64b5..0dd6e44 100644 --- a/azure_monitor/tests/metrics/test_metrics.py +++ b/azure_monitor/tests/metrics/test_metrics.py @@ -14,32 +14,38 @@ from opentelemetry.sdk.metrics.export.aggregate import CounterAggregator from opentelemetry.sdk.util import ns_to_iso_str -from azure_monitor.metrics import AzureMonitorMetricsExporter +from azure_monitor.export import ExportResult +from azure_monitor.export.metrics import AzureMonitorMetricsExporter +from azure_monitor.options import ExporterOptions from azure_monitor.protocol import Data, Envelope, MetricData -from azure_monitor.utils import ExportResult, Options class TestAzureMetricsExporter(unittest.TestCase): @classmethod - def setUpClass(self): + def setUpClass(cls): os.environ[ "APPINSIGHTS_INSTRUMENTATIONKEY" ] = "1234abcd-5678-4efa-8abc-1234567890ab" + cls._meter_defaults = (metrics._METER, metrics._METER_FACTORY) metrics.set_preferred_meter_implementation(lambda _: Meter()) - self._meter = metrics.meter() - self._test_metric = self._meter.create_metric( + cls._meter = metrics.meter() + cls._test_metric = cls._meter.create_metric( "testname", "testdesc", "unit", int, Counter, ["environment"] ) kvp = {"environment": "staging"} - self._test_label_set = self._meter.get_label_set(kvp) + cls._test_label_set = cls._meter.get_label_set(kvp) + + @classmethod + def tearDownClass(cls): + metrics._METER, metrics._METER_PROVIDER_FACTORY = cls._meter_defaults def test_constructor(self): """Test the constructor.""" exporter = AzureMonitorMetricsExporter( instrumentation_key="4321abcd-5678-4efa-8abc-1234567890ab" ) - self.assertIsInstance(exporter.options, Options) + self.assertIsInstance(exporter.options, ExporterOptions) self.assertEqual( exporter.options.instrumentation_key, "4321abcd-5678-4efa-8abc-1234567890ab", @@ -51,7 +57,7 @@ def test_export(self,): ) exporter = AzureMonitorMetricsExporter() with mock.patch( - "azure_monitor.metrics.AzureMonitorMetricsExporter._transmit" + "azure_monitor.export.metrics.AzureMonitorMetricsExporter._transmit" ) as transmit: # noqa: E501 transmit.return_value = ExportResult.SUCCESS result = exporter.export([record]) diff --git a/azure_monitor/tests/test_base_exporter.py b/azure_monitor/tests/test_base_exporter.py index fd74e46..72ceb15 100644 --- a/azure_monitor/tests/test_base_exporter.py +++ b/azure_monitor/tests/test_base_exporter.py @@ -4,9 +4,9 @@ import os import unittest -from azure_monitor.exporter import BaseExporter +from azure_monitor.export import BaseExporter +from azure_monitor.options import ExporterOptions from azure_monitor.protocol import Data, Envelope -from azure_monitor.utils import Options # pylint: disable=W0212 @@ -27,7 +27,7 @@ def test_constructor(self): storage_retention_period=4, timeout=5, ) - self.assertIsInstance(base.options, Options) + self.assertIsInstance(base.options, ExporterOptions) self.assertEqual( base.options.instrumentation_key, "4321abcd-5678-4efa-8abc-1234567890ab", diff --git a/azure_monitor/tests/test_options.py b/azure_monitor/tests/test_options.py new file mode 100644 index 0000000..69f119b --- /dev/null +++ b/azure_monitor/tests/test_options.py @@ -0,0 +1,255 @@ +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. + +import os +import unittest + +from azure_monitor.options import ExporterOptions + + +class TestOptions(unittest.TestCase): + def setUp(self): + os.environ.clear() + self._valid_instrumentation_key = ( + "1234abcd-5678-4efa-8abc-1234567890ab" + ) + + def test_validate_instrumentation_key(self): + options = ExporterOptions( + instrumentation_key=self._valid_instrumentation_key + ) + self.assertEqual( + options.instrumentation_key, self._valid_instrumentation_key + ) + + def test_invalid_key_none(self): + self.assertRaises( + ValueError, lambda: ExporterOptions(instrumentation_key=None) + ) + + def test_invalid_key_empty(self): + self.assertRaises( + ValueError, lambda: ExporterOptions(instrumentation_key="") + ) + + def test_invalid_key_prefix(self): + self.assertRaises( + ValueError, + lambda: ExporterOptions( + instrumentation_key="test1234abcd-5678-4efa-8abc-1234567890ab" + ), + ) + + def test_invalid_key_suffix(self): + self.assertRaises( + ValueError, + lambda: ExporterOptions( + instrumentation_key="1234abcd-5678-4efa-8abc-1234567890abtest" + ), + ) + + def test_invalid_key_length(self): + self.assertRaises( + ValueError, + lambda: ExporterOptions( + instrumentation_key="1234abcd-5678-4efa-8abc-12234567890ab" + ), + ) + + def test_invalid_key_dashes(self): + self.assertRaises( + ValueError, + lambda: ExporterOptions( + instrumentation_key="1234abcda5678-4efa-8abc-1234567890ab" + ), + ) + + def test_invalid_key_section1_length(self): + self.assertRaises( + ValueError, + lambda: ExporterOptions( + instrumentation_key="1234abcda-678-4efa-8abc-1234567890ab" + ), + ) + + def test_invalid_key_section2_length(self): + self.assertRaises( + ValueError, + lambda: ExporterOptions( + instrumentation_key="1234abcd-678-a4efa-8abc-1234567890ab" + ), + ) + + def test_invalid_key_section3_length(self): + self.assertRaises( + ValueError, + lambda: ExporterOptions( + instrumentation_key="1234abcd-6789-4ef-8cabc-1234567890ab" + ), + ) + + def test_invalid_key_section4_length(self): + self.assertRaises( + ValueError, + lambda: ExporterOptions( + instrumentation_key="1234abcd-678-4efa-8bc-11234567890ab" + ), + ) + + def test_invalid_key_section5_length(self): + self.assertRaises( + ValueError, + lambda: ExporterOptions( + instrumentation_key="234abcd-678-4efa-8abc-11234567890ab" + ), + ) + + def test_invalid_key_section1_hex(self): + self.assertRaises( + ValueError, + lambda: ExporterOptions( + instrumentation_key="x234abcd-5678-4efa-8abc-1234567890ab" + ), + ) + + def test_invalid_key_section2_hex(self): + self.assertRaises( + ValueError, + lambda: ExporterOptions( + instrumentation_key="1234abcd-x678-4efa-8abc-1234567890ab" + ), + ) + + def test_invalid_key_section3_hex(self): + self.assertRaises( + ValueError, + lambda: ExporterOptions( + instrumentation_key="1234abcd-5678-4xfa-8abc-1234567890ab" + ), + ) + + def test_invalid_key_section4_hex(self): + self.assertRaises( + ValueError, + lambda: ExporterOptions( + instrumentation_key="1234abcd-5678-4xfa-8abc-1234567890ab" + ), + ) + + def test_invalid_key_section5_hex(self): + self.assertRaises( + ValueError, + lambda: ExporterOptions( + instrumentation_key="1234abcd-5678-4xfa-8abc-1234567890ab" + ), + ) + + def test_invalid_key_version(self): + self.assertRaises( + ValueError, + lambda: ExporterOptions( + instrumentation_key="1234abcd-5678-6efa-8abc-1234567890ab" + ), + ) + + def test_invalid_key_variant(self): + self.assertRaises( + ValueError, + lambda: ExporterOptions( + instrumentation_key="1234abcd-5678-4efa-2abc-1234567890ab" + ), + ) + + def test_process_options_ikey_code_cs(self): + os.environ[ + "APPLICATIONINSIGHTS_CONNECTION_STRING" + ] = "Authorization=ikey;InstrumentationKey=789" + os.environ["APPINSIGHTS_INSTRUMENTATIONKEY"] = "101112" + options = ExporterOptions( + connection_string="Authorization=ikey;InstrumentationKey=" + + self._valid_instrumentation_key, + instrumentation_key="456", + ) + self.assertEqual( + options.instrumentation_key, self._valid_instrumentation_key + ) + + def test_process_options_ikey_code_ikey(self): + os.environ[ + "APPLICATIONINSIGHTS_CONNECTION_STRING" + ] = "Authorization=ikey;InstrumentationKey=789" + os.environ["APPINSIGHTS_INSTRUMENTATIONKEY"] = "101112" + options = ExporterOptions( + connection_string=None, + instrumentation_key=self._valid_instrumentation_key, + ) + self.assertEqual( + options.instrumentation_key, self._valid_instrumentation_key + ) + + def test_process_options_ikey_env_cs(self): + os.environ["APPLICATIONINSIGHTS_CONNECTION_STRING"] = ( + "Authorization=ikey;InstrumentationKey=" + + self._valid_instrumentation_key + ) + os.environ["APPINSIGHTS_INSTRUMENTATIONKEY"] = "101112" + options = ExporterOptions( + connection_string=None, instrumentation_key=None + ) + self.assertEqual( + options.instrumentation_key, self._valid_instrumentation_key + ) + + def test_process_options_ikey_env_ikey(self): + os.environ[ + "APPINSIGHTS_INSTRUMENTATIONKEY" + ] = self._valid_instrumentation_key + options = ExporterOptions( + connection_string=None, instrumentation_key=None + ) + self.assertEqual( + options.instrumentation_key, self._valid_instrumentation_key + ) + + def test_process_options_endpoint_code_cs(self): + os.environ[ + "APPLICATIONINSIGHTS_CONNECTION_STRING" + ] = "Authorization=ikey;IngestionEndpoint=456;InstrumentationKey=" + options = ExporterOptions( + connection_string="Authorization=ikey;IngestionEndpoint=123", + instrumentation_key=self._valid_instrumentation_key, + ) + self.assertEqual(options.endpoint, "123/v2/track") + + def test_process_options_endpoint_env_cs(self): + os.environ[ + "APPLICATIONINSIGHTS_CONNECTION_STRING" + ] = "Authorization=ikey;IngestionEndpoint=456" + options = ExporterOptions( + connection_string=None, + instrumentation_key=self._valid_instrumentation_key, + ) + self.assertEqual(options.endpoint, "456/v2/track") + + def test_process_options_endpoint_default(self): + options = ExporterOptions( + connection_string=None, + instrumentation_key=self._valid_instrumentation_key, + ) + self.assertEqual( + options.endpoint, "https://dc.services.visualstudio.com/v2/track" + ) + + def test_parse_connection_string_invalid(self): + self.assertRaises( + ValueError, lambda: ExporterOptions(connection_string="asd") + ) + + def test_parse_connection_string_invalid_auth(self): + self.assertRaises( + ValueError, + lambda: ExporterOptions( + connection_string="Authorization=asd", + instrumentation_key=self._valid_instrumentation_key, + ), + ) diff --git a/azure_monitor/tests/test_utils.py b/azure_monitor/tests/test_utils.py index a2e2152..bbebfaf 100644 --- a/azure_monitor/tests/test_utils.py +++ b/azure_monitor/tests/test_utils.py @@ -8,6 +8,11 @@ from opentelemetry.sdk.trace.export import SpanExportResult from azure_monitor import utils +from azure_monitor.export import ( + ExportResult, + get_metrics_export_result, + get_trace_export_result, +) class TestUtils(unittest.TestCase): @@ -26,278 +31,32 @@ def test_nanoseconds_to_duration(self): self.assertEqual(ns_to_duration(3600 * 1000000000), "0.01:00:00.000") self.assertEqual(ns_to_duration(86400 * 1000000000), "1.00:00:00.000") - def test_validate_instrumentation_key(self): - options = utils.Options( - instrumentation_key=self._valid_instrumentation_key - ) - self.assertEqual( - options.instrumentation_key, self._valid_instrumentation_key - ) - - def test_invalid_key_none(self): - self.assertRaises( - ValueError, lambda: utils.Options(instrumentation_key=None) - ) - - def test_invalid_key_empty(self): - self.assertRaises( - ValueError, lambda: utils.Options(instrumentation_key="") - ) - - def test_invalid_key_prefix(self): - self.assertRaises( - ValueError, - lambda: utils.Options( - instrumentation_key="test1234abcd-5678-4efa-8abc-1234567890ab" - ), - ) - - def test_invalid_key_suffix(self): - self.assertRaises( - ValueError, - lambda: utils.Options( - instrumentation_key="1234abcd-5678-4efa-8abc-1234567890abtest" - ), - ) - - def test_invalid_key_length(self): - self.assertRaises( - ValueError, - lambda: utils.Options( - instrumentation_key="1234abcd-5678-4efa-8abc-12234567890ab" - ), - ) - - def test_invalid_key_dashes(self): - self.assertRaises( - ValueError, - lambda: utils.Options( - instrumentation_key="1234abcda5678-4efa-8abc-1234567890ab" - ), - ) - - def test_invalid_key_section1_length(self): - self.assertRaises( - ValueError, - lambda: utils.Options( - instrumentation_key="1234abcda-678-4efa-8abc-1234567890ab" - ), - ) - - def test_invalid_key_section2_length(self): - self.assertRaises( - ValueError, - lambda: utils.Options( - instrumentation_key="1234abcd-678-a4efa-8abc-1234567890ab" - ), - ) - - def test_invalid_key_section3_length(self): - self.assertRaises( - ValueError, - lambda: utils.Options( - instrumentation_key="1234abcd-6789-4ef-8cabc-1234567890ab" - ), - ) - - def test_invalid_key_section4_length(self): - self.assertRaises( - ValueError, - lambda: utils.Options( - instrumentation_key="1234abcd-678-4efa-8bc-11234567890ab" - ), - ) - - def test_invalid_key_section5_length(self): - self.assertRaises( - ValueError, - lambda: utils.Options( - instrumentation_key="234abcd-678-4efa-8abc-11234567890ab" - ), - ) - - def test_invalid_key_section1_hex(self): - self.assertRaises( - ValueError, - lambda: utils.Options( - instrumentation_key="x234abcd-5678-4efa-8abc-1234567890ab" - ), - ) - - def test_invalid_key_section2_hex(self): - self.assertRaises( - ValueError, - lambda: utils.Options( - instrumentation_key="1234abcd-x678-4efa-8abc-1234567890ab" - ), - ) - - def test_invalid_key_section3_hex(self): - self.assertRaises( - ValueError, - lambda: utils.Options( - instrumentation_key="1234abcd-5678-4xfa-8abc-1234567890ab" - ), - ) - - def test_invalid_key_section4_hex(self): - self.assertRaises( - ValueError, - lambda: utils.Options( - instrumentation_key="1234abcd-5678-4xfa-8abc-1234567890ab" - ), - ) - - def test_invalid_key_section5_hex(self): - self.assertRaises( - ValueError, - lambda: utils.Options( - instrumentation_key="1234abcd-5678-4xfa-8abc-1234567890ab" - ), - ) - - def test_invalid_key_version(self): - self.assertRaises( - ValueError, - lambda: utils.Options( - instrumentation_key="1234abcd-5678-6efa-8abc-1234567890ab" - ), - ) - - def test_invalid_key_variant(self): - self.assertRaises( - ValueError, - lambda: utils.Options( - instrumentation_key="1234abcd-5678-4efa-2abc-1234567890ab" - ), - ) - - def test_process_options_ikey_code_cs(self): - os.environ[ - "APPLICATIONINSIGHTS_CONNECTION_STRING" - ] = "Authorization=ikey;InstrumentationKey=789" - os.environ["APPINSIGHTS_INSTRUMENTATIONKEY"] = "101112" - options = utils.Options( - connection_string="Authorization=ikey;InstrumentationKey=" - + self._valid_instrumentation_key, - instrumentation_key="456", - ) - self.assertEqual( - options.instrumentation_key, self._valid_instrumentation_key - ) - - def test_process_options_ikey_code_ikey(self): - os.environ[ - "APPLICATIONINSIGHTS_CONNECTION_STRING" - ] = "Authorization=ikey;InstrumentationKey=789" - os.environ["APPINSIGHTS_INSTRUMENTATIONKEY"] = "101112" - options = utils.Options( - connection_string=None, - instrumentation_key=self._valid_instrumentation_key, - ) - self.assertEqual( - options.instrumentation_key, self._valid_instrumentation_key - ) - - def test_process_options_ikey_env_cs(self): - os.environ["APPLICATIONINSIGHTS_CONNECTION_STRING"] = ( - "Authorization=ikey;InstrumentationKey=" - + self._valid_instrumentation_key - ) - os.environ["APPINSIGHTS_INSTRUMENTATIONKEY"] = "101112" - options = utils.Options( - connection_string=None, instrumentation_key=None - ) - self.assertEqual( - options.instrumentation_key, self._valid_instrumentation_key - ) - - def test_process_options_ikey_env_ikey(self): - os.environ[ - "APPINSIGHTS_INSTRUMENTATIONKEY" - ] = self._valid_instrumentation_key - options = utils.Options( - connection_string=None, instrumentation_key=None - ) - self.assertEqual( - options.instrumentation_key, self._valid_instrumentation_key - ) - - def test_process_options_endpoint_code_cs(self): - os.environ[ - "APPLICATIONINSIGHTS_CONNECTION_STRING" - ] = "Authorization=ikey;IngestionEndpoint=456;InstrumentationKey=" - options = utils.Options( - connection_string="Authorization=ikey;IngestionEndpoint=123", - instrumentation_key=self._valid_instrumentation_key, - ) - self.assertEqual(options.endpoint, "123/v2/track") - - def test_process_options_endpoint_env_cs(self): - os.environ[ - "APPLICATIONINSIGHTS_CONNECTION_STRING" - ] = "Authorization=ikey;IngestionEndpoint=456" - options = utils.Options( - connection_string=None, - instrumentation_key=self._valid_instrumentation_key, - ) - self.assertEqual(options.endpoint, "456/v2/track") - - def test_process_options_endpoint_default(self): - options = utils.Options( - connection_string=None, - instrumentation_key=self._valid_instrumentation_key, - ) - self.assertEqual( - options.endpoint, "https://dc.services.visualstudio.com/v2/track" - ) - - def test_parse_connection_string_invalid(self): - self.assertRaises( - ValueError, lambda: utils.Options(connection_string="asd") - ) - - def test_parse_connection_string_invalid_auth(self): - self.assertRaises( - ValueError, - lambda: utils.Options( - connection_string="Authorization=asd", - instrumentation_key=self._valid_instrumentation_key, - ), - ) - def test_get_trace_export_result(self): self.assertEqual( - utils.get_trace_export_result(utils.ExportResult.SUCCESS), + get_trace_export_result(ExportResult.SUCCESS), SpanExportResult.SUCCESS, ) self.assertEqual( - utils.get_trace_export_result( - utils.ExportResult.FAILED_NOT_RETRYABLE - ), + get_trace_export_result(ExportResult.FAILED_NOT_RETRYABLE), SpanExportResult.FAILED_NOT_RETRYABLE, ) self.assertEqual( - utils.get_trace_export_result(utils.ExportResult.FAILED_RETRYABLE), + get_trace_export_result(ExportResult.FAILED_RETRYABLE), SpanExportResult.FAILED_RETRYABLE, ) - self.assertEqual(utils.get_trace_export_result(None), None) + self.assertEqual(get_trace_export_result(None), None) def test_get_metrics_export_result(self): self.assertEqual( - utils.get_metrics_export_result(utils.ExportResult.SUCCESS), + get_metrics_export_result(ExportResult.SUCCESS), MetricsExportResult.SUCCESS, ) self.assertEqual( - utils.get_metrics_export_result( - utils.ExportResult.FAILED_NOT_RETRYABLE - ), + get_metrics_export_result(ExportResult.FAILED_NOT_RETRYABLE), MetricsExportResult.FAILED_NOT_RETRYABLE, ) self.assertEqual( - utils.get_metrics_export_result( - utils.ExportResult.FAILED_RETRYABLE - ), + get_metrics_export_result(ExportResult.FAILED_RETRYABLE), MetricsExportResult.FAILED_RETRYABLE, ) - self.assertEqual(utils.get_metrics_export_result(None), None) + self.assertEqual(get_metrics_export_result(None), None) diff --git a/azure_monitor/tests/trace/test_trace.py b/azure_monitor/tests/trace/test_trace.py index 7a1ceb1..7bf186f 100644 --- a/azure_monitor/tests/trace/test_trace.py +++ b/azure_monitor/tests/trace/test_trace.py @@ -13,9 +13,10 @@ from opentelemetry.trace import Link, SpanContext, SpanKind from opentelemetry.trace.status import Status, StatusCanonicalCode +from azure_monitor.export import ExportResult +from azure_monitor.export.trace import AzureMonitorSpanExporter +from azure_monitor.options import ExporterOptions from azure_monitor.protocol import Envelope -from azure_monitor.trace import AzureMonitorSpanExporter -from azure_monitor.utils import ExportResult, Options TEST_FOLDER = os.path.abspath(".test.exporter") @@ -50,7 +51,7 @@ def test_constructor(self): exporter = AzureMonitorSpanExporter( instrumentation_key="4321abcd-5678-4efa-8abc-1234567890ab" ) - self.assertIsInstance(exporter.options, Options) + self.assertIsInstance(exporter.options, ExporterOptions) self.assertEqual( exporter.options.instrumentation_key, "4321abcd-5678-4efa-8abc-1234567890ab", @@ -63,7 +64,7 @@ def test_export_empty(self): exporter.export([]) self.assertEqual(len(os.listdir(exporter.storage.path)), 0) - @mock.patch("azure_monitor.trace.logger") + @mock.patch("azure_monitor.export.trace.logger") def test_export_exception(self, mock_logger): exporter = AzureMonitorSpanExporter( storage_path=os.path.join(TEST_FOLDER, self.id()) @@ -72,7 +73,7 @@ def test_export_exception(self, mock_logger): mock_logger.exception.assert_called() @mock.patch( - "azure_monitor.trace.AzureMonitorSpanExporter.span_to_envelope" + "azure_monitor.export.trace.AzureMonitorSpanExporter.span_to_envelope" ) # noqa: E501 def test_export_failure(self, span_to_envelope_mock): span_to_envelope_mock.return_value = ["bar"] @@ -80,7 +81,7 @@ def test_export_failure(self, span_to_envelope_mock): storage_path=os.path.join(TEST_FOLDER, self.id()) ) with mock.patch( - "azure_monitor.trace.AzureMonitorSpanExporter._transmit" + "azure_monitor.export.trace.AzureMonitorSpanExporter._transmit" ) as transmit: # noqa: E501 test_span = Span( name="test", @@ -97,7 +98,7 @@ def test_export_failure(self, span_to_envelope_mock): self.assertIsNone(exporter.storage.get()) @mock.patch( - "azure_monitor.trace.AzureMonitorSpanExporter.span_to_envelope" + "azure_monitor.export.trace.AzureMonitorSpanExporter.span_to_envelope" ) # noqa: E501 def test_export_success(self, span_to_envelope_mock): span_to_envelope_mock.return_value = ["bar"] @@ -105,7 +106,7 @@ def test_export_success(self, span_to_envelope_mock): storage_path=os.path.join(TEST_FOLDER, self.id()) ) with mock.patch( - "azure_monitor.trace.AzureMonitorSpanExporter._transmit" + "azure_monitor.export.trace.AzureMonitorSpanExporter._transmit" ) as transmit: # noqa: E501 transmit.return_value = 0 exporter.export([]) From c3e4f73773f6ccd0adc1bc0d02df552a9339e711 Mon Sep 17 00:00:00 2001 From: Hector Hernandez Date: Tue, 10 Mar 2020 17:09:09 -0700 Subject: [PATCH 062/109] Added tests --- azure_monitor/src/azure_monitor/__init__.py | 1 - .../azure_monitor/auto_collection/__init__.py | 19 +- ...dency_metrics.py => dependency_metrics.py} | 22 +- ..._request_metrics.py => request_metrics.py} | 16 +- .../src/azure_monitor/export/__init__.py | 6 +- .../azure_monitor/export/metrics/__init__.py | 10 +- azure_monitor/src/azure_monitor/protocol.py | 10 +- .../auto_collection/test_auto_collection.py | 83 ++++++++ .../test_depencency_metrics.py | 101 +++++++++ .../test_performance_metrics.py | 137 +++++++++++-- .../auto_collection/test_request_metrics.py | 194 ++++++++++++++++++ azure_monitor/tests/metrics/test_metrics.py | 2 +- azure_monitor/tests/trace/test_trace.py | 1 - 13 files changed, 541 insertions(+), 61 deletions(-) rename azure_monitor/src/azure_monitor/auto_collection/{http_dependency_metrics.py => dependency_metrics.py} (80%) rename azure_monitor/src/azure_monitor/auto_collection/{http_request_metrics.py => request_metrics.py} (92%) create mode 100644 azure_monitor/tests/auto_collection/test_auto_collection.py create mode 100644 azure_monitor/tests/auto_collection/test_depencency_metrics.py create mode 100644 azure_monitor/tests/auto_collection/test_request_metrics.py diff --git a/azure_monitor/src/azure_monitor/__init__.py b/azure_monitor/src/azure_monitor/__init__.py index 66df8a4..63520fb 100644 --- a/azure_monitor/src/azure_monitor/__init__.py +++ b/azure_monitor/src/azure_monitor/__init__.py @@ -9,4 +9,3 @@ "AzureMonitorSpanExporter", "ExporterOptions", ] - diff --git a/azure_monitor/src/azure_monitor/auto_collection/__init__.py b/azure_monitor/src/azure_monitor/auto_collection/__init__.py index e90a66b..8d26d46 100644 --- a/azure_monitor/src/azure_monitor/auto_collection/__init__.py +++ b/azure_monitor/src/azure_monitor/auto_collection/__init__.py @@ -6,15 +6,16 @@ from azure_monitor.auto_collection.performance_metrics import ( PerformanceMetrics, ) -from azure_monitor.auto_collection.http_dependency_metrics import ( - HttpDependencyMetrics, -) -from azure_monitor.auto_collection.http_request_metrics import ( - HttpRequestMetrics, -) +from azure_monitor.auto_collection.dependency_metrics import DependencyMetrics +from azure_monitor.auto_collection.request_metrics import RequestMetrics from azure_monitor.utils import PeriodicTask -__all__ = ["AutoCollection", "HttpDependencyMetrics", "HttpRequestMetrics", "PerformanceMetrics"] +__all__ = [ + "AutoCollection", + "DependencyMetrics", + "RequestMetrics", + "PerformanceMetrics", +] class AutoCollection: @@ -22,8 +23,8 @@ def __init__( self, meter: Meter, label_set: LabelSet, collection_interval: int = 60 ): self._performance_metrics = PerformanceMetrics(meter, label_set) - self._dependency_metrics = HttpDependencyMetrics(meter, label_set) - self._request_metrics = HttpRequestMetrics(meter, label_set) + self._dependency_metrics = DependencyMetrics(meter, label_set) + self._request_metrics = RequestMetrics(meter, label_set) self._collect_task = PeriodicTask( interval=collection_interval, function=self.collect diff --git a/azure_monitor/src/azure_monitor/auto_collection/http_dependency_metrics.py b/azure_monitor/src/azure_monitor/auto_collection/dependency_metrics.py similarity index 80% rename from azure_monitor/src/azure_monitor/auto_collection/http_dependency_metrics.py rename to azure_monitor/src/azure_monitor/auto_collection/dependency_metrics.py index 7c7a9c4..159a1fe 100644 --- a/azure_monitor/src/azure_monitor/auto_collection/http_dependency_metrics.py +++ b/azure_monitor/src/azure_monitor/auto_collection/dependency_metrics.py @@ -9,21 +9,21 @@ from opentelemetry.metrics import LabelSet, Meter from opentelemetry.sdk.metrics import Gauge -logger = logging.getLogger(__name__) +dependency_map = dict() +_dependency_lock = threading.Lock() +ORIGINAL_REQUEST = requests.Session.request -def dependency_patch(self) -> None: +def dependency_patch(*args, **kwargs) -> None: result = ORIGINAL_REQUEST(*args, **kwargs) - # Only collect request metric if sent from non-exporter thread - if not execution_context.is_exporter(): - # We don't want multiple threads updating this at once - with _dependency_lock: - count = dependency_map.get("count", 0) - dependency_map["count"] = count + 1 + # We don't want multiple threads updating this at once + with _dependency_lock: + count = dependency_map.get("count", 0) + dependency_map["count"] = count + 1 return result -class HttpDependencyMetrics: +class DependencyMetrics: def __init__(self, meter: Meter, label_set: LabelSet): self._meter = meter self._label_set = label_set @@ -67,8 +67,8 @@ def _track_dependency_rate(self) -> None: dependency_map["last_time"] = current_time dependency_map["last_count"] = current_count dependency_map["last_result"] = result - self._dependency_rate_handle.set(result) + self._dependency_rate_handle.set(int(result)) except ZeroDivisionError: # If elapsed_seconds is 0, exporter call made too close to previous # Return the previous result if this is the case - self._dependency_rate_handle.set(last_result) + self._dependency_rate_handle.set(int(last_result)) diff --git a/azure_monitor/src/azure_monitor/auto_collection/http_request_metrics.py b/azure_monitor/src/azure_monitor/auto_collection/request_metrics.py similarity index 92% rename from azure_monitor/src/azure_monitor/auto_collection/http_request_metrics.py rename to azure_monitor/src/azure_monitor/auto_collection/request_metrics.py index 0531c75..c5b3011 100644 --- a/azure_monitor/src/azure_monitor/auto_collection/http_request_metrics.py +++ b/azure_monitor/src/azure_monitor/auto_collection/request_metrics.py @@ -10,6 +10,8 @@ logger = logging.getLogger(__name__) +_requests_lock = threading.Lock() +requests_map = dict() ORIGINAL_CONSTRUCTOR = HTTPServer.__init__ @@ -51,7 +53,7 @@ def server_patch(*args, **kwargs): return result -class HttpRequestMetrics: +class RequestMetrics: def __init__(self, meter: Meter, label_set: LabelSet): self._meter = meter self._label_set = label_set @@ -72,7 +74,7 @@ def __init__(self, meter: Meter, label_set: LabelSet): request_rate_metric = self._meter.create_metric( "\\ASP.NET Applications(??APP_W3SVC_PROC??)\\Requests/Sec", "Incoming Requests Average Execution Rate", - "milliseconds", + "rps", int, Gauge, ) @@ -101,11 +103,13 @@ def _track_request_duration(self) -> None: requests_map["last_average_duration"] = result requests_map["last_duration"] = requests_map.get("duration", 0) # Convert to milliseconds - self._request_duration_handle(result * 1000.0) + self._request_duration_handle.set(int(result * 1000.0)) except ZeroDivisionError: # If interval_count is 0, exporter call made too close to previous # Return the previous result if this is the case - self._request_duration_handle(last_average_duration * 1000.0) + self._request_duration_handle.set( + int(last_average_duration * 1000.0) + ) def _track_request_rate(self) -> None: """ Track Request execution rate @@ -131,8 +135,8 @@ def _track_request_rate(self) -> None: requests_map["last_time"] = current_time requests_map["last_count"] = requests_map.get("count", 0) requests_map["last_rate"] = result - self._request_rate_handle(result) + self._request_rate_handle.set(int(result)) except ZeroDivisionError: # If elapsed_seconds is 0, exporter call made too close to previous # Return the previous result if this is the case - self._request_rate_handle(last_rate) + self._request_rate_handle.set(int(last_rate)) diff --git a/azure_monitor/src/azure_monitor/export/__init__.py b/azure_monitor/src/azure_monitor/export/__init__.py index d98731a..31cc459 100644 --- a/azure_monitor/src/azure_monitor/export/__init__.py +++ b/azure_monitor/src/azure_monitor/export/__init__.py @@ -89,9 +89,7 @@ def _transmit_from_storage(self) -> None: else: blob.delete(silent=True) - def _transmit( - self, envelopes: typing.List[Envelope] - ) -> utils.ExportResult: + def _transmit(self, envelopes: typing.List[Envelope]) -> ExportResult: """ Transmit the data envelopes to the ingestion service. @@ -111,7 +109,7 @@ def _transmit( ) except Exception as ex: logger.warning("Transient client side error: %s.", ex) - return utils.ExportResult.FAILED_RETRYABLE + return ExportResult.FAILED_RETRYABLE text = "N/A" data = None diff --git a/azure_monitor/src/azure_monitor/export/metrics/__init__.py b/azure_monitor/src/azure_monitor/export/metrics/__init__.py index 082528e..90107df 100644 --- a/azure_monitor/src/azure_monitor/export/metrics/__init__.py +++ b/azure_monitor/src/azure_monitor/export/metrics/__init__.py @@ -31,10 +31,12 @@ def export( self, metric_records: Sequence[MetricRecord] ) -> MetricsExportResult: envelopes = list(map(self.metric_to_envelope, metric_records)) - envelopes = list(map( - lambda x: x.to_dict(), - self.apply_telemetry_processors(envelopes), - )) + envelopes = list( + map( + lambda x: x.to_dict(), + self.apply_telemetry_processors(envelopes), + ) + ) try: result = self._transmit(envelopes) if result == MetricsExportResult.FAILED_RETRYABLE: diff --git a/azure_monitor/src/azure_monitor/protocol.py b/azure_monitor/src/azure_monitor/protocol.py index cba995a..c70b142 100644 --- a/azure_monitor/src/azure_monitor/protocol.py +++ b/azure_monitor/src/azure_monitor/protocol.py @@ -37,7 +37,10 @@ def __init__(self, base_data: any = None, base_type: str = None) -> None: self.base_type = base_type def to_dict(self): - return {"baseData": self.base_data.to_dict(), "baseType": self.base_type} + return { + "baseData": self.base_data.to_dict(), + "baseType": self.base_type, + } class DataPointType(Enum): @@ -400,10 +403,7 @@ def __init__( def to_dict(self): return { "ver": self.ver, - "metrics": list(map( - lambda x: x.to_dict(), - self.metrics, - )), + "metrics": list(map(lambda x: x.to_dict(), self.metrics)), "properties": self.properties, } diff --git a/azure_monitor/tests/auto_collection/test_auto_collection.py b/azure_monitor/tests/auto_collection/test_auto_collection.py new file mode 100644 index 0000000..935f490 --- /dev/null +++ b/azure_monitor/tests/auto_collection/test_auto_collection.py @@ -0,0 +1,83 @@ +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. + + +import json +import os +import shutil +import unittest +from unittest import mock + +from opentelemetry import metrics +from opentelemetry.sdk.metrics import Meter + +from azure_monitor.auto_collection import AutoCollection +from azure_monitor.utils import PeriodicTask + + +class TestAutoCollection(unittest.TestCase): + @classmethod + def setUpClass(cls): + cls._meter_defaults = (metrics._METER, metrics._METER_FACTORY) + metrics.set_preferred_meter_implementation(lambda _: Meter()) + cls._meter = metrics.meter() + kvp = {"environment": "staging"} + cls._test_label_set = cls._meter.get_label_set(kvp) + + @classmethod + def tearDownClass(cls): + metrics._METER, metrics._METER_FACTORY = cls._meter_defaults + + @mock.patch( + "azure_monitor.auto_collection.PerformanceMetrics", autospec=True + ) + @mock.patch( + "azure_monitor.auto_collection.DependencyMetrics", autospec=True + ) + @mock.patch("azure_monitor.auto_collection.RequestMetrics", autospec=True) + def test_constructor( + self, mock_performance, mock_dependencies, mock_requests + ): + """Test the constructor.""" + + auto_collector = AutoCollection( + meter=self._meter, + label_set=self._test_label_set, + collection_interval=1000, + ) + self.assertEqual(mock_performance.called, True) + self.assertEqual(mock_dependencies.called, True) + self.assertEqual(mock_requests.called, True) + self.assertEqual(mock_performance.call_args[0][0], self._meter) + self.assertEqual( + mock_performance.call_args[0][1], self._test_label_set + ) + self.assertEqual(mock_dependencies.call_args[0][0], self._meter) + self.assertEqual( + mock_dependencies.call_args[0][1], self._test_label_set + ) + self.assertEqual(mock_requests.call_args[0][0], self._meter) + self.assertEqual(mock_requests.call_args[0][1], self._test_label_set) + + self.assertIsInstance(auto_collector._collect_task, PeriodicTask) + self.assertEqual(auto_collector._collect_task.interval, 1000) + + @mock.patch( + "azure_monitor.auto_collection.PerformanceMetrics.track", autospec=True + ) + @mock.patch( + "azure_monitor.auto_collection.DependencyMetrics.track", autospec=True + ) + @mock.patch( + "azure_monitor.auto_collection.RequestMetrics.track", autospec=True + ) + def test_collect(self, mock_performance, mock_dependencies, mock_requests): + """Test collect method.""" + + auto_collector = AutoCollection( + meter=self._meter, label_set=self._test_label_set + ) + auto_collector.collect() + self.assertEqual(mock_performance.called, True) + self.assertEqual(mock_dependencies.called, True) + self.assertEqual(mock_requests.called, True) diff --git a/azure_monitor/tests/auto_collection/test_depencency_metrics.py b/azure_monitor/tests/auto_collection/test_depencency_metrics.py new file mode 100644 index 0000000..2de3ff2 --- /dev/null +++ b/azure_monitor/tests/auto_collection/test_depencency_metrics.py @@ -0,0 +1,101 @@ +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. + +import collections +import json +import os +import shutil +import unittest + +import requests + +from http.server import HTTPServer +from unittest import mock + +from opentelemetry import metrics +from opentelemetry.sdk.metrics import Gauge, Meter +from opentelemetry.sdk.util import ns_to_iso_str + +from azure_monitor.auto_collection import dependency_metrics, DependencyMetrics + +ORIGINAL_FUNCTION = requests.Session.request +ORIGINAL_CONS = HTTPServer.__init__ + + +class TestDependencyMetrics(unittest.TestCase): + @classmethod + def setUpClass(cls): + cls._meter_defaults = (metrics._METER, metrics._METER_FACTORY) + metrics.set_preferred_meter_implementation(lambda _: Meter()) + cls._meter = metrics.meter() + kvp = {"environment": "staging"} + cls._test_label_set = cls._meter.get_label_set(kvp) + + @classmethod + def tearDown(cls): + metrics._METER, metrics._METER_FACTORY = cls._meter_defaults + + def setUp(self): + dependency_metrics.dependency_map.clear() + requests.Session.request = ORIGINAL_FUNCTION + dependency_metrics.ORIGINAL_CONSTRUCTOR = ORIGINAL_CONS + + def test_constructor(self): + mock_meter = mock.Mock() + metrics_collector = DependencyMetrics( + meter=mock_meter, label_set=self._test_label_set + ) + self.assertEqual(metrics_collector._meter, mock_meter) + self.assertEqual(metrics_collector._label_set, self._test_label_set) + + self.assertEqual(mock_meter.create_metric.call_count, 1) + + create_metric_calls = mock_meter.create_metric.call_args_list + + self.assertEqual( + create_metric_calls[0][0], + ( + "\\ApplicationInsights\\Dependency Calls/Sec", + "Outgoing Requests per second", + "rps", + int, + Gauge, + ), + ) + + @mock.patch("azure_monitor.auto_collection.dependency_metrics.time") + def test_track_depencency_rate(self, time_mock): + time_mock.time.return_value = 100 + metrics_collector = DependencyMetrics( + meter=self._meter, label_set=self._test_label_set + ) + map = dependency_metrics.dependency_map + map["last_time"] = 98 + map["count"] = 4 + metrics_collector._track_dependency_rate() + self.assertEqual( + metrics_collector._dependency_rate_handle.aggregator.current, 2 + ) + + @mock.patch("azure_monitor.auto_collection.dependency_metrics.time") + def test_track_depencency_rate_error(self, time_mock): + time_mock.time.return_value = 100 + metrics_collector = DependencyMetrics( + meter=self._meter, label_set=self._test_label_set + ) + map = dependency_metrics.dependency_map + map["last_time"] = 100 + map["last_result"] = 5 + metrics_collector._track_dependency_rate() + self.assertEqual( + metrics_collector._dependency_rate_handle.aggregator.current, 5 + ) + + def test_dependency_patch(self): + map = dependency_metrics.dependency_map + dependency_metrics.ORIGINAL_REQUEST = lambda x: None + session = requests.Session() + result = dependency_metrics.dependency_patch(session) + + self.assertEqual(map["count"], 1) + self.assertIsNone(result) diff --git a/azure_monitor/tests/auto_collection/test_performance_metrics.py b/azure_monitor/tests/auto_collection/test_performance_metrics.py index 2670714..11be69c 100644 --- a/azure_monitor/tests/auto_collection/test_performance_metrics.py +++ b/azure_monitor/tests/auto_collection/test_performance_metrics.py @@ -1,7 +1,7 @@ # Copyright (c) Microsoft Corporation. All rights reserved. # Licensed under the MIT License. - +import collections import json import os import shutil @@ -10,9 +10,6 @@ from opentelemetry import metrics from opentelemetry.sdk.metrics import Gauge, Meter -from opentelemetry.sdk.metrics.export import MetricRecord, MetricsExportResult -from opentelemetry.sdk.metrics.export.aggregate import CounterAggregator -from opentelemetry.sdk.util import ns_to_iso_str from azure_monitor.auto_collection import PerformanceMetrics @@ -31,26 +28,128 @@ def tearDownClass(cls): metrics._METER, metrics._METER_FACTORY = cls._meter_defaults def test_constructor(self): - """Test the constructor.""" - auto_collector = PerformanceMetrics( - meter=self._meter, label_set=self._test_label_set + mock_meter = mock.Mock() + performance_metrics_collector = PerformanceMetrics( + meter=mock_meter, label_set=self._test_label_set ) - self.assertEqual(auto_collector._meter, self._meter) - self.assertEqual(auto_collector._label_set, self._test_label_set) + self.assertEqual(performance_metrics_collector._meter, mock_meter) + self.assertEqual( + performance_metrics_collector._label_set, self._test_label_set + ) + + self.assertEqual(mock_meter.create_metric.call_count, 4) - result_metric = list(self._meter.metrics)[0] - self.assertIsInstance(result_metric, Gauge) + create_metric_calls = mock_meter.create_metric.call_args_list + + self.assertEqual( + create_metric_calls[0][0], + ( + "\\Processor(_Total)\\% Processor Time", + "Processor time as a percentage", + "percentage", + float, + Gauge, + ), + ) self.assertEqual( - result_metric.name, "\Processor(_Total)\% Processor Time" + create_metric_calls[1][0], + ( + "\\Memory\\Available Bytes", + "Amount of available memory in bytes", + "byte", + int, + Gauge, + ), ) self.assertEqual( - result_metric.description, "Processor time as a percentage" + create_metric_calls[2][0], + ( + "\\Process(??APP_WIN32_PROC??)\\% Processor Time", + "Process CPU usage as a percentage", + "percentage", + float, + Gauge, + ), ) - self.assertEqual(result_metric.unit, "percentage") - self.assertEqual(result_metric.value_type, float) self.assertEqual( - result_metric.get_handle( - self._test_label_set - ).aggregator.checkpoint, - 0, + create_metric_calls[3][0], + ( + "\\Process(??APP_WIN32_PROC??)\\Private Bytes", + "Amount of memory process has used in bytes", + "byte", + int, + Gauge, + ), ) + + def test_track_cpu(self): + performance_metrics_collector = PerformanceMetrics( + meter=self._meter, label_set=self._test_label_set + ) + with mock.patch("psutil.cpu_times_percent") as processor_mock: + cpu = collections.namedtuple("cpu", "idle") + cpu_times = cpu(idle=94.5) + processor_mock.return_value = cpu_times + performance_metrics_collector._track_cpu() + self.assertEqual( + performance_metrics_collector._cpu_handle.aggregator.current, + 5.5, + ) + + @mock.patch("azure_monitor.auto_collection.performance_metrics.psutil") + def test_track_process_cpu(self, psutil_mock): + with mock.patch( + "azure_monitor.auto_collection.performance_metrics.PROCESS" + ) as process_mock: + performance_metrics_collector = PerformanceMetrics( + meter=self._meter, label_set=self._test_label_set + ) + process_mock.cpu_percent.return_value = 44.4 + psutil_mock.cpu_count.return_value = 2 + performance_metrics_collector._track_process_cpu() + self.assertEqual( + performance_metrics_collector._process_cpu_handle.aggregator.current, + 22.2, + ) + + @mock.patch("azure_monitor.auto_collection.performance_metrics.logger") + def test_track_process_cpu_exception(self, logger_mock): + with mock.patch( + "azure_monitor.auto_collection.performance_metrics.psutil" + ) as psutil_mock: + performance_metrics_collector = PerformanceMetrics( + meter=self._meter, label_set=self._test_label_set + ) + psutil_mock.cpu_count.return_value = None + performance_metrics_collector._track_process_cpu() + logger_mock.exception.assert_called() + + @mock.patch("psutil.virtual_memory") + def test_track_memory(self, psutil_mock): + performance_metrics_collector = PerformanceMetrics( + meter=self._meter, label_set=self._test_label_set + ) + memory = collections.namedtuple("memory", "available") + vmem = memory(available=100) + psutil_mock.return_value = vmem + performance_metrics_collector._track_memory() + self.assertEqual( + performance_metrics_collector._memory_handle.aggregator.current, + 100, + ) + + def test_track_process_memory(self): + with mock.patch( + "azure_monitor.auto_collection.performance_metrics.PROCESS" + ) as process_mock: + performance_metrics_collector = PerformanceMetrics( + meter=self._meter, label_set=self._test_label_set + ) + memory = collections.namedtuple("memory", "rss") + pmem = memory(rss=100) + process_mock.memory_info.return_value = pmem + performance_metrics_collector._track_process_memory() + self.assertEqual( + performance_metrics_collector._process_memory_handle.aggregator.current, + 100, + ) diff --git a/azure_monitor/tests/auto_collection/test_request_metrics.py b/azure_monitor/tests/auto_collection/test_request_metrics.py new file mode 100644 index 0000000..1e4d629 --- /dev/null +++ b/azure_monitor/tests/auto_collection/test_request_metrics.py @@ -0,0 +1,194 @@ +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. + +import collections +import json +import os +import shutil +import unittest + +import requests + +from http.server import HTTPServer +from unittest import mock + +from opentelemetry import metrics +from opentelemetry.sdk.metrics import Gauge, Meter + +from azure_monitor.auto_collection import RequestMetrics, request_metrics + +ORIGINAL_FUNCTION = requests.Session.request +ORIGINAL_CONS = HTTPServer.__init__ + + +class TestRequestMetrics(unittest.TestCase): + @classmethod + def setUpClass(cls): + cls._meter_defaults = (metrics._METER, metrics._METER_FACTORY) + metrics.set_preferred_meter_implementation(lambda _: Meter()) + cls._meter = metrics.meter() + kvp = {"environment": "staging"} + cls._test_label_set = cls._meter.get_label_set(kvp) + + @classmethod + def tearDown(cls): + metrics._METER, metrics._METER_FACTORY = cls._meter_defaults + + def setUp(self): + request_metrics.requests_map.clear() + requests.Session.request = ORIGINAL_FUNCTION + request_metrics.ORIGINAL_CONSTRUCTOR = ORIGINAL_CONS + + def test_constructor(self): + mock_meter = mock.Mock() + request_metrics_collector = RequestMetrics( + meter=mock_meter, label_set=self._test_label_set + ) + self.assertEqual(request_metrics_collector._meter, mock_meter) + self.assertEqual( + request_metrics_collector._label_set, self._test_label_set + ) + + self.assertEqual(mock_meter.create_metric.call_count, 2) + + create_metric_calls = mock_meter.create_metric.call_args_list + + self.assertEqual( + create_metric_calls[0][0], + ( + "\\ASP.NET Applications(??APP_W3SVC_PROC??)\\Request Execution Time", + "Incoming Requests Average Execution Time", + "milliseconds", + int, + Gauge, + ), + ) + + self.assertEqual( + create_metric_calls[1][0], + ( + "\\ASP.NET Applications(??APP_W3SVC_PROC??)\\Requests/Sec", + "Incoming Requests Average Execution Rate", + "rps", + int, + Gauge, + ), + ) + + def test_track_request_duration(self): + request_metrics_collector = RequestMetrics( + meter=self._meter, label_set=self._test_label_set + ) + map = request_metrics.requests_map + map["duration"] = 0.1 + map["count"] = 10 + map["last_count"] = 5 + request_metrics_collector._track_request_duration() + self.assertEqual( + request_metrics_collector._request_duration_handle.aggregator.current, + 20, + ) + + def test_track_request_duration_error(self): + request_metrics_collector = RequestMetrics( + meter=self._meter, label_set=self._test_label_set + ) + map = request_metrics.requests_map + map["duration"] = 0.1 + map["count"] = 10 + map["last_count"] = 10 + request_metrics_collector._track_request_duration() + self.assertEqual( + request_metrics_collector._request_duration_handle.aggregator.current, + 0, + ) + + @mock.patch("azure_monitor.auto_collection.request_metrics.time") + def test_track_request_rate(self, time_mock): + request_metrics_collector = RequestMetrics( + meter=self._meter, label_set=self._test_label_set + ) + time_mock.time.return_value = 100 + request_metrics.requests_map["last_time"] = 98 + request_metrics.requests_map["count"] = 4 + request_metrics_collector._track_request_rate() + self.assertEqual( + request_metrics_collector._request_rate_handle.aggregator.current, + 2, + ) + + @mock.patch("azure_monitor.auto_collection.request_metrics.time") + def test_track_request_rate_error(self, time_mock): + request_metrics_collector = RequestMetrics( + meter=self._meter, label_set=self._test_label_set + ) + time_mock.time.return_value = 100 + request_metrics.requests_map["last_rate"] = 5 + request_metrics.requests_map["last_time"] = 100 + request_metrics_collector._track_request_rate() + self.assertEqual( + request_metrics_collector._request_rate_handle.aggregator.current, + 5, + ) + + def test_request_patch(self): + map = request_metrics.requests_map + func = mock.Mock() + new_func = request_metrics.request_patch(func) + new_func() + + self.assertEqual(map["count"], 1) + self.assertIsNotNone(map["duration"]) + self.assertEqual(len(func.call_args_list), 1) + + def test_server_patch(self): + request_metrics.ORIGINAL_CONSTRUCTOR = lambda x, y, z: None + with mock.patch( + "azure_monitor.auto_collection.request_metrics.request_patch" + ) as request_mock: + handler = mock.Mock() + handler.do_DELETE.return_value = None + handler.do_GET.return_value = None + handler.do_HEAD.return_value = None + handler.do_OPTIONS.return_value = None + handler.do_POST.return_value = None + handler.do_PUT.return_value = None + result = request_metrics.server_patch(None, None, handler) + handler.do_DELETE() + handler.do_GET() + handler.do_HEAD() + handler.do_OPTIONS() + handler.do_POST() + handler.do_PUT() + + self.assertEqual(result, None) + self.assertEqual(len(request_mock.call_args_list), 6) + + def test_server_patch_no_methods(self): + request_metrics.ORIGINAL_CONSTRUCTOR = lambda x, y, z: None + with mock.patch( + "azure_monitor.auto_collection.request_metrics.request_patch" + ) as request_mock: + handler = mock.Mock() + result = request_metrics.server_patch(None, None, handler) + handler.do_DELETE() + handler.do_GET() + handler.do_HEAD() + handler.do_OPTIONS() + handler.do_POST() + handler.do_PUT() + + self.assertEqual(result, None) + self.assertEqual(len(request_mock.call_args_list), 0) + + def test_server_patch_no_args(self): + request_metrics.ORIGINAL_CONSTRUCTOR = lambda x, y: None + r = request_metrics.server_patch(None, None) + + self.assertEqual(r, None) + + def test_server_patch_no_handler(self): + request_metrics.ORIGINAL_CONSTRUCTOR = lambda x, y, z: None + r = request_metrics.server_patch(None, None, None) + + self.assertEqual(r, None) diff --git a/azure_monitor/tests/metrics/test_metrics.py b/azure_monitor/tests/metrics/test_metrics.py index b358a7d..5111287 100644 --- a/azure_monitor/tests/metrics/test_metrics.py +++ b/azure_monitor/tests/metrics/test_metrics.py @@ -15,7 +15,7 @@ from azure_monitor.export import ExportResult from azure_monitor.export.metrics import AzureMonitorMetricsExporter from azure_monitor.options import ExporterOptions -from azure_monitor.protocol import Data, Envelope, MetricData +from azure_monitor.protocol import Data, DataPoint, Envelope, MetricData class TestAzureMetricsExporter(unittest.TestCase): diff --git a/azure_monitor/tests/trace/test_trace.py b/azure_monitor/tests/trace/test_trace.py index 7bf186f..6c024c7 100644 --- a/azure_monitor/tests/trace/test_trace.py +++ b/azure_monitor/tests/trace/test_trace.py @@ -18,7 +18,6 @@ from azure_monitor.options import ExporterOptions from azure_monitor.protocol import Envelope - TEST_FOLDER = os.path.abspath(".test.exporter") From 3d589bf4c460b69a820b7de95f50bda58d4a7b77 Mon Sep 17 00:00:00 2001 From: Hector Hernandez Date: Tue, 10 Mar 2020 17:28:03 -0700 Subject: [PATCH 063/109] Fixing lint --- .../src/azure_monitor/auto_collection/__init__.py | 2 +- .../azure_monitor/auto_collection/dependency_metrics.py | 1 - .../azure_monitor/auto_collection/performance_metrics.py | 4 ++-- .../src/azure_monitor/auto_collection/request_metrics.py | 2 +- azure_monitor/src/azure_monitor/export/__init__.py | 5 +---- azure_monitor/src/azure_monitor/options.py | 3 +-- .../tests/auto_collection/test_depencency_metrics.py | 7 ++----- .../tests/auto_collection/test_request_metrics.py | 4 +--- azure_monitor/tests/test_base_exporter.py | 2 +- 9 files changed, 10 insertions(+), 20 deletions(-) diff --git a/azure_monitor/src/azure_monitor/auto_collection/__init__.py b/azure_monitor/src/azure_monitor/auto_collection/__init__.py index 8d26d46..20cb0c3 100644 --- a/azure_monitor/src/azure_monitor/auto_collection/__init__.py +++ b/azure_monitor/src/azure_monitor/auto_collection/__init__.py @@ -3,10 +3,10 @@ # from opentelemetry.metrics import LabelSet, Meter +from azure_monitor.auto_collection.dependency_metrics import DependencyMetrics from azure_monitor.auto_collection.performance_metrics import ( PerformanceMetrics, ) -from azure_monitor.auto_collection.dependency_metrics import DependencyMetrics from azure_monitor.auto_collection.request_metrics import RequestMetrics from azure_monitor.utils import PeriodicTask diff --git a/azure_monitor/src/azure_monitor/auto_collection/dependency_metrics.py b/azure_monitor/src/azure_monitor/auto_collection/dependency_metrics.py index 159a1fe..4fac519 100644 --- a/azure_monitor/src/azure_monitor/auto_collection/dependency_metrics.py +++ b/azure_monitor/src/azure_monitor/auto_collection/dependency_metrics.py @@ -5,7 +5,6 @@ import time import requests - from opentelemetry.metrics import LabelSet, Meter from opentelemetry.sdk.metrics import Gauge diff --git a/azure_monitor/src/azure_monitor/auto_collection/performance_metrics.py b/azure_monitor/src/azure_monitor/auto_collection/performance_metrics.py index a56c6b6..3dcb055 100644 --- a/azure_monitor/src/azure_monitor/auto_collection/performance_metrics.py +++ b/azure_monitor/src/azure_monitor/auto_collection/performance_metrics.py @@ -2,11 +2,11 @@ # Licensed under the MIT License. import logging -import psutil - from opentelemetry.metrics import LabelSet, Meter from opentelemetry.sdk.metrics import Gauge +import psutil + logger = logging.getLogger(__name__) PROCESS = psutil.Process() diff --git a/azure_monitor/src/azure_monitor/auto_collection/request_metrics.py b/azure_monitor/src/azure_monitor/auto_collection/request_metrics.py index c5b3011..77e8bad 100644 --- a/azure_monitor/src/azure_monitor/auto_collection/request_metrics.py +++ b/azure_monitor/src/azure_monitor/auto_collection/request_metrics.py @@ -3,8 +3,8 @@ import logging import threading import time - from http.server import HTTPServer + from opentelemetry.metrics import LabelSet, Meter from opentelemetry.sdk.metrics import Gauge diff --git a/azure_monitor/src/azure_monitor/export/__init__.py b/azure_monitor/src/azure_monitor/export/__init__.py index 31cc459..ca44ed6 100644 --- a/azure_monitor/src/azure_monitor/export/__init__.py +++ b/azure_monitor/src/azure_monitor/export/__init__.py @@ -3,11 +3,9 @@ import json import logging import typing - -import requests - from enum import Enum +import requests from opentelemetry.sdk.metrics.export import MetricsExportResult from opentelemetry.sdk.trace.export import SpanExportResult @@ -15,7 +13,6 @@ from azure_monitor.protocol import Envelope from azure_monitor.storage import LocalFileStorage - logger = logging.getLogger(__name__) diff --git a/azure_monitor/src/azure_monitor/options.py b/azure_monitor/src/azure_monitor/options.py index a6896c5..9ee3d5d 100644 --- a/azure_monitor/src/azure_monitor/options.py +++ b/azure_monitor/src/azure_monitor/options.py @@ -1,13 +1,12 @@ # Copyright (c) Microsoft Corporation. All rights reserved. # Licensed under the MIT License. import os -import sys import re +import sys import typing from azure_monitor.protocol import BaseObject - INGESTION_ENDPOINT = "ingestionendpoint" INSTRUMENTATION_KEY = "instrumentationkey" diff --git a/azure_monitor/tests/auto_collection/test_depencency_metrics.py b/azure_monitor/tests/auto_collection/test_depencency_metrics.py index 2de3ff2..3a4fe6e 100644 --- a/azure_monitor/tests/auto_collection/test_depencency_metrics.py +++ b/azure_monitor/tests/auto_collection/test_depencency_metrics.py @@ -6,17 +6,14 @@ import os import shutil import unittest - -import requests - from http.server import HTTPServer from unittest import mock +import requests from opentelemetry import metrics from opentelemetry.sdk.metrics import Gauge, Meter -from opentelemetry.sdk.util import ns_to_iso_str -from azure_monitor.auto_collection import dependency_metrics, DependencyMetrics +from azure_monitor.auto_collection import DependencyMetrics, dependency_metrics ORIGINAL_FUNCTION = requests.Session.request ORIGINAL_CONS = HTTPServer.__init__ diff --git a/azure_monitor/tests/auto_collection/test_request_metrics.py b/azure_monitor/tests/auto_collection/test_request_metrics.py index 1e4d629..9e38ecc 100644 --- a/azure_monitor/tests/auto_collection/test_request_metrics.py +++ b/azure_monitor/tests/auto_collection/test_request_metrics.py @@ -6,12 +6,10 @@ import os import shutil import unittest - -import requests - from http.server import HTTPServer from unittest import mock +import requests from opentelemetry import metrics from opentelemetry.sdk.metrics import Gauge, Meter diff --git a/azure_monitor/tests/test_base_exporter.py b/azure_monitor/tests/test_base_exporter.py index 72ceb15..596a91c 100644 --- a/azure_monitor/tests/test_base_exporter.py +++ b/azure_monitor/tests/test_base_exporter.py @@ -41,7 +41,7 @@ def test_constructor(self): def test_constructor_wrong_options(self): """Test the constructor with wrong options.""" with self.assertRaises(TypeError): - base = BaseExporter(something_else=6) + BaseExporter(something_else=6) def test_telemetry_processor_add(self): base = BaseExporter() From d38739a0e137baabeaedbf1ef52b5c336830425b Mon Sep 17 00:00:00 2001 From: Hector Hernandez Date: Tue, 10 Mar 2020 19:04:48 -0700 Subject: [PATCH 064/109] Fixing lint for real --- .../auto_collection/dependency_metrics.py | 1 - .../auto_collection/performance_metrics.py | 7 +- .../src/azure_monitor/export/__init__.py | 21 +- .../azure_monitor/export/metrics/__init__.py | 5 +- .../azure_monitor/export/trace/__init__.py | 13 +- azure_monitor/src/azure_monitor/options.py | 71 +-- azure_monitor/src/azure_monitor/protocol.py | 22 +- azure_monitor/src/azure_monitor/storage.py | 7 +- azure_monitor/src/azure_monitor/utils.py | 1 - .../auto_collection/test_auto_collection.py | 5 +- .../test_depencency_metrics.py | 18 +- .../test_performance_metrics.py | 4 +- .../auto_collection/test_request_metrics.py | 30 +- azure_monitor/tests/metrics/test_metrics.py | 18 +- azure_monitor/tests/test_options.py | 1 + azure_monitor/tests/test_storage.py | 39 +- azure_monitor/tests/trace/test_trace.py | 25 +- eachdist.ini | 12 + scripts/eachdist.py | 508 ++++++++++++++++++ scripts/pylint.sh | 13 - tox.ini | 12 +- 21 files changed, 642 insertions(+), 191 deletions(-) create mode 100644 eachdist.ini create mode 100644 scripts/eachdist.py delete mode 100644 scripts/pylint.sh diff --git a/azure_monitor/src/azure_monitor/auto_collection/dependency_metrics.py b/azure_monitor/src/azure_monitor/auto_collection/dependency_metrics.py index 4fac519..f6c8d92 100644 --- a/azure_monitor/src/azure_monitor/auto_collection/dependency_metrics.py +++ b/azure_monitor/src/azure_monitor/auto_collection/dependency_metrics.py @@ -1,6 +1,5 @@ # Copyright (c) Microsoft Corporation. All rights reserved. # Licensed under the MIT License. -import logging import threading import time diff --git a/azure_monitor/src/azure_monitor/auto_collection/performance_metrics.py b/azure_monitor/src/azure_monitor/auto_collection/performance_metrics.py index 3dcb055..877252b 100644 --- a/azure_monitor/src/azure_monitor/auto_collection/performance_metrics.py +++ b/azure_monitor/src/azure_monitor/auto_collection/performance_metrics.py @@ -2,11 +2,10 @@ # Licensed under the MIT License. import logging +import psutil from opentelemetry.metrics import LabelSet, Meter from opentelemetry.sdk.metrics import Gauge -import psutil - logger = logging.getLogger(__name__) PROCESS = psutil.Process() @@ -85,7 +84,7 @@ def _track_process_cpu(self) -> None: # normalize the cpu process using the number of logical CPUs cpu_count = psutil.cpu_count(logical=True) self._process_cpu_handle.set(PROCESS.cpu_percent() / cpu_count) - except Exception: + except Exception: # pylint: disable=broad-except logger.exception("Error handling get process cpu usage.") def _track_memory(self) -> None: @@ -104,5 +103,5 @@ def _track_process_memory(self) -> None: """ try: self._process_memory_handle.set(PROCESS.memory_info().rss) - except Exception: + except Exception: # pylint: disable=broad-except logger.exception("Error handling get process private bytes.") diff --git a/azure_monitor/src/azure_monitor/export/__init__.py b/azure_monitor/src/azure_monitor/export/__init__.py index ca44ed6..a21bd8c 100644 --- a/azure_monitor/src/azure_monitor/export/__init__.py +++ b/azure_monitor/src/azure_monitor/export/__init__.py @@ -22,6 +22,7 @@ class ExportResult(Enum): FAILED_NOT_RETRYABLE = 2 +# pylint: disable=broad-except class BaseExporter: def __init__(self, **options): self._telemetry_processors = [] @@ -86,6 +87,8 @@ def _transmit_from_storage(self) -> None: else: blob.delete(silent=True) + # pylint: disable=too-many-branches + # pylint: disable=too-many-nested-blocks def _transmit(self, envelopes: typing.List[Envelope]) -> ExportResult: """ Transmit the data envelopes to the ingestion service. @@ -112,12 +115,12 @@ def _transmit(self, envelopes: typing.List[Envelope]) -> ExportResult: data = None try: text = response.text - except Exception as ex: # noqa pylint: disable=broad-except + except Exception as ex: logger.warning("Error while reading response body %s.", ex) else: try: data = json.loads(text) - except Exception: # noqa pylint: disable=broad-except + except Exception: pass if response.status_code == 200: @@ -172,20 +175,18 @@ def _transmit(self, envelopes: typing.List[Envelope]) -> ExportResult: def get_trace_export_result(result: ExportResult) -> SpanExportResult: if result == ExportResult.SUCCESS: return SpanExportResult.SUCCESS - elif result == ExportResult.FAILED_RETRYABLE: + if result == ExportResult.FAILED_RETRYABLE: return SpanExportResult.FAILED_RETRYABLE - elif result == ExportResult.FAILED_NOT_RETRYABLE: + if result == ExportResult.FAILED_NOT_RETRYABLE: return SpanExportResult.FAILED_NOT_RETRYABLE - else: - return None + return None def get_metrics_export_result(result: ExportResult) -> MetricsExportResult: if result == ExportResult.SUCCESS: return MetricsExportResult.SUCCESS - elif result == ExportResult.FAILED_RETRYABLE: + if result == ExportResult.FAILED_RETRYABLE: return MetricsExportResult.FAILED_RETRYABLE - elif result == ExportResult.FAILED_NOT_RETRYABLE: + if result == ExportResult.FAILED_NOT_RETRYABLE: return MetricsExportResult.FAILED_NOT_RETRYABLE - else: - return None + return None diff --git a/azure_monitor/src/azure_monitor/export/metrics/__init__.py b/azure_monitor/src/azure_monitor/export/metrics/__init__.py index 90107df..22ec365 100644 --- a/azure_monitor/src/azure_monitor/export/metrics/__init__.py +++ b/azure_monitor/src/azure_monitor/export/metrics/__init__.py @@ -24,9 +24,6 @@ class AzureMonitorMetricsExporter(BaseExporter, MetricsExporter): - def __init__(self, **options): - super(AzureMonitorMetricsExporter, self).__init__(**options) - def export( self, metric_records: Sequence[MetricRecord] ) -> MetricsExportResult: @@ -45,7 +42,7 @@ def export( # Try to send any cached events self._transmit_from_storage() return get_metrics_export_result(result) - except Exception: + except Exception: # pylint: disable=broad-except logger.exception("Exception occurred while exporting the data.") def metric_to_envelope( diff --git a/azure_monitor/src/azure_monitor/export/trace/__init__.py b/azure_monitor/src/azure_monitor/export/trace/__init__.py index a40aae2..e79998d 100644 --- a/azure_monitor/src/azure_monitor/export/trace/__init__.py +++ b/azure_monitor/src/azure_monitor/export/trace/__init__.py @@ -21,9 +21,6 @@ class AzureMonitorSpanExporter(BaseExporter, SpanExporter): - def __init__(self, **options): - super(AzureMonitorSpanExporter, self).__init__(**options) - def export(self, spans: Sequence[Span]) -> SpanExportResult: envelopes = map(self.span_to_envelope, spans) envelopes_to_export = map( @@ -38,16 +35,14 @@ def export(self, spans: Sequence[Span]) -> SpanExportResult: # Try to send any cached events self._transmit_from_storage() return get_trace_export_result(result) - except Exception: + except Exception: # pylint: disable=broad-except logger.exception("Exception occurred while exporting the data.") - def span_to_envelope( - self, span: Span - ) -> protocol.Envelope: # noqa pylint: disable=too-many-branches - + # pylint: disable=too-many-statements + # pylint: disable=too-many-branches + def span_to_envelope(self, span: Span) -> protocol.Envelope: if not span: return None - # pylint: disable=too-many-statements envelope = protocol.Envelope( ikey=self.options.instrumentation_key, tags=dict(utils.azure_monitor_context), diff --git a/azure_monitor/src/azure_monitor/options.py b/azure_monitor/src/azure_monitor/options.py index 9ee3d5d..19652f3 100644 --- a/azure_monitor/src/azure_monitor/options.py +++ b/azure_monitor/src/azure_monitor/options.py @@ -72,9 +72,9 @@ def __init__( self._validate_instrumentation_key() def _initialize(self) -> None: - code_cs = self._parse_connection_string(self.connection_string) + code_cs = parse_connection_string(self.connection_string) code_ikey = self.instrumentation_key - env_cs = self._parse_connection_string( + env_cs = parse_connection_string( os.getenv("APPLICATIONINSIGHTS_CONNECTION_STRING") ) env_ikey = os.getenv("APPINSIGHTS_INSTRUMENTATIONKEY") @@ -113,36 +113,37 @@ def _validate_instrumentation_key(self) -> None: if not match: raise ValueError("Invalid instrumentation key.") - def _parse_connection_string(self, connection_string) -> typing.Dict: - if connection_string is None: - return {} - try: - pairs = connection_string.split(";") - result = dict(s.split("=") for s in pairs) - # Convert keys to lower-case due to case type-insensitive checking - result = {key.lower(): value for key, value in result.items()} - except Exception: - raise ValueError("Invalid connection string") - # Validate authorization - auth = result.get("authorization") - if auth is not None and auth.lower() != "ikey": - raise ValueError("Invalid authorization mechanism") - # Construct the ingestion endpoint if not passed in explicitly - if result.get(INGESTION_ENDPOINT) is None: - endpoint_suffix = "" - location_prefix = "" - suffix = result.get("endpointsuffix") - if suffix is not None: - endpoint_suffix = suffix - # Get regional information if provided - prefix = result.get("location") - if prefix is not None: - location_prefix = prefix + "." - endpoint = "https://{0}dc.{1}".format( - location_prefix, endpoint_suffix - ) - result[INGESTION_ENDPOINT] = endpoint - else: - # Default to None if cannot construct - result[INGESTION_ENDPOINT] = None - return result + +def parse_connection_string(connection_string) -> typing.Dict: + if connection_string is None: + return {} + try: + pairs = connection_string.split(";") + result = dict(s.split("=") for s in pairs) + # Convert keys to lower-case due to case type-insensitive checking + result = {key.lower(): value for key, value in result.items()} + except Exception: + raise ValueError("Invalid connection string") + # Validate authorization + auth = result.get("authorization") + if auth is not None and auth.lower() != "ikey": + raise ValueError("Invalid authorization mechanism") + # Construct the ingestion endpoint if not passed in explicitly + if result.get(INGESTION_ENDPOINT) is None: + endpoint_suffix = "" + location_prefix = "" + suffix = result.get("endpointsuffix") + if suffix is not None: + endpoint_suffix = suffix + # Get regional information if provided + prefix = result.get("location") + if prefix is not None: + location_prefix = prefix + "." + endpoint = "https://{0}dc.{1}".format( + location_prefix, endpoint_suffix + ) + result[INGESTION_ENDPOINT] = endpoint + else: + # Default to None if cannot construct + result[INGESTION_ENDPOINT] = None + return result diff --git a/azure_monitor/src/azure_monitor/protocol.py b/azure_monitor/src/azure_monitor/protocol.py index c70b142..9f68f16 100644 --- a/azure_monitor/src/azure_monitor/protocol.py +++ b/azure_monitor/src/azure_monitor/protocol.py @@ -80,11 +80,11 @@ def __init__( kind: DataPointType = None, value: float = 0.0, count: float = None, - min: float = None, - max: float = None, + min: float = None, # pylint: disable=redefined-builtin + max: float = None, # pylint: disable=redefined-builtin std_dev: float = None, ) -> None: - self.ns = ns + self.ns = ns # pylint: disable=invalid-name self.name = name self.kind = kind self.value = value @@ -241,7 +241,7 @@ class ExceptionDetails(BaseObject): def __init__( self, - id: int = None, + id: int = None, # pylint: disable=redefined-builtin outer_id: int = None, type_name: str = None, message: str = None, @@ -249,7 +249,7 @@ def __init__( stack: str = None, parsed_stack: any = None, ) -> None: - self.id = id + self.id = id # pylint: disable=invalid-name self.outer_id = outer_id self.type_name = type_name self.message = message @@ -259,7 +259,7 @@ def __init__( def to_dict(self): return { - "id": self.ver, + "id": self.id, "outerId": self.outer_id, "typeName": self.type_name, "message": self.message, @@ -450,19 +450,19 @@ def __init__( self, ver: int = 2, name: str = "", - id: str = "", + id: str = "", # pylint: disable=redefined-builtin result_code: str = "", duration: str = "", success: bool = True, data: Data = None, - type: str = None, + type: str = None, # pylint: disable=redefined-builtin target: str = None, properties: typing.Dict[str, any] = None, measurements: typing.Dict[str, int] = None, ) -> None: self.ver = ver self.name = name - self.id = id + self.id = id # pylint: disable=invalid-name self.result_code = result_code self.duration = duration self.success = success @@ -527,7 +527,7 @@ class Request(BaseObject): def __init__( self, ver: int = 2, - id: str = "", + id: str = "", # pylint: disable=redefined-builtin duration: str = "", response_code: str = "", success: bool = True, @@ -538,7 +538,7 @@ def __init__( measurements: typing.Dict[str, int] = None, ) -> None: self.ver = ver - self.id = id + self.id = id # pylint: disable=invalid-name self.duration = duration self.response_code = response_code self.success = success diff --git a/azure_monitor/src/azure_monitor/storage.py b/azure_monitor/src/azure_monitor/storage.py index 10145f4..f8a265a 100644 --- a/azure_monitor/src/azure_monitor/storage.py +++ b/azure_monitor/src/azure_monitor/storage.py @@ -21,7 +21,8 @@ def _seconds(seconds): return datetime.timedelta(seconds=seconds) -class LocalFileBlob(object): +# pylint: disable=broad-except +class LocalFileBlob: def __init__(self, fullpath): self.fullpath = fullpath @@ -75,7 +76,8 @@ def lease(self, period): return self -class LocalFileStorage(object): +# pylint: disable=broad-except +class LocalFileStorage: def __init__( self, path, @@ -105,6 +107,7 @@ def close(self): def __enter__(self): return self + # pylint: disable=redefined-builtin def __exit__(self, type, value, traceback): self.close() diff --git a/azure_monitor/src/azure_monitor/utils.py b/azure_monitor/src/azure_monitor/utils.py index 0ad5da3..ad7eb93 100644 --- a/azure_monitor/src/azure_monitor/utils.py +++ b/azure_monitor/src/azure_monitor/utils.py @@ -9,7 +9,6 @@ from opentelemetry.sdk.version import __version__ as opentelemetry_version -from azure_monitor.protocol import BaseObject from azure_monitor.version import __version__ as ext_version azure_monitor_context = { diff --git a/azure_monitor/tests/auto_collection/test_auto_collection.py b/azure_monitor/tests/auto_collection/test_auto_collection.py index 935f490..2a73211 100644 --- a/azure_monitor/tests/auto_collection/test_auto_collection.py +++ b/azure_monitor/tests/auto_collection/test_auto_collection.py @@ -1,10 +1,6 @@ # Copyright (c) Microsoft Corporation. All rights reserved. # Licensed under the MIT License. - -import json -import os -import shutil import unittest from unittest import mock @@ -15,6 +11,7 @@ from azure_monitor.utils import PeriodicTask +# pylint: disable=protected-access class TestAutoCollection(unittest.TestCase): @classmethod def setUpClass(cls): diff --git a/azure_monitor/tests/auto_collection/test_depencency_metrics.py b/azure_monitor/tests/auto_collection/test_depencency_metrics.py index 3a4fe6e..186ddc4 100644 --- a/azure_monitor/tests/auto_collection/test_depencency_metrics.py +++ b/azure_monitor/tests/auto_collection/test_depencency_metrics.py @@ -1,10 +1,6 @@ # Copyright (c) Microsoft Corporation. All rights reserved. # Licensed under the MIT License. -import collections -import json -import os -import shutil import unittest from http.server import HTTPServer from unittest import mock @@ -19,6 +15,7 @@ ORIGINAL_CONS = HTTPServer.__init__ +# pylint: disable=protected-access class TestDependencyMetrics(unittest.TestCase): @classmethod def setUpClass(cls): @@ -66,9 +63,8 @@ def test_track_depencency_rate(self, time_mock): metrics_collector = DependencyMetrics( meter=self._meter, label_set=self._test_label_set ) - map = dependency_metrics.dependency_map - map["last_time"] = 98 - map["count"] = 4 + dependency_metrics.dependency_map["last_time"] = 98 + dependency_metrics.dependency_map["count"] = 4 metrics_collector._track_dependency_rate() self.assertEqual( metrics_collector._dependency_rate_handle.aggregator.current, 2 @@ -80,19 +76,17 @@ def test_track_depencency_rate_error(self, time_mock): metrics_collector = DependencyMetrics( meter=self._meter, label_set=self._test_label_set ) - map = dependency_metrics.dependency_map - map["last_time"] = 100 - map["last_result"] = 5 + dependency_metrics.dependency_map["last_time"] = 100 + dependency_metrics.dependency_map["last_result"] = 5 metrics_collector._track_dependency_rate() self.assertEqual( metrics_collector._dependency_rate_handle.aggregator.current, 5 ) def test_dependency_patch(self): - map = dependency_metrics.dependency_map dependency_metrics.ORIGINAL_REQUEST = lambda x: None session = requests.Session() result = dependency_metrics.dependency_patch(session) - self.assertEqual(map["count"], 1) + self.assertEqual(dependency_metrics.dependency_map["count"], 1) self.assertIsNone(result) diff --git a/azure_monitor/tests/auto_collection/test_performance_metrics.py b/azure_monitor/tests/auto_collection/test_performance_metrics.py index 11be69c..2b5c893 100644 --- a/azure_monitor/tests/auto_collection/test_performance_metrics.py +++ b/azure_monitor/tests/auto_collection/test_performance_metrics.py @@ -2,9 +2,6 @@ # Licensed under the MIT License. import collections -import json -import os -import shutil import unittest from unittest import mock @@ -14,6 +11,7 @@ from azure_monitor.auto_collection import PerformanceMetrics +# pylint: disable=protected-access class TestPerformanceMetrics(unittest.TestCase): @classmethod def setUpClass(cls): diff --git a/azure_monitor/tests/auto_collection/test_request_metrics.py b/azure_monitor/tests/auto_collection/test_request_metrics.py index 9e38ecc..93eec36 100644 --- a/azure_monitor/tests/auto_collection/test_request_metrics.py +++ b/azure_monitor/tests/auto_collection/test_request_metrics.py @@ -1,10 +1,6 @@ # Copyright (c) Microsoft Corporation. All rights reserved. # Licensed under the MIT License. -import collections -import json -import os -import shutil import unittest from http.server import HTTPServer from unittest import mock @@ -19,6 +15,7 @@ ORIGINAL_CONS = HTTPServer.__init__ +# pylint: disable=protected-access class TestRequestMetrics(unittest.TestCase): @classmethod def setUpClass(cls): @@ -77,10 +74,9 @@ def test_track_request_duration(self): request_metrics_collector = RequestMetrics( meter=self._meter, label_set=self._test_label_set ) - map = request_metrics.requests_map - map["duration"] = 0.1 - map["count"] = 10 - map["last_count"] = 5 + request_metrics.requests_map["duration"] = 0.1 + request_metrics.requests_map["count"] = 10 + request_metrics.requests_map["last_count"] = 5 request_metrics_collector._track_request_duration() self.assertEqual( request_metrics_collector._request_duration_handle.aggregator.current, @@ -91,10 +87,9 @@ def test_track_request_duration_error(self): request_metrics_collector = RequestMetrics( meter=self._meter, label_set=self._test_label_set ) - map = request_metrics.requests_map - map["duration"] = 0.1 - map["count"] = 10 - map["last_count"] = 10 + request_metrics.requests_map["duration"] = 0.1 + request_metrics.requests_map["count"] = 10 + request_metrics.requests_map["last_count"] = 10 request_metrics_collector._track_request_duration() self.assertEqual( request_metrics_collector._request_duration_handle.aggregator.current, @@ -130,7 +125,7 @@ def test_track_request_rate_error(self, time_mock): ) def test_request_patch(self): - map = request_metrics.requests_map + map = request_metrics.requests_map # pylint: disable=redefined-builtin func = mock.Mock() new_func = request_metrics.request_patch(func) new_func() @@ -181,12 +176,11 @@ def test_server_patch_no_methods(self): def test_server_patch_no_args(self): request_metrics.ORIGINAL_CONSTRUCTOR = lambda x, y: None - r = request_metrics.server_patch(None, None) + req = request_metrics.server_patch(None, None) - self.assertEqual(r, None) + self.assertEqual(req, None) def test_server_patch_no_handler(self): request_metrics.ORIGINAL_CONSTRUCTOR = lambda x, y, z: None - r = request_metrics.server_patch(None, None, None) - - self.assertEqual(r, None) + req = request_metrics.server_patch(None, None, None) + self.assertEqual(req, None) diff --git a/azure_monitor/tests/metrics/test_metrics.py b/azure_monitor/tests/metrics/test_metrics.py index 5111287..1109c1a 100644 --- a/azure_monitor/tests/metrics/test_metrics.py +++ b/azure_monitor/tests/metrics/test_metrics.py @@ -1,8 +1,6 @@ # Copyright (c) Microsoft Corporation. All rights reserved. # Licensed under the MIT License. -import json import os -import shutil import unittest from unittest import mock @@ -18,6 +16,7 @@ from azure_monitor.protocol import Data, DataPoint, Envelope, MetricData +# pylint: disable=protected-access class TestAzureMetricsExporter(unittest.TestCase): @classmethod def setUpClass(cls): @@ -104,18 +103,3 @@ def test_metric_to_envelope(self): self.assertIsNotNone(envelope.tags["ai.device.osVersion"]) self.assertIsNotNone(envelope.tags["ai.device.type"]) self.assertIsNotNone(envelope.tags["ai.internal.sdkVersion"]) - - -class MockResponse(object): - def __init__(self, status_code, text): - self.status_code = status_code - self.text = text - - -class MockTransport(object): - def __init__(self, exporter=None): - self.export_called = False - self.exporter = exporter - - def export(self, datas): - self.export_called = True diff --git a/azure_monitor/tests/test_options.py b/azure_monitor/tests/test_options.py index 69f119b..def6027 100644 --- a/azure_monitor/tests/test_options.py +++ b/azure_monitor/tests/test_options.py @@ -7,6 +7,7 @@ from azure_monitor.options import ExporterOptions +# pylint: disable=too-many-public-methods class TestOptions(unittest.TestCase): def setUp(self): os.environ.clear() diff --git a/azure_monitor/tests/test_storage.py b/azure_monitor/tests/test_storage.py index 1b30767..2a056f7 100644 --- a/azure_monitor/tests/test_storage.py +++ b/azure_monitor/tests/test_storage.py @@ -16,10 +16,12 @@ TEST_FOLDER = os.path.abspath(".test") +# pylint: disable=invalid-name def setUpModule(): os.makedirs(TEST_FOLDER) +# pylint: disable=invalid-name def tearDownModule(): shutil.rmtree(TEST_FOLDER) @@ -35,13 +37,13 @@ class TestLocalFileBlob(unittest.TestCase): def test_delete(self): blob = LocalFileBlob(os.path.join(TEST_FOLDER, "foobar")) blob.delete(silent=True) - self.assertRaises(Exception, lambda: blob.delete()) + self.assertRaises(Exception, blob.delete) self.assertRaises(Exception, lambda: blob.delete(silent=False)) def test_get(self): blob = LocalFileBlob(os.path.join(TEST_FOLDER, "foobar")) self.assertIsNone(blob.get(silent=True)) - self.assertRaises(Exception, lambda: blob.get()) + self.assertRaises(Exception, blob.get) self.assertRaises(Exception, lambda: blob.get(silent=False)) def test_put_error(self): @@ -51,18 +53,18 @@ def test_put_error(self): def test_put_without_lease(self): blob = LocalFileBlob(os.path.join(TEST_FOLDER, "foobar.blob")) - input = (1, 2, 3) + test_input = (1, 2, 3) blob.delete(silent=True) - blob.put(input) - self.assertEqual(blob.get(), input) + blob.put(test_input) + self.assertEqual(blob.get(), test_input) def test_put_with_lease(self): blob = LocalFileBlob(os.path.join(TEST_FOLDER, "foobar.blob")) - input = (1, 2, 3) + test_input = (1, 2, 3) blob.delete(silent=True) - blob.put(input, lease_period=0.01) + blob.put(test_input, lease_period=0.01) blob.lease(0.01) - self.assertEqual(blob.get(), input) + self.assertEqual(blob.get(), test_input) def test_lease_error(self): blob = LocalFileBlob(os.path.join(TEST_FOLDER, "foobar.blob")) @@ -70,6 +72,7 @@ def test_lease_error(self): self.assertEqual(blob.lease(0.01), None) +# pylint: disable=protected-access class TestLocalFileStorage(unittest.TestCase): def test_get_nothing(self): with LocalFileStorage(os.path.join(TEST_FOLDER, "test", "a")) as stor: @@ -95,15 +98,15 @@ def test_get(self): self.assertIsNone(stor.get()) def test_put(self): - input = (1, 2, 3) + test_input = (1, 2, 3) with LocalFileStorage(os.path.join(TEST_FOLDER, "bar")) as stor: - stor.put(input) - self.assertEqual(stor.get().get(), input) + stor.put(test_input) + self.assertEqual(stor.get().get(), test_input) with LocalFileStorage(os.path.join(TEST_FOLDER, "bar")) as stor: - self.assertEqual(stor.get().get(), input) + self.assertEqual(stor.get().get(), test_input) with mock.patch("os.rename", side_effect=throw(Exception)): - self.assertIsNone(stor.put(input, silent=True)) - self.assertRaises(Exception, lambda: stor.put(input)) + self.assertIsNone(stor.put(test_input, silent=True)) + self.assertRaises(Exception, lambda: stor.put(test_input)) def test_maintanence_routine(self): with mock.patch("os.makedirs") as m: @@ -125,11 +128,7 @@ def test_maintanence_routine(self): with LocalFileStorage(os.path.join(TEST_FOLDER, "baz")) as stor: with mock.patch("os.listdir", side_effect=throw(Exception)): stor._maintenance_routine(silent=True) - self.assertRaises( - Exception, lambda: stor._maintenance_routine() - ) + self.assertRaises(Exception, stor._maintenance_routine) with mock.patch("os.path.isdir", side_effect=throw(Exception)): stor._maintenance_routine(silent=True) - self.assertRaises( - Exception, lambda: stor._maintenance_routine() - ) + self.assertRaises(Exception, stor._maintenance_routine) diff --git a/azure_monitor/tests/trace/test_trace.py b/azure_monitor/tests/trace/test_trace.py index 6c024c7..2d55017 100644 --- a/azure_monitor/tests/trace/test_trace.py +++ b/azure_monitor/tests/trace/test_trace.py @@ -4,7 +4,6 @@ import json import os -import shutil import unittest from unittest import mock @@ -21,14 +20,6 @@ TEST_FOLDER = os.path.abspath(".test.exporter") -def setUpModule(): - os.makedirs(TEST_FOLDER) - - -def tearDownModule(): - shutil.rmtree(TEST_FOLDER) - - def throw(exc_type, *args, **kwargs): def func(*_args, **_kwargs): raise exc_type(*args, **kwargs) @@ -37,9 +28,11 @@ def func(*_args, **_kwargs): # pylint: disable=import-error +# pylint: disable=protected-access +# pylint: disable=too-many-lines class TestAzureExporter(unittest.TestCase): @classmethod - def setUpClass(self): + def setUpClass(cls): os.environ.clear() os.environ[ "APPINSIGHTS_INSTRUMENTATIONKEY" @@ -285,6 +278,7 @@ def test_transmission_500(self): self.assertIsNone(exporter.storage.get()) self.assertEqual(len(os.listdir(exporter.storage.path)), 1) + # pylint: disable=too-many-statements def test_span_to_envelope(self): options = { "instrumentation_key": "12345678-1234-5678-abcd-12345678abcd" @@ -1019,16 +1013,7 @@ def test_span_to_envelope(self): ) -class MockResponse(object): +class MockResponse: def __init__(self, status_code, text): self.status_code = status_code self.text = text - - -class MockTransport(object): - def __init__(self, exporter=None): - self.export_called = False - self.exporter = exporter - - def export(self, datas): - self.export_called = True diff --git a/eachdist.ini b/eachdist.ini new file mode 100644 index 0000000..fad7111 --- /dev/null +++ b/eachdist.ini @@ -0,0 +1,12 @@ +# These will be sorted first in that order. +# All packages that are depended upon by others should be listed here. +[DEFAULT] +sortfirst= + azure_monitor + +[lintroots] +extraroots=scripts/ +subglob=*.py,tests/,test/,src/* + +[testroots] +subglob=tests/,test/ diff --git a/scripts/eachdist.py b/scripts/eachdist.py new file mode 100644 index 0000000..406afb6 --- /dev/null +++ b/scripts/eachdist.py @@ -0,0 +1,508 @@ +#!/usr/bin/env python3 + +import argparse +import shlex +import shutil +import subprocess +import sys +from collections import namedtuple +from configparser import ConfigParser +from inspect import cleandoc +from itertools import chain +from pathlib import Path, PurePath + +DEFAULT_ALLSEP = " " +DEFAULT_ALLFMT = "{rel}" + + +def unique(elems): + seen = set() + for elem in elems: + if elem not in seen: + yield elem + seen.add(elem) + + +try: + subprocess_run = subprocess.run +except AttributeError: # Py < 3.5 compat + CompletedProcess = namedtuple("CompletedProcess", "returncode") + + def subprocess_run(*args, **kwargs): + check = kwargs.pop("check", False) + if check: + subprocess.check_call(*args, **kwargs) + return CompletedProcess(returncode=0) + return CompletedProcess(returncode=subprocess.call(*args, **kwargs)) + + +def extraargs_help(calledcmd): + return cleandoc( + """ + Additional arguments to pass on to {}. + + This is collected from any trailing arguments passed to `%(prog)s`. + Use an initial `--` to separate them from regular arguments. + """.format( + calledcmd + ) + ) + + +def parse_args(args=None): + parser = argparse.ArgumentParser(description="Development helper script.") + parser.set_defaults(parser=parser) + parser.add_argument( + "--dry-run", + action="store_true", + help="Only display what would be done, don't actually do anything.", + ) + subparsers = parser.add_subparsers(metavar="COMMAND") + subparsers.required = True + + excparser = subparsers.add_parser( + "exec", + help="Run a command for each or all targets.", + formatter_class=argparse.RawTextHelpFormatter, + description=cleandoc( + """Run a command according to the `format` argument for each or all targets. + + This is an advanced command that is used internally by other commands. + + For example, to install all distributions in this repository + editable, you could use: + + scripts/eachdist.py exec "python -m pip install -e {}" + + This will run pip for all distributions which is quite slow. It gets + a bit faster if we only invoke pip once but with all the paths + gathered together, which can be achieved by using `--all`: + + scripts/eachdist.py exec "python -m pip install {}" --all "-e {}" + + The sortfirst option in the DEFAULT section of eachdist.ini makes + sure that dependencies are installed before their dependents. + + Search for usages of `parse_subargs` in the source code of this script + to see more examples. + + This command first collects target paths and then executes + commands according to `format` and `--all`. + + Target paths are initially all Python distribution root paths + (as determined by the existence of setup.py, etc. files). + They are then augmented according to the section of the + `PROJECT_ROOT/eachdist.ini` config file specified by the `--mode` option. + + The following config options are available (and processed in that order): + + - `extraroots`: List of project root-relative glob expressions. + The resulting paths will be added. + - `sortfirst`: List of glob expressions. + Any matching paths will be put to the front of the path list, + in the same order they appear in this option. If more than one + glob matches, ordering is according to the first. + - `subglob`: List of glob expressions. Each path added so far is removed + and replaced with the result of all glob expressions relative to it (in + order of the glob expressions). + + After all this, any duplicate paths are removed (the first occurrence remains). + """ + ), + ) + excparser.set_defaults(func=execute_args) + excparser.add_argument( + "format", + help=cleandoc( + """Format string for the command to execute. + + The available replacements depend on whether `--all` is specified. + If `--all` was specified, there is only a single replacement, + `{}`, that is replaced with the string that is generated from + joining all targets formatted with `--all` to a single string + with the value of `--allsep` as separator. + + If `--all` was not specified, the following replacements are available: + + - `{}`: the absolute path to the current target in POSIX format + (with forward slashes) + - `{rel}`: like `{}` but relative to the project root. + - `{raw}`: the absolute path to the current target in native format + (thus exactly the same as `{}` on Unix but with backslashes on Windows). + - `{rawrel}`: like `{raw}` but relative to the project root. + + The resulting string is then split according to POSIX shell rules + (so you can use quotation marks or backslashes to handle arguments + containing spaces). + + The first token is the name of the executable to run, the remaining + tokens are the arguments. + + Note that a shell is *not* involved by default. + You can add bash/sh/cmd/powershell yourself to the format if you want. + + If `--all` was specified, the resulting command is simply executed once. + Otherwise, the command is executed for each found target. In both cases, + the project root is the working directory. + """ + ), + ) + excparser.add_argument( + "--all", + nargs="?", + const=DEFAULT_ALLFMT, + metavar="ALLFORMAT", + help=cleandoc( + """Instead of running the command for each target, join all target + paths together to run a single command. + + This option optionally takes a format string to apply to each path. The + available replacements are the ones that would be available for `format` + if `--all` was not specified. + + Default ALLFORMAT if this flag is specified: `%(const)s`. + """ + ), + ) + excparser.add_argument( + "--allsep", + help=cleandoc( + """Separator string for the strings resulting from `--all`. + Only valid if `--all` is specified. + """ + ), + ) + excparser.add_argument( + "--allowexitcode", + type=int, + action="append", + default=[0], + help=cleandoc( + """The given command exit code is treated as success and does not abort execution. + Can be specified multiple times. + """ + ), + ) + excparser.add_argument( + "--mode", + "-m", + default="DEFAULT", + help=cleandoc( + """Section of config file to use for target selection configuration. + See description of exec for available options.""" + ), + ) + + instparser = subparsers.add_parser( + "install", help="Install all distributions." + ) + + def setup_instparser(instparser): + instparser.set_defaults(func=install_args) + instparser.add_argument( + "pipargs", nargs=argparse.REMAINDER, help=extraargs_help("pip") + ) + + setup_instparser(instparser) + instparser.add_argument("--editable", "-e", action="store_true") + instparser.add_argument("--with-dev-deps", action="store_true") + instparser.add_argument("--eager-upgrades", action="store_true") + + devparser = subparsers.add_parser( + "develop", + help="Install all distributions editable + dev dependencies.", + ) + setup_instparser(devparser) + devparser.set_defaults( + editable=True, with_dev_deps=True, eager_upgrades=True + ) + + lintparser = subparsers.add_parser( + "lint", help="Lint everything, autofixing if possible." + ) + lintparser.add_argument("--check-only", action="store_true") + lintparser.set_defaults(func=lint_args) + + testparser = subparsers.add_parser( + "test", + help="Test everything (run pytest yourself for more complex operations).", + ) + testparser.set_defaults(func=test_args) + testparser.add_argument( + "pytestargs", nargs=argparse.REMAINDER, help=extraargs_help("pytest") + ) + + return parser.parse_args(args) + + +def find_projectroot(search_start=Path(".")): + root = search_start.resolve() + for root in chain((root,), root.parents): + if any((root / marker).exists() for marker in (".git", "tox.ini")): + return root + return None + + +def find_targets_unordered(rootpath): + for subdir in rootpath.iterdir(): + if not subdir.is_dir(): + continue + if subdir.name.startswith(".") or subdir.name.startswith("venv"): + continue + if any( + (subdir / marker).exists() + for marker in ("setup.py", "pyproject.toml") + ): + yield subdir + else: + yield from find_targets_unordered(subdir) + + +def getlistcfg(strval): + return [ + val.strip() + for line in strval.split("\n") + for val in line.split(",") + if val.strip() + ] + + +def find_targets(mode, rootpath): + if not rootpath: + sys.exit("Could not find a root directory.") + + cfg = ConfigParser() + cfg.read(str(rootpath / "eachdist.ini")) + mcfg = cfg[mode] + + targets = list(find_targets_unordered(rootpath)) + if "extraroots" in mcfg: + targets += [ + path + for extraglob in getlistcfg(mcfg["extraroots"]) + for path in rootpath.glob(extraglob) + ] + if "sortfirst" in mcfg: + sortfirst = getlistcfg(mcfg["sortfirst"]) + + def keyfunc(path): + path = path.relative_to(rootpath) + for idx, pattern in enumerate(sortfirst): + if path.match(pattern): + return idx + return float("inf") + + targets.sort(key=keyfunc) + + subglobs = getlistcfg(mcfg.get("subglob", "")) + if subglobs: + targets = [ + newentry + for newentry in ( + target / subdir + for target in targets + for subglob in subglobs + # We need to special-case the dot, because glob fails to parse that with an IndexError. + for subdir in ( + (target,) if subglob == "." else target.glob(subglob) + ) + ) + if ".egg-info" not in str(newentry) and newentry.exists() + ] + + return list(unique(targets)) + + +def runsubprocess(dry_run, params, *args, **kwargs): + cmdstr = join_args(params) + if dry_run: + print(cmdstr) + return None + + # Py < 3.6 compat. + cwd = kwargs.get("cwd") + if cwd and isinstance(cwd, PurePath): + kwargs["cwd"] = str(cwd) + + check = kwargs.pop("check") # Enforce specifying check + + print(">>>", cmdstr, file=sys.stderr) + + # This is a workaround for subprocess.run(['python']) leaving the virtualenv on Win32. + # The cause for this is that when running the python.exe in a virtualenv, + # the wrapper executable launches the global python as a subprocess and the search sequence + # for CreateProcessW which subprocess.run and Popen use is a follows + # (https://docs.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-createprocessw): + # > 1. The directory from which the application loaded. + # This will be the directory of the global python.exe, not the venv directory, due to the suprocess mechanism. + # > 6. The directories that are listed in the PATH environment variable. + # Only this would find the "correct" python.exe. + + params = list(params) + executable = shutil.which(params[0]) # On Win32, pytho + if executable: + params[0] = executable + try: + return subprocess_run(params, *args, check=check, **kwargs) + except OSError as exc: + raise ValueError( + "Failed executing " + repr(params) + ": " + str(exc) + ) from exc + + +def execute_args(args): + if args.allsep and not args.all: + args.parser.error("--allsep specified but not --all.") + + if args.all and not args.allsep: + args.allsep = DEFAULT_ALLSEP + + rootpath = find_projectroot() + targets = find_targets(args.mode, rootpath) + if not targets: + sys.exit("Error: No targets selected (root: {})".format(rootpath)) + + def fmt_for_path(fmt, path): + return fmt.format( + path.as_posix(), + rel=path.relative_to(rootpath).as_posix(), + raw=path, + rawrel=path.relative_to(rootpath), + ) + + def _runcmd(cmd): + result = runsubprocess( + args.dry_run, shlex.split(cmd), cwd=rootpath, check=False + ) + if result is not None and result.returncode not in args.allowexitcode: + print( + "'{}' failed with code {}".format(cmd, result.returncode), + file=sys.stderr, + ) + sys.exit(result.returncode) + + if args.all: + allstr = args.allsep.join( + fmt_for_path(args.all, path) for path in targets + ) + cmd = args.format.format(allstr) + _runcmd(cmd) + else: + for target in targets: + cmd = fmt_for_path(args.format, target) + _runcmd(cmd) + + +def clean_remainder_args(remainder_args): + if remainder_args and remainder_args[0] == "--": + del remainder_args[0] + + +def join_args(arglist): + return " ".join(map(shlex.quote, arglist)) + + +def install_args(args): + clean_remainder_args(args.pipargs) + if args.eager_upgrades: + args.pipargs += ["--upgrade-strategy=eager"] + + if args.with_dev_deps: + runsubprocess( + args.dry_run, + [ + "python", + "-m", + "pip", + "install", + "--upgrade", + "pip", + "setuptools", + "wheel", + ] + + args.pipargs, + check=True, + ) + + allfmt = "-e 'file://{}'" if args.editable else "'file://{}'" + execute_args( + parse_subargs( + args, + ( + "exec", + "python -m pip install {} " + join_args(args.pipargs), + "--all", + allfmt, + ), + ) + ) + if args.with_dev_deps: + rootpath = find_projectroot() + runsubprocess( + args.dry_run, + [ + "python", + "-m", + "pip", + "install", + "--upgrade", + "-r", + str(rootpath / "dev-requirements.txt"), + ] + + args.pipargs, + check=True, + ) + + +def parse_subargs(parentargs, args): + subargs = parse_args(args) + subargs.dry_run = parentargs.dry_run or subargs.dry_run + return subargs + + +def lint_args(args): + rootdir = str(find_projectroot()) + + runsubprocess( + args.dry_run, + ("black", ".") + (("--diff", "--check") if args.check_only else ()), + cwd=rootdir, + check=True, + ) + runsubprocess( + args.dry_run, + ("isort", "--recursive", ".") + + (("--diff", "--check-only") if args.check_only else ()), + cwd=rootdir, + check=True, + ) + runsubprocess(args.dry_run, ("flake8", rootdir), check=True) + execute_args( + parse_subargs( + args, ("exec", "pylint {}", "--all", "--mode", "lintroots") + ) + ) + + +def test_args(args): + clean_remainder_args(args.pytestargs) + execute_args( + parse_subargs( + args, + ( + "exec", + "pytest {} " + join_args(args.pytestargs), + "--mode", + "testroots", + ), + ) + ) + + +def main(): + args = parse_args() + args.func(args) + + +if __name__ == "__main__": + main() diff --git a/scripts/pylint.sh b/scripts/pylint.sh deleted file mode 100644 index f757fac..0000000 --- a/scripts/pylint.sh +++ /dev/null @@ -1,13 +0,0 @@ -# Copyright (c) Microsoft Corporation. All rights reserved. -# Licensed under the MIT License. - -set -ev - -# Run pylint on directories -function pylint_dir { - python -m pip install --upgrade pylint - pylint $(find azure_monitor -type f -name "*.py") - return $? -} - -pylint_dir diff --git a/tox.ini b/tox.ini index eccd753..1338d2d 100644 --- a/tox.ini +++ b/tox.ini @@ -9,7 +9,7 @@ envlist = [travis] python = - 3.7: py37, lint + 3.8: py38, lint [testenv] deps = @@ -34,7 +34,7 @@ commands = coverage: coverage report [testenv:lint] -basepython: python3.7 +basepython: python3.8 recreate = True deps = -c dev-requirements.txt @@ -42,12 +42,10 @@ deps = flake8 isort black + psutil commands_pre = - pip install {toxinidir}/azure_monitor + python scripts/eachdist.py install --editable commands = - black . --diff --check - isort --diff --check-only --recursive . - flake8 - bash ./scripts/pylint.sh + python scripts/eachdist.py lint --check-only From b9bfcbf256fdfc970077390f347908086462ab2d Mon Sep 17 00:00:00 2001 From: Hector Hernandez Date: Tue, 10 Mar 2020 19:17:51 -0700 Subject: [PATCH 065/109] Fixing python 3.5 issue --- .../tests/auto_collection/test_performance_metrics.py | 2 +- azure_monitor/tests/trace/test_trace.py | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/azure_monitor/tests/auto_collection/test_performance_metrics.py b/azure_monitor/tests/auto_collection/test_performance_metrics.py index 2b5c893..f6654af 100644 --- a/azure_monitor/tests/auto_collection/test_performance_metrics.py +++ b/azure_monitor/tests/auto_collection/test_performance_metrics.py @@ -120,7 +120,7 @@ def test_track_process_cpu_exception(self, logger_mock): ) psutil_mock.cpu_count.return_value = None performance_metrics_collector._track_process_cpu() - logger_mock.exception.assert_called() + self.assertEqual(logger_mock.exception.called, True) @mock.patch("psutil.virtual_memory") def test_track_memory(self, psutil_mock): diff --git a/azure_monitor/tests/trace/test_trace.py b/azure_monitor/tests/trace/test_trace.py index 2d55017..1e4a047 100644 --- a/azure_monitor/tests/trace/test_trace.py +++ b/azure_monitor/tests/trace/test_trace.py @@ -17,7 +17,7 @@ from azure_monitor.options import ExporterOptions from azure_monitor.protocol import Envelope -TEST_FOLDER = os.path.abspath(".test.exporter") +TEST_FOLDER = os.path.abspath(".test.exporter.trace") def throw(exc_type, *args, **kwargs): @@ -62,7 +62,7 @@ def test_export_exception(self, mock_logger): storage_path=os.path.join(TEST_FOLDER, self.id()) ) exporter.export([None]) - mock_logger.exception.assert_called() + self.assertEqual(mock_logger.exception.called, True) @mock.patch( "azure_monitor.export.trace.AzureMonitorSpanExporter.span_to_envelope" From 0dfc4703c22d4c89cb288c4d9fc1e43494220d8a Mon Sep 17 00:00:00 2001 From: Hector Hernandez Date: Tue, 10 Mar 2020 19:21:03 -0700 Subject: [PATCH 066/109] Adding test clean up back --- azure_monitor/tests/trace/test_trace.py | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/azure_monitor/tests/trace/test_trace.py b/azure_monitor/tests/trace/test_trace.py index 1e4a047..7d28b57 100644 --- a/azure_monitor/tests/trace/test_trace.py +++ b/azure_monitor/tests/trace/test_trace.py @@ -19,6 +19,15 @@ TEST_FOLDER = os.path.abspath(".test.exporter.trace") +# pylint: disable=invalid-name +def setUpModule(): + os.makedirs(TEST_FOLDER) + + +# pylint: disable=invalid-name +def tearDownModule(): + shutil.rmtree(TEST_FOLDER) + def throw(exc_type, *args, **kwargs): def func(*_args, **_kwargs): From 93330b86c8dbb8b7f586c2c7bcaaf963c21000b5 Mon Sep 17 00:00:00 2001 From: Hector Hernandez Date: Tue, 10 Mar 2020 19:23:40 -0700 Subject: [PATCH 067/109] Adding import shutil --- azure_monitor/tests/trace/test_trace.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/azure_monitor/tests/trace/test_trace.py b/azure_monitor/tests/trace/test_trace.py index 7d28b57..814d896 100644 --- a/azure_monitor/tests/trace/test_trace.py +++ b/azure_monitor/tests/trace/test_trace.py @@ -4,6 +4,7 @@ import json import os +import shutil import unittest from unittest import mock @@ -19,6 +20,7 @@ TEST_FOLDER = os.path.abspath(".test.exporter.trace") + # pylint: disable=invalid-name def setUpModule(): os.makedirs(TEST_FOLDER) From dc28f9caf3397028a43b2bc9132fdc35c6fa9f66 Mon Sep 17 00:00:00 2001 From: Hector Hernandez Date: Wed, 11 Mar 2020 16:59:33 -0700 Subject: [PATCH 068/109] Adding auto collection example --- azure_monitor/examples/auto_collector.py | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) create mode 100644 azure_monitor/examples/auto_collector.py diff --git a/azure_monitor/examples/auto_collector.py b/azure_monitor/examples/auto_collector.py new file mode 100644 index 0000000..3864b6b --- /dev/null +++ b/azure_monitor/examples/auto_collector.py @@ -0,0 +1,24 @@ +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. +from azure_monitor import AutoCollection, AzureMonitorMetricsExporter +from opentelemetry import metrics +from opentelemetry.sdk.metrics import Counter, Meter +from opentelemetry.sdk.metrics.export.controller import PushController + +metrics.set_preferred_meter_implementation(lambda _: Meter()) +meter = metrics.get_meter(__name__) +exporter = AzureMonitorMetricsExporter( + connection_string="InstrumentationKey=" +) +controller = PushController(meter, exporter, 5) + +testing_label_set = meter.get_label_set({"environment": "testing"}) + +# Automatically collect standard metrics +auto_collection = AutoCollection( + meter=meter, + label_set=testing_label_set, + collection_interval=120, # Collect every 2 minutes +) + +input("Press any key to exit...") From 5b28e50a28165dd4e9f278e664cd45d54527aa0c Mon Sep 17 00:00:00 2001 From: Hector Hernandez Date: Wed, 11 Mar 2020 17:09:53 -0700 Subject: [PATCH 069/109] Adding AutoCollection export --- azure_monitor/examples/auto_collector.py | 4 ++-- azure_monitor/src/azure_monitor/__init__.py | 2 ++ 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/azure_monitor/examples/auto_collector.py b/azure_monitor/examples/auto_collector.py index 3864b6b..a34a9b3 100644 --- a/azure_monitor/examples/auto_collector.py +++ b/azure_monitor/examples/auto_collector.py @@ -5,8 +5,8 @@ from opentelemetry.sdk.metrics import Counter, Meter from opentelemetry.sdk.metrics.export.controller import PushController -metrics.set_preferred_meter_implementation(lambda _: Meter()) -meter = metrics.get_meter(__name__) +metrics.set_preferred_meter_implementation(lambda T: Meter()) +meter = metrics.meter() exporter = AzureMonitorMetricsExporter( connection_string="InstrumentationKey=" ) diff --git a/azure_monitor/src/azure_monitor/__init__.py b/azure_monitor/src/azure_monitor/__init__.py index 63520fb..f0a8d37 100644 --- a/azure_monitor/src/azure_monitor/__init__.py +++ b/azure_monitor/src/azure_monitor/__init__.py @@ -1,10 +1,12 @@ # Copyright (c) Microsoft Corporation. All rights reserved. # Licensed under the MIT License. +from azure_monitor.auto_collection import AutoCollection from azure_monitor.export.metrics import AzureMonitorMetricsExporter from azure_monitor.export.trace import AzureMonitorSpanExporter from azure_monitor.options import ExporterOptions __all__ = [ + "AutoCollection", "AzureMonitorMetricsExporter", "AzureMonitorSpanExporter", "ExporterOptions", From 10495c6b277ee0328dac81d850c3857cb717147a Mon Sep 17 00:00:00 2001 From: Hector Hernandez Date: Wed, 11 Mar 2020 17:18:49 -0700 Subject: [PATCH 070/109] Fixing lint --- azure_monitor/examples/auto_collector.py | 3 ++- .../src/azure_monitor/auto_collection/request_metrics.py | 1 + 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/azure_monitor/examples/auto_collector.py b/azure_monitor/examples/auto_collector.py index a34a9b3..e816ff8 100644 --- a/azure_monitor/examples/auto_collector.py +++ b/azure_monitor/examples/auto_collector.py @@ -1,10 +1,11 @@ # Copyright (c) Microsoft Corporation. All rights reserved. # Licensed under the MIT License. -from azure_monitor import AutoCollection, AzureMonitorMetricsExporter from opentelemetry import metrics from opentelemetry.sdk.metrics import Counter, Meter from opentelemetry.sdk.metrics.export.controller import PushController +from azure_monitor import AutoCollection, AzureMonitorMetricsExporter + metrics.set_preferred_meter_implementation(lambda T: Meter()) meter = metrics.meter() exporter = AzureMonitorMetricsExporter( diff --git a/azure_monitor/src/azure_monitor/auto_collection/request_metrics.py b/azure_monitor/src/azure_monitor/auto_collection/request_metrics.py index 77e8bad..eba8074 100644 --- a/azure_monitor/src/azure_monitor/auto_collection/request_metrics.py +++ b/azure_monitor/src/azure_monitor/auto_collection/request_metrics.py @@ -84,6 +84,7 @@ def __init__(self, meter: Meter, label_set: LabelSet): def track(self) -> None: self._track_request_duration() + self._track_request_rate() def _track_request_duration(self) -> None: """ Track Request execution time From 5efa73c9e9e5e550efcfc3521e37c88afa6f8333 Mon Sep 17 00:00:00 2001 From: Hector Hernandez <39923391+hectorhdzg@users.noreply.github.com> Date: Wed, 11 Mar 2020 17:19:09 -0700 Subject: [PATCH 071/109] Update azure_monitor/src/azure_monitor/auto_collection/dependency_metrics.py Co-Authored-By: Leighton Chen --- .../src/azure_monitor/auto_collection/dependency_metrics.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/azure_monitor/src/azure_monitor/auto_collection/dependency_metrics.py b/azure_monitor/src/azure_monitor/auto_collection/dependency_metrics.py index f6c8d92..1f0b49e 100644 --- a/azure_monitor/src/azure_monitor/auto_collection/dependency_metrics.py +++ b/azure_monitor/src/azure_monitor/auto_collection/dependency_metrics.py @@ -44,7 +44,7 @@ def track(self) -> None: def _track_dependency_rate(self) -> None: """ Track Dependency rate - Calculated by obtaining by getting the number of outgoing requests made + Calculated by obtaining the number of outgoing requests made using the requests library within an elapsed time and dividing that value over the elapsed time. """ From 99c9178dff17191adb2ad3e7689cf1fcd813b26b Mon Sep 17 00:00:00 2001 From: Hector Hernandez Date: Thu, 12 Mar 2020 12:50:03 -0700 Subject: [PATCH 072/109] Removing eachdist.ini --- eachdist.ini | 12 -- scripts/eachdist.py | 508 -------------------------------------------- scripts/pylint.sh | 13 ++ tox.ini | 7 +- 4 files changed, 18 insertions(+), 522 deletions(-) delete mode 100644 eachdist.ini delete mode 100644 scripts/eachdist.py create mode 100644 scripts/pylint.sh diff --git a/eachdist.ini b/eachdist.ini deleted file mode 100644 index fad7111..0000000 --- a/eachdist.ini +++ /dev/null @@ -1,12 +0,0 @@ -# These will be sorted first in that order. -# All packages that are depended upon by others should be listed here. -[DEFAULT] -sortfirst= - azure_monitor - -[lintroots] -extraroots=scripts/ -subglob=*.py,tests/,test/,src/* - -[testroots] -subglob=tests/,test/ diff --git a/scripts/eachdist.py b/scripts/eachdist.py deleted file mode 100644 index 406afb6..0000000 --- a/scripts/eachdist.py +++ /dev/null @@ -1,508 +0,0 @@ -#!/usr/bin/env python3 - -import argparse -import shlex -import shutil -import subprocess -import sys -from collections import namedtuple -from configparser import ConfigParser -from inspect import cleandoc -from itertools import chain -from pathlib import Path, PurePath - -DEFAULT_ALLSEP = " " -DEFAULT_ALLFMT = "{rel}" - - -def unique(elems): - seen = set() - for elem in elems: - if elem not in seen: - yield elem - seen.add(elem) - - -try: - subprocess_run = subprocess.run -except AttributeError: # Py < 3.5 compat - CompletedProcess = namedtuple("CompletedProcess", "returncode") - - def subprocess_run(*args, **kwargs): - check = kwargs.pop("check", False) - if check: - subprocess.check_call(*args, **kwargs) - return CompletedProcess(returncode=0) - return CompletedProcess(returncode=subprocess.call(*args, **kwargs)) - - -def extraargs_help(calledcmd): - return cleandoc( - """ - Additional arguments to pass on to {}. - - This is collected from any trailing arguments passed to `%(prog)s`. - Use an initial `--` to separate them from regular arguments. - """.format( - calledcmd - ) - ) - - -def parse_args(args=None): - parser = argparse.ArgumentParser(description="Development helper script.") - parser.set_defaults(parser=parser) - parser.add_argument( - "--dry-run", - action="store_true", - help="Only display what would be done, don't actually do anything.", - ) - subparsers = parser.add_subparsers(metavar="COMMAND") - subparsers.required = True - - excparser = subparsers.add_parser( - "exec", - help="Run a command for each or all targets.", - formatter_class=argparse.RawTextHelpFormatter, - description=cleandoc( - """Run a command according to the `format` argument for each or all targets. - - This is an advanced command that is used internally by other commands. - - For example, to install all distributions in this repository - editable, you could use: - - scripts/eachdist.py exec "python -m pip install -e {}" - - This will run pip for all distributions which is quite slow. It gets - a bit faster if we only invoke pip once but with all the paths - gathered together, which can be achieved by using `--all`: - - scripts/eachdist.py exec "python -m pip install {}" --all "-e {}" - - The sortfirst option in the DEFAULT section of eachdist.ini makes - sure that dependencies are installed before their dependents. - - Search for usages of `parse_subargs` in the source code of this script - to see more examples. - - This command first collects target paths and then executes - commands according to `format` and `--all`. - - Target paths are initially all Python distribution root paths - (as determined by the existence of setup.py, etc. files). - They are then augmented according to the section of the - `PROJECT_ROOT/eachdist.ini` config file specified by the `--mode` option. - - The following config options are available (and processed in that order): - - - `extraroots`: List of project root-relative glob expressions. - The resulting paths will be added. - - `sortfirst`: List of glob expressions. - Any matching paths will be put to the front of the path list, - in the same order they appear in this option. If more than one - glob matches, ordering is according to the first. - - `subglob`: List of glob expressions. Each path added so far is removed - and replaced with the result of all glob expressions relative to it (in - order of the glob expressions). - - After all this, any duplicate paths are removed (the first occurrence remains). - """ - ), - ) - excparser.set_defaults(func=execute_args) - excparser.add_argument( - "format", - help=cleandoc( - """Format string for the command to execute. - - The available replacements depend on whether `--all` is specified. - If `--all` was specified, there is only a single replacement, - `{}`, that is replaced with the string that is generated from - joining all targets formatted with `--all` to a single string - with the value of `--allsep` as separator. - - If `--all` was not specified, the following replacements are available: - - - `{}`: the absolute path to the current target in POSIX format - (with forward slashes) - - `{rel}`: like `{}` but relative to the project root. - - `{raw}`: the absolute path to the current target in native format - (thus exactly the same as `{}` on Unix but with backslashes on Windows). - - `{rawrel}`: like `{raw}` but relative to the project root. - - The resulting string is then split according to POSIX shell rules - (so you can use quotation marks or backslashes to handle arguments - containing spaces). - - The first token is the name of the executable to run, the remaining - tokens are the arguments. - - Note that a shell is *not* involved by default. - You can add bash/sh/cmd/powershell yourself to the format if you want. - - If `--all` was specified, the resulting command is simply executed once. - Otherwise, the command is executed for each found target. In both cases, - the project root is the working directory. - """ - ), - ) - excparser.add_argument( - "--all", - nargs="?", - const=DEFAULT_ALLFMT, - metavar="ALLFORMAT", - help=cleandoc( - """Instead of running the command for each target, join all target - paths together to run a single command. - - This option optionally takes a format string to apply to each path. The - available replacements are the ones that would be available for `format` - if `--all` was not specified. - - Default ALLFORMAT if this flag is specified: `%(const)s`. - """ - ), - ) - excparser.add_argument( - "--allsep", - help=cleandoc( - """Separator string for the strings resulting from `--all`. - Only valid if `--all` is specified. - """ - ), - ) - excparser.add_argument( - "--allowexitcode", - type=int, - action="append", - default=[0], - help=cleandoc( - """The given command exit code is treated as success and does not abort execution. - Can be specified multiple times. - """ - ), - ) - excparser.add_argument( - "--mode", - "-m", - default="DEFAULT", - help=cleandoc( - """Section of config file to use for target selection configuration. - See description of exec for available options.""" - ), - ) - - instparser = subparsers.add_parser( - "install", help="Install all distributions." - ) - - def setup_instparser(instparser): - instparser.set_defaults(func=install_args) - instparser.add_argument( - "pipargs", nargs=argparse.REMAINDER, help=extraargs_help("pip") - ) - - setup_instparser(instparser) - instparser.add_argument("--editable", "-e", action="store_true") - instparser.add_argument("--with-dev-deps", action="store_true") - instparser.add_argument("--eager-upgrades", action="store_true") - - devparser = subparsers.add_parser( - "develop", - help="Install all distributions editable + dev dependencies.", - ) - setup_instparser(devparser) - devparser.set_defaults( - editable=True, with_dev_deps=True, eager_upgrades=True - ) - - lintparser = subparsers.add_parser( - "lint", help="Lint everything, autofixing if possible." - ) - lintparser.add_argument("--check-only", action="store_true") - lintparser.set_defaults(func=lint_args) - - testparser = subparsers.add_parser( - "test", - help="Test everything (run pytest yourself for more complex operations).", - ) - testparser.set_defaults(func=test_args) - testparser.add_argument( - "pytestargs", nargs=argparse.REMAINDER, help=extraargs_help("pytest") - ) - - return parser.parse_args(args) - - -def find_projectroot(search_start=Path(".")): - root = search_start.resolve() - for root in chain((root,), root.parents): - if any((root / marker).exists() for marker in (".git", "tox.ini")): - return root - return None - - -def find_targets_unordered(rootpath): - for subdir in rootpath.iterdir(): - if not subdir.is_dir(): - continue - if subdir.name.startswith(".") or subdir.name.startswith("venv"): - continue - if any( - (subdir / marker).exists() - for marker in ("setup.py", "pyproject.toml") - ): - yield subdir - else: - yield from find_targets_unordered(subdir) - - -def getlistcfg(strval): - return [ - val.strip() - for line in strval.split("\n") - for val in line.split(",") - if val.strip() - ] - - -def find_targets(mode, rootpath): - if not rootpath: - sys.exit("Could not find a root directory.") - - cfg = ConfigParser() - cfg.read(str(rootpath / "eachdist.ini")) - mcfg = cfg[mode] - - targets = list(find_targets_unordered(rootpath)) - if "extraroots" in mcfg: - targets += [ - path - for extraglob in getlistcfg(mcfg["extraroots"]) - for path in rootpath.glob(extraglob) - ] - if "sortfirst" in mcfg: - sortfirst = getlistcfg(mcfg["sortfirst"]) - - def keyfunc(path): - path = path.relative_to(rootpath) - for idx, pattern in enumerate(sortfirst): - if path.match(pattern): - return idx - return float("inf") - - targets.sort(key=keyfunc) - - subglobs = getlistcfg(mcfg.get("subglob", "")) - if subglobs: - targets = [ - newentry - for newentry in ( - target / subdir - for target in targets - for subglob in subglobs - # We need to special-case the dot, because glob fails to parse that with an IndexError. - for subdir in ( - (target,) if subglob == "." else target.glob(subglob) - ) - ) - if ".egg-info" not in str(newentry) and newentry.exists() - ] - - return list(unique(targets)) - - -def runsubprocess(dry_run, params, *args, **kwargs): - cmdstr = join_args(params) - if dry_run: - print(cmdstr) - return None - - # Py < 3.6 compat. - cwd = kwargs.get("cwd") - if cwd and isinstance(cwd, PurePath): - kwargs["cwd"] = str(cwd) - - check = kwargs.pop("check") # Enforce specifying check - - print(">>>", cmdstr, file=sys.stderr) - - # This is a workaround for subprocess.run(['python']) leaving the virtualenv on Win32. - # The cause for this is that when running the python.exe in a virtualenv, - # the wrapper executable launches the global python as a subprocess and the search sequence - # for CreateProcessW which subprocess.run and Popen use is a follows - # (https://docs.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-createprocessw): - # > 1. The directory from which the application loaded. - # This will be the directory of the global python.exe, not the venv directory, due to the suprocess mechanism. - # > 6. The directories that are listed in the PATH environment variable. - # Only this would find the "correct" python.exe. - - params = list(params) - executable = shutil.which(params[0]) # On Win32, pytho - if executable: - params[0] = executable - try: - return subprocess_run(params, *args, check=check, **kwargs) - except OSError as exc: - raise ValueError( - "Failed executing " + repr(params) + ": " + str(exc) - ) from exc - - -def execute_args(args): - if args.allsep and not args.all: - args.parser.error("--allsep specified but not --all.") - - if args.all and not args.allsep: - args.allsep = DEFAULT_ALLSEP - - rootpath = find_projectroot() - targets = find_targets(args.mode, rootpath) - if not targets: - sys.exit("Error: No targets selected (root: {})".format(rootpath)) - - def fmt_for_path(fmt, path): - return fmt.format( - path.as_posix(), - rel=path.relative_to(rootpath).as_posix(), - raw=path, - rawrel=path.relative_to(rootpath), - ) - - def _runcmd(cmd): - result = runsubprocess( - args.dry_run, shlex.split(cmd), cwd=rootpath, check=False - ) - if result is not None and result.returncode not in args.allowexitcode: - print( - "'{}' failed with code {}".format(cmd, result.returncode), - file=sys.stderr, - ) - sys.exit(result.returncode) - - if args.all: - allstr = args.allsep.join( - fmt_for_path(args.all, path) for path in targets - ) - cmd = args.format.format(allstr) - _runcmd(cmd) - else: - for target in targets: - cmd = fmt_for_path(args.format, target) - _runcmd(cmd) - - -def clean_remainder_args(remainder_args): - if remainder_args and remainder_args[0] == "--": - del remainder_args[0] - - -def join_args(arglist): - return " ".join(map(shlex.quote, arglist)) - - -def install_args(args): - clean_remainder_args(args.pipargs) - if args.eager_upgrades: - args.pipargs += ["--upgrade-strategy=eager"] - - if args.with_dev_deps: - runsubprocess( - args.dry_run, - [ - "python", - "-m", - "pip", - "install", - "--upgrade", - "pip", - "setuptools", - "wheel", - ] - + args.pipargs, - check=True, - ) - - allfmt = "-e 'file://{}'" if args.editable else "'file://{}'" - execute_args( - parse_subargs( - args, - ( - "exec", - "python -m pip install {} " + join_args(args.pipargs), - "--all", - allfmt, - ), - ) - ) - if args.with_dev_deps: - rootpath = find_projectroot() - runsubprocess( - args.dry_run, - [ - "python", - "-m", - "pip", - "install", - "--upgrade", - "-r", - str(rootpath / "dev-requirements.txt"), - ] - + args.pipargs, - check=True, - ) - - -def parse_subargs(parentargs, args): - subargs = parse_args(args) - subargs.dry_run = parentargs.dry_run or subargs.dry_run - return subargs - - -def lint_args(args): - rootdir = str(find_projectroot()) - - runsubprocess( - args.dry_run, - ("black", ".") + (("--diff", "--check") if args.check_only else ()), - cwd=rootdir, - check=True, - ) - runsubprocess( - args.dry_run, - ("isort", "--recursive", ".") - + (("--diff", "--check-only") if args.check_only else ()), - cwd=rootdir, - check=True, - ) - runsubprocess(args.dry_run, ("flake8", rootdir), check=True) - execute_args( - parse_subargs( - args, ("exec", "pylint {}", "--all", "--mode", "lintroots") - ) - ) - - -def test_args(args): - clean_remainder_args(args.pytestargs) - execute_args( - parse_subargs( - args, - ( - "exec", - "pytest {} " + join_args(args.pytestargs), - "--mode", - "testroots", - ), - ) - ) - - -def main(): - args = parse_args() - args.func(args) - - -if __name__ == "__main__": - main() diff --git a/scripts/pylint.sh b/scripts/pylint.sh new file mode 100644 index 0000000..5c412f8 --- /dev/null +++ b/scripts/pylint.sh @@ -0,0 +1,13 @@ +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. + +set -ev + +# Run pylint on directories +function pylint_dir { + python -m pip install --upgrade pylint + pylint $(find azure_monitor -type f -name "*.py") + return $? +} + +pylint_dir \ No newline at end of file diff --git a/tox.ini b/tox.ini index 1338d2d..84c7b5a 100644 --- a/tox.ini +++ b/tox.ini @@ -45,7 +45,10 @@ deps = psutil commands_pre = - python scripts/eachdist.py install --editable + pip install {toxinidir}/azure_monitor commands = - python scripts/eachdist.py lint --check-only + black . --diff --check + isort --diff --check-only --recursive . + flake8 + bash ./scripts/pylint.sh From 0113efc339c53cce443cb240c84fc6461608cf5a Mon Sep 17 00:00:00 2001 From: Hector Hernandez Date: Thu, 12 Mar 2020 13:25:26 -0700 Subject: [PATCH 073/109] Adding missing space --- scripts/pylint.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/pylint.sh b/scripts/pylint.sh index 5c412f8..f757fac 100644 --- a/scripts/pylint.sh +++ b/scripts/pylint.sh @@ -10,4 +10,4 @@ function pylint_dir { return $? } -pylint_dir \ No newline at end of file +pylint_dir From 1518fbad0d308b9d486e54e2d1e592c0d9341ace Mon Sep 17 00:00:00 2001 From: Hector Hernandez Date: Thu, 12 Mar 2020 13:45:50 -0700 Subject: [PATCH 074/109] Updating tox --- tox.ini | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tox.ini b/tox.ini index 84c7b5a..68fab29 100644 --- a/tox.ini +++ b/tox.ini @@ -45,7 +45,7 @@ deps = psutil commands_pre = - pip install {toxinidir}/azure_monitor + pip install ./azure_monitor commands = black . --diff --check From c2c5c0f97d8211e94ffb9e8f7f6e32879cc89538 Mon Sep 17 00:00:00 2001 From: Hector Hernandez Date: Thu, 12 Mar 2020 13:48:39 -0700 Subject: [PATCH 075/109] Fixing lint --- azure_monitor/examples/auto_collector.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/azure_monitor/examples/auto_collector.py b/azure_monitor/examples/auto_collector.py index e816ff8..676924c 100644 --- a/azure_monitor/examples/auto_collector.py +++ b/azure_monitor/examples/auto_collector.py @@ -1,7 +1,7 @@ # Copyright (c) Microsoft Corporation. All rights reserved. # Licensed under the MIT License. from opentelemetry import metrics -from opentelemetry.sdk.metrics import Counter, Meter +from opentelemetry.sdk.metrics import Meter from opentelemetry.sdk.metrics.export.controller import PushController from azure_monitor import AutoCollection, AzureMonitorMetricsExporter From 1a067486354eb9ad4c2b2783182cfcdb03fa368f Mon Sep 17 00:00:00 2001 From: Leighton Date: Thu, 12 Mar 2020 14:51:25 -0700 Subject: [PATCH 076/109] fix trace tests' --- azure_monitor/tests/test_base_exporter.py | 16 ++++++++++++++-- azure_monitor/tests/test_storage.py | 2 +- azure_monitor/tests/trace/test_trace.py | 6 +++--- 3 files changed, 18 insertions(+), 6 deletions(-) diff --git a/azure_monitor/tests/test_base_exporter.py b/azure_monitor/tests/test_base_exporter.py index 596a91c..d453a5a 100644 --- a/azure_monitor/tests/test_base_exporter.py +++ b/azure_monitor/tests/test_base_exporter.py @@ -2,6 +2,7 @@ # Licensed under the MIT License. import os +import shutil import unittest from azure_monitor.export import BaseExporter @@ -9,6 +10,17 @@ from azure_monitor.protocol import Data, Envelope +TEST_FOLDER = os.path.abspath(".test.exporter") + + +def setUpModule(): + os.makedirs(TEST_FOLDER) + + +def tearDownModule(): + shutil.rmtree(TEST_FOLDER) + + # pylint: disable=W0212 class TestBaseExporter(unittest.TestCase): @classmethod @@ -23,7 +35,7 @@ def test_constructor(self): instrumentation_key="4321abcd-5678-4efa-8abc-1234567890ab", storage_maintenance_period=2, storage_max_size=3, - storage_path="testStoragePath", + storage_path=os.path.join(TEST_FOLDER, self.id()), storage_retention_period=4, timeout=5, ) @@ -36,7 +48,7 @@ def test_constructor(self): self.assertEqual(base.options.storage_max_size, 3) self.assertEqual(base.options.storage_retention_period, 4) self.assertEqual(base.options.timeout, 5) - self.assertEqual(base.options.storage_path, "testStoragePath") + self.assertEqual(base.options.storage_path, os.path.join(TEST_FOLDER, self.id())) def test_constructor_wrong_options(self): """Test the constructor with wrong options.""" diff --git a/azure_monitor/tests/test_storage.py b/azure_monitor/tests/test_storage.py index 2a056f7..a24b49a 100644 --- a/azure_monitor/tests/test_storage.py +++ b/azure_monitor/tests/test_storage.py @@ -13,7 +13,7 @@ _seconds, ) -TEST_FOLDER = os.path.abspath(".test") +TEST_FOLDER = os.path.abspath(".test.storage") # pylint: disable=invalid-name diff --git a/azure_monitor/tests/trace/test_trace.py b/azure_monitor/tests/trace/test_trace.py index 814d896..d93ceb2 100644 --- a/azure_monitor/tests/trace/test_trace.py +++ b/azure_monitor/tests/trace/test_trace.py @@ -126,7 +126,7 @@ def test_transmission_nothing(self): post.return_value = None exporter._transmit_from_storage() - def test_transmission_request_exception(self): + def test_transmit_request_exception(self): exporter = AzureMonitorSpanExporter( storage_path=os.path.join(TEST_FOLDER, self.id()) ) @@ -152,7 +152,7 @@ def test_transmission_lease_failure(self, requests_mock): exporter._transmit_from_storage() self.assertTrue(exporter.storage.get()) - def test_transmission_response_exception(self): + def test_transmit_response_exception(self): exporter = AzureMonitorSpanExporter( storage_path=os.path.join(TEST_FOLDER, self.id()) ) @@ -223,7 +223,7 @@ def test_transmission_206_500(self): exporter.storage.get().get()[0]["name"], "testEnvelope" ) - def test_transmission_206_nothing_to_retry(self): + def test_transmission_206_no_retry(self): exporter = AzureMonitorSpanExporter( storage_path=os.path.join(TEST_FOLDER, self.id()) ) From f8506c6e98de65f33df2c8c9685fb9549ae0bfb9 Mon Sep 17 00:00:00 2001 From: Leighton Date: Thu, 12 Mar 2020 15:00:32 -0700 Subject: [PATCH 077/109] fix black --- azure_monitor/tests/test_base_exporter.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/azure_monitor/tests/test_base_exporter.py b/azure_monitor/tests/test_base_exporter.py index d453a5a..7c60542 100644 --- a/azure_monitor/tests/test_base_exporter.py +++ b/azure_monitor/tests/test_base_exporter.py @@ -48,7 +48,9 @@ def test_constructor(self): self.assertEqual(base.options.storage_max_size, 3) self.assertEqual(base.options.storage_retention_period, 4) self.assertEqual(base.options.timeout, 5) - self.assertEqual(base.options.storage_path, os.path.join(TEST_FOLDER, self.id())) + self.assertEqual( + base.options.storage_path, os.path.join(TEST_FOLDER, self.id()) + ) def test_constructor_wrong_options(self): """Test the constructor with wrong options.""" From a624cd30facdfd00ff4fb0b79d62709e3658727d Mon Sep 17 00:00:00 2001 From: Leighton Date: Thu, 12 Mar 2020 15:05:42 -0700 Subject: [PATCH 078/109] fix isort --- azure_monitor/tests/test_base_exporter.py | 1 - 1 file changed, 1 deletion(-) diff --git a/azure_monitor/tests/test_base_exporter.py b/azure_monitor/tests/test_base_exporter.py index 7c60542..1f85050 100644 --- a/azure_monitor/tests/test_base_exporter.py +++ b/azure_monitor/tests/test_base_exporter.py @@ -9,7 +9,6 @@ from azure_monitor.options import ExporterOptions from azure_monitor.protocol import Data, Envelope - TEST_FOLDER = os.path.abspath(".test.exporter") From 9f39a926887eb2547aa2b8be2bda44b1660b9524 Mon Sep 17 00:00:00 2001 From: Leighton Date: Thu, 12 Mar 2020 15:11:06 -0700 Subject: [PATCH 079/109] fix lint --- azure_monitor/tests/test_base_exporter.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/azure_monitor/tests/test_base_exporter.py b/azure_monitor/tests/test_base_exporter.py index 1f85050..240f174 100644 --- a/azure_monitor/tests/test_base_exporter.py +++ b/azure_monitor/tests/test_base_exporter.py @@ -12,10 +12,12 @@ TEST_FOLDER = os.path.abspath(".test.exporter") +# pylint: disable=invalid-name def setUpModule(): os.makedirs(TEST_FOLDER) +# pylint: disable=invalid-name def tearDownModule(): shutil.rmtree(TEST_FOLDER) From 4d2b1bd6d9baf60e35e6a5c22cbb975ac6859dd9 Mon Sep 17 00:00:00 2001 From: Leighton Date: Fri, 13 Mar 2020 12:51:43 -0700 Subject: [PATCH 080/109] add coverage --- .coveragerc | 2 +- .../azure_monitor/export/metrics/__init__.py | 3 +- .../azure_monitor/export/trace/__init__.py | 13 +- ..._metrics.py => test_dependency_metrics.py} | 26 +- .../test_performance_metrics.py | 38 +++ .../auto_collection/test_request_metrics.py | 26 ++ azure_monitor/tests/metrics/test_metrics.py | 40 +++ azure_monitor/tests/test_base_exporter.py | 232 ++++++++++++++++- azure_monitor/tests/test_options.py | 14 + azure_monitor/tests/test_protocol.py | 71 +++++ azure_monitor/tests/test_utils.py | 38 --- azure_monitor/tests/trace/test_trace.py | 242 ++++-------------- 12 files changed, 507 insertions(+), 238 deletions(-) rename azure_monitor/tests/auto_collection/{test_depencency_metrics.py => test_dependency_metrics.py} (76%) diff --git a/.coveragerc b/.coveragerc index f41a787..5f7e9aa 100644 --- a/.coveragerc +++ b/.coveragerc @@ -5,7 +5,7 @@ omit = azure_monitor/tests/* [report] -fail_under = 80 +fail_under = 98 show_missing = True omit = azure_monitor/setup.py diff --git a/azure_monitor/src/azure_monitor/export/metrics/__init__.py b/azure_monitor/src/azure_monitor/export/metrics/__init__.py index 22ec365..0f69740 100644 --- a/azure_monitor/src/azure_monitor/export/metrics/__init__.py +++ b/azure_monitor/src/azure_monitor/export/metrics/__init__.py @@ -36,7 +36,7 @@ def export( ) try: result = self._transmit(envelopes) - if result == MetricsExportResult.FAILED_RETRYABLE: + if result == ExportResult.FAILED_RETRYABLE: self.storage.put(envelopes, result) if result == ExportResult.SUCCESS: # Try to send any cached events @@ -44,6 +44,7 @@ def export( return get_metrics_export_result(result) except Exception: # pylint: disable=broad-except logger.exception("Exception occurred while exporting the data.") + return get_metrics_export_result(ExportResult.FAILED_NOT_RETRYABLE) def metric_to_envelope( self, metric_record: MetricRecord diff --git a/azure_monitor/src/azure_monitor/export/trace/__init__.py b/azure_monitor/src/azure_monitor/export/trace/__init__.py index e79998d..c9ceb32 100644 --- a/azure_monitor/src/azure_monitor/export/trace/__init__.py +++ b/azure_monitor/src/azure_monitor/export/trace/__init__.py @@ -22,13 +22,15 @@ class AzureMonitorSpanExporter(BaseExporter, SpanExporter): def export(self, spans: Sequence[Span]) -> SpanExportResult: - envelopes = map(self.span_to_envelope, spans) - envelopes_to_export = map( - lambda x: x.to_dict(), - tuple(self.apply_telemetry_processors(envelopes)), + envelopes = list(map(self.span_to_envelope, spans)) + envelopes = list( + map( + lambda x: x.to_dict(), + self.apply_telemetry_processors(envelopes), + ) ) try: - result = self._transmit(envelopes_to_export) + result = self._transmit(envelopes) if result == ExportResult.FAILED_RETRYABLE: self.storage.put(envelopes, result) if result == ExportResult.SUCCESS: @@ -37,6 +39,7 @@ def export(self, spans: Sequence[Span]) -> SpanExportResult: return get_trace_export_result(result) except Exception: # pylint: disable=broad-except logger.exception("Exception occurred while exporting the data.") + return get_trace_export_result(ExportResult.FAILED_NOT_RETRYABLE) # pylint: disable=too-many-statements # pylint: disable=too-many-branches diff --git a/azure_monitor/tests/auto_collection/test_depencency_metrics.py b/azure_monitor/tests/auto_collection/test_dependency_metrics.py similarity index 76% rename from azure_monitor/tests/auto_collection/test_depencency_metrics.py rename to azure_monitor/tests/auto_collection/test_dependency_metrics.py index 186ddc4..86ecb6c 100644 --- a/azure_monitor/tests/auto_collection/test_depencency_metrics.py +++ b/azure_monitor/tests/auto_collection/test_dependency_metrics.py @@ -57,8 +57,18 @@ def test_constructor(self): ), ) + def test_track(self): + mock_meter = mock.Mock() + metrics_collector = DependencyMetrics( + meter=mock_meter, label_set=self._test_label_set + ) + track_mock = mock.Mock() + metrics_collector._track_dependency_rate = track_mock + metrics_collector.track() + self.assertEqual(track_mock.call_count, 1) + @mock.patch("azure_monitor.auto_collection.dependency_metrics.time") - def test_track_depencency_rate(self, time_mock): + def test_track_dependency_rate(self, time_mock): time_mock.time.return_value = 100 metrics_collector = DependencyMetrics( meter=self._meter, label_set=self._test_label_set @@ -71,7 +81,19 @@ def test_track_depencency_rate(self, time_mock): ) @mock.patch("azure_monitor.auto_collection.dependency_metrics.time") - def test_track_depencency_rate_error(self, time_mock): + def test_track_dependency_rate_time_none(self, time_mock): + time_mock.time.return_value = 100 + metrics_collector = DependencyMetrics( + meter=self._meter, label_set=self._test_label_set + ) + dependency_metrics.dependency_map["last_time"] = None + metrics_collector._track_dependency_rate() + self.assertEqual( + metrics_collector._dependency_rate_handle.aggregator.current, 0 + ) + + @mock.patch("azure_monitor.auto_collection.dependency_metrics.time") + def test_track_dependency_rate_error(self, time_mock): time_mock.time.return_value = 100 metrics_collector = DependencyMetrics( meter=self._meter, label_set=self._test_label_set diff --git a/azure_monitor/tests/auto_collection/test_performance_metrics.py b/azure_monitor/tests/auto_collection/test_performance_metrics.py index f6654af..148a4b3 100644 --- a/azure_monitor/tests/auto_collection/test_performance_metrics.py +++ b/azure_monitor/tests/auto_collection/test_performance_metrics.py @@ -11,6 +11,13 @@ from azure_monitor.auto_collection import PerformanceMetrics +def throw(exc_type, *args, **kwargs): + def func(*_args, **_kwargs): + raise exc_type(*args, **kwargs) + + return func + + # pylint: disable=protected-access class TestPerformanceMetrics(unittest.TestCase): @classmethod @@ -80,6 +87,25 @@ def test_constructor(self): ), ) + def test_track(self): + mock_meter = mock.Mock() + performance_metrics_collector = PerformanceMetrics( + meter=mock_meter, label_set=self._test_label_set + ) + cpu_mock = mock.Mock() + process_mock = mock.Mock() + memory_mock = mock.Mock() + proc_memory_mock = mock.Mock() + performance_metrics_collector._track_cpu = cpu_mock + performance_metrics_collector._track_process_cpu = process_mock + performance_metrics_collector._track_memory = memory_mock + performance_metrics_collector._track_process_memory = proc_memory_mock + performance_metrics_collector.track() + self.assertEqual(cpu_mock.call_count, 1) + self.assertEqual(process_mock.call_count, 1) + self.assertEqual(memory_mock.call_count, 1) + self.assertEqual(proc_memory_mock.call_count, 1) + def test_track_cpu(self): performance_metrics_collector = PerformanceMetrics( meter=self._meter, label_set=self._test_label_set @@ -151,3 +177,15 @@ def test_track_process_memory(self): performance_metrics_collector._process_memory_handle.aggregator.current, 100, ) + + @mock.patch("azure_monitor.auto_collection.performance_metrics.logger") + def test_track_process_memory_exception(self, logger_mock): + with mock.patch( + "azure_monitor.auto_collection.performance_metrics.PROCESS", + throw(Exception) + ): + performance_metrics_collector = PerformanceMetrics( + meter=self._meter, label_set=self._test_label_set + ) + performance_metrics_collector._track_process_memory() + self.assertEqual(logger_mock.exception.called, True) diff --git a/azure_monitor/tests/auto_collection/test_request_metrics.py b/azure_monitor/tests/auto_collection/test_request_metrics.py index 93eec36..71c5d21 100644 --- a/azure_monitor/tests/auto_collection/test_request_metrics.py +++ b/azure_monitor/tests/auto_collection/test_request_metrics.py @@ -70,6 +70,19 @@ def test_constructor(self): ), ) + def test_track(self): + mock_meter = mock.Mock() + request_metrics_collector = RequestMetrics( + meter=mock_meter, label_set=self._test_label_set + ) + duration_mock = mock.Mock() + rate_mock = mock.Mock() + request_metrics_collector._track_request_duration = duration_mock + request_metrics_collector._track_request_rate = rate_mock + request_metrics_collector.track() + self.assertEqual(duration_mock.call_count, 1) + self.assertEqual(rate_mock.call_count, 1) + def test_track_request_duration(self): request_metrics_collector = RequestMetrics( meter=self._meter, label_set=self._test_label_set @@ -110,6 +123,19 @@ def test_track_request_rate(self, time_mock): 2, ) + @mock.patch("azure_monitor.auto_collection.request_metrics.time") + def test_track_request_rate_time_none(self, time_mock): + time_mock.time.return_value = 100 + request_metrics_collector = RequestMetrics( + meter=self._meter, label_set=self._test_label_set + ) + request_metrics.requests_map["last_time"] = None + request_metrics_collector._track_request_rate() + self.assertEqual( + request_metrics_collector._request_rate_handle.aggregator.current, + 0, + ) + @mock.patch("azure_monitor.auto_collection.request_metrics.time") def test_track_request_rate_error(self, time_mock): request_metrics_collector = RequestMetrics( diff --git a/azure_monitor/tests/metrics/test_metrics.py b/azure_monitor/tests/metrics/test_metrics.py index 1109c1a..32f22c4 100644 --- a/azure_monitor/tests/metrics/test_metrics.py +++ b/azure_monitor/tests/metrics/test_metrics.py @@ -16,6 +16,13 @@ from azure_monitor.protocol import Data, DataPoint, Envelope, MetricData +def throw(exc_type, *args, **kwargs): + def func(*_args, **_kwargs): + raise exc_type(*args, **kwargs) + + return func + + # pylint: disable=protected-access class TestAzureMetricsExporter(unittest.TestCase): @classmethod @@ -60,6 +67,39 @@ def test_export(self,): result = exporter.export([record]) self.assertEqual(result, MetricsExportResult.SUCCESS) + def test_export_failed_retryable(self): + record = MetricRecord( + CounterAggregator(), self._test_label_set, self._test_metric + ) + exporter = AzureMonitorMetricsExporter() + with mock.patch( + "azure_monitor.export.metrics.AzureMonitorMetricsExporter._transmit" + ) as transmit: # noqa: E501 + transmit.return_value = ExportResult.FAILED_RETRYABLE + storage_mock = mock.Mock() + exporter.storage.put = storage_mock + result = exporter.export([record]) + self.assertEqual(result, MetricsExportResult.FAILED_RETRYABLE) + self.assertEqual(storage_mock.call_count, 1) + + @mock.patch("azure_monitor.export.metrics.logger") + def test_export_exception(self, logger_mock): + record = MetricRecord( + CounterAggregator(), self._test_label_set, self._test_metric + ) + exporter = AzureMonitorMetricsExporter() + with mock.patch( + "azure_monitor.export.metrics.AzureMonitorMetricsExporter._transmit", + throw(Exception) + ): # noqa: E501 + result = exporter.export([record]) + self.assertEqual(result, MetricsExportResult.FAILED_NOT_RETRYABLE) + self.assertEqual(logger_mock.exception.called, True) + + def test_metric_to_envelope_none(self): + exporter = AzureMonitorMetricsExporter() + self.assertIsNone(exporter.metric_to_envelope(None)) + def test_metric_to_envelope(self): aggregator = CounterAggregator() aggregator.update(123) diff --git a/azure_monitor/tests/test_base_exporter.py b/azure_monitor/tests/test_base_exporter.py index 240f174..f306ee4 100644 --- a/azure_monitor/tests/test_base_exporter.py +++ b/azure_monitor/tests/test_base_exporter.py @@ -1,13 +1,22 @@ # Copyright (c) Microsoft Corporation. All rights reserved. # Licensed under the MIT License. +import json import os import shutil import unittest +from unittest import mock -from azure_monitor.export import BaseExporter +from azure_monitor.export import ( + BaseExporter, + ExportResult, + get_metrics_export_result, + get_trace_export_result, +) from azure_monitor.options import ExporterOptions from azure_monitor.protocol import Data, Envelope +from opentelemetry.sdk.metrics.export import MetricsExportResult +from opentelemetry.sdk.trace.export import SpanExportResult TEST_FOLDER = os.path.abspath(".test.exporter") @@ -22,6 +31,13 @@ def tearDownModule(): shutil.rmtree(TEST_FOLDER) +def throw(exc_type, *args, **kwargs): + def func(*_args, **_kwargs): + raise exc_type(*args, **kwargs) + + return func + + # pylint: disable=W0212 class TestBaseExporter(unittest.TestCase): @classmethod @@ -124,3 +140,217 @@ def callback_function(envelope): envelopes = base.apply_telemetry_processors([envelope, envelope2]) self.assertEqual(len(envelopes), 1) self.assertEqual(envelopes[0].data.base_type, "type2") + + def test_transmission_nothing(self): + exporter = BaseExporter( + storage_path=os.path.join(TEST_FOLDER, self.id()) + ) + with mock.patch("requests.post") as post: + post.return_value = None + exporter._transmit_from_storage() + + def test_transmit_request_exception(self): + exporter = BaseExporter( + storage_path=os.path.join(TEST_FOLDER, self.id()) + ) + envelopes_to_export = map(lambda x: x.to_dict(), tuple([Envelope()])) + exporter.storage.put(envelopes_to_export) + with mock.patch("requests.post", throw(Exception)): + exporter._transmit_from_storage() + self.assertIsNone(exporter.storage.get()) + self.assertEqual(len(os.listdir(exporter.storage.path)), 1) + + @mock.patch("requests.post", return_value=mock.Mock()) + def test_transmission_lease_failure(self, requests_mock): + requests_mock.return_value = MockResponse(200, "unknown") + exporter = BaseExporter( + storage_path=os.path.join(TEST_FOLDER, self.id()) + ) + envelopes_to_export = map(lambda x: x.to_dict(), tuple([Envelope()])) + exporter.storage.put(envelopes_to_export) + with mock.patch( + "azure_monitor.storage.LocalFileBlob.lease" + ) as lease: # noqa: E501 + lease.return_value = False + exporter._transmit_from_storage() + self.assertTrue(exporter.storage.get()) + + def test_transmit_response_exception(self): + exporter = BaseExporter( + storage_path=os.path.join(TEST_FOLDER, self.id()) + ) + envelopes_to_export = map(lambda x: x.to_dict(), tuple([Envelope()])) + exporter.storage.put(envelopes_to_export) + with mock.patch("requests.post") as post: + post.return_value = MockResponse(200, None) + del post.return_value.text + exporter._transmit_from_storage() + self.assertIsNone(exporter.storage.get()) + self.assertEqual(len(os.listdir(exporter.storage.path)), 0) + + def test_transmission_200(self): + exporter = BaseExporter( + storage_path=os.path.join(TEST_FOLDER, self.id()) + ) + envelopes_to_export = map(lambda x: x.to_dict(), tuple([Envelope()])) + exporter.storage.put(envelopes_to_export) + with mock.patch("requests.post") as post: + post.return_value = MockResponse(200, "unknown") + exporter._transmit_from_storage() + self.assertIsNone(exporter.storage.get()) + self.assertEqual(len(os.listdir(exporter.storage.path)), 0) + + def test_transmission_206(self): + exporter = BaseExporter( + storage_path=os.path.join(TEST_FOLDER, self.id()) + ) + envelopes_to_export = map(lambda x: x.to_dict(), tuple([Envelope()])) + exporter.storage.put(envelopes_to_export) + with mock.patch("requests.post") as post: + post.return_value = MockResponse(206, "unknown") + exporter._transmit_from_storage() + self.assertIsNone(exporter.storage.get()) + self.assertEqual(len(os.listdir(exporter.storage.path)), 1) + + def test_transmission_206_500(self): + exporter = BaseExporter( + storage_path=os.path.join(TEST_FOLDER, self.id()) + ) + test_envelope = Envelope(name="testEnvelope") + envelopes_to_export = map( + lambda x: x.to_dict(), + tuple([Envelope(), Envelope(), test_envelope]), + ) + exporter.storage.put(envelopes_to_export) + with mock.patch("requests.post") as post: + post.return_value = MockResponse( + 206, + json.dumps( + { + "itemsReceived": 5, + "itemsAccepted": 3, + "errors": [ + {"index": 0, "statusCode": 400, "message": ""}, + { + "index": 2, + "statusCode": 500, + "message": "Internal Server Error", + }, + ], + } + ), + ) + exporter._transmit_from_storage() + self.assertEqual(len(os.listdir(exporter.storage.path)), 1) + self.assertEqual( + exporter.storage.get().get()[0]["name"], "testEnvelope" + ) + + def test_transmission_206_no_retry(self): + exporter = BaseExporter( + storage_path=os.path.join(TEST_FOLDER, self.id()) + ) + envelopes_to_export = map(lambda x: x.to_dict(), tuple([Envelope()])) + exporter.storage.put(envelopes_to_export) + with mock.patch("requests.post") as post: + post.return_value = MockResponse( + 206, + json.dumps( + { + "itemsReceived": 3, + "itemsAccepted": 2, + "errors": [ + {"index": 0, "statusCode": 400, "message": ""} + ], + } + ), + ) + exporter._transmit_from_storage() + self.assertEqual(len(os.listdir(exporter.storage.path)), 0) + + def test_transmission_206_bogus(self): + exporter = BaseExporter( + storage_path=os.path.join(TEST_FOLDER, self.id()) + ) + envelopes_to_export = map(lambda x: x.to_dict(), tuple([Envelope()])) + exporter.storage.put(envelopes_to_export) + with mock.patch("requests.post") as post: + post.return_value = MockResponse( + 206, + json.dumps( + { + "itemsReceived": 5, + "itemsAccepted": 3, + "errors": [{"foo": 0, "bar": 1}], + } + ), + ) + exporter._transmit_from_storage() + self.assertIsNone(exporter.storage.get()) + self.assertEqual(len(os.listdir(exporter.storage.path)), 0) + + def test_transmission_400(self): + exporter = BaseExporter( + storage_path=os.path.join(TEST_FOLDER, self.id()) + ) + envelopes_to_export = map(lambda x: x.to_dict(), tuple([Envelope()])) + exporter.storage.put(envelopes_to_export) + with mock.patch("requests.post") as post: + post.return_value = MockResponse(400, "{}") + exporter._transmit_from_storage() + self.assertEqual(len(os.listdir(exporter.storage.path)), 0) + + def test_transmission_500(self): + exporter = BaseExporter( + storage_path=os.path.join(TEST_FOLDER, self.id()) + ) + envelopes_to_export = map(lambda x: x.to_dict(), tuple([Envelope()])) + exporter.storage.put(envelopes_to_export) + with mock.patch("requests.post") as post: + post.return_value = MockResponse(500, "{}") + exporter._transmit_from_storage() + self.assertIsNone(exporter.storage.get()) + self.assertEqual(len(os.listdir(exporter.storage.path)), 1) + + def test_transmission_empty(self): + exporter = BaseExporter( + storage_path=os.path.join(TEST_FOLDER, self.id()) + ) + status = exporter._transmit([]) + self.assertEqual(status, ExportResult.SUCCESS) + + def test_get_trace_export_result(self): + self.assertEqual( + get_trace_export_result(ExportResult.SUCCESS), + SpanExportResult.SUCCESS, + ) + self.assertEqual( + get_trace_export_result(ExportResult.FAILED_NOT_RETRYABLE), + SpanExportResult.FAILED_NOT_RETRYABLE, + ) + self.assertEqual( + get_trace_export_result(ExportResult.FAILED_RETRYABLE), + SpanExportResult.FAILED_RETRYABLE, + ) + self.assertEqual(get_trace_export_result(None), None) + + def test_get_metrics_export_result(self): + self.assertEqual( + get_metrics_export_result(ExportResult.SUCCESS), + MetricsExportResult.SUCCESS, + ) + self.assertEqual( + get_metrics_export_result(ExportResult.FAILED_NOT_RETRYABLE), + MetricsExportResult.FAILED_NOT_RETRYABLE, + ) + self.assertEqual( + get_metrics_export_result(ExportResult.FAILED_RETRYABLE), + MetricsExportResult.FAILED_RETRYABLE, + ) + self.assertEqual(get_metrics_export_result(None), None) + + +class MockResponse: + def __init__(self, status_code, text): + self.status_code = status_code + self.text = text diff --git a/azure_monitor/tests/test_options.py b/azure_monitor/tests/test_options.py index def6027..89b4368 100644 --- a/azure_monitor/tests/test_options.py +++ b/azure_monitor/tests/test_options.py @@ -254,3 +254,17 @@ def test_parse_connection_string_invalid_auth(self): instrumentation_key=self._valid_instrumentation_key, ), ) + + def test_parse_connection_string_suffix(self): + options = ExporterOptions( + connection_string="Authorization=ikey;EndpointSuffix=123;Location=US", + instrumentation_key=self._valid_instrumentation_key, + ) + self.assertEqual(options.endpoint, "https://US.dc.123/v2/track") + + def test_parse_connection_string_suffix_no_location(self): + options = ExporterOptions( + connection_string="Authorization=ikey;EndpointSuffix=123", + instrumentation_key=self._valid_instrumentation_key, + ) + self.assertEqual(options.endpoint, "https://dc.123/v2/track") diff --git a/azure_monitor/tests/test_protocol.py b/azure_monitor/tests/test_protocol.py index 0248b12..f4e6d90 100644 --- a/azure_monitor/tests/test_protocol.py +++ b/azure_monitor/tests/test_protocol.py @@ -28,14 +28,69 @@ def test_event(self): data = protocol.Event() self.assertEqual(data.ver, 2) + def test_event_to_dict(self): + data = protocol.Event() + to_dict = { + "ver": 2, + "name": "", + "properties": None, + "measurements": None, + } + self.assertEqual(data.to_dict(), to_dict) + + def test_exception_details(self): + data = protocol.ExceptionDetails() + self.assertEqual(data.id, None) + + def test_exception_details_to_dict(self): + data = protocol.ExceptionDetails() + to_dict = { + "id": None, + "outerId": None, + "typeName": None, + "message": None, + "hasFullStack ": None, + "stack": None, + "parsedStack": None, + } + self.assertEqual(data.to_dict(), to_dict) + def test_exception_data(self): data = protocol.ExceptionData() self.assertEqual(data.ver, 2) + def test_exception_data_details(self): + details = protocol.ExceptionDetails() + data = protocol.ExceptionData(exceptions=[details]) + self.assertEqual(len(data.exceptions), 1) + + def test_exception_data_to_dict(self): + data = protocol.ExceptionData() + to_dict = { + "ver": 2, + "exceptions": [], + "severityLevel": None, + "problemId": None, + "properties": None, + "measurements": None, + } + self.assertEqual(data.to_dict(), to_dict) + def test_message(self): data = protocol.Message() self.assertEqual(data.ver, 2) + def test_message_to_dict(self): + data = protocol.Message() + to_dict = { + "ver": 2, + "message": "", + "severityLevel": None, + "properties": None, + "measurements": None, + } + self.assertEqual(data.to_dict(), to_dict) + def test_metric_data(self): data = protocol.MetricData() self.assertEqual(data.ver, 2) @@ -47,3 +102,19 @@ def test_remote_dependency(self): def test_request(self): data = protocol.Request() self.assertEqual(data.ver, 2) + + def test_request_to_dict(self): + data = protocol.Request() + to_dict = { + "ver": 2, + "id": "", + "duration": "", + "responseCode": "", + "success": True, + "source": None, + "name": None, + "url": None, + "properties": None, + "measurements": None, + } + self.assertEqual(data.to_dict(), to_dict) diff --git a/azure_monitor/tests/test_utils.py b/azure_monitor/tests/test_utils.py index bbebfaf..cac3bc8 100644 --- a/azure_monitor/tests/test_utils.py +++ b/azure_monitor/tests/test_utils.py @@ -4,15 +4,7 @@ import os import unittest -from opentelemetry.sdk.metrics.export import MetricsExportResult -from opentelemetry.sdk.trace.export import SpanExportResult - from azure_monitor import utils -from azure_monitor.export import ( - ExportResult, - get_metrics_export_result, - get_trace_export_result, -) class TestUtils(unittest.TestCase): @@ -30,33 +22,3 @@ def test_nanoseconds_to_duration(self): self.assertEqual(ns_to_duration(60 * 1000000000), "0.00:01:00.000") self.assertEqual(ns_to_duration(3600 * 1000000000), "0.01:00:00.000") self.assertEqual(ns_to_duration(86400 * 1000000000), "1.00:00:00.000") - - def test_get_trace_export_result(self): - self.assertEqual( - get_trace_export_result(ExportResult.SUCCESS), - SpanExportResult.SUCCESS, - ) - self.assertEqual( - get_trace_export_result(ExportResult.FAILED_NOT_RETRYABLE), - SpanExportResult.FAILED_NOT_RETRYABLE, - ) - self.assertEqual( - get_trace_export_result(ExportResult.FAILED_RETRYABLE), - SpanExportResult.FAILED_RETRYABLE, - ) - self.assertEqual(get_trace_export_result(None), None) - - def test_get_metrics_export_result(self): - self.assertEqual( - get_metrics_export_result(ExportResult.SUCCESS), - MetricsExportResult.SUCCESS, - ) - self.assertEqual( - get_metrics_export_result(ExportResult.FAILED_NOT_RETRYABLE), - MetricsExportResult.FAILED_NOT_RETRYABLE, - ) - self.assertEqual( - get_metrics_export_result(ExportResult.FAILED_RETRYABLE), - MetricsExportResult.FAILED_RETRYABLE, - ) - self.assertEqual(get_metrics_export_result(None), None) diff --git a/azure_monitor/tests/trace/test_trace.py b/azure_monitor/tests/trace/test_trace.py index d93ceb2..36d6ddb 100644 --- a/azure_monitor/tests/trace/test_trace.py +++ b/azure_monitor/tests/trace/test_trace.py @@ -10,13 +10,13 @@ # pylint: disable=import-error from opentelemetry.sdk.trace import Span +from opentelemetry.sdk.trace.export import SpanExportResult from opentelemetry.trace import Link, SpanContext, SpanKind from opentelemetry.trace.status import Status, StatusCanonicalCode from azure_monitor.export import ExportResult from azure_monitor.export.trace import AzureMonitorSpanExporter from azure_monitor.options import ExporterOptions -from azure_monitor.protocol import Envelope TEST_FOLDER = os.path.abspath(".test.exporter.trace") @@ -67,19 +67,7 @@ def test_export_empty(self): exporter.export([]) self.assertEqual(len(os.listdir(exporter.storage.path)), 0) - @mock.patch("azure_monitor.export.trace.logger") - def test_export_exception(self, mock_logger): - exporter = AzureMonitorSpanExporter( - storage_path=os.path.join(TEST_FOLDER, self.id()) - ) - exporter.export([None]) - self.assertEqual(mock_logger.exception.called, True) - - @mock.patch( - "azure_monitor.export.trace.AzureMonitorSpanExporter.span_to_envelope" - ) # noqa: E501 - def test_export_failure(self, span_to_envelope_mock): - span_to_envelope_mock.return_value = ["bar"] + def test_export_failure(self): exporter = AzureMonitorSpanExporter( storage_path=os.path.join(TEST_FOLDER, self.id()) ) @@ -100,194 +88,74 @@ def test_export_failure(self, span_to_envelope_mock): self.assertEqual(len(os.listdir(exporter.storage.path)), 1) self.assertIsNone(exporter.storage.get()) - @mock.patch( - "azure_monitor.export.trace.AzureMonitorSpanExporter.span_to_envelope" - ) # noqa: E501 - def test_export_success(self, span_to_envelope_mock): - span_to_envelope_mock.return_value = ["bar"] + def test_export_success(self): exporter = AzureMonitorSpanExporter( storage_path=os.path.join(TEST_FOLDER, self.id()) ) + test_span = Span( + name="test", + context=SpanContext( + trace_id=36873507687745823477771305566750195431, + span_id=12030755672171557338, + ), + ) + test_span.start() + test_span.end() with mock.patch( "azure_monitor.export.trace.AzureMonitorSpanExporter._transmit" ) as transmit: # noqa: E501 - transmit.return_value = 0 - exporter.export([]) - exporter.export(["foo"]) - self.assertEqual(len(os.listdir(exporter.storage.path)), 0) - exporter.export(["foo"]) + transmit.return_value = ExportResult.SUCCESS + storage_mock = mock.Mock() + exporter._transmit_from_storage = storage_mock + exporter.export([test_span]) + self.assertEqual(storage_mock.call_count, 1) self.assertEqual(len(os.listdir(exporter.storage.path)), 0) - def test_transmission_nothing(self): - exporter = AzureMonitorSpanExporter( - storage_path=os.path.join(TEST_FOLDER, self.id()) - ) - with mock.patch("requests.post") as post: - post.return_value = None - exporter._transmit_from_storage() - - def test_transmit_request_exception(self): - exporter = AzureMonitorSpanExporter( - storage_path=os.path.join(TEST_FOLDER, self.id()) - ) - envelopes_to_export = map(lambda x: x.to_dict(), tuple([Envelope()])) - exporter.storage.put(envelopes_to_export) - with mock.patch("requests.post", throw(Exception)): - exporter._transmit_from_storage() - self.assertIsNone(exporter.storage.get()) - self.assertEqual(len(os.listdir(exporter.storage.path)), 1) - - @mock.patch("requests.post", return_value=mock.Mock()) - def test_transmission_lease_failure(self, requests_mock): - requests_mock.return_value = MockResponse(200, "unknown") - exporter = AzureMonitorSpanExporter( - storage_path=os.path.join(TEST_FOLDER, self.id()) - ) - envelopes_to_export = map(lambda x: x.to_dict(), tuple([Envelope()])) - exporter.storage.put(envelopes_to_export) - with mock.patch( - "azure_monitor.storage.LocalFileBlob.lease" - ) as lease: # noqa: E501 - lease.return_value = False - exporter._transmit_from_storage() - self.assertTrue(exporter.storage.get()) - - def test_transmit_response_exception(self): - exporter = AzureMonitorSpanExporter( - storage_path=os.path.join(TEST_FOLDER, self.id()) - ) - envelopes_to_export = map(lambda x: x.to_dict(), tuple([Envelope()])) - exporter.storage.put(envelopes_to_export) - with mock.patch("requests.post") as post: - post.return_value = MockResponse(200, None) - del post.return_value.text - exporter._transmit_from_storage() - self.assertIsNone(exporter.storage.get()) - self.assertEqual(len(os.listdir(exporter.storage.path)), 0) - - def test_transmission_200(self): - exporter = AzureMonitorSpanExporter( - storage_path=os.path.join(TEST_FOLDER, self.id()) - ) - envelopes_to_export = map(lambda x: x.to_dict(), tuple([Envelope()])) - exporter.storage.put(envelopes_to_export) - with mock.patch("requests.post") as post: - post.return_value = MockResponse(200, "unknown") - exporter._transmit_from_storage() - self.assertIsNone(exporter.storage.get()) - self.assertEqual(len(os.listdir(exporter.storage.path)), 0) - - def test_transmission_206(self): - exporter = AzureMonitorSpanExporter( - storage_path=os.path.join(TEST_FOLDER, self.id()) - ) - envelopes_to_export = map(lambda x: x.to_dict(), tuple([Envelope()])) - exporter.storage.put(envelopes_to_export) - with mock.patch("requests.post") as post: - post.return_value = MockResponse(206, "unknown") - exporter._transmit_from_storage() - self.assertIsNone(exporter.storage.get()) - self.assertEqual(len(os.listdir(exporter.storage.path)), 1) - - def test_transmission_206_500(self): - exporter = AzureMonitorSpanExporter( - storage_path=os.path.join(TEST_FOLDER, self.id()) - ) - test_envelope = Envelope(name="testEnvelope") - envelopes_to_export = map( - lambda x: x.to_dict(), - tuple([Envelope(), Envelope(), test_envelope]), - ) - exporter.storage.put(envelopes_to_export) - with mock.patch("requests.post") as post: - post.return_value = MockResponse( - 206, - json.dumps( - { - "itemsReceived": 5, - "itemsAccepted": 3, - "errors": [ - {"index": 0, "statusCode": 400, "message": ""}, - { - "index": 2, - "statusCode": 500, - "message": "Internal Server Error", - }, - ], - } + @mock.patch("azure_monitor.export.trace.logger") + def test_export_exception(self, logger_mock): + test_span = Span( + name="test", + context=SpanContext( + trace_id=36873507687745823477771305566750195431, + span_id=12030755672171557338, ), ) - exporter._transmit_from_storage() - self.assertEqual(len(os.listdir(exporter.storage.path)), 1) - self.assertEqual( - exporter.storage.get().get()[0]["name"], "testEnvelope" - ) - - def test_transmission_206_no_retry(self): + test_span.start() + test_span.end() exporter = AzureMonitorSpanExporter( storage_path=os.path.join(TEST_FOLDER, self.id()) ) - envelopes_to_export = map(lambda x: x.to_dict(), tuple([Envelope()])) - exporter.storage.put(envelopes_to_export) - with mock.patch("requests.post") as post: - post.return_value = MockResponse( - 206, - json.dumps( - { - "itemsReceived": 3, - "itemsAccepted": 2, - "errors": [ - {"index": 0, "statusCode": 400, "message": ""} - ], - } - ), - ) - exporter._transmit_from_storage() - self.assertEqual(len(os.listdir(exporter.storage.path)), 0) - - def test_transmission_206_bogus(self): + with mock.patch( + "azure_monitor.export.trace.AzureMonitorSpanExporter._transmit", + throw(Exception) + ): # noqa: E501 + result = exporter.export([test_span]) + self.assertEqual(result, SpanExportResult.FAILED_NOT_RETRYABLE) + self.assertEqual(logger_mock.exception.called, True) + + def test_export_not_retryable(self): exporter = AzureMonitorSpanExporter( storage_path=os.path.join(TEST_FOLDER, self.id()) ) - envelopes_to_export = map(lambda x: x.to_dict(), tuple([Envelope()])) - exporter.storage.put(envelopes_to_export) - with mock.patch("requests.post") as post: - post.return_value = MockResponse( - 206, - json.dumps( - { - "itemsReceived": 5, - "itemsAccepted": 3, - "errors": [{"foo": 0, "bar": 1}], - } - ), - ) - exporter._transmit_from_storage() - self.assertIsNone(exporter.storage.get()) - self.assertEqual(len(os.listdir(exporter.storage.path)), 0) - - def test_transmission_400(self): - exporter = AzureMonitorSpanExporter( - storage_path=os.path.join(TEST_FOLDER, self.id()) + test_span = Span( + name="test", + context=SpanContext( + trace_id=36873507687745823477771305566750195431, + span_id=12030755672171557338, + ), ) - envelopes_to_export = map(lambda x: x.to_dict(), tuple([Envelope()])) - exporter.storage.put(envelopes_to_export) - with mock.patch("requests.post") as post: - post.return_value = MockResponse(400, "{}") - exporter._transmit_from_storage() - self.assertEqual(len(os.listdir(exporter.storage.path)), 0) + test_span.start() + test_span.end() + with mock.patch( + "azure_monitor.export.trace.AzureMonitorSpanExporter._transmit" + ) as transmit: # noqa: E501 + transmit.return_value = ExportResult.FAILED_NOT_RETRYABLE + result = exporter.export([test_span]) + self.assertEqual(result, SpanExportResult.FAILED_NOT_RETRYABLE) - def test_transmission_500(self): - exporter = AzureMonitorSpanExporter( - storage_path=os.path.join(TEST_FOLDER, self.id()) - ) - envelopes_to_export = map(lambda x: x.to_dict(), tuple([Envelope()])) - exporter.storage.put(envelopes_to_export) - with mock.patch("requests.post") as post: - post.return_value = MockResponse(500, "{}") - exporter._transmit_from_storage() - self.assertIsNone(exporter.storage.get()) - self.assertEqual(len(os.listdir(exporter.storage.path)), 1) + def test_span_to_envelope_none(self): + exporter = AzureMonitorSpanExporter() + self.assertIsNone(exporter.span_to_envelope(None)) # pylint: disable=too-many-statements def test_span_to_envelope(self): @@ -1022,9 +890,3 @@ def test_span_to_envelope(self): self.assertIsNone( envelope.data.base_data.properties.get("request.url") ) - - -class MockResponse: - def __init__(self, status_code, text): - self.status_code = status_code - self.text = text From 775e452f8062b4803e0f3c3a135c5c316e65f68b Mon Sep 17 00:00:00 2001 From: Leighton Date: Fri, 13 Mar 2020 12:59:33 -0700 Subject: [PATCH 081/109] Fix lint --- .../auto_collection/test_performance_metrics.py | 2 +- azure_monitor/tests/metrics/test_metrics.py | 2 +- azure_monitor/tests/test_base_exporter.py | 7 ++++--- azure_monitor/tests/trace/test_trace.py | 14 +++++++------- 4 files changed, 13 insertions(+), 12 deletions(-) diff --git a/azure_monitor/tests/auto_collection/test_performance_metrics.py b/azure_monitor/tests/auto_collection/test_performance_metrics.py index 148a4b3..73f508d 100644 --- a/azure_monitor/tests/auto_collection/test_performance_metrics.py +++ b/azure_monitor/tests/auto_collection/test_performance_metrics.py @@ -182,7 +182,7 @@ def test_track_process_memory(self): def test_track_process_memory_exception(self, logger_mock): with mock.patch( "azure_monitor.auto_collection.performance_metrics.PROCESS", - throw(Exception) + throw(Exception), ): performance_metrics_collector = PerformanceMetrics( meter=self._meter, label_set=self._test_label_set diff --git a/azure_monitor/tests/metrics/test_metrics.py b/azure_monitor/tests/metrics/test_metrics.py index 32f22c4..15b4754 100644 --- a/azure_monitor/tests/metrics/test_metrics.py +++ b/azure_monitor/tests/metrics/test_metrics.py @@ -90,7 +90,7 @@ def test_export_exception(self, logger_mock): exporter = AzureMonitorMetricsExporter() with mock.patch( "azure_monitor.export.metrics.AzureMonitorMetricsExporter._transmit", - throw(Exception) + throw(Exception), ): # noqa: E501 result = exporter.export([record]) self.assertEqual(result, MetricsExportResult.FAILED_NOT_RETRYABLE) diff --git a/azure_monitor/tests/test_base_exporter.py b/azure_monitor/tests/test_base_exporter.py index f306ee4..b1859c6 100644 --- a/azure_monitor/tests/test_base_exporter.py +++ b/azure_monitor/tests/test_base_exporter.py @@ -7,6 +7,9 @@ import unittest from unittest import mock +from opentelemetry.sdk.metrics.export import MetricsExportResult +from opentelemetry.sdk.trace.export import SpanExportResult + from azure_monitor.export import ( BaseExporter, ExportResult, @@ -15,8 +18,6 @@ ) from azure_monitor.options import ExporterOptions from azure_monitor.protocol import Data, Envelope -from opentelemetry.sdk.metrics.export import MetricsExportResult -from opentelemetry.sdk.trace.export import SpanExportResult TEST_FOLDER = os.path.abspath(".test.exporter") @@ -160,7 +161,7 @@ def test_transmit_request_exception(self): self.assertIsNone(exporter.storage.get()) self.assertEqual(len(os.listdir(exporter.storage.path)), 1) - @mock.patch("requests.post", return_value=mock.Mock()) + @mock.patch("requests.post", return_value=mock.Mock()) def test_transmission_lease_failure(self, requests_mock): requests_mock.return_value = MockResponse(200, "unknown") exporter = BaseExporter( diff --git a/azure_monitor/tests/trace/test_trace.py b/azure_monitor/tests/trace/test_trace.py index 36d6ddb..6b85d83 100644 --- a/azure_monitor/tests/trace/test_trace.py +++ b/azure_monitor/tests/trace/test_trace.py @@ -114,12 +114,12 @@ def test_export_success(self): @mock.patch("azure_monitor.export.trace.logger") def test_export_exception(self, logger_mock): test_span = Span( - name="test", - context=SpanContext( - trace_id=36873507687745823477771305566750195431, - span_id=12030755672171557338, - ), - ) + name="test", + context=SpanContext( + trace_id=36873507687745823477771305566750195431, + span_id=12030755672171557338, + ), + ) test_span.start() test_span.end() exporter = AzureMonitorSpanExporter( @@ -127,7 +127,7 @@ def test_export_exception(self, logger_mock): ) with mock.patch( "azure_monitor.export.trace.AzureMonitorSpanExporter._transmit", - throw(Exception) + throw(Exception), ): # noqa: E501 result = exporter.export([test_span]) self.assertEqual(result, SpanExportResult.FAILED_NOT_RETRYABLE) From 83224a76c2d6d1961e94885c40520c9c8dd2cc90 Mon Sep 17 00:00:00 2001 From: Leighton Date: Fri, 13 Mar 2020 13:04:12 -0700 Subject: [PATCH 082/109] pylint --- azure_monitor/tests/test_base_exporter.py | 1 + 1 file changed, 1 insertion(+) diff --git a/azure_monitor/tests/test_base_exporter.py b/azure_monitor/tests/test_base_exporter.py index b1859c6..481e1ba 100644 --- a/azure_monitor/tests/test_base_exporter.py +++ b/azure_monitor/tests/test_base_exporter.py @@ -40,6 +40,7 @@ def func(*_args, **_kwargs): # pylint: disable=W0212 +# pylint: disable=R0904 class TestBaseExporter(unittest.TestCase): @classmethod def setUpClass(cls): From a6ea6e51fbcd57d313336b0b44c50af3ec75b5b2 Mon Sep 17 00:00:00 2001 From: Leighton Date: Thu, 19 Mar 2020 12:12:09 -0700 Subject: [PATCH 083/109] deps --- README.md | 23 +-- .../examples/{ => metrics}/auto_collector.py | 9 +- azure_monitor/examples/metrics/simple.py | 47 +++-- azure_monitor/examples/traces/README.md | 1 - azure_monitor/examples/traces/client.py | 45 ++--- azure_monitor/examples/traces/request.py | 46 ++--- azure_monitor/examples/traces/server.py | 92 ++++----- azure_monitor/examples/traces/trace.py | 38 ++-- azure_monitor/setup.cfg | 4 +- .../azure_monitor/auto_collection/__init__.py | 12 +- .../auto_collection/dependency_metrics.py | 28 ++- .../auto_collection/performance_metrics.py | 100 ++++------ .../auto_collection/request_metrics.py | 50 ++--- .../azure_monitor/export/metrics/__init__.py | 30 ++- .../auto_collection/test_auto_collection.py | 36 +--- .../test_dependency_metrics.py | 76 +++---- .../test_performance_metrics.py | 187 ++++++++++-------- .../auto_collection/test_request_metrics.py | 117 ++++++----- azure_monitor/tests/metrics/test_metrics.py | 11 +- 19 files changed, 468 insertions(+), 484 deletions(-) rename azure_monitor/examples/{ => metrics}/auto_collector.py (74%) diff --git a/README.md b/README.md index eea2dc6..955fede 100644 --- a/README.md +++ b/README.md @@ -28,9 +28,7 @@ from opentelemetry import trace from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import BatchExportSpanProcessor -# The preferred tracer implementation must be set, as the opentelemetry-api -# defines the interface with a no-op implementation. -trace.set_preferred_tracer_provider_implementation(lambda T: TracerProvider()) +trace.set_tracer_provider(TracerProvider()) # We tell OpenTelemetry who it is that is creating spans. In this case, we have # no real name (no setup.py), so we make one up. If we had a version, we would @@ -43,7 +41,7 @@ exporter = AzureMonitorSpanExporter( # SpanExporter receives the spans and send them to the target location. span_processor = BatchExportSpanProcessor(exporter) -trace.tracer_provider().add_span_processor(span_processor) +trace.get_tracer_provider().add_span_processor(span_processor) with tracer.start_as_current_span('hello'): print('Hello World!') @@ -67,15 +65,10 @@ from azure_monitor import AzureMonitorSpanExporter from opentelemetry import trace from opentelemetry.ext import http_requests from opentelemetry.sdk.trace import TracerProvider -from opentelemetry.sdk.trace.export import ( - BatchExportSpanProcessor, - ConsoleSpanExporter, -) +from opentelemetry.sdk.trace.export import BatchExportSpanProcessor -# The preferred tracer implementation must be set, as the opentelemetry-api -# defines the interface with a no-op implementation. -trace.set_preferred_tracer_provider_implementation(lambda T: TracerProvider()) -tracer_provider = trace.tracer_provider() +trace.set_tracer_provider(TracerProvider()) +tracer_provider = trace.get_tracer_provider() exporter = AzureMonitorSpanExporter( connection_string='InstrumentationKey=', @@ -111,10 +104,10 @@ exporter = AzureMonitorSpanExporter( ) exporter.add_telemetry_processor(callback_function) -trace.set_preferred_tracer_provider_implementation(lambda T: TracerProvider()) +trace.set_tracer_provider(TracerProvider()) tracer = trace.get_tracer(__name__) span_processor = BatchExportSpanProcessor(exporter) -trace.tracer_provider().add_span_processor(span_processor) +trace.get_tracer_provider().add_span_processor(span_processor) with tracer.start_as_current_span('hello'): print('Hello World!') @@ -138,7 +131,7 @@ from opentelemetry import metrics from opentelemetry.sdk.metrics import Counter, MeterProvider from opentelemetry.sdk.metrics.export.controller import PushController -metrics.set_preferred_meter_provider_implementation(lambda _: MeterProvider()) +metrics.set_meter_provider(MeterProvider()) meter = metrics.get_meter(__name__) exporter = AzureMonitorMetricsExporter( connection_string='InstrumentationKey=' diff --git a/azure_monitor/examples/auto_collector.py b/azure_monitor/examples/metrics/auto_collector.py similarity index 74% rename from azure_monitor/examples/auto_collector.py rename to azure_monitor/examples/metrics/auto_collector.py index 676924c..906940d 100644 --- a/azure_monitor/examples/auto_collector.py +++ b/azure_monitor/examples/metrics/auto_collector.py @@ -1,13 +1,13 @@ # Copyright (c) Microsoft Corporation. All rights reserved. # Licensed under the MIT License. from opentelemetry import metrics -from opentelemetry.sdk.metrics import Meter +from opentelemetry.sdk.metrics import MeterProvider from opentelemetry.sdk.metrics.export.controller import PushController from azure_monitor import AutoCollection, AzureMonitorMetricsExporter -metrics.set_preferred_meter_implementation(lambda T: Meter()) -meter = metrics.meter() +metrics.set_meter_provider(MeterProvider()) +meter = metrics.get_meter(__name__) exporter = AzureMonitorMetricsExporter( connection_string="InstrumentationKey=" ) @@ -18,8 +18,7 @@ # Automatically collect standard metrics auto_collection = AutoCollection( meter=meter, - label_set=testing_label_set, - collection_interval=120, # Collect every 2 minutes + label_set=testing_label_set ) input("Press any key to exit...") diff --git a/azure_monitor/examples/metrics/simple.py b/azure_monitor/examples/metrics/simple.py index 14d586a..6b3a0cc 100644 --- a/azure_monitor/examples/metrics/simple.py +++ b/azure_monitor/examples/metrics/simple.py @@ -1,29 +1,28 @@ -# # Copyright (c) Microsoft Corporation. All rights reserved. -# # Licensed under the MIT License. -# import time +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. +from azure_monitor import AzureMonitorMetricsExporter +from opentelemetry import metrics +from opentelemetry.sdk.metrics import Counter, MeterProvider +from opentelemetry.sdk.metrics.export.controller import PushController -# from azure_monitor import AzureMonitorMetricsExporter -# from opentelemetry import metrics -# from opentelemetry.sdk.metrics import Counter, MeterProvider -# from opentelemetry.sdk.metrics.export.controller import PushController +metrics.set_meter_provider(MeterProvider()) +meter = metrics.get_meter(__name__) +exporter = AzureMonitorMetricsExporter( + connection_string="InstrumentationKey=" +) +controller = PushController(meter, exporter, 5) -# metrics.set_preferred_meter_provider_implementation(lambda _: MeterProvider()) -# meter = metrics.get_meter(__name__) -# exporter = AzureMonitorMetricsExporter( -# connection_string="InstrumentationKey=" -# ) -# controller = PushController(meter, exporter, 5) +requests_counter = meter.create_metric( + name="requests", + description="number of requests", + unit="1", + value_type=int, + metric_type=Counter, + label_keys=("environment",), +) -# requests_counter = meter.create_metric( -# name="requests", -# description="number of requests", -# unit="1", -# value_type=int, -# metric_type=Counter, -# label_keys=("environment",), -# ) +testing_label_set = meter.get_label_set({"environment": "testing"}) -# testing_label_set = meter.get_label_set({"environment": "testing"}) +requests_counter.add(25, testing_label_set) -# requests_counter.add(25, testing_label_set) -# time.sleep(100) +input("Press any key to exit...") diff --git a/azure_monitor/examples/traces/README.md b/azure_monitor/examples/traces/README.md index df64170..383bf92 100644 --- a/azure_monitor/examples/traces/README.md +++ b/azure_monitor/examples/traces/README.md @@ -50,4 +50,3 @@ $ python server.py After running the applications, data would be available in [Azure]( https://docs.microsoft.com/en-us/azure/azure-monitor/app/app-insights-overview#where-do-i-see-my-telemetry) - diff --git a/azure_monitor/examples/traces/client.py b/azure_monitor/examples/traces/client.py index f51ad10..37908dd 100644 --- a/azure_monitor/examples/traces/client.py +++ b/azure_monitor/examples/traces/client.py @@ -1,25 +1,26 @@ -# # Copyright (c) Microsoft Corporation. All rights reserved. -# # Licensed under the MIT License. -# # pylint: disable=import-error -# # pylint: disable=no-member -# # pylint: disable=no-name-in-module -# import requests -# from opentelemetry import trace -# from opentelemetry.ext import http_requests -# from opentelemetry.sdk.trace import TracerProvider -# from opentelemetry.sdk.trace.export import BatchExportSpanProcessor +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. +# pylint: disable=import-error +# pylint: disable=no-member +# pylint: disable=no-name-in-module +import requests +from opentelemetry import trace +from opentelemetry.ext import http_requests +from opentelemetry.sdk.trace import TracerProvider +from opentelemetry.sdk.trace.export import BatchExportSpanProcessor -# from azure_monitor import AzureMonitorSpanExporter +from azure_monitor import AzureMonitorSpanExporter -# trace.set_preferred_tracer_provider_implementation(lambda T: TracerProvider()) -# tracer = trace.get_tracer(__name__) -# http_requests.enable(trace.tracer_provider()) -# span_processor = BatchExportSpanProcessor( -# AzureMonitorSpanExporter( -# connection_string="InstrumentationKey=" -# ) -# ) -# trace.tracer_provider().add_span_processor(span_processor) +trace.set_tracer_provider(TracerProvider()) +tracer = trace.get_tracer(__name__) +http_requests.enable(trace.get_tracer_provider()) +span_processor = BatchExportSpanProcessor( + AzureMonitorSpanExporter( + # connection_string="InstrumentationKey=" + ) +) +trace.get_tracer_provider().add_span_processor(span_processor) -# response = requests.get(url="http://127.0.0.1:8080/") -# span_processor.shutdown() +response = requests.get(url="http://127.0.0.1:8080/") + +input("Press any key to exit...") diff --git a/azure_monitor/examples/traces/request.py b/azure_monitor/examples/traces/request.py index 20d68f0..b30e3ac 100644 --- a/azure_monitor/examples/traces/request.py +++ b/azure_monitor/examples/traces/request.py @@ -1,26 +1,28 @@ -# # Copyright (c) Microsoft Corporation. All rights reserved. -# # Licensed under the MIT License. -# # pylint: disable=import-error -# # pylint: disable=no-member -# # pylint: disable=no-name-in-module -# import requests -# from opentelemetry import trace -# from opentelemetry.ext import http_requests -# from opentelemetry.sdk.trace import TracerProvider -# from opentelemetry.sdk.trace.export import SimpleExportSpanProcessor +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. +# pylint: disable=import-error +# pylint: disable=no-member +# pylint: disable=no-name-in-module +import requests +from opentelemetry import trace +from opentelemetry.ext import http_requests +from opentelemetry.sdk.trace import TracerProvider +from opentelemetry.sdk.trace.export import SimpleExportSpanProcessor -# from azure_monitor import AzureMonitorSpanExporter +from azure_monitor import AzureMonitorSpanExporter -# trace.set_preferred_tracer_provider_implementation(lambda T: TracerProvider()) +trace.set_tracer_provider(TracerProvider()) -# http_requests.enable(trace.tracer_provider()) -# span_processor = SimpleExportSpanProcessor( -# AzureMonitorSpanExporter( -# connection_string="InstrumentationKey=" -# ) -# ) -# trace.tracer_provider().add_span_processor(span_processor) -# tracer = trace.get_tracer(__name__) +http_requests.enable(trace.get_tracer_provider()) +span_processor = SimpleExportSpanProcessor( + AzureMonitorSpanExporter( + connection_string="InstrumentationKey=" + ) +) +trace.get_tracer_provider().add_span_processor(span_processor) +tracer = trace.get_tracer(__name__) -# with tracer.start_as_current_span("parent"): -# response = requests.get("https://azure.microsoft.com/", timeout=5) +with tracer.start_as_current_span("parent"): + response = requests.get("https://azure.microsoft.com/", timeout=5) + +input("Press any key to exit...") \ No newline at end of file diff --git a/azure_monitor/examples/traces/server.py b/azure_monitor/examples/traces/server.py index 62ae190..cc78f1a 100644 --- a/azure_monitor/examples/traces/server.py +++ b/azure_monitor/examples/traces/server.py @@ -1,46 +1,46 @@ -# # Copyright (c) Microsoft Corporation. All rights reserved. -# # Licensed under the MIT License. -# # pylint: disable=import-error -# # pylint: disable=no-member -# # pylint: disable=no-name-in-module -# import requests -# from opentelemetry import trace -# from opentelemetry.ext import http_requests -# from opentelemetry.ext.wsgi import OpenTelemetryMiddleware -# from opentelemetry.sdk.trace import TracerProvider -# from opentelemetry.sdk.trace.export import BatchExportSpanProcessor - -# import flask -# from azure_monitor import AzureMonitorSpanExporter - -# # The preferred tracer implementation must be set, as the opentelemetry-api -# # defines the interface with a no-op implementation. -# trace.set_preferred_tracer_provider_implementation(lambda T: TracerProvider()) -# tracer = trace.get_tracer(__name__) - -# exporter = AzureMonitorSpanExporter( -# connection_string="InstrumentationKey=" -# ) - -# # SpanExporter receives the spans and send them to the target location. -# span_processor = BatchExportSpanProcessor(exporter) -# trace.tracer_provider().add_span_processor(span_processor) - -# # Integrations are the glue that binds the OpenTelemetry API and the -# # frameworks and libraries that are used together, automatically creating -# # Spans and propagating context as appropriate. -# http_requests.enable(trace.tracer_provider()) -# app = flask.Flask(__name__) -# app.wsgi_app = OpenTelemetryMiddleware(app.wsgi_app) - - -# @app.route("/") -# def hello(): -# with tracer.start_as_current_span("parent"): -# requests.get("https://www.wikipedia.org/wiki/Rabbit") -# return "hello" - - -# if __name__ == "__main__": -# app.run(host="localhost", port=8080, threaded=True) -# span_processor.shutdown() +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. +# pylint: disable=import-error +# pylint: disable=no-member +# pylint: disable=no-name-in-module +import requests +from opentelemetry import trace +from opentelemetry.ext import http_requests +from opentelemetry.ext.wsgi import OpenTelemetryMiddleware +from opentelemetry.sdk.trace import TracerProvider +from opentelemetry.sdk.trace.export import BatchExportSpanProcessor + +import flask +from azure_monitor import AzureMonitorSpanExporter + +# The preferred tracer implementation must be set, as the opentelemetry-api +# defines the interface with a no-op implementation. +trace.set_tracer_provider(TracerProvider()) +tracer = trace.get_tracer(__name__) + +exporter = AzureMonitorSpanExporter( + connection_string="InstrumentationKey=" +) + +# SpanExporter receives the spans and send them to the target location. +span_processor = BatchExportSpanProcessor(exporter) +trace.get_tracer_provider().add_span_processor(span_processor) + +# Integrations are the glue that binds the OpenTelemetry API and the +# frameworks and libraries that are used together, automatically creating +# Spans and propagating context as appropriate. +http_requests.enable(trace.get_tracer_provider()) +app = flask.Flask(__name__) +app.wsgi_app = OpenTelemetryMiddleware(app.wsgi_app) + + +@app.route("/") +def hello(): + with tracer.start_as_current_span("parent"): + requests.get("https://www.wikipedia.org/wiki/Rabbit") + return "hello" + + +if __name__ == "__main__": + app.run(host="localhost", port=8080, threaded=True) + diff --git a/azure_monitor/examples/traces/trace.py b/azure_monitor/examples/traces/trace.py index 889ca7b..a01cd25 100644 --- a/azure_monitor/examples/traces/trace.py +++ b/azure_monitor/examples/traces/trace.py @@ -1,24 +1,26 @@ -# from opentelemetry import trace -# from opentelemetry.sdk.trace import TracerProvider -# from opentelemetry.sdk.trace.export import BatchExportSpanProcessor +from opentelemetry import trace +from opentelemetry.sdk.trace import TracerProvider +from opentelemetry.sdk.trace.export import BatchExportSpanProcessor -# from azure_monitor import AzureMonitorSpanExporter +from azure_monitor import AzureMonitorSpanExporter -# # Callback function to add os_type: linux to span properties -# def callback_function(envelope): -# envelope.data.baseData.properties["os_type"] = "linux" -# return True +# Callback function to add os_type: linux to span properties +def callback_function(envelope): + envelope.data.base_data.properties["os_type"] = "linux" + return True -# exporter = AzureMonitorSpanExporter( -# connection_string="InstrumentationKey=" -# ) -# exporter.add_telemetry_processor(callback_function) +exporter = AzureMonitorSpanExporter( + connection_string="InstrumentationKey=" +) +exporter.add_telemetry_processor(callback_function) -# trace.set_preferred_tracer_provider_implementation(lambda T: TracerProvider()) -# tracer = trace.get_tracer(__name__) -# span_processor = BatchExportSpanProcessor(exporter) -# trace.tracer_provider().add_span_processor(span_processor) +trace.set_tracer_provider(TracerProvider()) +tracer = trace.get_tracer(__name__) +span_processor = BatchExportSpanProcessor(exporter) +trace.get_tracer_provider().add_span_processor(span_processor) -# with tracer.start_as_current_span("hello"): -# print("Hello, World!") +with tracer.start_as_current_span("hello"): + print("Hello, World!") + +input("Press any key to exit...") \ No newline at end of file diff --git a/azure_monitor/setup.cfg b/azure_monitor/setup.cfg index 0660283..bbb0ddb 100644 --- a/azure_monitor/setup.cfg +++ b/azure_monitor/setup.cfg @@ -27,8 +27,8 @@ package_dir= =src packages=find_namespace: install_requires = - opentelemetry-api >= 0.4a0 - opentelemetry-sdk >= 0.4a0 + opentelemetry-api >= 0.5b0 + opentelemetry-sdk >= 0.5b0 psutil >= 5.6.3 requests ~= 2.0 diff --git a/azure_monitor/src/azure_monitor/auto_collection/__init__.py b/azure_monitor/src/azure_monitor/auto_collection/__init__.py index 20cb0c3..72b72e0 100644 --- a/azure_monitor/src/azure_monitor/auto_collection/__init__.py +++ b/azure_monitor/src/azure_monitor/auto_collection/__init__.py @@ -8,7 +8,6 @@ PerformanceMetrics, ) from azure_monitor.auto_collection.request_metrics import RequestMetrics -from azure_monitor.utils import PeriodicTask __all__ = [ "AutoCollection", @@ -20,17 +19,8 @@ class AutoCollection: def __init__( - self, meter: Meter, label_set: LabelSet, collection_interval: int = 60 + self, meter: Meter, label_set: LabelSet ): self._performance_metrics = PerformanceMetrics(meter, label_set) self._dependency_metrics = DependencyMetrics(meter, label_set) self._request_metrics = RequestMetrics(meter, label_set) - - self._collect_task = PeriodicTask( - interval=collection_interval, function=self.collect - ) - - def collect(self): - self._performance_metrics.track() - self._dependency_metrics.track() - self._request_metrics.track() diff --git a/azure_monitor/src/azure_monitor/auto_collection/dependency_metrics.py b/azure_monitor/src/azure_monitor/auto_collection/dependency_metrics.py index 1f0b49e..f223751 100644 --- a/azure_monitor/src/azure_monitor/auto_collection/dependency_metrics.py +++ b/azure_monitor/src/azure_monitor/auto_collection/dependency_metrics.py @@ -4,8 +4,8 @@ import time import requests -from opentelemetry.metrics import LabelSet, Meter -from opentelemetry.sdk.metrics import Gauge +from opentelemetry.metrics import Meter +from opentelemetry.sdk.metrics import LabelSet dependency_map = dict() _dependency_lock = threading.Lock() @@ -27,21 +27,15 @@ def __init__(self, meter: Meter, label_set: LabelSet): self._label_set = label_set # Patch requests requests.Session.request = dependency_patch - dependency_rate_metric = self._meter.create_metric( - "\\ApplicationInsights\\Dependency Calls/Sec", - "Outgoing Requests per second", - "rps", - int, - Gauge, + meter.register_observer( + callback=self._track_dependency_rate, + name="\\ApplicationInsights\\Dependency Calls/Sec", + description="Outgoing Requests per second", + unit="rps", + value_type=int, ) - self._dependency_rate_handle = dependency_rate_metric.get_handle( - self._label_set - ) - - def track(self) -> None: - self._track_dependency_rate() - def _track_dependency_rate(self) -> None: + def _track_dependency_rate(self, observer) -> None: """ Track Dependency rate Calculated by obtaining the number of outgoing requests made @@ -65,8 +59,8 @@ def _track_dependency_rate(self) -> None: dependency_map["last_time"] = current_time dependency_map["last_count"] = current_count dependency_map["last_result"] = result - self._dependency_rate_handle.set(int(result)) + observer.observe(int(result), self._label_set) except ZeroDivisionError: # If elapsed_seconds is 0, exporter call made too close to previous # Return the previous result if this is the case - self._dependency_rate_handle.set(int(last_result)) + observer.observe(int(last_result), self._label_set) diff --git a/azure_monitor/src/azure_monitor/auto_collection/performance_metrics.py b/azure_monitor/src/azure_monitor/auto_collection/performance_metrics.py index 877252b..6a2e279 100644 --- a/azure_monitor/src/azure_monitor/auto_collection/performance_metrics.py +++ b/azure_monitor/src/azure_monitor/auto_collection/performance_metrics.py @@ -3,8 +3,8 @@ import logging import psutil -from opentelemetry.metrics import LabelSet, Meter -from opentelemetry.sdk.metrics import Gauge +from opentelemetry.metrics import Meter +from opentelemetry.sdk.metrics import LabelSet logger = logging.getLogger(__name__) PROCESS = psutil.Process() @@ -15,53 +15,36 @@ def __init__(self, meter: Meter, label_set: LabelSet): self._meter = meter self._label_set = label_set # Create performance metrics - cpu_metric = self._meter.create_metric( - "\\Processor(_Total)\\% Processor Time", - "Processor time as a percentage", - "percentage", - float, - Gauge, + meter.register_observer( + callback=self._track_cpu, + name="\\Processor(_Total)\\% Processor Time", + description="Processor time as a percentage", + unit="percentage", + value_type=float, ) - self._cpu_handle = cpu_metric.get_handle(self._label_set) - - memory_metric = self._meter.create_metric( - "\\Memory\\Available Bytes", - "Amount of available memory in bytes", - "byte", - int, - Gauge, - ) - self._memory_handle = memory_metric.get_handle(self._label_set) - - process_cpu_metric = self._meter.create_metric( - "\\Process(??APP_WIN32_PROC??)\\% Processor Time", - "Process CPU usage as a percentage", - "percentage", - float, - Gauge, + meter.register_observer( + callback=self._track_memory, + name="\\Memory\\Available Bytes", + description="Amount of available memory in bytes", + unit="byte", + value_type=int, ) - self._process_cpu_handle = process_cpu_metric.get_handle( - self._label_set - ) - - process_memory_metric = self._meter.create_metric( - "\\Process(??APP_WIN32_PROC??)\\Private Bytes", - "Amount of memory process has used in bytes", - "byte", - int, - Gauge, + meter.register_observer( + callback=self._track_process_cpu, + name="\\Process(??APP_WIN32_PROC??)\\% Processor Time", + description="Process CPU usage as a percentage", + unit="percentage", + value_type=float, ) - self._process_memory_handle = process_memory_metric.get_handle( - self._label_set + meter.register_observer( + callback=self._track_process_memory, + name="\\Process(??APP_WIN32_PROC??)\\Private Bytes", + description="Amount of memory process has used in bytes", + unit="byte", + value_type=int, ) - def track(self) -> None: - self._track_cpu() - self._track_process_cpu() - self._track_memory() - self._track_process_memory() - - def _track_cpu(self) -> None: + def _track_cpu(self, observer) -> None: """ Track CPU time Processor time is defined as a float representing the current system @@ -70,9 +53,17 @@ def _track_cpu(self) -> None: from 0.0 to 100.0 inclusive. """ cpu_times_percent = psutil.cpu_times_percent() - self._cpu_handle.set(100 - cpu_times_percent.idle) + observer.observe(100.0 - cpu_times_percent.idle, self._label_set) + + def _track_memory(self, observer) -> None: + """ Track Memory + + Available memory is defined as memory that can be given instantly to + processes without the system going into swap. + """ + observer.observe(psutil.virtual_memory().available, self._label_set) - def _track_process_cpu(self) -> None: + def _track_process_cpu(self, observer) -> None: """ Track Process CPU time Returns a derived gauge for the CPU usage for the current process. @@ -83,25 +74,20 @@ def _track_process_cpu(self) -> None: # CPU cores, the returned value of cpu_percent() can be > 100.0. We # normalize the cpu process using the number of logical CPUs cpu_count = psutil.cpu_count(logical=True) - self._process_cpu_handle.set(PROCESS.cpu_percent() / cpu_count) + observer.observe( + PROCESS.cpu_percent() / cpu_count, + self._label_set + ) except Exception: # pylint: disable=broad-except logger.exception("Error handling get process cpu usage.") - def _track_memory(self) -> None: - """ Track Memory - - Available memory is defined as memory that can be given instantly to - processes without the system going into swap. - """ - self._memory_handle.set(psutil.virtual_memory().available) - - def _track_process_memory(self) -> None: + def _track_process_memory(self, observer) -> None: """ Track Memory Available memory is defined as memory that can be given instantly to processes without the system going into swap. """ try: - self._process_memory_handle.set(PROCESS.memory_info().rss) + observer.observe(PROCESS.memory_info().rss, self._label_set) except Exception: # pylint: disable=broad-except logger.exception("Error handling get process private bytes.") diff --git a/azure_monitor/src/azure_monitor/auto_collection/request_metrics.py b/azure_monitor/src/azure_monitor/auto_collection/request_metrics.py index eba8074..b901c4f 100644 --- a/azure_monitor/src/azure_monitor/auto_collection/request_metrics.py +++ b/azure_monitor/src/azure_monitor/auto_collection/request_metrics.py @@ -6,7 +6,6 @@ from http.server import HTTPServer from opentelemetry.metrics import LabelSet, Meter -from opentelemetry.sdk.metrics import Gauge logger = logging.getLogger(__name__) @@ -60,33 +59,23 @@ def __init__(self, meter: Meter, label_set: LabelSet): # Patch the HTTPServer handler to track request information HTTPServer.__init__ = server_patch - request_duration_metric = self._meter.create_metric( - "\\ASP.NET Applications(??APP_W3SVC_PROC??)\\Request Execution Time", - "Incoming Requests Average Execution Time", - "milliseconds", - int, - Gauge, - ) - self._request_duration_handle = request_duration_metric.get_handle( - self._label_set + meter.register_observer( + callback=self._track_request_duration, + name="\\ASP.NET Applications(??APP_W3SVC_PROC??)\\Request Execution Time", + description="Incoming Requests Average Execution Time", + unit="milliseconds", + value_type=int, ) - request_rate_metric = self._meter.create_metric( - "\\ASP.NET Applications(??APP_W3SVC_PROC??)\\Requests/Sec", - "Incoming Requests Average Execution Rate", - "rps", - int, - Gauge, - ) - self._request_rate_handle = request_rate_metric.get_handle( - self._label_set + meter.register_observer( + callback=self._track_request_rate, + name="\\ASP.NET Applications(??APP_W3SVC_PROC??)\\Requests/Sec", + description="Incoming Requests Average Execution Rate", + unit="rps", + value_type=int, ) - def track(self) -> None: - self._track_request_duration() - self._track_request_rate() - - def _track_request_duration(self) -> None: + def _track_request_duration(self, observer) -> None: """ Track Request execution time Calculated by getting the time it takes to make an incoming request @@ -104,15 +93,16 @@ def _track_request_duration(self) -> None: requests_map["last_average_duration"] = result requests_map["last_duration"] = requests_map.get("duration", 0) # Convert to milliseconds - self._request_duration_handle.set(int(result * 1000.0)) + observer.observe(int(result * 1000.0), self._label_set) except ZeroDivisionError: # If interval_count is 0, exporter call made too close to previous # Return the previous result if this is the case - self._request_duration_handle.set( - int(last_average_duration * 1000.0) + observer.observe( + int(last_average_duration * 1000.0), + self._label_set ) - def _track_request_rate(self) -> None: + def _track_request_rate(self, observer) -> None: """ Track Request execution rate Calculated by obtaining by getting the number of incoming requests @@ -136,8 +126,8 @@ def _track_request_rate(self) -> None: requests_map["last_time"] = current_time requests_map["last_count"] = requests_map.get("count", 0) requests_map["last_rate"] = result - self._request_rate_handle.set(int(result)) + observer.observe(int(result), self._label_set) except ZeroDivisionError: # If elapsed_seconds is 0, exporter call made too close to previous # Return the previous result if this is the case - self._request_rate_handle.set(int(last_rate)) + observer.observe(int(last_rate), self._label_set) diff --git a/azure_monitor/src/azure_monitor/export/metrics/__init__.py b/azure_monitor/src/azure_monitor/export/metrics/__init__.py index 0f69740..f8a95f0 100644 --- a/azure_monitor/src/azure_monitor/export/metrics/__init__.py +++ b/azure_monitor/src/azure_monitor/export/metrics/__init__.py @@ -5,7 +5,9 @@ from typing import Sequence from urllib.parse import urlparse -from opentelemetry.metrics import Counter, Measure, Metric +from opentelemetry.metrics import Metric +from opentelemetry.util import time_ns +from opentelemetry.sdk.metrics import Counter, Observer from opentelemetry.sdk.metrics.export import ( MetricRecord, MetricsExporter, @@ -52,21 +54,35 @@ def metric_to_envelope( if not metric_record: return None + # TODO: Opentelemetry does not have last updated timestamp for observer + # type metrics yet. + _time = time_ns() + if isinstance(metric_record.metric, Metric): + _time = metric_record.metric.bind( + metric_record.label_set + ).last_update_timestamp envelope = protocol.Envelope( ikey=self.options.instrumentation_key, tags=dict(utils.azure_monitor_context), - time=ns_to_iso_str( - metric_record.metric.get_handle( - metric_record.label_set - ).last_update_timestamp - ), + time=ns_to_iso_str(_time), ) envelope.name = "Microsoft.ApplicationInsights.Metric" + value = 0 + metric = metric_record.metric + if isinstance(metric, Counter): + value = metric_record.aggregator.checkpoint + elif isinstance(metric, Observer): + value = metric_record.aggregator.checkpoint.last + if not value: + value = 0 + else: + # TODO: What do measure aggregations look like in AI? + logger.warning("Measure metric recorded.") data_point = protocol.DataPoint( ns=metric_record.metric.description, name=metric_record.metric.name, - value=metric_record.aggregator.checkpoint, + value=value, kind=protocol.DataPointType.MEASUREMENT.value, ) diff --git a/azure_monitor/tests/auto_collection/test_auto_collection.py b/azure_monitor/tests/auto_collection/test_auto_collection.py index 2a73211..9d370f6 100644 --- a/azure_monitor/tests/auto_collection/test_auto_collection.py +++ b/azure_monitor/tests/auto_collection/test_auto_collection.py @@ -5,7 +5,7 @@ from unittest import mock from opentelemetry import metrics -from opentelemetry.sdk.metrics import Meter +from opentelemetry.sdk.metrics import MeterProvider from azure_monitor.auto_collection import AutoCollection from azure_monitor.utils import PeriodicTask @@ -15,15 +15,14 @@ class TestAutoCollection(unittest.TestCase): @classmethod def setUpClass(cls): - cls._meter_defaults = (metrics._METER, metrics._METER_FACTORY) - metrics.set_preferred_meter_implementation(lambda _: Meter()) - cls._meter = metrics.meter() + metrics.set_meter_provider(MeterProvider()) + cls._meter = metrics.get_meter(__name__) kvp = {"environment": "staging"} cls._test_label_set = cls._meter.get_label_set(kvp) @classmethod def tearDownClass(cls): - metrics._METER, metrics._METER_FACTORY = cls._meter_defaults + metrics._METER_PROVIDER = None @mock.patch( "azure_monitor.auto_collection.PerformanceMetrics", autospec=True @@ -36,11 +35,9 @@ def test_constructor( self, mock_performance, mock_dependencies, mock_requests ): """Test the constructor.""" - - auto_collector = AutoCollection( + AutoCollection( meter=self._meter, label_set=self._test_label_set, - collection_interval=1000, ) self.assertEqual(mock_performance.called, True) self.assertEqual(mock_dependencies.called, True) @@ -55,26 +52,3 @@ def test_constructor( ) self.assertEqual(mock_requests.call_args[0][0], self._meter) self.assertEqual(mock_requests.call_args[0][1], self._test_label_set) - - self.assertIsInstance(auto_collector._collect_task, PeriodicTask) - self.assertEqual(auto_collector._collect_task.interval, 1000) - - @mock.patch( - "azure_monitor.auto_collection.PerformanceMetrics.track", autospec=True - ) - @mock.patch( - "azure_monitor.auto_collection.DependencyMetrics.track", autospec=True - ) - @mock.patch( - "azure_monitor.auto_collection.RequestMetrics.track", autospec=True - ) - def test_collect(self, mock_performance, mock_dependencies, mock_requests): - """Test collect method.""" - - auto_collector = AutoCollection( - meter=self._meter, label_set=self._test_label_set - ) - auto_collector.collect() - self.assertEqual(mock_performance.called, True) - self.assertEqual(mock_dependencies.called, True) - self.assertEqual(mock_requests.called, True) diff --git a/azure_monitor/tests/auto_collection/test_dependency_metrics.py b/azure_monitor/tests/auto_collection/test_dependency_metrics.py index 86ecb6c..43757a9 100644 --- a/azure_monitor/tests/auto_collection/test_dependency_metrics.py +++ b/azure_monitor/tests/auto_collection/test_dependency_metrics.py @@ -7,7 +7,7 @@ import requests from opentelemetry import metrics -from opentelemetry.sdk.metrics import Gauge, Meter +from opentelemetry.sdk.metrics import MeterProvider, Observer from azure_monitor.auto_collection import DependencyMetrics, dependency_metrics @@ -19,15 +19,14 @@ class TestDependencyMetrics(unittest.TestCase): @classmethod def setUpClass(cls): - cls._meter_defaults = (metrics._METER, metrics._METER_FACTORY) - metrics.set_preferred_meter_implementation(lambda _: Meter()) - cls._meter = metrics.meter() + metrics.set_meter_provider(MeterProvider()) + cls._meter = metrics.get_meter(__name__) kvp = {"environment": "staging"} cls._test_label_set = cls._meter.get_label_set(kvp) @classmethod def tearDown(cls): - metrics._METER, metrics._METER_FACTORY = cls._meter_defaults + metrics._METER_PROVIDER = None def setUp(self): dependency_metrics.dependency_map.clear() @@ -41,31 +40,14 @@ def test_constructor(self): ) self.assertEqual(metrics_collector._meter, mock_meter) self.assertEqual(metrics_collector._label_set, self._test_label_set) - - self.assertEqual(mock_meter.create_metric.call_count, 1) - - create_metric_calls = mock_meter.create_metric.call_args_list - - self.assertEqual( - create_metric_calls[0][0], - ( - "\\ApplicationInsights\\Dependency Calls/Sec", - "Outgoing Requests per second", - "rps", - int, - Gauge, - ), - ) - - def test_track(self): - mock_meter = mock.Mock() - metrics_collector = DependencyMetrics( - meter=mock_meter, label_set=self._test_label_set + self.assertEqual(mock_meter.register_observer.call_count, 1) + mock_meter.register_observer.assert_called_with( + callback=metrics_collector._track_dependency_rate, + name="\\ApplicationInsights\\Dependency Calls/Sec", + description="Outgoing Requests per second", + unit="rps", + value_type=int, ) - track_mock = mock.Mock() - metrics_collector._track_dependency_rate = track_mock - metrics_collector.track() - self.assertEqual(track_mock.call_count, 1) @mock.patch("azure_monitor.auto_collection.dependency_metrics.time") def test_track_dependency_rate(self, time_mock): @@ -73,11 +55,19 @@ def test_track_dependency_rate(self, time_mock): metrics_collector = DependencyMetrics( meter=self._meter, label_set=self._test_label_set ) + obs = Observer( + callback=metrics_collector._track_dependency_rate, + name="\\ApplicationInsights\\Dependency Calls/Sec", + description="Outgoing Requests per second", + unit="rps", + value_type=int, + meter=self._meter + ) dependency_metrics.dependency_map["last_time"] = 98 dependency_metrics.dependency_map["count"] = 4 - metrics_collector._track_dependency_rate() + metrics_collector._track_dependency_rate(obs) self.assertEqual( - metrics_collector._dependency_rate_handle.aggregator.current, 2 + obs.aggregators[self._test_label_set].current, 2 ) @mock.patch("azure_monitor.auto_collection.dependency_metrics.time") @@ -87,9 +77,17 @@ def test_track_dependency_rate_time_none(self, time_mock): meter=self._meter, label_set=self._test_label_set ) dependency_metrics.dependency_map["last_time"] = None - metrics_collector._track_dependency_rate() + obs = Observer( + callback=metrics_collector._track_dependency_rate, + name="\\ApplicationInsights\\Dependency Calls/Sec", + description="Outgoing Requests per second", + unit="rps", + value_type=int, + meter=self._meter + ) + metrics_collector._track_dependency_rate(obs) self.assertEqual( - metrics_collector._dependency_rate_handle.aggregator.current, 0 + obs.aggregators[self._test_label_set].current, 0 ) @mock.patch("azure_monitor.auto_collection.dependency_metrics.time") @@ -100,9 +98,17 @@ def test_track_dependency_rate_error(self, time_mock): ) dependency_metrics.dependency_map["last_time"] = 100 dependency_metrics.dependency_map["last_result"] = 5 - metrics_collector._track_dependency_rate() + obs = Observer( + callback=metrics_collector._track_dependency_rate, + name="\\ApplicationInsights\\Dependency Calls/Sec", + description="Outgoing Requests per second", + unit="rps", + value_type=int, + meter=self._meter + ) + metrics_collector._track_dependency_rate(obs) self.assertEqual( - metrics_collector._dependency_rate_handle.aggregator.current, 5 + obs.aggregators[self._test_label_set].current, 5 ) def test_dependency_patch(self): diff --git a/azure_monitor/tests/auto_collection/test_performance_metrics.py b/azure_monitor/tests/auto_collection/test_performance_metrics.py index 73f508d..cce3cdc 100644 --- a/azure_monitor/tests/auto_collection/test_performance_metrics.py +++ b/azure_monitor/tests/auto_collection/test_performance_metrics.py @@ -6,7 +6,7 @@ from unittest import mock from opentelemetry import metrics -from opentelemetry.sdk.metrics import Gauge, Meter +from opentelemetry.sdk.metrics import MeterProvider, Observer from azure_monitor.auto_collection import PerformanceMetrics @@ -22,15 +22,14 @@ def func(*_args, **_kwargs): class TestPerformanceMetrics(unittest.TestCase): @classmethod def setUpClass(cls): - cls._meter_defaults = (metrics._METER, metrics._METER_FACTORY) - metrics.set_preferred_meter_implementation(lambda _: Meter()) - cls._meter = metrics.meter() + metrics.set_meter_provider(MeterProvider()) + cls._meter = metrics.get_meter(__name__) kvp = {"environment": "staging"} cls._test_label_set = cls._meter.get_label_set(kvp) @classmethod def tearDownClass(cls): - metrics._METER, metrics._METER_FACTORY = cls._meter_defaults + metrics._METER_PROVIDER = None def test_constructor(self): mock_meter = mock.Mock() @@ -41,71 +40,37 @@ def test_constructor(self): self.assertEqual( performance_metrics_collector._label_set, self._test_label_set ) - - self.assertEqual(mock_meter.create_metric.call_count, 4) - - create_metric_calls = mock_meter.create_metric.call_args_list - - self.assertEqual( - create_metric_calls[0][0], - ( - "\\Processor(_Total)\\% Processor Time", - "Processor time as a percentage", - "percentage", - float, - Gauge, - ), + self.assertEqual(mock_meter.register_observer.call_count, 4) + reg_obs_calls = mock_meter.register_observer.call_args_list + reg_obs_calls[0].assert_called_with( + callback=performance_metrics_collector._track_cpu, + name="\\Processor(_Total)\\% Processor Time", + description="Processor time as a percentage", + unit="percentage", + value_type=float, ) - self.assertEqual( - create_metric_calls[1][0], - ( - "\\Memory\\Available Bytes", - "Amount of available memory in bytes", - "byte", - int, - Gauge, - ), + reg_obs_calls[1].assert_called_with( + callback=performance_metrics_collector._track_memory, + name="\\Memory\\Available Bytes", + description="Amount of available memory in bytes", + unit="byte", + value_type=int, ) - self.assertEqual( - create_metric_calls[2][0], - ( - "\\Process(??APP_WIN32_PROC??)\\% Processor Time", - "Process CPU usage as a percentage", - "percentage", - float, - Gauge, - ), + reg_obs_calls[2].assert_called_with( + callback=performance_metrics_collector._track_process_cpu, + name="\\Process(??APP_WIN32_PROC??)\\% Processor Time", + description="Process CPU usage as a percentage", + unit="percentage", + value_type=float, ) - self.assertEqual( - create_metric_calls[3][0], - ( - "\\Process(??APP_WIN32_PROC??)\\Private Bytes", - "Amount of memory process has used in bytes", - "byte", - int, - Gauge, - ), + reg_obs_calls[3].assert_called_with( + callback=performance_metrics_collector._track_process_memory, + name="\\Process(??APP_WIN32_PROC??)\\Private Bytes", + description="Amount of memory process has used in bytes", + unit="byte", + value_type=int, ) - def test_track(self): - mock_meter = mock.Mock() - performance_metrics_collector = PerformanceMetrics( - meter=mock_meter, label_set=self._test_label_set - ) - cpu_mock = mock.Mock() - process_mock = mock.Mock() - memory_mock = mock.Mock() - proc_memory_mock = mock.Mock() - performance_metrics_collector._track_cpu = cpu_mock - performance_metrics_collector._track_process_cpu = process_mock - performance_metrics_collector._track_memory = memory_mock - performance_metrics_collector._track_process_memory = proc_memory_mock - performance_metrics_collector.track() - self.assertEqual(cpu_mock.call_count, 1) - self.assertEqual(process_mock.call_count, 1) - self.assertEqual(memory_mock.call_count, 1) - self.assertEqual(proc_memory_mock.call_count, 1) - def test_track_cpu(self): performance_metrics_collector = PerformanceMetrics( meter=self._meter, label_set=self._test_label_set @@ -114,12 +79,42 @@ def test_track_cpu(self): cpu = collections.namedtuple("cpu", "idle") cpu_times = cpu(idle=94.5) processor_mock.return_value = cpu_times - performance_metrics_collector._track_cpu() + obs = Observer( + callback=performance_metrics_collector._track_cpu, + name="\\Processor(_Total)\\% Processor Time", + description="Processor time as a percentage", + unit="percentage", + value_type=float, + meter=self._meter + ) + performance_metrics_collector._track_cpu(obs) self.assertEqual( - performance_metrics_collector._cpu_handle.aggregator.current, + obs.aggregators[self._test_label_set].current, 5.5, ) + @mock.patch("psutil.virtual_memory") + def test_track_memory(self, psutil_mock): + performance_metrics_collector = PerformanceMetrics( + meter=self._meter, label_set=self._test_label_set + ) + memory = collections.namedtuple("memory", "available") + vmem = memory(available=100) + psutil_mock.return_value = vmem + obs = Observer( + callback=performance_metrics_collector._track_memory, + name="\\Memory\\Available Bytes", + description="Amount of available memory in bytes", + unit="byte", + value_type=int, + meter=self._meter + ) + performance_metrics_collector._track_memory(obs) + self.assertEqual( + obs.aggregators[self._test_label_set].current, + 100, + ) + @mock.patch("azure_monitor.auto_collection.performance_metrics.psutil") def test_track_process_cpu(self, psutil_mock): with mock.patch( @@ -130,9 +125,17 @@ def test_track_process_cpu(self, psutil_mock): ) process_mock.cpu_percent.return_value = 44.4 psutil_mock.cpu_count.return_value = 2 - performance_metrics_collector._track_process_cpu() + obs = Observer( + callback=performance_metrics_collector._track_process_cpu, + name="\\Process(??APP_WIN32_PROC??)\\% Processor Time", + description="Process CPU usage as a percentage", + unit="percentage", + value_type=float, + meter=self._meter + ) + performance_metrics_collector._track_process_cpu(obs) self.assertEqual( - performance_metrics_collector._process_cpu_handle.aggregator.current, + obs.aggregators[self._test_label_set].current, 22.2, ) @@ -145,23 +148,17 @@ def test_track_process_cpu_exception(self, logger_mock): meter=self._meter, label_set=self._test_label_set ) psutil_mock.cpu_count.return_value = None - performance_metrics_collector._track_process_cpu() + obs = Observer( + callback=performance_metrics_collector._track_process_cpu, + name="\\Process(??APP_WIN32_PROC??)\\% Processor Time", + description="Process CPU usage as a percentage", + unit="percentage", + value_type=float, + meter=self._meter + ) + performance_metrics_collector._track_process_cpu(obs) self.assertEqual(logger_mock.exception.called, True) - @mock.patch("psutil.virtual_memory") - def test_track_memory(self, psutil_mock): - performance_metrics_collector = PerformanceMetrics( - meter=self._meter, label_set=self._test_label_set - ) - memory = collections.namedtuple("memory", "available") - vmem = memory(available=100) - psutil_mock.return_value = vmem - performance_metrics_collector._track_memory() - self.assertEqual( - performance_metrics_collector._memory_handle.aggregator.current, - 100, - ) - def test_track_process_memory(self): with mock.patch( "azure_monitor.auto_collection.performance_metrics.PROCESS" @@ -172,9 +169,17 @@ def test_track_process_memory(self): memory = collections.namedtuple("memory", "rss") pmem = memory(rss=100) process_mock.memory_info.return_value = pmem - performance_metrics_collector._track_process_memory() + obs = Observer( + callback=performance_metrics_collector._track_process_memory, + name="\\Process(??APP_WIN32_PROC??)\\Private Bytes", + description="Amount of memory process has used in bytes", + unit="byte", + value_type=int, + meter=self._meter + ) + performance_metrics_collector._track_process_memory(obs) self.assertEqual( - performance_metrics_collector._process_memory_handle.aggregator.current, + obs.aggregators[self._test_label_set].current, 100, ) @@ -187,5 +192,13 @@ def test_track_process_memory_exception(self, logger_mock): performance_metrics_collector = PerformanceMetrics( meter=self._meter, label_set=self._test_label_set ) - performance_metrics_collector._track_process_memory() + obs = Observer( + callback=performance_metrics_collector._track_process_memory, + name="\\Process(??APP_WIN32_PROC??)\\Private Bytes", + description="Amount of memory process has used in bytes", + unit="byte", + value_type=int, + meter=self._meter + ) + performance_metrics_collector._track_process_memory(obs) self.assertEqual(logger_mock.exception.called, True) diff --git a/azure_monitor/tests/auto_collection/test_request_metrics.py b/azure_monitor/tests/auto_collection/test_request_metrics.py index 71c5d21..11d6e68 100644 --- a/azure_monitor/tests/auto_collection/test_request_metrics.py +++ b/azure_monitor/tests/auto_collection/test_request_metrics.py @@ -7,7 +7,7 @@ import requests from opentelemetry import metrics -from opentelemetry.sdk.metrics import Gauge, Meter +from opentelemetry.sdk.metrics import MeterProvider, Observer from azure_monitor.auto_collection import RequestMetrics, request_metrics @@ -19,15 +19,14 @@ class TestRequestMetrics(unittest.TestCase): @classmethod def setUpClass(cls): - cls._meter_defaults = (metrics._METER, metrics._METER_FACTORY) - metrics.set_preferred_meter_implementation(lambda _: Meter()) - cls._meter = metrics.meter() + metrics.set_meter_provider(MeterProvider()) + cls._meter = metrics.get_meter(__name__) kvp = {"environment": "staging"} cls._test_label_set = cls._meter.get_label_set(kvp) @classmethod def tearDown(cls): - metrics._METER, metrics._METER_FACTORY = cls._meter_defaults + metrics._METER_PROVIDER = None def setUp(self): request_metrics.requests_map.clear() @@ -44,44 +43,26 @@ def test_constructor(self): request_metrics_collector._label_set, self._test_label_set ) - self.assertEqual(mock_meter.create_metric.call_count, 2) + self.assertEqual(mock_meter.register_observer.call_count, 2) - create_metric_calls = mock_meter.create_metric.call_args_list + create_metric_calls = mock_meter.register_observer.call_args_list - self.assertEqual( - create_metric_calls[0][0], - ( - "\\ASP.NET Applications(??APP_W3SVC_PROC??)\\Request Execution Time", - "Incoming Requests Average Execution Time", - "milliseconds", - int, - Gauge, - ), + create_metric_calls[0].assert_called_with( + callback=request_metrics_collector._track_request_duration, + name="\\ASP.NET Applications(??APP_W3SVC_PROC??)\\Request Execution Time", + description="Incoming Requests Average Execution Time", + unit="milliseconds", + value_type=int, ) - self.assertEqual( - create_metric_calls[1][0], - ( - "\\ASP.NET Applications(??APP_W3SVC_PROC??)\\Requests/Sec", - "Incoming Requests Average Execution Rate", - "rps", - int, - Gauge, - ), - ) + create_metric_calls[1].assert_called_with( + callback=request_metrics_collector._track_request_rate, + name="\\ASP.NET Applications(??APP_W3SVC_PROC??)\\Requests/Sec", + description="Incoming Requests Average Execution Rate", + unit="rps", + value_type=int, + ) - def test_track(self): - mock_meter = mock.Mock() - request_metrics_collector = RequestMetrics( - meter=mock_meter, label_set=self._test_label_set - ) - duration_mock = mock.Mock() - rate_mock = mock.Mock() - request_metrics_collector._track_request_duration = duration_mock - request_metrics_collector._track_request_rate = rate_mock - request_metrics_collector.track() - self.assertEqual(duration_mock.call_count, 1) - self.assertEqual(rate_mock.call_count, 1) def test_track_request_duration(self): request_metrics_collector = RequestMetrics( @@ -90,9 +71,17 @@ def test_track_request_duration(self): request_metrics.requests_map["duration"] = 0.1 request_metrics.requests_map["count"] = 10 request_metrics.requests_map["last_count"] = 5 - request_metrics_collector._track_request_duration() + obs = Observer( + callback=request_metrics_collector._track_request_duration, + name="\\ASP.NET Applications(??APP_W3SVC_PROC??)\\Request Execution Time", + description="Incoming Requests Average Execution Time", + unit="milliseconds", + value_type=int, + meter=self._meter + ) + request_metrics_collector._track_request_duration(obs) self.assertEqual( - request_metrics_collector._request_duration_handle.aggregator.current, + obs.aggregators[self._test_label_set].current, 20, ) @@ -103,9 +92,17 @@ def test_track_request_duration_error(self): request_metrics.requests_map["duration"] = 0.1 request_metrics.requests_map["count"] = 10 request_metrics.requests_map["last_count"] = 10 - request_metrics_collector._track_request_duration() + obs = Observer( + callback=request_metrics_collector._track_request_duration, + name="\\ASP.NET Applications(??APP_W3SVC_PROC??)\\Request Execution Time", + description="Incoming Requests Average Execution Time", + unit="milliseconds", + value_type=int, + meter=self._meter + ) + request_metrics_collector._track_request_duration(obs) self.assertEqual( - request_metrics_collector._request_duration_handle.aggregator.current, + obs.aggregators[self._test_label_set].current, 0, ) @@ -117,9 +114,17 @@ def test_track_request_rate(self, time_mock): time_mock.time.return_value = 100 request_metrics.requests_map["last_time"] = 98 request_metrics.requests_map["count"] = 4 - request_metrics_collector._track_request_rate() + obs = Observer( + callback=request_metrics_collector._track_request_rate, + name="\\ASP.NET Applications(??APP_W3SVC_PROC??)\\Requests/Sec", + description="Incoming Requests Average Execution Rate", + unit="rps", + value_type=int, + meter=self._meter + ) + request_metrics_collector._track_request_rate(obs) self.assertEqual( - request_metrics_collector._request_rate_handle.aggregator.current, + obs.aggregators[self._test_label_set].current, 2, ) @@ -130,9 +135,17 @@ def test_track_request_rate_time_none(self, time_mock): meter=self._meter, label_set=self._test_label_set ) request_metrics.requests_map["last_time"] = None - request_metrics_collector._track_request_rate() + obs = Observer( + callback=request_metrics_collector._track_request_rate, + name="\\ASP.NET Applications(??APP_W3SVC_PROC??)\\Requests/Sec", + description="Incoming Requests Average Execution Rate", + unit="rps", + value_type=int, + meter=self._meter + ) + request_metrics_collector._track_request_rate(obs) self.assertEqual( - request_metrics_collector._request_rate_handle.aggregator.current, + obs.aggregators[self._test_label_set].current, 0, ) @@ -144,9 +157,17 @@ def test_track_request_rate_error(self, time_mock): time_mock.time.return_value = 100 request_metrics.requests_map["last_rate"] = 5 request_metrics.requests_map["last_time"] = 100 - request_metrics_collector._track_request_rate() + obs = Observer( + callback=request_metrics_collector._track_request_rate, + name="\\ASP.NET Applications(??APP_W3SVC_PROC??)\\Requests/Sec", + description="Incoming Requests Average Execution Rate", + unit="rps", + value_type=int, + meter=self._meter + ) + request_metrics_collector._track_request_rate(obs) self.assertEqual( - request_metrics_collector._request_rate_handle.aggregator.current, + obs.aggregators[self._test_label_set].current, 5, ) diff --git a/azure_monitor/tests/metrics/test_metrics.py b/azure_monitor/tests/metrics/test_metrics.py index 15b4754..5f61c07 100644 --- a/azure_monitor/tests/metrics/test_metrics.py +++ b/azure_monitor/tests/metrics/test_metrics.py @@ -5,7 +5,7 @@ from unittest import mock from opentelemetry import metrics -from opentelemetry.sdk.metrics import Counter, Meter +from opentelemetry.sdk.metrics import Counter, MeterProvider from opentelemetry.sdk.metrics.export import MetricRecord, MetricsExportResult from opentelemetry.sdk.metrics.export.aggregate import CounterAggregator from opentelemetry.sdk.util import ns_to_iso_str @@ -31,9 +31,8 @@ def setUpClass(cls): "APPINSIGHTS_INSTRUMENTATIONKEY" ] = "1234abcd-5678-4efa-8abc-1234567890ab" - cls._meter_defaults = (metrics._METER, metrics._METER_FACTORY) - metrics.set_preferred_meter_implementation(lambda _: Meter()) - cls._meter = metrics.meter() + metrics.set_meter_provider(MeterProvider()) + cls._meter = metrics.get_meter(__name__) cls._test_metric = cls._meter.create_metric( "testname", "testdesc", "unit", int, Counter, ["environment"] ) @@ -42,7 +41,7 @@ def setUpClass(cls): @classmethod def tearDownClass(cls): - metrics._METER, metrics._METER_PROVIDER_FACTORY = cls._meter_defaults + metrics._METER_PROVIDER = None def test_constructor(self): """Test the constructor.""" @@ -115,7 +114,7 @@ def test_metric_to_envelope(self): self.assertEqual( envelope.time, ns_to_iso_str( - record.metric.get_handle( + record.metric.bind( record.label_set ).last_update_timestamp ), From 27eb8ea3d5eab787a82ee0421b59c7a75e36ccb6 Mon Sep 17 00:00:00 2001 From: Leighton Date: Thu, 19 Mar 2020 12:39:31 -0700 Subject: [PATCH 084/109] coverage --- azure_monitor/tests/metrics/test_metrics.py | 105 +++++++++++++++++++- 1 file changed, 103 insertions(+), 2 deletions(-) diff --git a/azure_monitor/tests/metrics/test_metrics.py b/azure_monitor/tests/metrics/test_metrics.py index 5f61c07..1e97539 100644 --- a/azure_monitor/tests/metrics/test_metrics.py +++ b/azure_monitor/tests/metrics/test_metrics.py @@ -5,9 +5,13 @@ from unittest import mock from opentelemetry import metrics -from opentelemetry.sdk.metrics import Counter, MeterProvider +from opentelemetry.sdk.metrics import Counter, Measure, MeterProvider from opentelemetry.sdk.metrics.export import MetricRecord, MetricsExportResult -from opentelemetry.sdk.metrics.export.aggregate import CounterAggregator +from opentelemetry.sdk.metrics.export.aggregate import ( + CounterAggregator, + MinMaxSumCountAggregator, + ObserverAggregator +) from opentelemetry.sdk.util import ns_to_iso_str from azure_monitor.export import ExportResult @@ -36,6 +40,12 @@ def setUpClass(cls): cls._test_metric = cls._meter.create_metric( "testname", "testdesc", "unit", int, Counter, ["environment"] ) + cls._test_measure = cls._meter.create_metric( + "testname", "testdesc", "unit", int, Measure, ["environment"] + ) + cls._test_obs = cls._meter.register_observer( + lambda x: x, "testname", "testdesc", "unit", int, Counter, ["environment"] + ) kvp = {"environment": "staging"} cls._test_label_set = cls._meter.get_label_set(kvp) @@ -142,3 +152,94 @@ def test_metric_to_envelope(self): self.assertIsNotNone(envelope.tags["ai.device.osVersion"]) self.assertIsNotNone(envelope.tags["ai.device.type"]) self.assertIsNotNone(envelope.tags["ai.internal.sdkVersion"]) + + def test_observer_to_envelope(self): + aggregator = ObserverAggregator() + aggregator.update(123) + aggregator.take_checkpoint() + record = MetricRecord( + aggregator, self._test_label_set, self._test_obs + ) + exporter = AzureMonitorMetricsExporter() + envelope = exporter.metric_to_envelope(record) + self.assertIsInstance(envelope, Envelope) + self.assertEqual(envelope.ver, 1) + self.assertEqual(envelope.name, "Microsoft.ApplicationInsights.Metric") + # TODO: implement last updated timestamp for observer + # self.assertEqual( + # envelope.time, + # ns_to_iso_str( + # record.metric.bind( + # record.label_set + # ).last_update_timestamp + # ), + # ) + self.assertEqual(envelope.sample_rate, None) + self.assertEqual(envelope.seq, None) + self.assertEqual(envelope.ikey, "1234abcd-5678-4efa-8abc-1234567890ab") + self.assertEqual(envelope.flags, None) + + self.assertIsInstance(envelope.data, Data) + self.assertIsInstance(envelope.data.base_data, MetricData) + self.assertEqual(envelope.data.base_data.ver, 2) + self.assertEqual(len(envelope.data.base_data.metrics), 1) + self.assertIsInstance(envelope.data.base_data.metrics[0], DataPoint) + self.assertEqual(envelope.data.base_data.metrics[0].ns, "testdesc") + self.assertEqual(envelope.data.base_data.metrics[0].name, "testname") + self.assertEqual(envelope.data.base_data.metrics[0].value, 123) + self.assertEqual( + envelope.data.base_data.properties["environment"], "staging" + ) + self.assertIsNotNone(envelope.tags["ai.cloud.role"]) + self.assertIsNotNone(envelope.tags["ai.cloud.roleInstance"]) + self.assertIsNotNone(envelope.tags["ai.device.id"]) + self.assertIsNotNone(envelope.tags["ai.device.locale"]) + self.assertIsNotNone(envelope.tags["ai.device.osVersion"]) + self.assertIsNotNone(envelope.tags["ai.device.type"]) + self.assertIsNotNone(envelope.tags["ai.internal.sdkVersion"]) + + @mock.patch("azure_monitor.export.metrics.logger") + def test_measure_to_envelope(self, logger_mock): + aggregator = MinMaxSumCountAggregator() + aggregator.update(123) + aggregator.take_checkpoint() + record = MetricRecord( + aggregator, self._test_label_set, self._test_measure + ) + exporter = AzureMonitorMetricsExporter() + envelope = exporter.metric_to_envelope(record) + self.assertIsInstance(envelope, Envelope) + self.assertEqual(envelope.ver, 1) + self.assertEqual(envelope.name, "Microsoft.ApplicationInsights.Metric") + self.assertEqual( + envelope.time, + ns_to_iso_str( + record.metric.bind( + record.label_set + ).last_update_timestamp + ), + ) + self.assertEqual(envelope.sample_rate, None) + self.assertEqual(envelope.seq, None) + self.assertEqual(envelope.ikey, "1234abcd-5678-4efa-8abc-1234567890ab") + self.assertEqual(envelope.flags, None) + + self.assertIsInstance(envelope.data, Data) + self.assertIsInstance(envelope.data.base_data, MetricData) + self.assertEqual(envelope.data.base_data.ver, 2) + self.assertEqual(len(envelope.data.base_data.metrics), 1) + self.assertIsInstance(envelope.data.base_data.metrics[0], DataPoint) + self.assertEqual(envelope.data.base_data.metrics[0].ns, "testdesc") + self.assertEqual(envelope.data.base_data.metrics[0].name, "testname") + self.assertEqual(envelope.data.base_data.metrics[0].value, 0) + self.assertEqual( + envelope.data.base_data.properties["environment"], "staging" + ) + self.assertIsNotNone(envelope.tags["ai.cloud.role"]) + self.assertIsNotNone(envelope.tags["ai.cloud.roleInstance"]) + self.assertIsNotNone(envelope.tags["ai.device.id"]) + self.assertIsNotNone(envelope.tags["ai.device.locale"]) + self.assertIsNotNone(envelope.tags["ai.device.osVersion"]) + self.assertIsNotNone(envelope.tags["ai.device.type"]) + self.assertIsNotNone(envelope.tags["ai.internal.sdkVersion"]) + self.assertEqual(logger_mock.warning.called, True) From bddf070d1e50db4e31455735529135efdc522308 Mon Sep 17 00:00:00 2001 From: Leighton Date: Thu, 19 Mar 2020 12:43:38 -0700 Subject: [PATCH 085/109] lint --- .../examples/metrics/auto_collector.py | 5 +-- azure_monitor/examples/metrics/simple.py | 3 +- azure_monitor/examples/traces/request.py | 2 +- azure_monitor/examples/traces/server.py | 3 +- azure_monitor/examples/traces/trace.py | 3 +- .../azure_monitor/auto_collection/__init__.py | 4 +- .../auto_collection/performance_metrics.py | 5 +-- .../auto_collection/request_metrics.py | 3 +- .../azure_monitor/export/metrics/__init__.py | 6 +-- .../auto_collection/test_auto_collection.py | 5 +-- .../test_dependency_metrics.py | 18 +++------ .../test_performance_metrics.py | 26 +++++-------- .../auto_collection/test_request_metrics.py | 38 ++++++------------- azure_monitor/tests/metrics/test_metrics.py | 22 +++++------ 14 files changed, 53 insertions(+), 90 deletions(-) diff --git a/azure_monitor/examples/metrics/auto_collector.py b/azure_monitor/examples/metrics/auto_collector.py index 906940d..6ec7c00 100644 --- a/azure_monitor/examples/metrics/auto_collector.py +++ b/azure_monitor/examples/metrics/auto_collector.py @@ -16,9 +16,6 @@ testing_label_set = meter.get_label_set({"environment": "testing"}) # Automatically collect standard metrics -auto_collection = AutoCollection( - meter=meter, - label_set=testing_label_set -) +auto_collection = AutoCollection(meter=meter, label_set=testing_label_set) input("Press any key to exit...") diff --git a/azure_monitor/examples/metrics/simple.py b/azure_monitor/examples/metrics/simple.py index 6b3a0cc..3547764 100644 --- a/azure_monitor/examples/metrics/simple.py +++ b/azure_monitor/examples/metrics/simple.py @@ -1,10 +1,11 @@ # Copyright (c) Microsoft Corporation. All rights reserved. # Licensed under the MIT License. -from azure_monitor import AzureMonitorMetricsExporter from opentelemetry import metrics from opentelemetry.sdk.metrics import Counter, MeterProvider from opentelemetry.sdk.metrics.export.controller import PushController +from azure_monitor import AzureMonitorMetricsExporter + metrics.set_meter_provider(MeterProvider()) meter = metrics.get_meter(__name__) exporter = AzureMonitorMetricsExporter( diff --git a/azure_monitor/examples/traces/request.py b/azure_monitor/examples/traces/request.py index b30e3ac..5b26868 100644 --- a/azure_monitor/examples/traces/request.py +++ b/azure_monitor/examples/traces/request.py @@ -25,4 +25,4 @@ with tracer.start_as_current_span("parent"): response = requests.get("https://azure.microsoft.com/", timeout=5) -input("Press any key to exit...") \ No newline at end of file +input("Press any key to exit...") diff --git a/azure_monitor/examples/traces/server.py b/azure_monitor/examples/traces/server.py index cc78f1a..60d3170 100644 --- a/azure_monitor/examples/traces/server.py +++ b/azure_monitor/examples/traces/server.py @@ -3,6 +3,7 @@ # pylint: disable=import-error # pylint: disable=no-member # pylint: disable=no-name-in-module +import flask import requests from opentelemetry import trace from opentelemetry.ext import http_requests @@ -10,7 +11,6 @@ from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import BatchExportSpanProcessor -import flask from azure_monitor import AzureMonitorSpanExporter # The preferred tracer implementation must be set, as the opentelemetry-api @@ -43,4 +43,3 @@ def hello(): if __name__ == "__main__": app.run(host="localhost", port=8080, threaded=True) - diff --git a/azure_monitor/examples/traces/trace.py b/azure_monitor/examples/traces/trace.py index a01cd25..b5bd7a4 100644 --- a/azure_monitor/examples/traces/trace.py +++ b/azure_monitor/examples/traces/trace.py @@ -4,6 +4,7 @@ from azure_monitor import AzureMonitorSpanExporter + # Callback function to add os_type: linux to span properties def callback_function(envelope): envelope.data.base_data.properties["os_type"] = "linux" @@ -23,4 +24,4 @@ def callback_function(envelope): with tracer.start_as_current_span("hello"): print("Hello, World!") -input("Press any key to exit...") \ No newline at end of file +input("Press any key to exit...") diff --git a/azure_monitor/src/azure_monitor/auto_collection/__init__.py b/azure_monitor/src/azure_monitor/auto_collection/__init__.py index 72b72e0..2d8e55c 100644 --- a/azure_monitor/src/azure_monitor/auto_collection/__init__.py +++ b/azure_monitor/src/azure_monitor/auto_collection/__init__.py @@ -18,9 +18,7 @@ class AutoCollection: - def __init__( - self, meter: Meter, label_set: LabelSet - ): + def __init__(self, meter: Meter, label_set: LabelSet): self._performance_metrics = PerformanceMetrics(meter, label_set) self._dependency_metrics = DependencyMetrics(meter, label_set) self._request_metrics = RequestMetrics(meter, label_set) diff --git a/azure_monitor/src/azure_monitor/auto_collection/performance_metrics.py b/azure_monitor/src/azure_monitor/auto_collection/performance_metrics.py index 6a2e279..d997333 100644 --- a/azure_monitor/src/azure_monitor/auto_collection/performance_metrics.py +++ b/azure_monitor/src/azure_monitor/auto_collection/performance_metrics.py @@ -75,9 +75,8 @@ def _track_process_cpu(self, observer) -> None: # normalize the cpu process using the number of logical CPUs cpu_count = psutil.cpu_count(logical=True) observer.observe( - PROCESS.cpu_percent() / cpu_count, - self._label_set - ) + PROCESS.cpu_percent() / cpu_count, self._label_set + ) except Exception: # pylint: disable=broad-except logger.exception("Error handling get process cpu usage.") diff --git a/azure_monitor/src/azure_monitor/auto_collection/request_metrics.py b/azure_monitor/src/azure_monitor/auto_collection/request_metrics.py index b901c4f..6a8df2b 100644 --- a/azure_monitor/src/azure_monitor/auto_collection/request_metrics.py +++ b/azure_monitor/src/azure_monitor/auto_collection/request_metrics.py @@ -98,8 +98,7 @@ def _track_request_duration(self, observer) -> None: # If interval_count is 0, exporter call made too close to previous # Return the previous result if this is the case observer.observe( - int(last_average_duration * 1000.0), - self._label_set + int(last_average_duration * 1000.0), self._label_set ) def _track_request_rate(self, observer) -> None: diff --git a/azure_monitor/src/azure_monitor/export/metrics/__init__.py b/azure_monitor/src/azure_monitor/export/metrics/__init__.py index f8a95f0..41a22ab 100644 --- a/azure_monitor/src/azure_monitor/export/metrics/__init__.py +++ b/azure_monitor/src/azure_monitor/export/metrics/__init__.py @@ -6,7 +6,6 @@ from urllib.parse import urlparse from opentelemetry.metrics import Metric -from opentelemetry.util import time_ns from opentelemetry.sdk.metrics import Counter, Observer from opentelemetry.sdk.metrics.export import ( MetricRecord, @@ -14,6 +13,7 @@ MetricsExportResult, ) from opentelemetry.sdk.util import ns_to_iso_str +from opentelemetry.util import time_ns from azure_monitor import protocol, utils from azure_monitor.export import ( @@ -59,8 +59,8 @@ def metric_to_envelope( _time = time_ns() if isinstance(metric_record.metric, Metric): _time = metric_record.metric.bind( - metric_record.label_set - ).last_update_timestamp + metric_record.label_set + ).last_update_timestamp envelope = protocol.Envelope( ikey=self.options.instrumentation_key, tags=dict(utils.azure_monitor_context), diff --git a/azure_monitor/tests/auto_collection/test_auto_collection.py b/azure_monitor/tests/auto_collection/test_auto_collection.py index 9d370f6..0e308ff 100644 --- a/azure_monitor/tests/auto_collection/test_auto_collection.py +++ b/azure_monitor/tests/auto_collection/test_auto_collection.py @@ -35,10 +35,7 @@ def test_constructor( self, mock_performance, mock_dependencies, mock_requests ): """Test the constructor.""" - AutoCollection( - meter=self._meter, - label_set=self._test_label_set, - ) + AutoCollection(meter=self._meter, label_set=self._test_label_set) self.assertEqual(mock_performance.called, True) self.assertEqual(mock_dependencies.called, True) self.assertEqual(mock_requests.called, True) diff --git a/azure_monitor/tests/auto_collection/test_dependency_metrics.py b/azure_monitor/tests/auto_collection/test_dependency_metrics.py index 43757a9..dfedc74 100644 --- a/azure_monitor/tests/auto_collection/test_dependency_metrics.py +++ b/azure_monitor/tests/auto_collection/test_dependency_metrics.py @@ -61,14 +61,12 @@ def test_track_dependency_rate(self, time_mock): description="Outgoing Requests per second", unit="rps", value_type=int, - meter=self._meter + meter=self._meter, ) dependency_metrics.dependency_map["last_time"] = 98 dependency_metrics.dependency_map["count"] = 4 metrics_collector._track_dependency_rate(obs) - self.assertEqual( - obs.aggregators[self._test_label_set].current, 2 - ) + self.assertEqual(obs.aggregators[self._test_label_set].current, 2) @mock.patch("azure_monitor.auto_collection.dependency_metrics.time") def test_track_dependency_rate_time_none(self, time_mock): @@ -83,12 +81,10 @@ def test_track_dependency_rate_time_none(self, time_mock): description="Outgoing Requests per second", unit="rps", value_type=int, - meter=self._meter + meter=self._meter, ) metrics_collector._track_dependency_rate(obs) - self.assertEqual( - obs.aggregators[self._test_label_set].current, 0 - ) + self.assertEqual(obs.aggregators[self._test_label_set].current, 0) @mock.patch("azure_monitor.auto_collection.dependency_metrics.time") def test_track_dependency_rate_error(self, time_mock): @@ -104,12 +100,10 @@ def test_track_dependency_rate_error(self, time_mock): description="Outgoing Requests per second", unit="rps", value_type=int, - meter=self._meter + meter=self._meter, ) metrics_collector._track_dependency_rate(obs) - self.assertEqual( - obs.aggregators[self._test_label_set].current, 5 - ) + self.assertEqual(obs.aggregators[self._test_label_set].current, 5) def test_dependency_patch(self): dependency_metrics.ORIGINAL_REQUEST = lambda x: None diff --git a/azure_monitor/tests/auto_collection/test_performance_metrics.py b/azure_monitor/tests/auto_collection/test_performance_metrics.py index cce3cdc..03a9b61 100644 --- a/azure_monitor/tests/auto_collection/test_performance_metrics.py +++ b/azure_monitor/tests/auto_collection/test_performance_metrics.py @@ -85,12 +85,11 @@ def test_track_cpu(self): description="Processor time as a percentage", unit="percentage", value_type=float, - meter=self._meter + meter=self._meter, ) performance_metrics_collector._track_cpu(obs) self.assertEqual( - obs.aggregators[self._test_label_set].current, - 5.5, + obs.aggregators[self._test_label_set].current, 5.5 ) @mock.patch("psutil.virtual_memory") @@ -107,13 +106,10 @@ def test_track_memory(self, psutil_mock): description="Amount of available memory in bytes", unit="byte", value_type=int, - meter=self._meter + meter=self._meter, ) performance_metrics_collector._track_memory(obs) - self.assertEqual( - obs.aggregators[self._test_label_set].current, - 100, - ) + self.assertEqual(obs.aggregators[self._test_label_set].current, 100) @mock.patch("azure_monitor.auto_collection.performance_metrics.psutil") def test_track_process_cpu(self, psutil_mock): @@ -131,12 +127,11 @@ def test_track_process_cpu(self, psutil_mock): description="Process CPU usage as a percentage", unit="percentage", value_type=float, - meter=self._meter + meter=self._meter, ) performance_metrics_collector._track_process_cpu(obs) self.assertEqual( - obs.aggregators[self._test_label_set].current, - 22.2, + obs.aggregators[self._test_label_set].current, 22.2 ) @mock.patch("azure_monitor.auto_collection.performance_metrics.logger") @@ -154,7 +149,7 @@ def test_track_process_cpu_exception(self, logger_mock): description="Process CPU usage as a percentage", unit="percentage", value_type=float, - meter=self._meter + meter=self._meter, ) performance_metrics_collector._track_process_cpu(obs) self.assertEqual(logger_mock.exception.called, True) @@ -175,12 +170,11 @@ def test_track_process_memory(self): description="Amount of memory process has used in bytes", unit="byte", value_type=int, - meter=self._meter + meter=self._meter, ) performance_metrics_collector._track_process_memory(obs) self.assertEqual( - obs.aggregators[self._test_label_set].current, - 100, + obs.aggregators[self._test_label_set].current, 100 ) @mock.patch("azure_monitor.auto_collection.performance_metrics.logger") @@ -198,7 +192,7 @@ def test_track_process_memory_exception(self, logger_mock): description="Amount of memory process has used in bytes", unit="byte", value_type=int, - meter=self._meter + meter=self._meter, ) performance_metrics_collector._track_process_memory(obs) self.assertEqual(logger_mock.exception.called, True) diff --git a/azure_monitor/tests/auto_collection/test_request_metrics.py b/azure_monitor/tests/auto_collection/test_request_metrics.py index 11d6e68..389e5c2 100644 --- a/azure_monitor/tests/auto_collection/test_request_metrics.py +++ b/azure_monitor/tests/auto_collection/test_request_metrics.py @@ -61,8 +61,7 @@ def test_constructor(self): description="Incoming Requests Average Execution Rate", unit="rps", value_type=int, - ) - + ) def test_track_request_duration(self): request_metrics_collector = RequestMetrics( @@ -77,13 +76,10 @@ def test_track_request_duration(self): description="Incoming Requests Average Execution Time", unit="milliseconds", value_type=int, - meter=self._meter + meter=self._meter, ) request_metrics_collector._track_request_duration(obs) - self.assertEqual( - obs.aggregators[self._test_label_set].current, - 20, - ) + self.assertEqual(obs.aggregators[self._test_label_set].current, 20) def test_track_request_duration_error(self): request_metrics_collector = RequestMetrics( @@ -98,13 +94,10 @@ def test_track_request_duration_error(self): description="Incoming Requests Average Execution Time", unit="milliseconds", value_type=int, - meter=self._meter + meter=self._meter, ) request_metrics_collector._track_request_duration(obs) - self.assertEqual( - obs.aggregators[self._test_label_set].current, - 0, - ) + self.assertEqual(obs.aggregators[self._test_label_set].current, 0) @mock.patch("azure_monitor.auto_collection.request_metrics.time") def test_track_request_rate(self, time_mock): @@ -120,13 +113,10 @@ def test_track_request_rate(self, time_mock): description="Incoming Requests Average Execution Rate", unit="rps", value_type=int, - meter=self._meter + meter=self._meter, ) request_metrics_collector._track_request_rate(obs) - self.assertEqual( - obs.aggregators[self._test_label_set].current, - 2, - ) + self.assertEqual(obs.aggregators[self._test_label_set].current, 2) @mock.patch("azure_monitor.auto_collection.request_metrics.time") def test_track_request_rate_time_none(self, time_mock): @@ -141,13 +131,10 @@ def test_track_request_rate_time_none(self, time_mock): description="Incoming Requests Average Execution Rate", unit="rps", value_type=int, - meter=self._meter + meter=self._meter, ) request_metrics_collector._track_request_rate(obs) - self.assertEqual( - obs.aggregators[self._test_label_set].current, - 0, - ) + self.assertEqual(obs.aggregators[self._test_label_set].current, 0) @mock.patch("azure_monitor.auto_collection.request_metrics.time") def test_track_request_rate_error(self, time_mock): @@ -163,13 +150,10 @@ def test_track_request_rate_error(self, time_mock): description="Incoming Requests Average Execution Rate", unit="rps", value_type=int, - meter=self._meter + meter=self._meter, ) request_metrics_collector._track_request_rate(obs) - self.assertEqual( - obs.aggregators[self._test_label_set].current, - 5, - ) + self.assertEqual(obs.aggregators[self._test_label_set].current, 5) def test_request_patch(self): map = request_metrics.requests_map # pylint: disable=redefined-builtin diff --git a/azure_monitor/tests/metrics/test_metrics.py b/azure_monitor/tests/metrics/test_metrics.py index 1e97539..13e5433 100644 --- a/azure_monitor/tests/metrics/test_metrics.py +++ b/azure_monitor/tests/metrics/test_metrics.py @@ -10,7 +10,7 @@ from opentelemetry.sdk.metrics.export.aggregate import ( CounterAggregator, MinMaxSumCountAggregator, - ObserverAggregator + ObserverAggregator, ) from opentelemetry.sdk.util import ns_to_iso_str @@ -44,7 +44,13 @@ def setUpClass(cls): "testname", "testdesc", "unit", int, Measure, ["environment"] ) cls._test_obs = cls._meter.register_observer( - lambda x: x, "testname", "testdesc", "unit", int, Counter, ["environment"] + lambda x: x, + "testname", + "testdesc", + "unit", + int, + Counter, + ["environment"], ) kvp = {"environment": "staging"} cls._test_label_set = cls._meter.get_label_set(kvp) @@ -124,9 +130,7 @@ def test_metric_to_envelope(self): self.assertEqual( envelope.time, ns_to_iso_str( - record.metric.bind( - record.label_set - ).last_update_timestamp + record.metric.bind(record.label_set).last_update_timestamp ), ) self.assertEqual(envelope.sample_rate, None) @@ -157,9 +161,7 @@ def test_observer_to_envelope(self): aggregator = ObserverAggregator() aggregator.update(123) aggregator.take_checkpoint() - record = MetricRecord( - aggregator, self._test_label_set, self._test_obs - ) + record = MetricRecord(aggregator, self._test_label_set, self._test_obs) exporter = AzureMonitorMetricsExporter() envelope = exporter.metric_to_envelope(record) self.assertIsInstance(envelope, Envelope) @@ -214,9 +216,7 @@ def test_measure_to_envelope(self, logger_mock): self.assertEqual( envelope.time, ns_to_iso_str( - record.metric.bind( - record.label_set - ).last_update_timestamp + record.metric.bind(record.label_set).last_update_timestamp ), ) self.assertEqual(envelope.sample_rate, None) From 6f40f93c26eb1bc73ce231e4cf1bc981425f0117 Mon Sep 17 00:00:00 2001 From: Leighton Date: Thu, 19 Mar 2020 12:46:26 -0700 Subject: [PATCH 086/109] isort --- azure_monitor/examples/traces/server.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/azure_monitor/examples/traces/server.py b/azure_monitor/examples/traces/server.py index 60d3170..7be59b0 100644 --- a/azure_monitor/examples/traces/server.py +++ b/azure_monitor/examples/traces/server.py @@ -3,7 +3,6 @@ # pylint: disable=import-error # pylint: disable=no-member # pylint: disable=no-name-in-module -import flask import requests from opentelemetry import trace from opentelemetry.ext import http_requests @@ -11,6 +10,7 @@ from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import BatchExportSpanProcessor +import flask from azure_monitor import AzureMonitorSpanExporter # The preferred tracer implementation must be set, as the opentelemetry-api From eecdd43b1812149a3614bb1b84db0d49444241c6 Mon Sep 17 00:00:00 2001 From: Leighton Date: Thu, 19 Mar 2020 12:51:12 -0700 Subject: [PATCH 087/109] pylint --- azure_monitor/tests/auto_collection/test_auto_collection.py | 1 - 1 file changed, 1 deletion(-) diff --git a/azure_monitor/tests/auto_collection/test_auto_collection.py b/azure_monitor/tests/auto_collection/test_auto_collection.py index 0e308ff..8266b88 100644 --- a/azure_monitor/tests/auto_collection/test_auto_collection.py +++ b/azure_monitor/tests/auto_collection/test_auto_collection.py @@ -8,7 +8,6 @@ from opentelemetry.sdk.metrics import MeterProvider from azure_monitor.auto_collection import AutoCollection -from azure_monitor.utils import PeriodicTask # pylint: disable=protected-access From 0b79d596acd2961b4c12b626757f98ea71090288 Mon Sep 17 00:00:00 2001 From: Leighton Date: Fri, 20 Mar 2020 10:05:57 -0700 Subject: [PATCH 088/109] add commment on seperate export interval --- azure_monitor/examples/metrics/auto_collector.py | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/azure_monitor/examples/metrics/auto_collector.py b/azure_monitor/examples/metrics/auto_collector.py index 6ec7c00..65ed70d 100644 --- a/azure_monitor/examples/metrics/auto_collector.py +++ b/azure_monitor/examples/metrics/auto_collector.py @@ -18,4 +18,9 @@ # Automatically collect standard metrics auto_collection = AutoCollection(meter=meter, label_set=testing_label_set) +# To configure a separate export interval specific for standard metrics +# meter_standard = metrics.get_meter(__name__ + "_standard") +# controller _standard = PushController(meter_standard, exporter, 30) +# _auto_collection = AutoCollection(meter=meter_standard, label_set=testing_label_set) + input("Press any key to exit...") From b3bf8ad9fc37c4c4a1147fc37c1f86833a285364 Mon Sep 17 00:00:00 2001 From: Hector Hernandez Guzman Date: Mon, 23 Mar 2020 12:55:59 -0700 Subject: [PATCH 089/109] Adding docs --- .../azure_monitor/auto_collection/__init__.py | 7 ++ .../auto_collection/dependency_metrics.py | 7 ++ .../auto_collection/performance_metrics.py | 9 +++ .../auto_collection/request_metrics.py | 8 ++ .../src/azure_monitor/export/__init__.py | 16 +++- .../azure_monitor/export/metrics/__init__.py | 12 ++- .../azure_monitor/export/trace/__init__.py | 12 ++- azure_monitor/src/azure_monitor/options.py | 14 ++-- azure_monitor/tests/metrics/test_metrics.py | 8 +- azure_monitor/tests/test_base_exporter.py | 8 +- azure_monitor/tests/trace/test_trace.py | 42 +++++----- docs/Makefile | 20 +++++ .../auto-collection.standard-metrics.rst | 10 +++ docs/azure_monitor/auto-collection/main.rst | 9 +++ .../export/export.base-exporter.rst | 8 ++ docs/azure_monitor/export/export.metrics.rst | 9 +++ docs/azure_monitor/export/export.options.rst | 9 +++ docs/azure_monitor/export/export.trace.rst | 9 +++ docs/azure_monitor/export/main.rst | 12 +++ docs/conf.py | 81 +++++++++++++++++++ docs/index.rst | 38 +++++++++ docs/make.bat | 35 ++++++++ tox.ini | 19 ++++- 23 files changed, 357 insertions(+), 45 deletions(-) create mode 100644 docs/Makefile create mode 100644 docs/azure_monitor/auto-collection/auto-collection.standard-metrics.rst create mode 100644 docs/azure_monitor/auto-collection/main.rst create mode 100644 docs/azure_monitor/export/export.base-exporter.rst create mode 100644 docs/azure_monitor/export/export.metrics.rst create mode 100644 docs/azure_monitor/export/export.options.rst create mode 100644 docs/azure_monitor/export/export.trace.rst create mode 100644 docs/azure_monitor/export/main.rst create mode 100644 docs/conf.py create mode 100644 docs/index.rst create mode 100644 docs/make.bat diff --git a/azure_monitor/src/azure_monitor/auto_collection/__init__.py b/azure_monitor/src/azure_monitor/auto_collection/__init__.py index 2d8e55c..9b44f42 100644 --- a/azure_monitor/src/azure_monitor/auto_collection/__init__.py +++ b/azure_monitor/src/azure_monitor/auto_collection/__init__.py @@ -18,6 +18,13 @@ class AutoCollection: + """Starts auto collection of standard metrics, including performance, dependency and request metrics. + + Args: + meter: OpenTelemetry Meter + label_set: OpenTelemetry label set + """ + def __init__(self, meter: Meter, label_set: LabelSet): self._performance_metrics = PerformanceMetrics(meter, label_set) self._dependency_metrics = DependencyMetrics(meter, label_set) diff --git a/azure_monitor/src/azure_monitor/auto_collection/dependency_metrics.py b/azure_monitor/src/azure_monitor/auto_collection/dependency_metrics.py index f223751..bbe408d 100644 --- a/azure_monitor/src/azure_monitor/auto_collection/dependency_metrics.py +++ b/azure_monitor/src/azure_monitor/auto_collection/dependency_metrics.py @@ -22,6 +22,13 @@ def dependency_patch(*args, **kwargs) -> None: class DependencyMetrics: + """Starts auto collection of dependency metrics, including "Outgoing Requests per second" metric. + + Args: + meter: OpenTelemetry Meter + label_set: OpenTelemetry label set + """ + def __init__(self, meter: Meter, label_set: LabelSet): self._meter = meter self._label_set = label_set diff --git a/azure_monitor/src/azure_monitor/auto_collection/performance_metrics.py b/azure_monitor/src/azure_monitor/auto_collection/performance_metrics.py index d997333..4fa83c3 100644 --- a/azure_monitor/src/azure_monitor/auto_collection/performance_metrics.py +++ b/azure_monitor/src/azure_monitor/auto_collection/performance_metrics.py @@ -11,6 +11,15 @@ class PerformanceMetrics: + """Starts auto collection of performance metrics, including "Processor time as a percentage", + "Amount of available memory in bytes", "Process CPU usage as a percentage" and "Amount of memory + process has used in bytes metrics". + + Args: + meter: OpenTelemetry Meter + label_set: OpenTelemetry label set + """ + def __init__(self, meter: Meter, label_set: LabelSet): self._meter = meter self._label_set = label_set diff --git a/azure_monitor/src/azure_monitor/auto_collection/request_metrics.py b/azure_monitor/src/azure_monitor/auto_collection/request_metrics.py index 6a8df2b..befda80 100644 --- a/azure_monitor/src/azure_monitor/auto_collection/request_metrics.py +++ b/azure_monitor/src/azure_monitor/auto_collection/request_metrics.py @@ -53,6 +53,14 @@ def server_patch(*args, **kwargs): class RequestMetrics: + """Starts auto collection of request metrics, including "Incoming Requests Average Execution Time" + and "Incoming Requests Average Execution Rate" metrics. + + Args: + meter: OpenTelemetry Meter + label_set: OpenTelemetry label set + """ + def __init__(self, meter: Meter, label_set: LabelSet): self._meter = meter self._label_set = label_set diff --git a/azure_monitor/src/azure_monitor/export/__init__.py b/azure_monitor/src/azure_monitor/export/__init__.py index a21bd8c..6829753 100644 --- a/azure_monitor/src/azure_monitor/export/__init__.py +++ b/azure_monitor/src/azure_monitor/export/__init__.py @@ -24,6 +24,12 @@ class ExportResult(Enum): # pylint: disable=broad-except class BaseExporter: + """Azure Monitor base exporter for OpenTelemetry. + + Args: + options: Configuration options for the exporter + """ + def __init__(self, **options): self._telemetry_processors = [] self.options = ExporterOptions(**options) @@ -41,7 +47,9 @@ def add_telemetry_processor( Telemetry processors will be called one by one before telemetry item is pushed for sending and in the order they were added. - :param processor: The processor to add. + + Args: + processor: Processor to add """ self._telemetry_processors.append(processor) @@ -49,7 +57,7 @@ def clear_telemetry_processors(self) -> None: """Removes all telemetry processors""" self._telemetry_processors = [] - def apply_telemetry_processors( + def _apply_telemetry_processors( self, envelopes: typing.List[Envelope] ) -> typing.List[Envelope]: """Applies all telemetry processors in the order they were added. @@ -59,7 +67,9 @@ def apply_telemetry_processors( throw exceptions and fail, but the applying of all telemetry processors will proceed (not fast fail). Processors also return True if envelope should be included for exporting, False otherwise. - :param envelopes: The envelopes to apply each processor to. + + Args: + envelopes: The envelopes to apply each processor to. """ filtered_envelopes = [] for envelope in envelopes: diff --git a/azure_monitor/src/azure_monitor/export/metrics/__init__.py b/azure_monitor/src/azure_monitor/export/metrics/__init__.py index 41a22ab..b4826a7 100644 --- a/azure_monitor/src/azure_monitor/export/metrics/__init__.py +++ b/azure_monitor/src/azure_monitor/export/metrics/__init__.py @@ -26,14 +26,20 @@ class AzureMonitorMetricsExporter(BaseExporter, MetricsExporter): + """Azure Monitor metrics exporter for OpenTelemetry. + + Args: + options: Configuration options for the exporter + """ + def export( self, metric_records: Sequence[MetricRecord] ) -> MetricsExportResult: - envelopes = list(map(self.metric_to_envelope, metric_records)) + envelopes = list(map(self._metric_to_envelope, metric_records)) envelopes = list( map( lambda x: x.to_dict(), - self.apply_telemetry_processors(envelopes), + self._apply_telemetry_processors(envelopes), ) ) try: @@ -48,7 +54,7 @@ def export( logger.exception("Exception occurred while exporting the data.") return get_metrics_export_result(ExportResult.FAILED_NOT_RETRYABLE) - def metric_to_envelope( + def _metric_to_envelope( self, metric_record: MetricRecord ) -> protocol.Envelope: diff --git a/azure_monitor/src/azure_monitor/export/trace/__init__.py b/azure_monitor/src/azure_monitor/export/trace/__init__.py index c9ceb32..dd4483a 100644 --- a/azure_monitor/src/azure_monitor/export/trace/__init__.py +++ b/azure_monitor/src/azure_monitor/export/trace/__init__.py @@ -21,12 +21,18 @@ class AzureMonitorSpanExporter(BaseExporter, SpanExporter): + """Azure Monitor span exporter for OpenTelemetry. + + Args: + options: Configuration options for the exporter + """ + def export(self, spans: Sequence[Span]) -> SpanExportResult: - envelopes = list(map(self.span_to_envelope, spans)) + envelopes = list(map(self._span_to_envelope, spans)) envelopes = list( map( lambda x: x.to_dict(), - self.apply_telemetry_processors(envelopes), + self._apply_telemetry_processors(envelopes), ) ) try: @@ -43,7 +49,7 @@ def export(self, spans: Sequence[Span]) -> SpanExportResult: # pylint: disable=too-many-statements # pylint: disable=too-many-branches - def span_to_envelope(self, span: Span) -> protocol.Envelope: + def _span_to_envelope(self, span: Span) -> protocol.Envelope: if not span: return None envelope = protocol.Envelope( diff --git a/azure_monitor/src/azure_monitor/options.py b/azure_monitor/src/azure_monitor/options.py index 19652f3..a8e6c91 100644 --- a/azure_monitor/src/azure_monitor/options.py +++ b/azure_monitor/src/azure_monitor/options.py @@ -51,15 +51,17 @@ def __init__( instrumentation_key: str = None, storage_maintenance_period: int = 60, storage_max_size: int = 100 * 1024 * 1024, - storage_path: str = os.path.join( - os.path.expanduser("~"), - ".opentelemetry", - ".azure", - os.path.basename(sys.argv[0]) or ".console", - ), + storage_path: str = None, storage_retention_period: int = 7 * 24 * 60 * 60, timeout: int = 10.0, # networking timeout in seconds ) -> None: + if storage_path is None: + storage_path = os.path.join( + os.path.expanduser("~"), + ".opentelemetry", + ".azure", + os.path.basename(sys.argv[0]) or ".console", + ) self.connection_string = connection_string self.instrumentation_key = instrumentation_key self.storage_maintenance_period = storage_maintenance_period diff --git a/azure_monitor/tests/metrics/test_metrics.py b/azure_monitor/tests/metrics/test_metrics.py index 13e5433..b8b4711 100644 --- a/azure_monitor/tests/metrics/test_metrics.py +++ b/azure_monitor/tests/metrics/test_metrics.py @@ -113,7 +113,7 @@ def test_export_exception(self, logger_mock): def test_metric_to_envelope_none(self): exporter = AzureMonitorMetricsExporter() - self.assertIsNone(exporter.metric_to_envelope(None)) + self.assertIsNone(exporter._metric_to_envelope(None)) def test_metric_to_envelope(self): aggregator = CounterAggregator() @@ -123,7 +123,7 @@ def test_metric_to_envelope(self): aggregator, self._test_label_set, self._test_metric ) exporter = AzureMonitorMetricsExporter() - envelope = exporter.metric_to_envelope(record) + envelope = exporter._metric_to_envelope(record) self.assertIsInstance(envelope, Envelope) self.assertEqual(envelope.ver, 1) self.assertEqual(envelope.name, "Microsoft.ApplicationInsights.Metric") @@ -163,7 +163,7 @@ def test_observer_to_envelope(self): aggregator.take_checkpoint() record = MetricRecord(aggregator, self._test_label_set, self._test_obs) exporter = AzureMonitorMetricsExporter() - envelope = exporter.metric_to_envelope(record) + envelope = exporter._metric_to_envelope(record) self.assertIsInstance(envelope, Envelope) self.assertEqual(envelope.ver, 1) self.assertEqual(envelope.name, "Microsoft.ApplicationInsights.Metric") @@ -209,7 +209,7 @@ def test_measure_to_envelope(self, logger_mock): aggregator, self._test_label_set, self._test_measure ) exporter = AzureMonitorMetricsExporter() - envelope = exporter.metric_to_envelope(record) + envelope = exporter._metric_to_envelope(record) self.assertIsInstance(envelope, Envelope) self.assertEqual(envelope.ver, 1) self.assertEqual(envelope.name, "Microsoft.ApplicationInsights.Metric") diff --git a/azure_monitor/tests/test_base_exporter.py b/azure_monitor/tests/test_base_exporter.py index 481e1ba..f328759 100644 --- a/azure_monitor/tests/test_base_exporter.py +++ b/azure_monitor/tests/test_base_exporter.py @@ -96,7 +96,7 @@ def callback_function(envelope): base.add_telemetry_processor(callback_function) envelope = Envelope(data=Data(base_type="type1")) - base.apply_telemetry_processors([envelope]) + base._apply_telemetry_processors([envelope]) self.assertEqual(envelope.data.base_type, "type1_world") def test_telemetry_processor_apply_multiple(self): @@ -112,7 +112,7 @@ def callback_function2(envelope): base.add_telemetry_processor(callback_function) base.add_telemetry_processor(callback_function2) envelope = Envelope(data=Data(base_type="type1")) - base.apply_telemetry_processors([envelope]) + base._apply_telemetry_processors([envelope]) self.assertEqual(envelope.data.base_type, "type1_world_world2") def test_telemetry_processor_apply_exception(self): @@ -127,7 +127,7 @@ def callback_function2(envelope): base.add_telemetry_processor(callback_function) base.add_telemetry_processor(callback_function2) envelope = Envelope(data=Data(base_type="type1")) - base.apply_telemetry_processors([envelope]) + base._apply_telemetry_processors([envelope]) self.assertEqual(envelope.data.base_type, "type1_world2") def test_telemetry_processor_apply_not_accepted(self): @@ -139,7 +139,7 @@ def callback_function(envelope): base.add_telemetry_processor(callback_function) envelope = Envelope(data=Data(base_type="type1")) envelope2 = Envelope(data=Data(base_type="type2")) - envelopes = base.apply_telemetry_processors([envelope, envelope2]) + envelopes = base._apply_telemetry_processors([envelope, envelope2]) self.assertEqual(len(envelopes), 1) self.assertEqual(envelopes[0].data.base_type, "type2") diff --git a/azure_monitor/tests/trace/test_trace.py b/azure_monitor/tests/trace/test_trace.py index 6b85d83..513d990 100644 --- a/azure_monitor/tests/trace/test_trace.py +++ b/azure_monitor/tests/trace/test_trace.py @@ -155,7 +155,7 @@ def test_export_not_retryable(self): def test_span_to_envelope_none(self): exporter = AzureMonitorSpanExporter() - self.assertIsNone(exporter.span_to_envelope(None)) + self.assertIsNone(exporter._span_to_envelope(None)) # pylint: disable=too-many-statements def test_span_to_envelope(self): @@ -199,7 +199,7 @@ def test_span_to_envelope(self): span.start(start_time=start_time) span.end(end_time=end_time) span.status = Status(canonical_code=StatusCanonicalCode.OK) - envelope = exporter.span_to_envelope(span) + envelope = exporter._span_to_envelope(span) self.assertEqual(envelope.ikey, "12345678-1234-5678-abcd-12345678abcd") self.assertEqual( envelope.name, "Microsoft.ApplicationInsights.RemoteDependency" @@ -243,7 +243,7 @@ def test_span_to_envelope(self): span.status = Status(canonical_code=StatusCanonicalCode.OK) span.start(start_time=start_time) span.end(end_time=end_time) - envelope = exporter.span_to_envelope(span) + envelope = exporter._span_to_envelope(span) self.assertEqual(envelope.ikey, "12345678-1234-5678-abcd-12345678abcd") self.assertEqual( envelope.name, "Microsoft.ApplicationInsights.RemoteDependency" @@ -285,7 +285,7 @@ def test_span_to_envelope(self): span.status = Status(canonical_code=StatusCanonicalCode.OK) span.start(start_time=start_time) span.end(end_time=end_time) - envelope = exporter.span_to_envelope(span) + envelope = exporter._span_to_envelope(span) self.assertEqual(envelope.ikey, "12345678-1234-5678-abcd-12345678abcd") self.assertEqual( envelope.name, "Microsoft.ApplicationInsights.RemoteDependency" @@ -336,7 +336,7 @@ def test_span_to_envelope(self): span.status = Status(canonical_code=StatusCanonicalCode.OK) span.start(start_time=start_time) span.end(end_time=end_time) - envelope = exporter.span_to_envelope(span) + envelope = exporter._span_to_envelope(span) self.assertEqual(envelope.ikey, "12345678-1234-5678-abcd-12345678abcd") self.assertEqual( envelope.name, "Microsoft.ApplicationInsights.Request" @@ -389,7 +389,7 @@ def test_span_to_envelope(self): span.status = Status(canonical_code=StatusCanonicalCode.OK) span.start(start_time=start_time) span.end(end_time=end_time) - envelope = exporter.span_to_envelope(span) + envelope = exporter._span_to_envelope(span) self.assertEqual(envelope.ikey, "12345678-1234-5678-abcd-12345678abcd") self.assertEqual( envelope.name, "Microsoft.ApplicationInsights.Request" @@ -442,7 +442,7 @@ def test_span_to_envelope(self): span.status = Status(canonical_code=StatusCanonicalCode.OK) span.start(start_time=start_time) span.end(end_time=end_time) - envelope = exporter.span_to_envelope(span) + envelope = exporter._span_to_envelope(span) self.assertEqual(envelope.ikey, "12345678-1234-5678-abcd-12345678abcd") self.assertEqual( envelope.name, "Microsoft.ApplicationInsights.Request" @@ -478,7 +478,7 @@ def test_span_to_envelope(self): span.status = Status(canonical_code=StatusCanonicalCode.OK) span.start(start_time=start_time) span.end(end_time=end_time) - envelope = exporter.span_to_envelope(span) + envelope = exporter._span_to_envelope(span) self.assertEqual(envelope.ikey, "12345678-1234-5678-abcd-12345678abcd") self.assertEqual( envelope.name, "Microsoft.ApplicationInsights.RemoteDependency" @@ -522,7 +522,7 @@ def test_span_to_envelope(self): span.status = Status(canonical_code=StatusCanonicalCode.OK) span.start(start_time=start_time) span.end(end_time=end_time) - envelope = exporter.span_to_envelope(span) + envelope = exporter._span_to_envelope(span) self.assertEqual(len(envelope.data.base_data.properties), 2) self.assertEqual( envelope.data.base_data.properties["component"], "http" @@ -562,7 +562,7 @@ def test_span_to_envelope(self): span.status = Status(canonical_code=StatusCanonicalCode.OK) span.start(start_time=start_time) span.end(end_time=end_time) - envelope = exporter.span_to_envelope(span) + envelope = exporter._span_to_envelope(span) self.assertEqual(len(envelope.data.base_data.properties), 2) json_dict = json.loads( envelope.data.base_data.properties["_MS.links"] @@ -593,7 +593,7 @@ def test_span_to_envelope(self): span.status = Status(canonical_code=StatusCanonicalCode.OK) span.start(start_time=start_time) span.end(end_time=end_time) - envelope = exporter.span_to_envelope(span) + envelope = exporter._span_to_envelope(span) self.assertEqual(envelope.data.base_data.response_code, "500") self.assertFalse(envelope.data.base_data.success) @@ -620,7 +620,7 @@ def test_span_to_envelope(self): span.status = Status(canonical_code=StatusCanonicalCode.OK) span.start(start_time=start_time) span.end(end_time=end_time) - envelope = exporter.span_to_envelope(span) + envelope = exporter._span_to_envelope(span) self.assertEqual(envelope.data.base_data.result_code, "500") self.assertFalse(envelope.data.base_data.success) @@ -646,7 +646,7 @@ def test_span_to_envelope(self): span.status = Status(canonical_code=StatusCanonicalCode.OK) span.start(start_time=start_time) span.end(end_time=end_time) - envelope = exporter.span_to_envelope(span) + envelope = exporter._span_to_envelope(span) self.assertEqual(envelope.data.base_data.response_code, "0") self.assertTrue(envelope.data.base_data.success) @@ -672,7 +672,7 @@ def test_span_to_envelope(self): span.status = Status(canonical_code=StatusCanonicalCode.OK) span.start(start_time=start_time) span.end(end_time=end_time) - envelope = exporter.span_to_envelope(span) + envelope = exporter._span_to_envelope(span) self.assertEqual(envelope.data.base_data.result_code, "0") self.assertTrue(envelope.data.base_data.success) @@ -698,7 +698,7 @@ def test_span_to_envelope(self): span.start(start_time=start_time) span.end(end_time=end_time) span.status = Status(canonical_code=StatusCanonicalCode.UNKNOWN) - envelope = exporter.span_to_envelope(span) + envelope = exporter._span_to_envelope(span) self.assertEqual(envelope.data.base_data.response_code, "2") self.assertFalse(envelope.data.base_data.success) @@ -724,7 +724,7 @@ def test_span_to_envelope(self): span.start(start_time=start_time) span.end(end_time=end_time) span.status = Status(canonical_code=StatusCanonicalCode.UNKNOWN) - envelope = exporter.span_to_envelope(span) + envelope = exporter._span_to_envelope(span) self.assertEqual(envelope.data.base_data.result_code, "2") self.assertFalse(envelope.data.base_data.success) @@ -754,7 +754,7 @@ def test_span_to_envelope(self): span.start(start_time=start_time) span.end(end_time=end_time) span.status = Status(canonical_code=StatusCanonicalCode.OK) - envelope = exporter.span_to_envelope(span) + envelope = exporter._span_to_envelope(span) self.assertEqual( envelope.data.base_data.properties["request.name"], "GET /wiki/Rabbit", @@ -788,7 +788,7 @@ def test_span_to_envelope(self): span.start(start_time=start_time) span.end(end_time=end_time) span.status = Status(canonical_code=StatusCanonicalCode.OK) - envelope = exporter.span_to_envelope(span) + envelope = exporter._span_to_envelope(span) self.assertIsNone(envelope.data.base_data.name) # Server route attribute missing @@ -816,7 +816,7 @@ def test_span_to_envelope(self): span.start(start_time=start_time) span.end(end_time=end_time) span.status = Status(canonical_code=StatusCanonicalCode.OK) - envelope = exporter.span_to_envelope(span) + envelope = exporter._span_to_envelope(span) self.assertEqual(envelope.data.base_data.name, "GET") self.assertEqual( envelope.data.base_data.properties["request.name"], @@ -851,7 +851,7 @@ def test_span_to_envelope(self): span.start(start_time=start_time) span.end(end_time=end_time) span.status = Status(canonical_code=StatusCanonicalCode.OK) - envelope = exporter.span_to_envelope(span) + envelope = exporter._span_to_envelope(span) self.assertIsNone( envelope.data.base_data.properties.get("request.name") ) @@ -885,7 +885,7 @@ def test_span_to_envelope(self): span.start(start_time=start_time) span.end(end_time=end_time) span.status = Status(canonical_code=StatusCanonicalCode.OK) - envelope = exporter.span_to_envelope(span) + envelope = exporter._span_to_envelope(span) self.assertIsNone(envelope.data.base_data.url) self.assertIsNone( envelope.data.base_data.properties.get("request.url") diff --git a/docs/Makefile b/docs/Makefile new file mode 100644 index 0000000..d4bb2cb --- /dev/null +++ b/docs/Makefile @@ -0,0 +1,20 @@ +# Minimal makefile for Sphinx documentation +# + +# You can set these variables from the command line, and also +# from the environment for the first two. +SPHINXOPTS ?= +SPHINXBUILD ?= sphinx-build +SOURCEDIR = . +BUILDDIR = _build + +# Put it first so that "make" without argument is like "make help". +help: + @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) + +.PHONY: help Makefile + +# Catch-all target: route all unknown targets to Sphinx using the new +# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). +%: Makefile + @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) diff --git a/docs/azure_monitor/auto-collection/auto-collection.standard-metrics.rst b/docs/azure_monitor/auto-collection/auto-collection.standard-metrics.rst new file mode 100644 index 0000000..3c1dba3 --- /dev/null +++ b/docs/azure_monitor/auto-collection/auto-collection.standard-metrics.rst @@ -0,0 +1,10 @@ +Standard Metrics +================ + + +.. automodule:: azure_monitor.auto_collection + :members: + :undoc-members: + + + diff --git a/docs/azure_monitor/auto-collection/main.rst b/docs/azure_monitor/auto-collection/main.rst new file mode 100644 index 0000000..c2bc1ec --- /dev/null +++ b/docs/azure_monitor/auto-collection/main.rst @@ -0,0 +1,9 @@ +Auto Collection +=============== + + +.. toctree:: + + auto-collection.standard-metrics + + diff --git a/docs/azure_monitor/export/export.base-exporter.rst b/docs/azure_monitor/export/export.base-exporter.rst new file mode 100644 index 0000000..12a3fb6 --- /dev/null +++ b/docs/azure_monitor/export/export.base-exporter.rst @@ -0,0 +1,8 @@ +Base Exporter +============= + + + +.. autoclass:: azure_monitor.export.BaseExporter + :members: + :undoc-members: diff --git a/docs/azure_monitor/export/export.metrics.rst b/docs/azure_monitor/export/export.metrics.rst new file mode 100644 index 0000000..04d3d26 --- /dev/null +++ b/docs/azure_monitor/export/export.metrics.rst @@ -0,0 +1,9 @@ +Metrics Exporter +================ + + + +.. automodule:: azure_monitor.export.metrics + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/azure_monitor/export/export.options.rst b/docs/azure_monitor/export/export.options.rst new file mode 100644 index 0000000..287e828 --- /dev/null +++ b/docs/azure_monitor/export/export.options.rst @@ -0,0 +1,9 @@ +Exporter Options +================ + + + +.. autoclass:: azure_monitor.options.ExporterOptions + :members: + :undoc-members: + diff --git a/docs/azure_monitor/export/export.trace.rst b/docs/azure_monitor/export/export.trace.rst new file mode 100644 index 0000000..8761371 --- /dev/null +++ b/docs/azure_monitor/export/export.trace.rst @@ -0,0 +1,9 @@ +Span Exporter +============= + + + +.. automodule:: azure_monitor.export.trace + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/azure_monitor/export/main.rst b/docs/azure_monitor/export/main.rst new file mode 100644 index 0000000..c44ed09 --- /dev/null +++ b/docs/azure_monitor/export/main.rst @@ -0,0 +1,12 @@ +Exporters +========= + + +.. toctree:: + + export.trace + export.metrics + export.options + export.base-exporter + + diff --git a/docs/conf.py b/docs/conf.py new file mode 100644 index 0000000..5b9d428 --- /dev/null +++ b/docs/conf.py @@ -0,0 +1,81 @@ +# Configuration file for the Sphinx documentation builder. +# +# This file only contains a selection of the most common options. For a full +# list see the documentation: +# https://www.sphinx-doc.org/en/master/usage/configuration.html + +# -- Path setup -------------------------------------------------------------- + +# If extensions (or modules to document with autodoc) are in another directory, +# add these directories to sys.path here. If the directory is relative to the +# documentation root, use os.path.abspath to make it absolute, like shown here. + +import os +import sys +from os import listdir +from os.path import isdir, join + +source_dirs = [ + os.path.abspath("../azure_monitor/src/"), +] + + +sys.path[:0] = source_dirs + + +# -- Project information ----------------------------------------------------- + +project = 'OpenTelemetry Azure Monitor Python' +copyright = '2020, Microsoft' +author = 'Microsoft' + +# The full version, including alpha/beta/rc tags +release = '0.0.1b' + + +# -- General configuration --------------------------------------------------- + +# Add any Sphinx extension module names here, as strings. They can be +# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom +# ones. +extensions = [ + # API doc generation + "sphinx.ext.autodoc", + # Support for google-style docstrings + "sphinx.ext.napoleon", + # Infer types from hints instead of docstrings + "sphinx_autodoc_typehints", + # Add links to source from generated docs + "sphinx.ext.viewcode", + # Link to other sphinx docs + "sphinx.ext.intersphinx", + # Add a .nojekyll file to the generated HTML docs + # https://help.github.com/en/articles/files-that-start-with-an-underscore-are-missing + "sphinx.ext.githubpages", + # Support external links to different versions in the Github repo + "sphinx.ext.extlinks", +] + + +intersphinx_mapping = {'python': ('https://opentelemetry-python.readthedocs.io/en/stable',None)} + +# Add any paths that contain templates here, relative to this directory. +templates_path = ['_templates'] + +# List of patterns, relative to source directory, that match files and +# directories to ignore when looking for source files. +# This pattern also affects html_static_path and html_extra_path. +exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store'] + + +# -- Options for HTML output ------------------------------------------------- + +# The theme to use for HTML and HTML Help pages. See the documentation for +# a list of builtin themes. +# +html_theme = 'sphinx_rtd_theme' + +# Add any paths that contain custom static files (such as style sheets) here, +# relative to this directory. They are copied after the builtin static files, +# so a file named "default.css" will overwrite the builtin "default.css". +html_static_path = ['_static'] \ No newline at end of file diff --git a/docs/index.rst b/docs/index.rst new file mode 100644 index 0000000..0958181 --- /dev/null +++ b/docs/index.rst @@ -0,0 +1,38 @@ + +OpenTelemetry Azure Monitor Python +================================== + +The Python `OpenTelemetry `_ Azure Monitor library. + +.. image:: https://img.shields.io/gitter/room/Microsoft/azure-monitor-python + :target: https://gitter.im/Microsoft/azure-monitor-python + :alt: Gitter Chat + + +**Please note** that this library is currently in beta, and shouldn't be +used in production environments. + +Installation +------------ + +The package is available on PyPI, and can installed via pip: + +.. code-block:: sh + + pip install opentelemetry-azure-monitor-exporter + +.. toctree:: + :maxdepth: 1 + :caption: Documentation + :name: documentation + + azure_monitor/export/main + azure_monitor/auto-collection/main + + +Indices and tables +================== + +* :ref:`genindex` +* :ref:`modindex` +* :ref:`search` diff --git a/docs/make.bat b/docs/make.bat new file mode 100644 index 0000000..922152e --- /dev/null +++ b/docs/make.bat @@ -0,0 +1,35 @@ +@ECHO OFF + +pushd %~dp0 + +REM Command file for Sphinx documentation + +if "%SPHINXBUILD%" == "" ( + set SPHINXBUILD=sphinx-build +) +set SOURCEDIR=. +set BUILDDIR=_build + +if "%1" == "" goto help + +%SPHINXBUILD% >NUL 2>NUL +if errorlevel 9009 ( + echo. + echo.The 'sphinx-build' command was not found. Make sure you have Sphinx + echo.installed, then set the SPHINXBUILD environment variable to point + echo.to the full path of the 'sphinx-build' executable. Alternatively you + echo.may add the Sphinx directory to PATH. + echo. + echo.If you don't have Sphinx installed, grab it from + echo.http://sphinx-doc.org/ + exit /b 1 +) + +%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O% +goto end + +:help +%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O% + +:end +popd diff --git a/tox.ini b/tox.ini index 68fab29..880b2a1 100644 --- a/tox.ini +++ b/tox.ini @@ -6,10 +6,11 @@ envlist = py3{4,5,6,7,8}-coverage lint + docs [travis] python = - 3.8: py38, lint + 3.8: py38, lint, docs [testenv] deps = @@ -52,3 +53,19 @@ commands = isort --diff --check-only --recursive . flake8 bash ./scripts/pylint.sh + +[testenv:docs] +deps = + -c dev-requirements.txt + sphinx + sphinx-rtd-theme + sphinx-autodoc-typehints + # External + opentelemetry-api + opentelemetry-sdk + psutil + +changedir = docs + +commands = + sphinx-build -E -a --keep-going -b html -T . _build/html \ No newline at end of file From f64dcbbb65b592d4a8bd0998de1c1f6af0ee1a9e Mon Sep 17 00:00:00 2001 From: Hector Hernandez Guzman Date: Mon, 23 Mar 2020 13:28:28 -0700 Subject: [PATCH 090/109] Fix format --- docs/conf.py | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/docs/conf.py b/docs/conf.py index 5b9d428..c21be60 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -15,9 +15,7 @@ from os import listdir from os.path import isdir, join -source_dirs = [ - os.path.abspath("../azure_monitor/src/"), -] +source_dirs = [os.path.abspath("../azure_monitor/src/")] sys.path[:0] = source_dirs @@ -25,12 +23,12 @@ # -- Project information ----------------------------------------------------- -project = 'OpenTelemetry Azure Monitor Python' -copyright = '2020, Microsoft' -author = 'Microsoft' +project = "OpenTelemetry Azure Monitor Python" +copyright = "2020, Microsoft" +author = "Microsoft" # The full version, including alpha/beta/rc tags -release = '0.0.1b' +release = "0.0.1b" # -- General configuration --------------------------------------------------- @@ -57,15 +55,17 @@ ] -intersphinx_mapping = {'python': ('https://opentelemetry-python.readthedocs.io/en/stable',None)} +intersphinx_mapping = { + "python": ("https://opentelemetry-python.readthedocs.io/en/stable", None) +} # Add any paths that contain templates here, relative to this directory. -templates_path = ['_templates'] +templates_path = ["_templates"] # List of patterns, relative to source directory, that match files and # directories to ignore when looking for source files. # This pattern also affects html_static_path and html_extra_path. -exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store'] +exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"] # -- Options for HTML output ------------------------------------------------- @@ -73,9 +73,9 @@ # The theme to use for HTML and HTML Help pages. See the documentation for # a list of builtin themes. # -html_theme = 'sphinx_rtd_theme' +html_theme = "sphinx_rtd_theme" # Add any paths that contain custom static files (such as style sheets) here, # relative to this directory. They are copied after the builtin static files, # so a file named "default.css" will overwrite the builtin "default.css". -html_static_path = ['_static'] \ No newline at end of file +html_static_path = ["_static"] From 5be9da19b5f429c5fe98a4d3c1ba7e5670d50043 Mon Sep 17 00:00:00 2001 From: Hector Hernandez Guzman Date: Mon, 23 Mar 2020 13:57:09 -0700 Subject: [PATCH 091/109] Adding check to avoid metrics of telemetry calls --- .../auto_collection/dependency_metrics.py | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/azure_monitor/src/azure_monitor/auto_collection/dependency_metrics.py b/azure_monitor/src/azure_monitor/auto_collection/dependency_metrics.py index f223751..c17e41f 100644 --- a/azure_monitor/src/azure_monitor/auto_collection/dependency_metrics.py +++ b/azure_monitor/src/azure_monitor/auto_collection/dependency_metrics.py @@ -4,6 +4,7 @@ import time import requests +from opentelemetry import context from opentelemetry.metrics import Meter from opentelemetry.sdk.metrics import LabelSet @@ -14,10 +15,12 @@ def dependency_patch(*args, **kwargs) -> None: result = ORIGINAL_REQUEST(*args, **kwargs) - # We don't want multiple threads updating this at once - with _dependency_lock: - count = dependency_map.get("count", 0) - dependency_map["count"] = count + 1 + # Only collect request metric if sent from non-exporter thread + if context.get_value("suppress_instrumentation") is None: + # We don't want multiple threads updating this at once + with _dependency_lock: + count = dependency_map.get("count", 0) + dependency_map["count"] = count + 1 return result From deb0b5e01bf0859dc1144597c042fb75717d30fa Mon Sep 17 00:00:00 2001 From: Hector Hernandez Guzman Date: Tue, 24 Mar 2020 12:28:02 -0700 Subject: [PATCH 092/109] Addressing comments --- .../src/azure_monitor/auto_collection/__init__.py | 3 ++- .../azure_monitor/auto_collection/dependency_metrics.py | 7 ++++--- .../azure_monitor/auto_collection/performance_metrics.py | 7 ++++--- .../src/azure_monitor/auto_collection/request_metrics.py | 5 +++-- azure_monitor/src/azure_monitor/export/metrics/__init__.py | 2 +- azure_monitor/src/azure_monitor/export/trace/__init__.py | 2 +- docs/azure_monitor/export/export.options.rst | 2 +- tox.ini | 2 +- 8 files changed, 17 insertions(+), 13 deletions(-) diff --git a/azure_monitor/src/azure_monitor/auto_collection/__init__.py b/azure_monitor/src/azure_monitor/auto_collection/__init__.py index 9b44f42..89437bc 100644 --- a/azure_monitor/src/azure_monitor/auto_collection/__init__.py +++ b/azure_monitor/src/azure_monitor/auto_collection/__init__.py @@ -18,7 +18,8 @@ class AutoCollection: - """Starts auto collection of standard metrics, including performance, dependency and request metrics. + """Starts auto collection of standard metrics, including performance, + dependency and request metrics. Args: meter: OpenTelemetry Meter diff --git a/azure_monitor/src/azure_monitor/auto_collection/dependency_metrics.py b/azure_monitor/src/azure_monitor/auto_collection/dependency_metrics.py index bbe408d..3137da8 100644 --- a/azure_monitor/src/azure_monitor/auto_collection/dependency_metrics.py +++ b/azure_monitor/src/azure_monitor/auto_collection/dependency_metrics.py @@ -22,7 +22,8 @@ def dependency_patch(*args, **kwargs) -> None: class DependencyMetrics: - """Starts auto collection of dependency metrics, including "Outgoing Requests per second" metric. + """Starts auto collection of dependency metrics, including + "Outgoing Requests per second" metric. Args: meter: OpenTelemetry Meter @@ -46,8 +47,8 @@ def _track_dependency_rate(self, observer) -> None: """ Track Dependency rate Calculated by obtaining the number of outgoing requests made - using the requests library within an elapsed time and dividing that - value over the elapsed time. + using the requests library within an elapsed time and dividing + that value over the elapsed time. """ current_count = dependency_map.get("count", 0) current_time = time.time() diff --git a/azure_monitor/src/azure_monitor/auto_collection/performance_metrics.py b/azure_monitor/src/azure_monitor/auto_collection/performance_metrics.py index 4fa83c3..0a66fce 100644 --- a/azure_monitor/src/azure_monitor/auto_collection/performance_metrics.py +++ b/azure_monitor/src/azure_monitor/auto_collection/performance_metrics.py @@ -11,9 +11,10 @@ class PerformanceMetrics: - """Starts auto collection of performance metrics, including "Processor time as a percentage", - "Amount of available memory in bytes", "Process CPU usage as a percentage" and "Amount of memory - process has used in bytes metrics". + """Starts auto collection of performance metrics, including + "Processor time as a percentage", "Amount of available memory + in bytes", "Process CPU usage as a percentage" and "Amount of + memory process has used in bytes" metrics. Args: meter: OpenTelemetry Meter diff --git a/azure_monitor/src/azure_monitor/auto_collection/request_metrics.py b/azure_monitor/src/azure_monitor/auto_collection/request_metrics.py index befda80..c1f5cac 100644 --- a/azure_monitor/src/azure_monitor/auto_collection/request_metrics.py +++ b/azure_monitor/src/azure_monitor/auto_collection/request_metrics.py @@ -53,8 +53,9 @@ def server_patch(*args, **kwargs): class RequestMetrics: - """Starts auto collection of request metrics, including "Incoming Requests Average Execution Time" - and "Incoming Requests Average Execution Rate" metrics. + """Starts auto collection of request metrics, including + "Incoming Requests Average Execution Time" and + "Incoming Requests Average Execution Rate" metrics. Args: meter: OpenTelemetry Meter diff --git a/azure_monitor/src/azure_monitor/export/metrics/__init__.py b/azure_monitor/src/azure_monitor/export/metrics/__init__.py index b4826a7..98d7369 100644 --- a/azure_monitor/src/azure_monitor/export/metrics/__init__.py +++ b/azure_monitor/src/azure_monitor/export/metrics/__init__.py @@ -29,7 +29,7 @@ class AzureMonitorMetricsExporter(BaseExporter, MetricsExporter): """Azure Monitor metrics exporter for OpenTelemetry. Args: - options: Configuration options for the exporter + options: :doc:`export.options` to allow configuration for the exporter """ def export( diff --git a/azure_monitor/src/azure_monitor/export/trace/__init__.py b/azure_monitor/src/azure_monitor/export/trace/__init__.py index dd4483a..805d271 100644 --- a/azure_monitor/src/azure_monitor/export/trace/__init__.py +++ b/azure_monitor/src/azure_monitor/export/trace/__init__.py @@ -24,7 +24,7 @@ class AzureMonitorSpanExporter(BaseExporter, SpanExporter): """Azure Monitor span exporter for OpenTelemetry. Args: - options: Configuration options for the exporter + options: :doc:`export.options` to allow configuration for the exporter """ def export(self, spans: Sequence[Span]) -> SpanExportResult: diff --git a/docs/azure_monitor/export/export.options.rst b/docs/azure_monitor/export/export.options.rst index 287e828..c54dc5f 100644 --- a/docs/azure_monitor/export/export.options.rst +++ b/docs/azure_monitor/export/export.options.rst @@ -5,5 +5,5 @@ Exporter Options .. autoclass:: azure_monitor.options.ExporterOptions :members: - :undoc-members: + diff --git a/tox.ini b/tox.ini index 880b2a1..1e23387 100644 --- a/tox.ini +++ b/tox.ini @@ -68,4 +68,4 @@ deps = changedir = docs commands = - sphinx-build -E -a --keep-going -b html -T . _build/html \ No newline at end of file + sphinx-build -E -a --keep-going -b html -T . _build/html From aae6c36603834136c49759388d895f9b48d8241a Mon Sep 17 00:00:00 2001 From: Hector Hernandez Guzman Date: Tue, 24 Mar 2020 12:37:11 -0700 Subject: [PATCH 093/109] Fixing lint --- .../src/azure_monitor/auto_collection/dependency_metrics.py | 2 +- .../src/azure_monitor/auto_collection/request_metrics.py | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/azure_monitor/src/azure_monitor/auto_collection/dependency_metrics.py b/azure_monitor/src/azure_monitor/auto_collection/dependency_metrics.py index 3137da8..3c7530a 100644 --- a/azure_monitor/src/azure_monitor/auto_collection/dependency_metrics.py +++ b/azure_monitor/src/azure_monitor/auto_collection/dependency_metrics.py @@ -22,7 +22,7 @@ def dependency_patch(*args, **kwargs) -> None: class DependencyMetrics: - """Starts auto collection of dependency metrics, including + """Starts auto collection of dependency metrics, including "Outgoing Requests per second" metric. Args: diff --git a/azure_monitor/src/azure_monitor/auto_collection/request_metrics.py b/azure_monitor/src/azure_monitor/auto_collection/request_metrics.py index c1f5cac..2505cbe 100644 --- a/azure_monitor/src/azure_monitor/auto_collection/request_metrics.py +++ b/azure_monitor/src/azure_monitor/auto_collection/request_metrics.py @@ -53,8 +53,8 @@ def server_patch(*args, **kwargs): class RequestMetrics: - """Starts auto collection of request metrics, including - "Incoming Requests Average Execution Time" and + """Starts auto collection of request metrics, including + "Incoming Requests Average Execution Time" and "Incoming Requests Average Execution Rate" metrics. Args: From 0b73ae4b73f7f00727fb81ae0139bffed70afc22 Mon Sep 17 00:00:00 2001 From: Hector Hernandez Guzman Date: Tue, 24 Mar 2020 13:34:44 -0700 Subject: [PATCH 094/109] Address comment --- azure_monitor/src/azure_monitor/export/__init__.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/azure_monitor/src/azure_monitor/export/__init__.py b/azure_monitor/src/azure_monitor/export/__init__.py index 6829753..e705a90 100644 --- a/azure_monitor/src/azure_monitor/export/__init__.py +++ b/azure_monitor/src/azure_monitor/export/__init__.py @@ -27,7 +27,7 @@ class BaseExporter: """Azure Monitor base exporter for OpenTelemetry. Args: - options: Configuration options for the exporter + options: :doc:`export.options` to allow configuration for the exporter """ def __init__(self, **options): From cacee9da5454beb3f590d70273f930f6588325fd Mon Sep 17 00:00:00 2001 From: Leighton Date: Wed, 25 Mar 2020 13:39:57 -0700 Subject: [PATCH 095/109] Refactor --- README.md | 6 +-- .../examples/metrics/auto_collector.py | 3 +- azure_monitor/examples/metrics/client.py | 26 +++++++++++ azure_monitor/examples/metrics/simple.py | 2 +- azure_monitor/examples/traces/server.py | 2 +- azure_monitor/src/azure_monitor/__init__.py | 9 +--- .../src/azure_monitor/sdk/__init__.py | 5 +++ .../{ => sdk}/auto_collection/__init__.py | 8 ++-- .../auto_collection/dependency_metrics.py | 0 .../auto_collection/performance_metrics.py | 0 .../auto_collection/request_metrics.py | 0 .../auto_collection/test_auto_collection.py | 10 +++-- .../test_dependency_metrics.py | 40 +++++++++++------ .../test_performance_metrics.py | 16 +++---- .../auto_collection/test_request_metrics.py | 24 +++++------ azure_monitor/tests/metrics/test_metrics.py | 43 +++++++++++++++++++ .../auto-collection.standard-metrics.rst | 2 +- .../{ => sdk}/auto-collection/main.rst | 0 docs/index.rst | 2 +- 19 files changed, 142 insertions(+), 56 deletions(-) create mode 100644 azure_monitor/examples/metrics/client.py create mode 100644 azure_monitor/src/azure_monitor/sdk/__init__.py rename azure_monitor/src/azure_monitor/{ => sdk}/auto_collection/__init__.py (76%) rename azure_monitor/src/azure_monitor/{ => sdk}/auto_collection/dependency_metrics.py (100%) rename azure_monitor/src/azure_monitor/{ => sdk}/auto_collection/performance_metrics.py (100%) rename azure_monitor/src/azure_monitor/{ => sdk}/auto_collection/request_metrics.py (100%) rename docs/azure_monitor/{ => sdk}/auto-collection/auto-collection.standard-metrics.rst (59%) rename docs/azure_monitor/{ => sdk}/auto-collection/main.rst (100%) diff --git a/README.md b/README.md index 955fede..6b2b4b8 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,7 @@ -# OpenTelemetry Azure Monitor Exporters +# OpenTelemetry Azure Monitor SDKs and Exporters -[![Gitter chat](https://img.shields.io/gitter/room/opentelemetry/opentelemetry-python)](https://gitter.im/Microsoft/azure-monitor-python) -[![Build status](https://travis-ci.org/microsoft/opentelemetry-exporters-python.svg?branch=master)](https://travis-ci.org/microsoft/opentelemetry-exporters-python) +[![Gitter chat](https://img.shields.io/gitter/room/Microsoft/azure-monitor-python)](https://gitter.im/Microsoft/azure-monitor-python) +[![Build status](https://travis-ci.org/microsoft/opentelemetry-azure-monitor-python.svg?branch=master)](https://travis-ci.org/microsoft/opentelemetry-exporters-python) [![PyPI version](https://badge.fury.io/py/opentelemetry-azure-monitor-exporter.svg)](https://badge.fury.io/py/opentelemetry-azure-monitor-exporter) ## Installation diff --git a/azure_monitor/examples/metrics/auto_collector.py b/azure_monitor/examples/metrics/auto_collector.py index 65ed70d..cd899d7 100644 --- a/azure_monitor/examples/metrics/auto_collector.py +++ b/azure_monitor/examples/metrics/auto_collector.py @@ -4,7 +4,8 @@ from opentelemetry.sdk.metrics import MeterProvider from opentelemetry.sdk.metrics.export.controller import PushController -from azure_monitor import AutoCollection, AzureMonitorMetricsExporter +from azure_monitor import AzureMonitorMetricsExporter +from azure_monitor.sdk.auto_collection import AutoCollection metrics.set_meter_provider(MeterProvider()) meter = metrics.get_meter(__name__) diff --git a/azure_monitor/examples/metrics/client.py b/azure_monitor/examples/metrics/client.py new file mode 100644 index 0000000..57bec03 --- /dev/null +++ b/azure_monitor/examples/metrics/client.py @@ -0,0 +1,26 @@ +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. +# pylint: disable=import-error +# pylint: disable=no-member +# pylint: disable=no-name-in-module +import requests +from opentelemetry import trace +from opentelemetry.ext import http_requests +from opentelemetry.sdk.trace import TracerProvider +from opentelemetry.sdk.trace.export import BatchExportSpanProcessor + +from azure_monitor import AzureMonitorSpanExporter + +trace.set_tracer_provider(TracerProvider()) +tracer = trace.get_tracer(__name__) +http_requests.enable(trace.get_tracer_provider()) +span_processor = BatchExportSpanProcessor( + AzureMonitorSpanExporter( + connection_string="InstrumentationKey=" + ) +) +trace.get_tracer_provider().add_span_processor(span_processor) + +response = requests.get(url="http://google.com") + +input("Press any key to exit...") diff --git a/azure_monitor/examples/metrics/simple.py b/azure_monitor/examples/metrics/simple.py index 3547764..1e60b3e 100644 --- a/azure_monitor/examples/metrics/simple.py +++ b/azure_monitor/examples/metrics/simple.py @@ -9,7 +9,7 @@ metrics.set_meter_provider(MeterProvider()) meter = metrics.get_meter(__name__) exporter = AzureMonitorMetricsExporter( - connection_string="InstrumentationKey=" + # connection_string="InstrumentationKey=" ) controller = PushController(meter, exporter, 5) diff --git a/azure_monitor/examples/traces/server.py b/azure_monitor/examples/traces/server.py index 7be59b0..60d3170 100644 --- a/azure_monitor/examples/traces/server.py +++ b/azure_monitor/examples/traces/server.py @@ -3,6 +3,7 @@ # pylint: disable=import-error # pylint: disable=no-member # pylint: disable=no-name-in-module +import flask import requests from opentelemetry import trace from opentelemetry.ext import http_requests @@ -10,7 +11,6 @@ from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import BatchExportSpanProcessor -import flask from azure_monitor import AzureMonitorSpanExporter # The preferred tracer implementation must be set, as the opentelemetry-api diff --git a/azure_monitor/src/azure_monitor/__init__.py b/azure_monitor/src/azure_monitor/__init__.py index f0a8d37..4a963e4 100644 --- a/azure_monitor/src/azure_monitor/__init__.py +++ b/azure_monitor/src/azure_monitor/__init__.py @@ -1,13 +1,6 @@ # Copyright (c) Microsoft Corporation. All rights reserved. # Licensed under the MIT License. -from azure_monitor.auto_collection import AutoCollection from azure_monitor.export.metrics import AzureMonitorMetricsExporter from azure_monitor.export.trace import AzureMonitorSpanExporter -from azure_monitor.options import ExporterOptions -__all__ = [ - "AutoCollection", - "AzureMonitorMetricsExporter", - "AzureMonitorSpanExporter", - "ExporterOptions", -] +__all__ = ["AzureMonitorMetricsExporter", "AzureMonitorSpanExporter"] diff --git a/azure_monitor/src/azure_monitor/sdk/__init__.py b/azure_monitor/src/azure_monitor/sdk/__init__.py new file mode 100644 index 0000000..4eeadf2 --- /dev/null +++ b/azure_monitor/src/azure_monitor/sdk/__init__.py @@ -0,0 +1,5 @@ +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. +from . import auto_collection + +__all__ = ["auto_collection"] diff --git a/azure_monitor/src/azure_monitor/auto_collection/__init__.py b/azure_monitor/src/azure_monitor/sdk/auto_collection/__init__.py similarity index 76% rename from azure_monitor/src/azure_monitor/auto_collection/__init__.py rename to azure_monitor/src/azure_monitor/sdk/auto_collection/__init__.py index 89437bc..b7ebe41 100644 --- a/azure_monitor/src/azure_monitor/auto_collection/__init__.py +++ b/azure_monitor/src/azure_monitor/sdk/auto_collection/__init__.py @@ -3,11 +3,13 @@ # from opentelemetry.metrics import LabelSet, Meter -from azure_monitor.auto_collection.dependency_metrics import DependencyMetrics -from azure_monitor.auto_collection.performance_metrics import ( +from azure_monitor.sdk.auto_collection.dependency_metrics import ( + DependencyMetrics, +) +from azure_monitor.sdk.auto_collection.performance_metrics import ( PerformanceMetrics, ) -from azure_monitor.auto_collection.request_metrics import RequestMetrics +from azure_monitor.sdk.auto_collection.request_metrics import RequestMetrics __all__ = [ "AutoCollection", diff --git a/azure_monitor/src/azure_monitor/auto_collection/dependency_metrics.py b/azure_monitor/src/azure_monitor/sdk/auto_collection/dependency_metrics.py similarity index 100% rename from azure_monitor/src/azure_monitor/auto_collection/dependency_metrics.py rename to azure_monitor/src/azure_monitor/sdk/auto_collection/dependency_metrics.py diff --git a/azure_monitor/src/azure_monitor/auto_collection/performance_metrics.py b/azure_monitor/src/azure_monitor/sdk/auto_collection/performance_metrics.py similarity index 100% rename from azure_monitor/src/azure_monitor/auto_collection/performance_metrics.py rename to azure_monitor/src/azure_monitor/sdk/auto_collection/performance_metrics.py diff --git a/azure_monitor/src/azure_monitor/auto_collection/request_metrics.py b/azure_monitor/src/azure_monitor/sdk/auto_collection/request_metrics.py similarity index 100% rename from azure_monitor/src/azure_monitor/auto_collection/request_metrics.py rename to azure_monitor/src/azure_monitor/sdk/auto_collection/request_metrics.py diff --git a/azure_monitor/tests/auto_collection/test_auto_collection.py b/azure_monitor/tests/auto_collection/test_auto_collection.py index 8266b88..301d249 100644 --- a/azure_monitor/tests/auto_collection/test_auto_collection.py +++ b/azure_monitor/tests/auto_collection/test_auto_collection.py @@ -7,7 +7,7 @@ from opentelemetry import metrics from opentelemetry.sdk.metrics import MeterProvider -from azure_monitor.auto_collection import AutoCollection +from azure_monitor.sdk.auto_collection import AutoCollection # pylint: disable=protected-access @@ -24,12 +24,14 @@ def tearDownClass(cls): metrics._METER_PROVIDER = None @mock.patch( - "azure_monitor.auto_collection.PerformanceMetrics", autospec=True + "azure_monitor.sdk.auto_collection.PerformanceMetrics", autospec=True ) @mock.patch( - "azure_monitor.auto_collection.DependencyMetrics", autospec=True + "azure_monitor.sdk.auto_collection.DependencyMetrics", autospec=True + ) + @mock.patch( + "azure_monitor.sdk.auto_collection.RequestMetrics", autospec=True ) - @mock.patch("azure_monitor.auto_collection.RequestMetrics", autospec=True) def test_constructor( self, mock_performance, mock_dependencies, mock_requests ): diff --git a/azure_monitor/tests/auto_collection/test_dependency_metrics.py b/azure_monitor/tests/auto_collection/test_dependency_metrics.py index dfedc74..fe273df 100644 --- a/azure_monitor/tests/auto_collection/test_dependency_metrics.py +++ b/azure_monitor/tests/auto_collection/test_dependency_metrics.py @@ -9,7 +9,7 @@ from opentelemetry import metrics from opentelemetry.sdk.metrics import MeterProvider, Observer -from azure_monitor.auto_collection import DependencyMetrics, dependency_metrics +from azure_monitor.sdk.auto_collection import dependency_metrics ORIGINAL_FUNCTION = requests.Session.request ORIGINAL_CONS = HTTPServer.__init__ @@ -27,6 +27,8 @@ def setUpClass(cls): @classmethod def tearDown(cls): metrics._METER_PROVIDER = None + requests.Session.request = ORIGINAL_FUNCTION + dependency_metrics.ORIGINAL_CONSTRUCTOR = ORIGINAL_CONS def setUp(self): dependency_metrics.dependency_map.clear() @@ -35,7 +37,7 @@ def setUp(self): def test_constructor(self): mock_meter = mock.Mock() - metrics_collector = DependencyMetrics( + metrics_collector = dependency_metrics.DependencyMetrics( meter=mock_meter, label_set=self._test_label_set ) self.assertEqual(metrics_collector._meter, mock_meter) @@ -49,10 +51,10 @@ def test_constructor(self): value_type=int, ) - @mock.patch("azure_monitor.auto_collection.dependency_metrics.time") + @mock.patch("azure_monitor.sdk.auto_collection.dependency_metrics.time") def test_track_dependency_rate(self, time_mock): time_mock.time.return_value = 100 - metrics_collector = DependencyMetrics( + metrics_collector = dependency_metrics.DependencyMetrics( meter=self._meter, label_set=self._test_label_set ) obs = Observer( @@ -68,10 +70,10 @@ def test_track_dependency_rate(self, time_mock): metrics_collector._track_dependency_rate(obs) self.assertEqual(obs.aggregators[self._test_label_set].current, 2) - @mock.patch("azure_monitor.auto_collection.dependency_metrics.time") + @mock.patch("azure_monitor.sdk.auto_collection.dependency_metrics.time") def test_track_dependency_rate_time_none(self, time_mock): time_mock.time.return_value = 100 - metrics_collector = DependencyMetrics( + metrics_collector = dependency_metrics.DependencyMetrics( meter=self._meter, label_set=self._test_label_set ) dependency_metrics.dependency_map["last_time"] = None @@ -86,10 +88,10 @@ def test_track_dependency_rate_time_none(self, time_mock): metrics_collector._track_dependency_rate(obs) self.assertEqual(obs.aggregators[self._test_label_set].current, 0) - @mock.patch("azure_monitor.auto_collection.dependency_metrics.time") + @mock.patch("azure_monitor.sdk.auto_collection.dependency_metrics.time") def test_track_dependency_rate_error(self, time_mock): time_mock.time.return_value = 100 - metrics_collector = DependencyMetrics( + metrics_collector = dependency_metrics.DependencyMetrics( meter=self._meter, label_set=self._test_label_set ) dependency_metrics.dependency_map["last_time"] = 100 @@ -105,10 +107,22 @@ def test_track_dependency_rate_error(self, time_mock): metrics_collector._track_dependency_rate(obs) self.assertEqual(obs.aggregators[self._test_label_set].current, 5) - def test_dependency_patch(self): - dependency_metrics.ORIGINAL_REQUEST = lambda x: None + @mock.patch( + "azure_monitor.sdk.auto_collection.dependency_metrics.ORIGINAL_REQUEST" + ) + def test_dependency_patch(self, request_mock): session = requests.Session() - result = dependency_metrics.dependency_patch(session) - + dependency_metrics.dependency_patch(session) self.assertEqual(dependency_metrics.dependency_map["count"], 1) - self.assertIsNone(result) + request_mock.assert_called_with(session) + + @mock.patch( + "azure_monitor.sdk.auto_collection.dependency_metrics.ORIGINAL_REQUEST" + ) + @mock.patch("azure_monitor.sdk.auto_collection.dependency_metrics.context") + def test_dependency_patch_suppress(self, context_mock, request_mock): + context_mock.get_value.return_value = {} + session = requests.Session() + dependency_metrics.dependency_patch(session) + self.assertEqual(dependency_metrics.dependency_map.get("count"), None) + request_mock.assert_called_with(session) diff --git a/azure_monitor/tests/auto_collection/test_performance_metrics.py b/azure_monitor/tests/auto_collection/test_performance_metrics.py index 03a9b61..6637913 100644 --- a/azure_monitor/tests/auto_collection/test_performance_metrics.py +++ b/azure_monitor/tests/auto_collection/test_performance_metrics.py @@ -8,7 +8,7 @@ from opentelemetry import metrics from opentelemetry.sdk.metrics import MeterProvider, Observer -from azure_monitor.auto_collection import PerformanceMetrics +from azure_monitor.sdk.auto_collection import PerformanceMetrics def throw(exc_type, *args, **kwargs): @@ -111,10 +111,10 @@ def test_track_memory(self, psutil_mock): performance_metrics_collector._track_memory(obs) self.assertEqual(obs.aggregators[self._test_label_set].current, 100) - @mock.patch("azure_monitor.auto_collection.performance_metrics.psutil") + @mock.patch("azure_monitor.sdk.auto_collection.performance_metrics.psutil") def test_track_process_cpu(self, psutil_mock): with mock.patch( - "azure_monitor.auto_collection.performance_metrics.PROCESS" + "azure_monitor.sdk.auto_collection.performance_metrics.PROCESS" ) as process_mock: performance_metrics_collector = PerformanceMetrics( meter=self._meter, label_set=self._test_label_set @@ -134,10 +134,10 @@ def test_track_process_cpu(self, psutil_mock): obs.aggregators[self._test_label_set].current, 22.2 ) - @mock.patch("azure_monitor.auto_collection.performance_metrics.logger") + @mock.patch("azure_monitor.sdk.auto_collection.performance_metrics.logger") def test_track_process_cpu_exception(self, logger_mock): with mock.patch( - "azure_monitor.auto_collection.performance_metrics.psutil" + "azure_monitor.sdk.auto_collection.performance_metrics.psutil" ) as psutil_mock: performance_metrics_collector = PerformanceMetrics( meter=self._meter, label_set=self._test_label_set @@ -156,7 +156,7 @@ def test_track_process_cpu_exception(self, logger_mock): def test_track_process_memory(self): with mock.patch( - "azure_monitor.auto_collection.performance_metrics.PROCESS" + "azure_monitor.sdk.auto_collection.performance_metrics.PROCESS" ) as process_mock: performance_metrics_collector = PerformanceMetrics( meter=self._meter, label_set=self._test_label_set @@ -177,10 +177,10 @@ def test_track_process_memory(self): obs.aggregators[self._test_label_set].current, 100 ) - @mock.patch("azure_monitor.auto_collection.performance_metrics.logger") + @mock.patch("azure_monitor.sdk.auto_collection.performance_metrics.logger") def test_track_process_memory_exception(self, logger_mock): with mock.patch( - "azure_monitor.auto_collection.performance_metrics.PROCESS", + "azure_monitor.sdk.auto_collection.performance_metrics.PROCESS", throw(Exception), ): performance_metrics_collector = PerformanceMetrics( diff --git a/azure_monitor/tests/auto_collection/test_request_metrics.py b/azure_monitor/tests/auto_collection/test_request_metrics.py index 389e5c2..63030d1 100644 --- a/azure_monitor/tests/auto_collection/test_request_metrics.py +++ b/azure_monitor/tests/auto_collection/test_request_metrics.py @@ -9,7 +9,7 @@ from opentelemetry import metrics from opentelemetry.sdk.metrics import MeterProvider, Observer -from azure_monitor.auto_collection import RequestMetrics, request_metrics +from azure_monitor.sdk.auto_collection import request_metrics ORIGINAL_FUNCTION = requests.Session.request ORIGINAL_CONS = HTTPServer.__init__ @@ -35,7 +35,7 @@ def setUp(self): def test_constructor(self): mock_meter = mock.Mock() - request_metrics_collector = RequestMetrics( + request_metrics_collector = request_metrics.RequestMetrics( meter=mock_meter, label_set=self._test_label_set ) self.assertEqual(request_metrics_collector._meter, mock_meter) @@ -64,7 +64,7 @@ def test_constructor(self): ) def test_track_request_duration(self): - request_metrics_collector = RequestMetrics( + request_metrics_collector = request_metrics.RequestMetrics( meter=self._meter, label_set=self._test_label_set ) request_metrics.requests_map["duration"] = 0.1 @@ -82,7 +82,7 @@ def test_track_request_duration(self): self.assertEqual(obs.aggregators[self._test_label_set].current, 20) def test_track_request_duration_error(self): - request_metrics_collector = RequestMetrics( + request_metrics_collector = request_metrics.RequestMetrics( meter=self._meter, label_set=self._test_label_set ) request_metrics.requests_map["duration"] = 0.1 @@ -99,9 +99,9 @@ def test_track_request_duration_error(self): request_metrics_collector._track_request_duration(obs) self.assertEqual(obs.aggregators[self._test_label_set].current, 0) - @mock.patch("azure_monitor.auto_collection.request_metrics.time") + @mock.patch("azure_monitor.sdk.auto_collection.request_metrics.time") def test_track_request_rate(self, time_mock): - request_metrics_collector = RequestMetrics( + request_metrics_collector = request_metrics.RequestMetrics( meter=self._meter, label_set=self._test_label_set ) time_mock.time.return_value = 100 @@ -118,10 +118,10 @@ def test_track_request_rate(self, time_mock): request_metrics_collector._track_request_rate(obs) self.assertEqual(obs.aggregators[self._test_label_set].current, 2) - @mock.patch("azure_monitor.auto_collection.request_metrics.time") + @mock.patch("azure_monitor.sdk.auto_collection.request_metrics.time") def test_track_request_rate_time_none(self, time_mock): time_mock.time.return_value = 100 - request_metrics_collector = RequestMetrics( + request_metrics_collector = request_metrics.RequestMetrics( meter=self._meter, label_set=self._test_label_set ) request_metrics.requests_map["last_time"] = None @@ -136,9 +136,9 @@ def test_track_request_rate_time_none(self, time_mock): request_metrics_collector._track_request_rate(obs) self.assertEqual(obs.aggregators[self._test_label_set].current, 0) - @mock.patch("azure_monitor.auto_collection.request_metrics.time") + @mock.patch("azure_monitor.sdk.auto_collection.request_metrics.time") def test_track_request_rate_error(self, time_mock): - request_metrics_collector = RequestMetrics( + request_metrics_collector = request_metrics.RequestMetrics( meter=self._meter, label_set=self._test_label_set ) time_mock.time.return_value = 100 @@ -168,7 +168,7 @@ def test_request_patch(self): def test_server_patch(self): request_metrics.ORIGINAL_CONSTRUCTOR = lambda x, y, z: None with mock.patch( - "azure_monitor.auto_collection.request_metrics.request_patch" + "azure_monitor.sdk.auto_collection.request_metrics.request_patch" ) as request_mock: handler = mock.Mock() handler.do_DELETE.return_value = None @@ -191,7 +191,7 @@ def test_server_patch(self): def test_server_patch_no_methods(self): request_metrics.ORIGINAL_CONSTRUCTOR = lambda x, y, z: None with mock.patch( - "azure_monitor.auto_collection.request_metrics.request_patch" + "azure_monitor.sdk.auto_collection.request_metrics.request_patch" ) as request_mock: handler = mock.Mock() result = request_metrics.server_patch(None, None, handler) diff --git a/azure_monitor/tests/metrics/test_metrics.py b/azure_monitor/tests/metrics/test_metrics.py index b8b4711..7053dba 100644 --- a/azure_monitor/tests/metrics/test_metrics.py +++ b/azure_monitor/tests/metrics/test_metrics.py @@ -200,6 +200,49 @@ def test_observer_to_envelope(self): self.assertIsNotNone(envelope.tags["ai.device.type"]) self.assertIsNotNone(envelope.tags["ai.internal.sdkVersion"]) + def test_observer_to_envelope_value_none(self): + aggregator = ObserverAggregator() + aggregator.update(None) + aggregator.take_checkpoint() + record = MetricRecord(aggregator, self._test_label_set, self._test_obs) + exporter = AzureMonitorMetricsExporter() + envelope = exporter._metric_to_envelope(record) + self.assertIsInstance(envelope, Envelope) + self.assertEqual(envelope.ver, 1) + self.assertEqual(envelope.name, "Microsoft.ApplicationInsights.Metric") + # TODO: implement last updated timestamp for observer + # self.assertEqual( + # envelope.time, + # ns_to_iso_str( + # record.metric.bind( + # record.label_set + # ).last_update_timestamp + # ), + # ) + self.assertEqual(envelope.sample_rate, None) + self.assertEqual(envelope.seq, None) + self.assertEqual(envelope.ikey, "1234abcd-5678-4efa-8abc-1234567890ab") + self.assertEqual(envelope.flags, None) + + self.assertIsInstance(envelope.data, Data) + self.assertIsInstance(envelope.data.base_data, MetricData) + self.assertEqual(envelope.data.base_data.ver, 2) + self.assertEqual(len(envelope.data.base_data.metrics), 1) + self.assertIsInstance(envelope.data.base_data.metrics[0], DataPoint) + self.assertEqual(envelope.data.base_data.metrics[0].ns, "testdesc") + self.assertEqual(envelope.data.base_data.metrics[0].name, "testname") + self.assertEqual(envelope.data.base_data.metrics[0].value, 0) + self.assertEqual( + envelope.data.base_data.properties["environment"], "staging" + ) + self.assertIsNotNone(envelope.tags["ai.cloud.role"]) + self.assertIsNotNone(envelope.tags["ai.cloud.roleInstance"]) + self.assertIsNotNone(envelope.tags["ai.device.id"]) + self.assertIsNotNone(envelope.tags["ai.device.locale"]) + self.assertIsNotNone(envelope.tags["ai.device.osVersion"]) + self.assertIsNotNone(envelope.tags["ai.device.type"]) + self.assertIsNotNone(envelope.tags["ai.internal.sdkVersion"]) + @mock.patch("azure_monitor.export.metrics.logger") def test_measure_to_envelope(self, logger_mock): aggregator = MinMaxSumCountAggregator() diff --git a/docs/azure_monitor/auto-collection/auto-collection.standard-metrics.rst b/docs/azure_monitor/sdk/auto-collection/auto-collection.standard-metrics.rst similarity index 59% rename from docs/azure_monitor/auto-collection/auto-collection.standard-metrics.rst rename to docs/azure_monitor/sdk/auto-collection/auto-collection.standard-metrics.rst index 3c1dba3..f825c79 100644 --- a/docs/azure_monitor/auto-collection/auto-collection.standard-metrics.rst +++ b/docs/azure_monitor/sdk/auto-collection/auto-collection.standard-metrics.rst @@ -2,7 +2,7 @@ Standard Metrics ================ -.. automodule:: azure_monitor.auto_collection +.. automodule:: azure_monitor.sdk.auto_collection :members: :undoc-members: diff --git a/docs/azure_monitor/auto-collection/main.rst b/docs/azure_monitor/sdk/auto-collection/main.rst similarity index 100% rename from docs/azure_monitor/auto-collection/main.rst rename to docs/azure_monitor/sdk/auto-collection/main.rst diff --git a/docs/index.rst b/docs/index.rst index 0958181..e360c22 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -27,7 +27,7 @@ The package is available on PyPI, and can installed via pip: :name: documentation azure_monitor/export/main - azure_monitor/auto-collection/main + azure_monitor/sdk/auto-collection/main Indices and tables From ffa0b70a25fb141dbdcbb0c4615a49203c4b9dd3 Mon Sep 17 00:00:00 2001 From: Leighton Date: Wed, 25 Mar 2020 13:49:41 -0700 Subject: [PATCH 096/109] fix lint --- azure_monitor/examples/traces/server.py | 2 +- azure_monitor/examples/traces/trace.py | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/azure_monitor/examples/traces/server.py b/azure_monitor/examples/traces/server.py index 60d3170..7be59b0 100644 --- a/azure_monitor/examples/traces/server.py +++ b/azure_monitor/examples/traces/server.py @@ -3,7 +3,6 @@ # pylint: disable=import-error # pylint: disable=no-member # pylint: disable=no-name-in-module -import flask import requests from opentelemetry import trace from opentelemetry.ext import http_requests @@ -11,6 +10,7 @@ from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import BatchExportSpanProcessor +import flask from azure_monitor import AzureMonitorSpanExporter # The preferred tracer implementation must be set, as the opentelemetry-api diff --git a/azure_monitor/examples/traces/trace.py b/azure_monitor/examples/traces/trace.py index b5bd7a4..6bbce67 100644 --- a/azure_monitor/examples/traces/trace.py +++ b/azure_monitor/examples/traces/trace.py @@ -1,3 +1,5 @@ +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. from opentelemetry import trace from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import BatchExportSpanProcessor From d6aa63905b0f4a8fdca5a9c134fb0afdbd7de570 Mon Sep 17 00:00:00 2001 From: Leighton Chen Date: Thu, 26 Mar 2020 08:17:42 -0700 Subject: [PATCH 097/109] Update azure_monitor/examples/metrics/simple.py Co-Authored-By: Hector Hernandez <39923391+hectorhdzg@users.noreply.github.com> --- azure_monitor/examples/metrics/simple.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/azure_monitor/examples/metrics/simple.py b/azure_monitor/examples/metrics/simple.py index 1e60b3e..3547764 100644 --- a/azure_monitor/examples/metrics/simple.py +++ b/azure_monitor/examples/metrics/simple.py @@ -9,7 +9,7 @@ metrics.set_meter_provider(MeterProvider()) meter = metrics.get_meter(__name__) exporter = AzureMonitorMetricsExporter( - # connection_string="InstrumentationKey=" + connection_string="InstrumentationKey=" ) controller = PushController(meter, exporter, 5) From e8595ab0071709bf8d56616ce83a13d140f675fa Mon Sep 17 00:00:00 2001 From: Hector Hernandez Guzman Date: Thu, 26 Mar 2020 11:38:59 -0700 Subject: [PATCH 098/109] Adding doc requirements --- .readthedocs.yml | 14 ++++++++++++++ docs-requirements.txt | 4 ++++ tox.ini | 1 + 3 files changed, 19 insertions(+) create mode 100644 .readthedocs.yml create mode 100644 docs-requirements.txt diff --git a/.readthedocs.yml b/.readthedocs.yml new file mode 100644 index 0000000..3dcf0e5 --- /dev/null +++ b/.readthedocs.yml @@ -0,0 +1,14 @@ +# Read the Docs configuration file +# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details +version: 2 + +sphinx: + configuration: docs/conf.py + +build: + image: latest + +python: + version: 3.8 + install: + - requirements: docs-requirements.txt diff --git a/docs-requirements.txt b/docs-requirements.txt new file mode 100644 index 0000000..92377a0 --- /dev/null +++ b/docs-requirements.txt @@ -0,0 +1,4 @@ +sphinx~=2.4 +sphinx-rtd-theme~=0.4 +sphinx-autodoc-typehints~=1.10.2 + diff --git a/tox.ini b/tox.ini index 1e23387..5a9de73 100644 --- a/tox.ini +++ b/tox.ini @@ -57,6 +57,7 @@ commands = [testenv:docs] deps = -c dev-requirements.txt + -c docs-requirements.txt sphinx sphinx-rtd-theme sphinx-autodoc-typehints From c0219207e03f7195ad7a4b599594ccbeb6ce68d6 Mon Sep 17 00:00:00 2001 From: Hector Hernandez Guzman Date: Thu, 26 Mar 2020 11:44:10 -0700 Subject: [PATCH 099/109] Updating requirements --- docs-requirements.txt | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs-requirements.txt b/docs-requirements.txt index 92377a0..4a4bbd8 100644 --- a/docs-requirements.txt +++ b/docs-requirements.txt @@ -1,4 +1,8 @@ sphinx~=2.4 sphinx-rtd-theme~=0.4 sphinx-autodoc-typehints~=1.10.2 +# External +opentelemetry-api +opentelemetry-sdk +psutil From c0a7ac67ff465c6af71f717280aee4e3e9d82565 Mon Sep 17 00:00:00 2001 From: Leighton Date: Thu, 26 Mar 2020 21:36:58 -0700 Subject: [PATCH 100/109] tests --- azure_monitor/tests/metrics/test_metrics.py | 48 +++++++++++++++++---- azure_monitor/tests/test_base_exporter.py | 14 +++--- azure_monitor/tests/trace/test_trace.py | 18 +++++--- tox.ini | 1 - 4 files changed, 58 insertions(+), 23 deletions(-) diff --git a/azure_monitor/tests/metrics/test_metrics.py b/azure_monitor/tests/metrics/test_metrics.py index 7053dba..f079aca 100644 --- a/azure_monitor/tests/metrics/test_metrics.py +++ b/azure_monitor/tests/metrics/test_metrics.py @@ -1,6 +1,7 @@ # Copyright (c) Microsoft Corporation. All rights reserved. # Licensed under the MIT License. import os +import shutil import unittest from unittest import mock @@ -19,6 +20,18 @@ from azure_monitor.options import ExporterOptions from azure_monitor.protocol import Data, DataPoint, Envelope, MetricData +TEST_FOLDER = os.path.abspath(".test.exporter.trace") + + +# pylint: disable=invalid-name +def setUpModule(): + os.makedirs(TEST_FOLDER) + + +# pylint: disable=invalid-name +def tearDownModule(): + shutil.rmtree(TEST_FOLDER) + def throw(exc_type, *args, **kwargs): def func(*_args, **_kwargs): @@ -62,7 +75,8 @@ def tearDownClass(cls): def test_constructor(self): """Test the constructor.""" exporter = AzureMonitorMetricsExporter( - instrumentation_key="4321abcd-5678-4efa-8abc-1234567890ab" + instrumentation_key="4321abcd-5678-4efa-8abc-1234567890ab", + storage_path=os.path.join(TEST_FOLDER, self.id()), ) self.assertIsInstance(exporter.options, ExporterOptions) self.assertEqual( @@ -74,7 +88,9 @@ def test_export(self,): record = MetricRecord( CounterAggregator(), self._test_label_set, self._test_metric ) - exporter = AzureMonitorMetricsExporter() + exporter = AzureMonitorMetricsExporter( + storage_path=os.path.join(TEST_FOLDER, self.id()) + ) with mock.patch( "azure_monitor.export.metrics.AzureMonitorMetricsExporter._transmit" ) as transmit: # noqa: E501 @@ -86,7 +102,9 @@ def test_export_failed_retryable(self): record = MetricRecord( CounterAggregator(), self._test_label_set, self._test_metric ) - exporter = AzureMonitorMetricsExporter() + exporter = AzureMonitorMetricsExporter( + storage_path=os.path.join(TEST_FOLDER, self.id()) + ) with mock.patch( "azure_monitor.export.metrics.AzureMonitorMetricsExporter._transmit" ) as transmit: # noqa: E501 @@ -102,7 +120,9 @@ def test_export_exception(self, logger_mock): record = MetricRecord( CounterAggregator(), self._test_label_set, self._test_metric ) - exporter = AzureMonitorMetricsExporter() + exporter = AzureMonitorMetricsExporter( + storage_path=os.path.join(TEST_FOLDER, self.id()) + ) with mock.patch( "azure_monitor.export.metrics.AzureMonitorMetricsExporter._transmit", throw(Exception), @@ -112,7 +132,9 @@ def test_export_exception(self, logger_mock): self.assertEqual(logger_mock.exception.called, True) def test_metric_to_envelope_none(self): - exporter = AzureMonitorMetricsExporter() + exporter = AzureMonitorMetricsExporter( + storage_path=os.path.join(TEST_FOLDER, self.id()) + ) self.assertIsNone(exporter._metric_to_envelope(None)) def test_metric_to_envelope(self): @@ -122,7 +144,9 @@ def test_metric_to_envelope(self): record = MetricRecord( aggregator, self._test_label_set, self._test_metric ) - exporter = AzureMonitorMetricsExporter() + exporter = AzureMonitorMetricsExporter( + storage_path=os.path.join(TEST_FOLDER, self.id()) + ) envelope = exporter._metric_to_envelope(record) self.assertIsInstance(envelope, Envelope) self.assertEqual(envelope.ver, 1) @@ -162,7 +186,9 @@ def test_observer_to_envelope(self): aggregator.update(123) aggregator.take_checkpoint() record = MetricRecord(aggregator, self._test_label_set, self._test_obs) - exporter = AzureMonitorMetricsExporter() + exporter = AzureMonitorMetricsExporter( + storage_path=os.path.join(TEST_FOLDER, self.id()) + ) envelope = exporter._metric_to_envelope(record) self.assertIsInstance(envelope, Envelope) self.assertEqual(envelope.ver, 1) @@ -205,7 +231,9 @@ def test_observer_to_envelope_value_none(self): aggregator.update(None) aggregator.take_checkpoint() record = MetricRecord(aggregator, self._test_label_set, self._test_obs) - exporter = AzureMonitorMetricsExporter() + exporter = AzureMonitorMetricsExporter( + storage_path=os.path.join(TEST_FOLDER, self.id()) + ) envelope = exporter._metric_to_envelope(record) self.assertIsInstance(envelope, Envelope) self.assertEqual(envelope.ver, 1) @@ -251,7 +279,9 @@ def test_measure_to_envelope(self, logger_mock): record = MetricRecord( aggregator, self._test_label_set, self._test_measure ) - exporter = AzureMonitorMetricsExporter() + exporter = AzureMonitorMetricsExporter( + storage_path=os.path.join(TEST_FOLDER, self.id()) + ) envelope = exporter._metric_to_envelope(record) self.assertIsInstance(envelope, Envelope) self.assertEqual(envelope.ver, 1) diff --git a/azure_monitor/tests/test_base_exporter.py b/azure_monitor/tests/test_base_exporter.py index f328759..fe17fa9 100644 --- a/azure_monitor/tests/test_base_exporter.py +++ b/azure_monitor/tests/test_base_exporter.py @@ -77,19 +77,19 @@ def test_constructor_wrong_options(self): BaseExporter(something_else=6) def test_telemetry_processor_add(self): - base = BaseExporter() + base = BaseExporter(storage_path=os.path.join(TEST_FOLDER, self.id())) base.add_telemetry_processor(lambda: True) self.assertEqual(len(base._telemetry_processors), 1) def test_telemetry_processor_clear(self): - base = BaseExporter() + base = BaseExporter(storage_path=os.path.join(TEST_FOLDER, self.id())) base.add_telemetry_processor(lambda: True) self.assertEqual(len(base._telemetry_processors), 1) base.clear_telemetry_processors() self.assertEqual(len(base._telemetry_processors), 0) def test_telemetry_processor_apply(self): - base = BaseExporter() + base = BaseExporter(storage_path=os.path.join(TEST_FOLDER, self.id())) def callback_function(envelope): envelope.data.base_type += "_world" @@ -100,7 +100,7 @@ def callback_function(envelope): self.assertEqual(envelope.data.base_type, "type1_world") def test_telemetry_processor_apply_multiple(self): - base = BaseExporter() + base = BaseExporter(storage_path=os.path.join(TEST_FOLDER, self.id())) base._telemetry_processors = [] def callback_function(envelope): @@ -116,7 +116,7 @@ def callback_function2(envelope): self.assertEqual(envelope.data.base_type, "type1_world_world2") def test_telemetry_processor_apply_exception(self): - base = BaseExporter() + base = BaseExporter(storage_path=os.path.join(TEST_FOLDER, self.id())) def callback_function(envelope): raise ValueError() @@ -131,7 +131,7 @@ def callback_function2(envelope): self.assertEqual(envelope.data.base_type, "type1_world2") def test_telemetry_processor_apply_not_accepted(self): - base = BaseExporter() + base = BaseExporter(storage_path=os.path.join(TEST_FOLDER, self.id())) def callback_function(envelope): return envelope.data.base_type == "type2" @@ -177,7 +177,7 @@ def test_transmission_lease_failure(self, requests_mock): exporter._transmit_from_storage() self.assertTrue(exporter.storage.get()) - def test_transmit_response_exception(self): + def test_(self): exporter = BaseExporter( storage_path=os.path.join(TEST_FOLDER, self.id()) ) diff --git a/azure_monitor/tests/trace/test_trace.py b/azure_monitor/tests/trace/test_trace.py index 513d990..c64b5a2 100644 --- a/azure_monitor/tests/trace/test_trace.py +++ b/azure_monitor/tests/trace/test_trace.py @@ -52,13 +52,17 @@ def setUpClass(cls): def test_constructor(self): """Test the constructor.""" exporter = AzureMonitorSpanExporter( - instrumentation_key="4321abcd-5678-4efa-8abc-1234567890ab" + instrumentation_key="4321abcd-5678-4efa-8abc-1234567890ab", + storage_path=os.path.join(TEST_FOLDER, self.id()), ) self.assertIsInstance(exporter.options, ExporterOptions) self.assertEqual( exporter.options.instrumentation_key, "4321abcd-5678-4efa-8abc-1234567890ab", ) + self.assertEqual( + exporter.options.storage_path, os.path.join(TEST_FOLDER, self.id()) + ) def test_export_empty(self): exporter = AzureMonitorSpanExporter( @@ -154,15 +158,17 @@ def test_export_not_retryable(self): self.assertEqual(result, SpanExportResult.FAILED_NOT_RETRYABLE) def test_span_to_envelope_none(self): - exporter = AzureMonitorSpanExporter() + exporter = AzureMonitorSpanExporter( + storage_path=os.path.join(TEST_FOLDER, self.id()) + ) self.assertIsNone(exporter._span_to_envelope(None)) # pylint: disable=too-many-statements def test_span_to_envelope(self): - options = { - "instrumentation_key": "12345678-1234-5678-abcd-12345678abcd" - } - exporter = AzureMonitorSpanExporter(**options) + exporter = AzureMonitorSpanExporter( + instrumentation_key="12345678-1234-5678-abcd-12345678abcd", + storage_path=os.path.join(TEST_FOLDER, self.id()), + ) parent_span = Span( name="test", diff --git a/tox.ini b/tox.ini index 1e23387..66b65be 100644 --- a/tox.ini +++ b/tox.ini @@ -43,7 +43,6 @@ deps = flake8 isort black - psutil commands_pre = pip install ./azure_monitor From 05555381c60ca97a37812b173e82685a9bef184e Mon Sep 17 00:00:00 2001 From: Hector Hernandez Guzman Date: Fri, 27 Mar 2020 11:29:24 -0700 Subject: [PATCH 101/109] Adding missing versions in req file --- docs-requirements.txt | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs-requirements.txt b/docs-requirements.txt index 4a4bbd8..b9ffedb 100644 --- a/docs-requirements.txt +++ b/docs-requirements.txt @@ -2,7 +2,7 @@ sphinx~=2.4 sphinx-rtd-theme~=0.4 sphinx-autodoc-typehints~=1.10.2 # External -opentelemetry-api -opentelemetry-sdk -psutil +opentelemetry-api>= 0.5b0 +opentelemetry-sdk>= 0.5b0 +psutil>= 5.6.3 From ba531244fa1c6c842e9686bf7b4a3cd85ed06561 Mon Sep 17 00:00:00 2001 From: Hector Hernandez Guzman Date: Fri, 27 Mar 2020 11:33:50 -0700 Subject: [PATCH 102/109] Addressing comments --- dev-requirements.txt | 3 --- tox.ini | 1 - 2 files changed, 4 deletions(-) diff --git a/dev-requirements.txt b/dev-requirements.txt index 6ce5479..3ffaba9 100644 --- a/dev-requirements.txt +++ b/dev-requirements.txt @@ -3,8 +3,5 @@ flake8~=3.7 isort~=4.3 black>=19.3b0,==19.* mypy==0.740 -sphinx~=2.1 -sphinx-rtd-theme~=0.4 -sphinx-autodoc-typehints~=1.10.2 pytest!=5.2.3 pytest-cov>=2.8 diff --git a/tox.ini b/tox.ini index 5a9de73..3968186 100644 --- a/tox.ini +++ b/tox.ini @@ -56,7 +56,6 @@ commands = [testenv:docs] deps = - -c dev-requirements.txt -c docs-requirements.txt sphinx sphinx-rtd-theme From 6e1dde53f6968407d5f290d0ec72bb580b4f9093 Mon Sep 17 00:00:00 2001 From: Leighton Date: Fri, 27 Mar 2020 14:41:11 -0700 Subject: [PATCH 103/109] address comments --- azure_monitor/tests/metrics/test_metrics.py | 46 ++++++++++----------- azure_monitor/tests/test_base_exporter.py | 28 +++++++++---- azure_monitor/tests/trace/test_trace.py | 37 +++++++++-------- 3 files changed, 61 insertions(+), 50 deletions(-) diff --git a/azure_monitor/tests/metrics/test_metrics.py b/azure_monitor/tests/metrics/test_metrics.py index f079aca..e6f9691 100644 --- a/azure_monitor/tests/metrics/test_metrics.py +++ b/azure_monitor/tests/metrics/test_metrics.py @@ -21,7 +21,7 @@ from azure_monitor.protocol import Data, DataPoint, Envelope, MetricData TEST_FOLDER = os.path.abspath(".test.exporter.trace") - +STORAGE_PATH = os.path.join(TEST_FOLDER) # pylint: disable=invalid-name def setUpModule(): @@ -47,6 +47,7 @@ def setUpClass(cls): os.environ[ "APPINSIGHTS_INSTRUMENTATIONKEY" ] = "1234abcd-5678-4efa-8abc-1234567890ab" + cls._exporter = AzureMonitorMetricsExporter(storage_path=STORAGE_PATH) metrics.set_meter_provider(MeterProvider()) cls._meter = metrics.get_meter(__name__) @@ -68,6 +69,17 @@ def setUpClass(cls): kvp = {"environment": "staging"} cls._test_label_set = cls._meter.get_label_set(kvp) + def setUp(self): + for filename in os.listdir(STORAGE_PATH): + file_path = os.path.join(STORAGE_PATH, filename) + try: + if os.path.isfile(file_path) or os.path.islink(file_path): + os.unlink(file_path) + elif os.path.isdir(file_path): + shutil.rmtree(file_path) + except Exception as e: + print("Failed to delete %s. Reason: %s" % (file_path, e)) + @classmethod def tearDownClass(cls): metrics._METER_PROVIDER = None @@ -88,9 +100,7 @@ def test_export(self,): record = MetricRecord( CounterAggregator(), self._test_label_set, self._test_metric ) - exporter = AzureMonitorMetricsExporter( - storage_path=os.path.join(TEST_FOLDER, self.id()) - ) + exporter = self._exporter with mock.patch( "azure_monitor.export.metrics.AzureMonitorMetricsExporter._transmit" ) as transmit: # noqa: E501 @@ -102,9 +112,7 @@ def test_export_failed_retryable(self): record = MetricRecord( CounterAggregator(), self._test_label_set, self._test_metric ) - exporter = AzureMonitorMetricsExporter( - storage_path=os.path.join(TEST_FOLDER, self.id()) - ) + exporter = self._exporter with mock.patch( "azure_monitor.export.metrics.AzureMonitorMetricsExporter._transmit" ) as transmit: # noqa: E501 @@ -120,9 +128,7 @@ def test_export_exception(self, logger_mock): record = MetricRecord( CounterAggregator(), self._test_label_set, self._test_metric ) - exporter = AzureMonitorMetricsExporter( - storage_path=os.path.join(TEST_FOLDER, self.id()) - ) + exporter = self._exporter with mock.patch( "azure_monitor.export.metrics.AzureMonitorMetricsExporter._transmit", throw(Exception), @@ -132,9 +138,7 @@ def test_export_exception(self, logger_mock): self.assertEqual(logger_mock.exception.called, True) def test_metric_to_envelope_none(self): - exporter = AzureMonitorMetricsExporter( - storage_path=os.path.join(TEST_FOLDER, self.id()) - ) + exporter = self._exporter self.assertIsNone(exporter._metric_to_envelope(None)) def test_metric_to_envelope(self): @@ -144,9 +148,7 @@ def test_metric_to_envelope(self): record = MetricRecord( aggregator, self._test_label_set, self._test_metric ) - exporter = AzureMonitorMetricsExporter( - storage_path=os.path.join(TEST_FOLDER, self.id()) - ) + exporter = self._exporter envelope = exporter._metric_to_envelope(record) self.assertIsInstance(envelope, Envelope) self.assertEqual(envelope.ver, 1) @@ -186,9 +188,7 @@ def test_observer_to_envelope(self): aggregator.update(123) aggregator.take_checkpoint() record = MetricRecord(aggregator, self._test_label_set, self._test_obs) - exporter = AzureMonitorMetricsExporter( - storage_path=os.path.join(TEST_FOLDER, self.id()) - ) + exporter = self._exporter envelope = exporter._metric_to_envelope(record) self.assertIsInstance(envelope, Envelope) self.assertEqual(envelope.ver, 1) @@ -231,9 +231,7 @@ def test_observer_to_envelope_value_none(self): aggregator.update(None) aggregator.take_checkpoint() record = MetricRecord(aggregator, self._test_label_set, self._test_obs) - exporter = AzureMonitorMetricsExporter( - storage_path=os.path.join(TEST_FOLDER, self.id()) - ) + exporter = self._exporter envelope = exporter._metric_to_envelope(record) self.assertIsInstance(envelope, Envelope) self.assertEqual(envelope.ver, 1) @@ -279,9 +277,7 @@ def test_measure_to_envelope(self, logger_mock): record = MetricRecord( aggregator, self._test_label_set, self._test_measure ) - exporter = AzureMonitorMetricsExporter( - storage_path=os.path.join(TEST_FOLDER, self.id()) - ) + exporter = self._exporter envelope = exporter._metric_to_envelope(record) self.assertIsInstance(envelope, Envelope) self.assertEqual(envelope.ver, 1) diff --git a/azure_monitor/tests/test_base_exporter.py b/azure_monitor/tests/test_base_exporter.py index fe17fa9..a4655ff 100644 --- a/azure_monitor/tests/test_base_exporter.py +++ b/azure_monitor/tests/test_base_exporter.py @@ -19,7 +19,8 @@ from azure_monitor.options import ExporterOptions from azure_monitor.protocol import Data, Envelope -TEST_FOLDER = os.path.abspath(".test.exporter") +TEST_FOLDER = os.path.abspath(".test.exporter.base") +STORAGE_PATH = os.path.join(TEST_FOLDER) # pylint: disable=invalid-name @@ -47,6 +48,19 @@ def setUpClass(cls): os.environ[ "APPINSIGHTS_INSTRUMENTATIONKEY" ] = "1234abcd-5678-4efa-8abc-1234567890ab" + cls._base = BaseExporter(storage_path=STORAGE_PATH) + + def setUp(self): + for filename in os.listdir(STORAGE_PATH): + file_path = os.path.join(STORAGE_PATH, filename) + try: + if os.path.isfile(file_path) or os.path.islink(file_path): + os.unlink(file_path) + elif os.path.isdir(file_path): + shutil.rmtree(file_path) + except Exception as e: + print("Failed to delete %s. Reason: %s" % (file_path, e)) + self._base.clear_telemetry_processors() def test_constructor(self): """Test the constructor.""" @@ -77,19 +91,19 @@ def test_constructor_wrong_options(self): BaseExporter(something_else=6) def test_telemetry_processor_add(self): - base = BaseExporter(storage_path=os.path.join(TEST_FOLDER, self.id())) + base = self._base base.add_telemetry_processor(lambda: True) self.assertEqual(len(base._telemetry_processors), 1) def test_telemetry_processor_clear(self): - base = BaseExporter(storage_path=os.path.join(TEST_FOLDER, self.id())) + base = self._base base.add_telemetry_processor(lambda: True) self.assertEqual(len(base._telemetry_processors), 1) base.clear_telemetry_processors() self.assertEqual(len(base._telemetry_processors), 0) def test_telemetry_processor_apply(self): - base = BaseExporter(storage_path=os.path.join(TEST_FOLDER, self.id())) + base = self._base def callback_function(envelope): envelope.data.base_type += "_world" @@ -100,7 +114,7 @@ def callback_function(envelope): self.assertEqual(envelope.data.base_type, "type1_world") def test_telemetry_processor_apply_multiple(self): - base = BaseExporter(storage_path=os.path.join(TEST_FOLDER, self.id())) + base = self._base base._telemetry_processors = [] def callback_function(envelope): @@ -116,7 +130,7 @@ def callback_function2(envelope): self.assertEqual(envelope.data.base_type, "type1_world_world2") def test_telemetry_processor_apply_exception(self): - base = BaseExporter(storage_path=os.path.join(TEST_FOLDER, self.id())) + base = self._base def callback_function(envelope): raise ValueError() @@ -131,7 +145,7 @@ def callback_function2(envelope): self.assertEqual(envelope.data.base_type, "type1_world2") def test_telemetry_processor_apply_not_accepted(self): - base = BaseExporter(storage_path=os.path.join(TEST_FOLDER, self.id())) + base = self._base def callback_function(envelope): return envelope.data.base_type == "type2" diff --git a/azure_monitor/tests/trace/test_trace.py b/azure_monitor/tests/trace/test_trace.py index c64b5a2..7e7d21d 100644 --- a/azure_monitor/tests/trace/test_trace.py +++ b/azure_monitor/tests/trace/test_trace.py @@ -19,6 +19,7 @@ from azure_monitor.options import ExporterOptions TEST_FOLDER = os.path.abspath(".test.exporter.trace") +STORAGE_PATH = os.path.join(TEST_FOLDER) # pylint: disable=invalid-name @@ -48,6 +49,18 @@ def setUpClass(cls): os.environ[ "APPINSIGHTS_INSTRUMENTATIONKEY" ] = "1234abcd-5678-4efa-8abc-1234567890ab" + cls._exporter = AzureMonitorSpanExporter(storage_path=STORAGE_PATH) + + def setUp(self): + for filename in os.listdir(STORAGE_PATH): + file_path = os.path.join(STORAGE_PATH, filename) + try: + if os.path.isfile(file_path) or os.path.islink(file_path): + os.unlink(file_path) + elif os.path.isdir(file_path): + shutil.rmtree(file_path) + except Exception as e: + print("Failed to delete %s. Reason: %s" % (file_path, e)) def test_constructor(self): """Test the constructor.""" @@ -65,16 +78,12 @@ def test_constructor(self): ) def test_export_empty(self): - exporter = AzureMonitorSpanExporter( - storage_path=os.path.join(TEST_FOLDER, self.id()) - ) + exporter = self._exporter exporter.export([]) self.assertEqual(len(os.listdir(exporter.storage.path)), 0) def test_export_failure(self): - exporter = AzureMonitorSpanExporter( - storage_path=os.path.join(TEST_FOLDER, self.id()) - ) + exporter = self._exporter with mock.patch( "azure_monitor.export.trace.AzureMonitorSpanExporter._transmit" ) as transmit: # noqa: E501 @@ -93,9 +102,7 @@ def test_export_failure(self): self.assertIsNone(exporter.storage.get()) def test_export_success(self): - exporter = AzureMonitorSpanExporter( - storage_path=os.path.join(TEST_FOLDER, self.id()) - ) + exporter = self._exporter test_span = Span( name="test", context=SpanContext( @@ -126,9 +133,7 @@ def test_export_exception(self, logger_mock): ) test_span.start() test_span.end() - exporter = AzureMonitorSpanExporter( - storage_path=os.path.join(TEST_FOLDER, self.id()) - ) + exporter = self._exporter with mock.patch( "azure_monitor.export.trace.AzureMonitorSpanExporter._transmit", throw(Exception), @@ -138,9 +143,7 @@ def test_export_exception(self, logger_mock): self.assertEqual(logger_mock.exception.called, True) def test_export_not_retryable(self): - exporter = AzureMonitorSpanExporter( - storage_path=os.path.join(TEST_FOLDER, self.id()) - ) + exporter = self._exporter test_span = Span( name="test", context=SpanContext( @@ -158,9 +161,7 @@ def test_export_not_retryable(self): self.assertEqual(result, SpanExportResult.FAILED_NOT_RETRYABLE) def test_span_to_envelope_none(self): - exporter = AzureMonitorSpanExporter( - storage_path=os.path.join(TEST_FOLDER, self.id()) - ) + exporter = self._exporter self.assertIsNone(exporter._span_to_envelope(None)) # pylint: disable=too-many-statements From 9ef32622ff6312b7021a68da5460d8c0a174575b Mon Sep 17 00:00:00 2001 From: Leighton Date: Fri, 27 Mar 2020 14:56:30 -0700 Subject: [PATCH 104/109] build --- tox.ini | 1 + 1 file changed, 1 insertion(+) diff --git a/tox.ini b/tox.ini index 66b65be..8c74a24 100644 --- a/tox.ini +++ b/tox.ini @@ -8,6 +8,7 @@ envlist = lint docs + [travis] python = 3.8: py38, lint, docs From 594f4757e1c769c9f1008395ba621ca699928bb7 Mon Sep 17 00:00:00 2001 From: Hector Hernandez Guzman Date: Fri, 27 Mar 2020 15:20:53 -0700 Subject: [PATCH 105/109] Adding docs link in readme --- README.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/README.md b/README.md index 6b2b4b8..80688d3 100644 --- a/README.md +++ b/README.md @@ -10,6 +10,11 @@ pip install opentelemetry-azure-monitor-exporter ``` +## Documentation + +The online documentation is available at https://opentelemetry-azure-monitor-python.readthedocs.io/. + + ## Usage ### Trace From 9ce006f8eb43cd67ffe259c7f77b43ef18b2ea37 Mon Sep 17 00:00:00 2001 From: Hector Hernandez Guzman Date: Fri, 27 Mar 2020 15:26:07 -0700 Subject: [PATCH 106/109] Adding space between references --- README.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/README.md b/README.md index 80688d3..91fd8ee 100644 --- a/README.md +++ b/README.md @@ -161,6 +161,9 @@ time.sleep(100) # References [Azure Monitor](https://docs.microsoft.com/azure/azure-monitor/) + [OpenTelemetry Project](https://opentelemetry.io/) + [OpenTelemetry Python Client](https://github.com/open-telemetry/opentelemetry-python) + [Azure Monitor Python Gitter](https://gitter.im/Microsoft/azure-monitor-python) From 54e69e969d1742d9d1c22618c15bd8b3e9a0685d Mon Sep 17 00:00:00 2001 From: Leighton Date: Fri, 27 Mar 2020 15:39:28 -0700 Subject: [PATCH 107/109] Fix lint --- azure_monitor/tests/metrics/test_metrics.py | 1 + tox.ini | 1 - 2 files changed, 1 insertion(+), 1 deletion(-) diff --git a/azure_monitor/tests/metrics/test_metrics.py b/azure_monitor/tests/metrics/test_metrics.py index e6f9691..553d3b7 100644 --- a/azure_monitor/tests/metrics/test_metrics.py +++ b/azure_monitor/tests/metrics/test_metrics.py @@ -23,6 +23,7 @@ TEST_FOLDER = os.path.abspath(".test.exporter.trace") STORAGE_PATH = os.path.join(TEST_FOLDER) + # pylint: disable=invalid-name def setUpModule(): os.makedirs(TEST_FOLDER) diff --git a/tox.ini b/tox.ini index 8c74a24..66b65be 100644 --- a/tox.ini +++ b/tox.ini @@ -8,7 +8,6 @@ envlist = lint docs - [travis] python = 3.8: py38, lint, docs From c1e90c03718ebf81a08fbb22c3407f2e84d18a53 Mon Sep 17 00:00:00 2001 From: Leighton Date: Fri, 27 Mar 2020 16:00:01 -0700 Subject: [PATCH 108/109] fix lint --- azure_monitor/tests/metrics/test_metrics.py | 2 +- azure_monitor/tests/test_base_exporter.py | 2 +- azure_monitor/tests/trace/test_trace.py | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/azure_monitor/tests/metrics/test_metrics.py b/azure_monitor/tests/metrics/test_metrics.py index 553d3b7..1984c3b 100644 --- a/azure_monitor/tests/metrics/test_metrics.py +++ b/azure_monitor/tests/metrics/test_metrics.py @@ -78,7 +78,7 @@ def setUp(self): os.unlink(file_path) elif os.path.isdir(file_path): shutil.rmtree(file_path) - except Exception as e: + except OSError as e: print("Failed to delete %s. Reason: %s" % (file_path, e)) @classmethod diff --git a/azure_monitor/tests/test_base_exporter.py b/azure_monitor/tests/test_base_exporter.py index a4655ff..8cbb6ab 100644 --- a/azure_monitor/tests/test_base_exporter.py +++ b/azure_monitor/tests/test_base_exporter.py @@ -58,7 +58,7 @@ def setUp(self): os.unlink(file_path) elif os.path.isdir(file_path): shutil.rmtree(file_path) - except Exception as e: + except OSError as e: print("Failed to delete %s. Reason: %s" % (file_path, e)) self._base.clear_telemetry_processors() diff --git a/azure_monitor/tests/trace/test_trace.py b/azure_monitor/tests/trace/test_trace.py index 7e7d21d..3e3f161 100644 --- a/azure_monitor/tests/trace/test_trace.py +++ b/azure_monitor/tests/trace/test_trace.py @@ -59,7 +59,7 @@ def setUp(self): os.unlink(file_path) elif os.path.isdir(file_path): shutil.rmtree(file_path) - except Exception as e: + except OSError as e: print("Failed to delete %s. Reason: %s" % (file_path, e)) def test_constructor(self): From e0bd6f08e489fa34190e2406ec561cdf8a6360ab Mon Sep 17 00:00:00 2001 From: Leighton Date: Tue, 31 Mar 2020 14:09:00 -0700 Subject: [PATCH 109/109] for release beta v0.2.0 --- CHANGELOG.md | 13 +++++++++++++ README.md | 6 +++--- azure_monitor/setup.cfg | 6 +++--- azure_monitor/src/azure_monitor/version.py | 2 +- 4 files changed, 20 insertions(+), 7 deletions(-) create mode 100644 CHANGELOG.md diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..ca2f99e --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,13 @@ +# Changelog + +## Unreleased + +## 0.2.0 +Released 2020-03-31 + +- Initial beta release + +## 0.1.0 +Released 2019-11-06 + +- Initial alpha release diff --git a/README.md b/README.md index 91fd8ee..112d3ff 100644 --- a/README.md +++ b/README.md @@ -1,13 +1,13 @@ # OpenTelemetry Azure Monitor SDKs and Exporters [![Gitter chat](https://img.shields.io/gitter/room/Microsoft/azure-monitor-python)](https://gitter.im/Microsoft/azure-monitor-python) -[![Build status](https://travis-ci.org/microsoft/opentelemetry-azure-monitor-python.svg?branch=master)](https://travis-ci.org/microsoft/opentelemetry-exporters-python) -[![PyPI version](https://badge.fury.io/py/opentelemetry-azure-monitor-exporter.svg)](https://badge.fury.io/py/opentelemetry-azure-monitor-exporter) +[![Build status](https://travis-ci.org/microsoft/opentelemetry-azure-monitor-python.svg?branch=master)](https://travis-ci.org/microsoft/opentelemetry-azure-monitor-python) +[![PyPI version](https://badge.fury.io/py/opentelemetry-azure-monitor.svg)](https://badge.fury.io/py/opentelemetry-azure-monitor) ## Installation ```sh -pip install opentelemetry-azure-monitor-exporter +pip install opentelemetry-azure-monitor ``` ## Documentation diff --git a/azure_monitor/setup.cfg b/azure_monitor/setup.cfg index bbb0ddb..0157b80 100644 --- a/azure_monitor/setup.cfg +++ b/azure_monitor/setup.cfg @@ -1,17 +1,17 @@ # Copyright (c) Microsoft Corporation. All rights reserved. # Licensed under the MIT License. [metadata] -name = opentelemetry-azure-monitor-exporter +name = opentelemetry-azure-monitor description = Azure Monitor integration for OpenTelemetry long_description = file: README.rst long_description_content_type = text/x-rst author = Microsoft author_email = appinsightssdk@microsoft.com -url = https://github.com/microsoft/opentelemetry-exporters-python +url = https://github.com/microsoft/opentelemetry-azure-monitor-python platforms = any license = MIT classifiers = - Development Status :: 3 - Alpha + Development Status :: 4 - Beta Intended Audience :: Developers License :: OSI Approved :: MIT License Programming Language :: Python diff --git a/azure_monitor/src/azure_monitor/version.py b/azure_monitor/src/azure_monitor/version.py index 7a5fe00..3e3cc8e 100644 --- a/azure_monitor/src/azure_monitor/version.py +++ b/azure_monitor/src/azure_monitor/version.py @@ -1,3 +1,3 @@ # Copyright (c) Microsoft Corporation. All rights reserved. # Licensed under the MIT License. -__version__ = "0.2.dev0" +__version__ = "0.2b.0"