parent cca8025

author zkscpqm <qmcpzks@naver.com> 1595084266 +0300
committer kezhenxu94 <kezhenxu94@163.com> 1596458512 +0800
gpgsig -----BEGIN PGP SIGNATURE-----

 iQIzBAABCAAdFiEE/Eo3KEa/qP7Oqvu/Ng+pfz8mM0wFAl8oBhAACgkQNg+pfz8m
 M0y/bBAAoBBwE54/GJ00JQ8tgn5xqNo1bfqlBcNa6obvXJbkvFwt/knVEU+fuKxm
 p/jGahr2EyHzhXTh4XUo5xxlK+xgTid5JHRopjgTY3MgbEG/397tcS2+1P8NQUCB
 Lh+OxX5V4Gm0GqvFP02zrsHjBiJYL1TVz4R8PomuE+w7fuNyuqPtNMUt5ILsZz/3
 X3X7JJnbJqjKK5fEcerPKPUL3Ee+muazoCuR5KFGJ6PbxrVu/YBG3UTTeOhVonOb
 r7VKicGJSKQZD3nCH8cE56Qwt3uoQE1W0lQ/L1RwE++ixa+vk19FpPlwoup/eYac
 vYTGjMSgloCpiksGmhLhPU5bOlMxoArqJQg2zHayPfrdFNfgu1qfu+fMciU+8A11
 OxhLEjLxAnlEO7wtjcUuVD+siVlmkKeRBxffsoiZTXWFQVbbNnM9Sgrc70ql9KwK
 uCi3Bcfr2gJctC+vrHEBO2AJX4HtSuhYtpGzKAPc0SHb2niKqqFFZHWa+t5db7zg
 ixxUrT5j9OSAi4TzDi4/QLEjCt0ECOxJJpeUFMXn4SLHYiSaU5lgV+H5VOxEKotj
 w3dpHU1J1/mylZ9TauFX703vIc/il8iqKeqc0JFgSa25F0ZjlyfI8MIkwfHA8CNS
 MTMjnwmcDpOgc/AADPqAz+kwLYLARlvBUkz5uhwqI8slXkpwnrY=
 =4v88
 -----END PGP SIGNATURE-----

Created venv builders for linux/windows and req flashers + use documentation (#38)

* Created venv builders for linux/windows and req flashers + use documentation

* aadd license headers to scripts

* moved scripts, updated paths, changed how reqs are installed, updated linux venv bin path

* update license checker to accomodate for shebangs and batchfile comments

Add ability to collect SQL Parameters in PyMsql plugin (#39)

Add plugin for Django Agent (#37)

Add ignore_suffix Config (#40)

Add Django in document (#42)

Add Flask Http Params (#43)

* Add Flask Http Params

* Add Flask Http Params

Co-authored-by: huawei <huawei@bit-s.cn>

Add Redis Plugin (#44)

Update Redis plugin Component Id (#45)

Test: print the diff list when validation failed (#46)

feature: add Tornado Plugin (#48)

feature: add Kafka Plugin (#50)

Enable Django collect http parameters (#49)

Add changelog for 0.2.0 (#51)

Make plugin compatible with Django < 2.2 (#52)

Add Rabbitmq Plugin (#53)

chore: add make goal to package release tar ball (#54)

Support propagating correlation context (#55)

Sync

Add changelog for 0.2.0 (#51)

Make plugin compatible with Django < 2.2 (#52)

Add Rabbitmq Plugin (#53)

chore: add make goal to package release tar ball (#54)

Support propagating correlation context (#55)

Sync

.gitignore

@@ -7,4 +7,5 @@ __pycache__
 build/
 dist/
 *.pyc
-/apache_skywalking.egg-info/
+/apache_skywalking.egg-info/
+**/venv/

CHANGELOG.md

@@ -1,5 +1,25 @@
 ## Change Logs
 
+### 0.2.0
+
+- New plugins
+    - Kafka Plugin (#50)
+    - Tornado Plugin (#48)
+    - Redis Plugin (#44)
+    - Django Plugin (#37)
+    - PyMsql Plugin (#35)
+    - Flask plugin (#31)
+
+- API
+    - Add ignore_suffix Config (#40)
+    - Add missing `log` method and simplify test codes (#34)
+    - Add content equality of SegmentRef (#30)
+    - Validate carrier before using it (#29)
+
+- Chores and tests
+    - Test: print the diff list when validation failed (#46)
+    - Created venv builders for linux/windows and req flashers + use documentation (#38)
+
 ### 0.1.0
 
 - API: agent core APIs, check [the APIs and the examples](https://github.com/apache/skywalking-python/blob/3892cab9d5d2c03107cfb2b1c59a6c77c5c3cc35/README.md#api)

Makefile

@@ -14,6 +14,8 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+VERSION ?= latest
+
 .PHONY: license
 
 setup:
@@ -59,4 +61,11 @@ upload: package
 clean:
 	rm -rf browser common language_agent management profile service_mesh_probe
 	rm -rf skywalking_python.egg-info dist build
-	find . -type d -name  "__pycache__" -exec rm -r {} +
+	rm -rf skywalking-python*.tgz*
+	find . -name "__pycache__" -exec rm -r {} +
+	find . -name "*.pyc" -exec rm -r {} +
+
+release: clean lint license
+	-tar -zcvf skywalking-python-src-$(VERSION).tgz *
+	gpg --batch --yes --armor --detach-sig skywalking-python-src-$(VERSION).tgz
+	shasum -a 512 skywalking-python-src-$(VERSION).tgz > skywalking-python-src-$(VERSION).tgz.tgz.sha512

README.md

@@ -58,6 +58,15 @@ Environment Variable | Description | Default
 | `SW_AGENT_AUTHENTICATION` | The authentication token to verify that the agent is trusted by the backend OAP, as for how to configure the backend, refer to [the yaml](https://github.com/apache/skywalking/blob/4f0f39ffccdc9b41049903cc540b8904f7c9728e/oap-server/server-bootstrap/src/main/resources/application.yml#L155-L158). | not set |
 | `SW_AGENT_LOGGING_LEVEL` | The logging level, could be one of `CRITICAL`, `FATAL`, `ERROR`, `WARN`(`WARNING`), `INFO`, `DEBUG` | `INFO` |
 | `SW_AGENT_DISABLE_PLUGINS` | The name patterns in CSV pattern, plugins whose name matches one of the pattern won't be installed | `''` |
+| `SW_MYSQL_TRACE_SQL_PARAMETERS` | Indicates whether to collect the sql parameters or not | `False` |
+| `SW_MYSQL_SQL_PARAMETERS_MAX_LENGTH` | The maximum length of the collected parameter, parameters longer than the specified length will be truncated | `512` |
+| `SW_IGNORE_SUFFIX` | If the operation name of the first span is included in this set, this segment should be ignored. | `.jpg,.jpeg,.js,.css,.png,.bmp,.gif,.ico,.mp3,.mp4,.html,.svg` |
+| `SW_FLASK_COLLECT_HTTP_PARAMS`| This config item controls that whether the Flask plugin should collect the parameters of the request.| `false` |
+| `SW_DJANGO_COLLECT_HTTP_PARAMS`| This config item controls that whether the Django plugin should collect the parameters of the request.| `false` |
+| `SW_HTTP_PARAMS_LENGTH_THRESHOLD`| When `COLLECT_HTTP_PARAMS` is enabled, how many characters to keep and send to the OAP backend, use negative values to keep and send the complete parameters, NB. this config item is added for the sake of performance.  | `1024` |
+| `SW_CORRELATION_ELEMENT_MAX_NUMBER`|Max element count of the correlation context.| `3` |
+| `SW_CORRELATION_VALUE_MAX_LENGTH`| Max value length of correlation context element.| `128` |
+
 
 ## Supported Libraries
 
@@ -70,6 +79,11 @@ Library | Plugin Name
 | [requests](https://requests.readthedocs.io/en/master/) | `sw_requests` |
 | [Flask](https://flask.palletsprojects.com/en/1.1.x/) | `sw_flask` |
 | [PyMySQL](https://pymysql.readthedocs.io/en/latest/) | `sw_pymysql` |
+| [Django](https://www.djangoproject.com/) | `sw_django` |
+| [redis-py](https://github.com/andymccurdy/redis-py/) | `sw_redis` |
+| [kafka-python](https://kafka-python.readthedocs.io/en/master/) | `sw_kafka` |
+| [tornado](https://www.tornadoweb.org/en/stable/) | `sw_tornado` |
+| [pika](https://pika.readthedocs.io/en/stable/) | `sw_rabbitmq` |
 
 ## API
 
@@ -127,5 +141,22 @@ with context.new_entry_span(op=str('https://github.com/apache/skywalking')) as s
 
 Check [the FAQ page](docs/FAQ.md) or add the FAQs there.
 
+## For Developers
+
+### Steps to get an operational virtual environment:
+
+1. `git clone https://github.com/apache/skywalking-python.git`
+2. `cd skywalking-python/tools/env` (*make sure you actually go into the directory since the scripts use relative paths*)
+3. Run the script for your relevant OS to create a virtual environment folder in the project root (*skywalking-python/venv*) and install all the necessary requirements
+
+**Make sure that when the `python` command is executed on your workstation, the binary it references is python 3.5 or newer!**
+
+### Steps after contributing
+
+If your PR introduces the need for a new non-standard library which needs to be pulled via pip or if it removes the need for a previously-used library:
+1. navigate to `/path/to/skywalking/agent/tools/env`
+2. Execute the `build_requirements` script relevant to your OS.
+3. Double check the `requirements.txt` file in the project root to ensure that the changes have been reflected. 
+
 ## License
 Apache 2.0

requirements.txt

@@ -0,0 +1,23 @@
+blindspin==2.0.1
+certifi==2020.6.20
+chardet==3.0.4
+colorama==0.4.3
+crayons==0.3.1
+deprecation==2.1.0
+docker==4.2.2
+grpcio==1.30.0
+grpcio-tools==1.30.0
+idna==2.10
+packaging==20.4
+protobuf==3.12.2
+PyMySQL==0.9.3
+pyparsing==2.4.7
+pypiwin32==223
+pywin32==228
+requests==2.24.0
+six==1.15.0
+testcontainers==3.0.3
+urllib3==1.25.9
+websocket-client==0.57.0
+Werkzeug==1.0.1
+wrapt==1.12.1

setup.py

@@ -25,7 +25,7 @@
 
 setup(
     name="apache-skywalking",
-    version="0.1.0",
+    version="0.2.0",
     description="Python Agent for Apache SkyWalking",
     long_description=README,
     long_description_content_type="text/markdown",
@@ -43,8 +43,15 @@
     extras_require={
         "test": [
             "testcontainers",
+            "pyyaml",
+            "django",
+            "flask",
             "Werkzeug",
             "pymysql",
+            "redis",
+            "kafka-python",
+            "tornado",
+            "pika",
         ],
     },
     classifiers=[

skywalking/__init__.py

@@ -27,6 +27,13 @@ class Component(Enum):
     Flask = 7001
     Requests = 7002
     PyMysql = 7003
+    Django = 7004
+    Tornado = 7005
+    Redis = 7
+    KafkaProducer = 40
+    KafkaConsumer = 41
+    RabbitmqProducer = 52
+    RabbitmqConsumer = 53
 
 
 class Layer(Enum):

skywalking/config.py

@@ -26,6 +26,18 @@
 authentication = os.getenv('SW_AGENT_AUTHENTICATION')  # type: str
 logging_level = os.getenv('SW_AGENT_LOGGING_LEVEL') or 'INFO'  # type: str
 disable_plugins = (os.getenv('SW_AGENT_DISABLE_PLUGINS') or '').split(',')  # type: List[str]
+mysql_trace_sql_parameters = True if os.getenv('SW_MYSQL_TRACE_SQL_PARAMETERS') and \
+                                     os.getenv('SW_MYSQL_TRACE_SQL_PARAMETERS') == 'True' else False  # type: bool
+mysql_sql_parameters_max_length = int(os.getenv('SW_MYSQL_SQL_PARAMETERS_MAX_LENGTH') or '512')  # type: int
+ignore_suffix = os.getenv('SW_IGNORE_SUFFIX') or '.jpg,.jpeg,.js,.css,.png,.bmp,.gif,.ico,.mp3,' \
+                                                 '.mp4,.html,.svg '  # type: str
+flask_collect_http_params = True if os.getenv('SW_FLASK_COLLECT_HTTP_PARAMS') and \
+                                    os.getenv('SW_FLASK_COLLECT_HTTP_PARAMS') == 'True' else False   # type: bool
+http_params_length_threshold = int(os.getenv('SW_HTTP_PARAMS_LENGTH_THRESHOLD') or '1024')  # type: int
+django_collect_http_params = True if os.getenv('SW_DJANGO_COLLECT_HTTP_PARAMS') and \
+                                    os.getenv('SW_DJANGO_COLLECT_HTTP_PARAMS') == 'True' else False   # type: bool
+correlation_element_max_number = int(os.getenv('SW_CORRELATION_ELEMENT_MAX_NUMBER') or '3')  # type: int
+correlation_value_max_length = int(os.getenv('SW_CORRELATION_VALUE_MAX_LENGTH') or '128')  # type: int
 
 
 def init(

skywalking/plugins/sw_django.py

@@ -0,0 +1,85 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+import logging
+
+from skywalking import Layer, Component, config
+from skywalking.trace import tags
+from skywalking.trace.carrier import Carrier
+from skywalking.trace.context import get_context
+from skywalking.trace.tags import Tag
+
+logger = logging.getLogger(__name__)
+
+
+def install():
+    try:
+        from django.core.handlers.base import BaseHandler
+        from django.core.handlers import exception
+
+        _get_response = BaseHandler.get_response
+        _handle_uncaught_exception = exception.handle_uncaught_exception
+
+        def _sw_get_response(this, request):
+            if request is None:
+                resp = _get_response(this, request)
+                return resp
+
+            context = get_context()
+            carrier = Carrier()
+            for item in carrier:
+                # Any HTTP headers in the request are converted to META keys by converting all characters to uppercase,
+                # replacing any hyphens with underscores and adding an HTTP_ prefix to the name.
+                # https://docs.djangoproject.com/en/3.0/ref/request-response/#django.http.HttpRequest.META
+                sw_http_header_key = 'HTTP_%s' % item.key.upper().replace('-', '_')
+                if sw_http_header_key in request.META:
+                    item.val = request.META[sw_http_header_key]
+
+            with context.new_entry_span(op=request.path, carrier=carrier) as span:
+                span.layer = Layer.Http
+                span.component = Component.Django
+                span.peer = '%s:%s' % (request.META.get('REMOTE_ADDR'), request.META.get('REMOTE_PORT') or "80")
+
+                span.tag(Tag(key=tags.HttpMethod, val=request.method))
+                span.tag(Tag(key=tags.HttpUrl, val=request.build_absolute_uri().split("?")[0]))
+
+                # you can get request parameters by `request.GET` even though client are using POST or other methods
+                if config.django_collect_http_params and request.GET:
+                    span.tag(Tag(key=tags.HttpParams,
+                                 val=params_tostring(request.GET)[0:config.http_params_length_threshold]))
+
+                resp = _get_response(this, request)
+                span.tag(Tag(key=tags.HttpStatus, val=resp.status_code))
+                if resp.status_code >= 400:
+                    span.error_occurred = True
+                return resp
+
+        def _sw_handle_uncaught_exception(request, resolver, exc_info):
+            if exc_info is not None:
+                entry_span = get_context().active_span()
+                if entry_span is not None:
+                    entry_span.raised()
+
+            return _handle_uncaught_exception(request, resolver, exc_info)
+
+        BaseHandler.get_response = _sw_get_response
+        exception.handle_uncaught_exception = _sw_handle_uncaught_exception
+    except Exception:
+        logger.warning('failed to install plugin %s', __name__)
+
+
+def params_tostring(params):
+    return "\n".join([k + '=[' + ",".join(params.getlist(k)) + ']' for k, _ in params.items()])

skywalking/plugins/sw_flask.py

@@ -16,7 +16,7 @@
 #
 import logging
 
-from skywalking import Layer, Component
+from skywalking import Layer, Component, config
 from skywalking.trace import tags
 from skywalking.trace.carrier import Carrier
 from skywalking.trace.context import get_context
@@ -34,6 +34,9 @@ def install():
 
         _handle_user_exception = Flask.handle_user_exception
 
+        def params_tostring(params):
+            return "\n".join([k + '=[' + ",".join(params.getlist(k)) + ']' for k, _ in params.items()])
+
         def _sw_full_dispatch_request(this: Flask):
             import flask
             req = flask.request
@@ -48,7 +51,10 @@ def _sw_full_dispatch_request(this: Flask):
                 span.component = Component.Flask
                 span.peer = '%s:%s' % (req.environ["REMOTE_ADDR"], req.environ["REMOTE_PORT"])
                 span.tag(Tag(key=tags.HttpMethod, val=req.method))
-                span.tag(Tag(key=tags.HttpUrl, val=req.url))
+                span.tag(Tag(key=tags.HttpUrl, val=req.url.split("?")[0]))
+                if config.flask_collect_http_params and req.values:
+                    span.tag(Tag(key=tags.HttpParams,
+                             val=params_tostring(req.values)[0:config.http_params_length_threshold]))
                 resp = _full_dispatch_request(this)
 
                 if resp.status_code >= 400: