New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Carbon returner failing when run through scheduler on log.trace #47999

Closed
arthurlogilab opened this Issue Jun 6, 2018 · 3 comments

Comments

Projects
None yet
4 participants
@arthurlogilab
Contributor

arthurlogilab commented Jun 6, 2018

Description of Issue/Question

After upgrading to 2018.3.0 the scheduled module executions with carbon returners stopped sending their returner.

Steps to Reproduce Issue

(Include debug logs if possible and relevant.)

In the minion logs I get :

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/utils/schedule.py", line 729, in handle_func
    self.returners[ret_str](ret)
  File "/usr/lib/python2.7/dist-packages/salt/returners/carbon_return.py", line 302, in returner
    _send(saltdata, metric_base, opts)
  File "/usr/lib/python2.7/dist-packages/salt/returners/carbon_return.py", line 253, in _send
    _walk(metric_base, saltdata, metrics, timestamp, skip)
  File "/usr/lib/python2.7/dist-packages/salt/returners/carbon_return.py", line 204, in _walk
    'timestamp: %s', path, value, metrics, timestamp
  File "/usr/lib/python2.7/dist-packages/salt/log/mixins.py", line 35, in trace
    self.log(getattr(logging, 'TRACE', 5), msg, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/celery/utils/log.py", line 278, in log
    return Logger.log(self, *args, **kwargs)
  File "/usr/lib/python2.7/logging/__init__.py", line 1211, in log
    Log 'msg % args' with the integer severity 'level'.
  File "/usr/lib/python2.7/logging/__init__.py", line 1224, in log
    self._log(level, msg, args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/salt/log/setup.py", line 328, in _log
    self, level, msg, args, exc_info=exc_info, extra=extra
  File "/usr/lib/python2.7/logging/__init__.py", line 1278, in _log
    record = self.makeRecord(self.name, level, fn, lno, msg, args, exc_info, func, extra)
  File "/usr/lib/python2.7/dist-packages/celery/utils/log.py", line 271, in makeRecord
    record = Logger.makeRecord(self, *args, **kwds)
  File "/usr/lib/python2.7/dist-packages/salt/log/setup.py", line 375, in makeRecord
    exc_info, func)
  File "/usr/lib/python2.7/dist-packages/salt/log/setup.py", line 209, in __init__
    self.colormsg = '%s%s%s' % (cmsg, self.getMessage(), reset)
  File "/usr/lib/python2.7/logging/__init__.py", line 335, in getMessage
    msg = msg % self.args
TypeError: not all arguments converted during string formatting

Commenting the log.trace lines fixes solves my problem (insert pink sombrero here).

Versions Report

# salt --versions-report
Salt Version:
           Salt: 2018.3.0
 
Dependency Versions:
           cffi: 0.8.6
       cherrypy: Not Installed
       dateutil: 2.4.2
      docker-py: 1.7.2
          gitdb: 2.0.0
      gitpython: 2.1.1
          ioflo: Not Installed
         Jinja2: 2.9.4
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: 2.10
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.9 (default, Jun 29 2016, 13:08:31)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 14.4.0
           RAET: Not Installed
          smmap: 2.0.1
        timelib: Not Installed
        Tornado: 4.4.3
            ZMQ: 4.0.5
 
System Versions:
           dist: debian 8.10 
         locale: UTF-8
        machine: x86_64
        release: 4.9.0-0.bpo.6-amd64
         system: Linux
        version: debian 8.10 


@garethgreenaway

This comment has been minimized.

Member

garethgreenaway commented Jun 6, 2018

@arthurlogilab Thanks for the report. Looks like there is an issue with that log statement that you commented out, too many arguments perhaps. Which one was it exactly in which file?

@kuetrzi

This comment has been minimized.

Contributor

kuetrzi commented Jul 19, 2018

from
#c65d471

it looks like an extra comma was added.

-    log.trace('Carbon return walking path: {0}, value: {1}, metrics: {2}, timestamp: {3}'.format(path, value, metrics, timestamp))
+    log.trace(
+        'Carbon return walking path: %s, value: %s, metrics: %s, ',
+        'timestamp: %s', path, value, metrics, timestamp
+    )

code should be:
    log.trace(
        'Carbon return walking path: %s, value: %s, metrics: %s, '             
         'timestamp: %s', path, value, metrics, timestamp 
        )

patch.txt

@rallytime

This comment has been minimized.

Contributor

rallytime commented Jul 20, 2018

@kuetrzi Would you like to make a PR to fix that?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment