Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

syslog presence detection does not work #9317

Closed
2 tasks done
weijiang1994 opened this issue May 26, 2023 · 8 comments
Closed
2 tasks done

syslog presence detection does not work #9317

weijiang1994 opened this issue May 26, 2023 · 8 comments
Assignees
@nijel
Labels
bug Something is broken.
Milestone
4.18

Comments

@weijiang1994
Copy link
Contributor

Describe the issue

[2023-05-26 16:38:29,950: WARNING/ForkPoolWorker-7] --- Logging error ---
[2023-05-26 16:38:29,951: WARNING/ForkPoolWorker-7] Traceback (most recent call last):
[2023-05-26 16:38:29,951: WARNING/ForkPoolWorker-7] File "/usr/lib/python3.8/logging/handlers.py", line 934, in emit
self.socket.send(msg)
[2023-05-26 16:38:29,951: WARNING/ForkPoolWorker-7] OSError: [Errno 9] Bad file descriptor
[2023-05-26 16:38:29,951: WARNING/ForkPoolWorker-7]
During handling of the above exception, another exception occurred:
[2023-05-26 16:38:29,951: WARNING/ForkPoolWorker-7] Traceback (most recent call last):
[2023-05-26 16:38:29,951: WARNING/ForkPoolWorker-7] File "/usr/lib/python3.8/logging/handlers.py", line 855, in _connect_unixsocket
self.socket.connect(address)
[2023-05-26 16:38:29,951: WARNING/ForkPoolWorker-7] FileNotFoundError: [Errno 2] No such file or directory
[2023-05-26 16:38:29,951: WARNING/ForkPoolWorker-7]
During handling of the above exception, another exception occurred:
[2023-05-26 16:38:29,951: WARNING/ForkPoolWorker-7] Traceback (most recent call last):
[2023-05-26 16:38:29,951: WARNING/ForkPoolWorker-7] File "/usr/lib/python3.8/logging/handlers.py", line 937, in emit
self._connect_unixsocket(self.address)
[2023-05-26 16:38:29,951: WARNING/ForkPoolWorker-7] File "/usr/lib/python3.8/logging/handlers.py", line 866, in _connect_unixsocket
self.socket.connect(address)
[2023-05-26 16:38:29,952: WARNING/ForkPoolWorker-7] FileNotFoundError: [Errno 2] No such file or directory
[2023-05-26 16:38:29,952: WARNING/ForkPoolWorker-7] Call stack:
[2023-05-26 16:38:29,954: WARNING/ForkPoolWorker-7] File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
[2023-05-26 16:38:29,954: WARNING/ForkPoolWorker-7] File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
[2023-05-26 16:38:29,954: WARNING/ForkPoolWorker-7] File "/root/weblate/venv/lib/python3.8/site-packages/celery/main.py", line 19, in
main()
[2023-05-26 16:38:29,954: WARNING/ForkPoolWorker-7] File "/root/weblate/venv/lib/python3.8/site-packages/celery/main.py", line 15, in main
sys.exit(_main())
[2023-05-26 16:38:29,954: WARNING/ForkPoolWorker-7] File "/root/weblate/venv/lib/python3.8/site-packages/celery/bin/celery.py", line 217, in main
return celery(auto_envvar_prefix="CELERY")
[2023-05-26 16:38:29,954: WARNING/ForkPoolWorker-7] File "/root/weblate/venv/lib/python3.8/site-packages/click/core.py", line 1130, in call
return self.main(*args, **kwargs)
[2023-05-26 16:38:29,954: WARNING/ForkPoolWorker-7] File "/root/weblate/venv/lib/python3.8/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
[2023-05-26 16:38:29,954: WARNING/ForkPoolWorker-7] File "/root/weblate/venv/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
[2023-05-26 16:38:29,955: WARNING/ForkPoolWorker-7] File "/root/weblate/venv/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
[2023-05-26 16:38:29,955: WARNING/ForkPoolWorker-7] File "/root/weblate/venv/lib/python3.8/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
[2023-05-26 16:38:29,955: WARNING/ForkPoolWorker-7] File "/root/weblate/venv/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
return f(get_current_context(), *args, **kwargs)
[2023-05-26 16:38:29,955: WARNING/ForkPoolWorker-7] File "/root/weblate/venv/lib/python3.8/site-packages/celery/bin/base.py", line 134, in caller
return f(ctx, *args, **kwargs)
[2023-05-26 16:38:29,955: WARNING/ForkPoolWorker-7] File "/root/weblate/venv/lib/python3.8/site-packages/celery/bin/worker.py", line 351, in worker
worker.start()
[2023-05-26 16:38:29,955: WARNING/ForkPoolWorker-7] File "/root/weblate/venv/lib/python3.8/site-packages/celery/worker/worker.py", line 203, in start
self.blueprint.start(self)
[2023-05-26 16:38:29,955: WARNING/ForkPoolWorker-7] File "/root/weblate/venv/lib/python3.8/site-packages/celery/bootsteps.py", line 116, in start
step.start(parent)
[2023-05-26 16:38:29,955: WARNING/ForkPoolWorker-7] File "/root/weblate/venv/lib/python3.8/site-packages/celery/bootsteps.py", line 365, in start
return self.obj.start()
[2023-05-26 16:38:29,955: WARNING/ForkPoolWorker-7] File "/root/weblate/venv/lib/python3.8/site-packages/celery/concurrency/base.py", line 129, in start
self.on_start()
[2023-05-26 16:38:29,955: WARNING/ForkPoolWorker-7] File "/root/weblate/venv/lib/python3.8/site-packages/celery/concurrency/prefork.py", line 109, in on_start
P = self._pool = Pool(processes=self.limit,
[2023-05-26 16:38:29,955: WARNING/ForkPoolWorker-7] File "/root/weblate/venv/lib/python3.8/site-packages/celery/concurrency/asynpool.py", line 463, in init
super().init(processes, *args, **kwargs)
[2023-05-26 16:38:29,955: WARNING/ForkPoolWorker-7] File "/root/weblate/venv/lib/python3.8/site-packages/billiard/pool.py", line 1046, in init
self._create_worker_process(i)
[2023-05-26 16:38:29,955: WARNING/ForkPoolWorker-7] File "/root/weblate/venv/lib/python3.8/site-packages/celery/concurrency/asynpool.py", line 480, in _create_worker_process
return super()._create_worker_process(i)
[2023-05-26 16:38:29,955: WARNING/ForkPoolWorker-7] File "/root/weblate/venv/lib/python3.8/site-packages/billiard/pool.py", line 1158, in _create_worker_process
w.start()
[2023-05-26 16:38:29,955: WARNING/ForkPoolWorker-7] File "/root/weblate/venv/lib/python3.8/site-packages/billiard/process.py", line 124, in start
self._popen = self._Popen(self)
[2023-05-26 16:38:29,955: WARNING/ForkPoolWorker-7] File "/root/weblate/venv/lib/python3.8/site-packages/billiard/context.py", line 333, in _Popen
return Popen(process_obj)
[2023-05-26 16:38:29,956: WARNING/ForkPoolWorker-7] File "/root/weblate/venv/lib/python3.8/site-packages/billiard/popen_fork.py", line 24, in init
self._launch(process_obj)
[2023-05-26 16:38:29,956: WARNING/ForkPoolWorker-7] File "/root/weblate/venv/lib/python3.8/site-packages/billiard/popen_fork.py", line 79, in _launch
code = process_obj._bootstrap()
[2023-05-26 16:38:29,956: WARNING/ForkPoolWorker-7] File "/root/weblate/venv/lib/python3.8/site-packages/billiard/process.py", line 327, in _bootstrap
self.run()
[2023-05-26 16:38:29,956: WARNING/ForkPoolWorker-7] File "/root/weblate/venv/lib/python3.8/site-packages/billiard/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
[2023-05-26 16:38:29,956: WARNING/ForkPoolWorker-7] File "/root/weblate/venv/lib/python3.8/site-packages/billiard/pool.py", line 292, in call
sys.exit(self.workloop(pid=pid))
[2023-05-26 16:38:29,956: WARNING/ForkPoolWorker-7] File "/root/weblate/venv/lib/python3.8/site-packages/billiard/pool.py", line 362, in workloop
result = (True, prepare_result(fun(*args, **kwargs)))
[2023-05-26 16:38:29,956: WARNING/ForkPoolWorker-7] File "/root/weblate/venv/lib/python3.8/site-packages/celery/app/trace.py", line 649, in fast_trace_task
R, I, T, Rstr = tasks[task].trace(
[2023-05-26 16:38:29,956: WARNING/ForkPoolWorker-7] File "/root/weblate/venv/lib/python3.8/site-packages/celery/app/trace.py", line 451, in trace_task
R = retval = fun(*args, **kwargs)
[2023-05-26 16:38:29,956: WARNING/ForkPoolWorker-7] File "/root/weblate/venv/lib/python3.8/site-packages/celery/app/trace.py", line 734, in protected_call
return self.run(*args, **kwargs)
[2023-05-26 16:38:29,956: WARNING/ForkPoolWorker-7] File "/root/weblate/venv/lib/python3.8/site-packages/celery/app/autoretry.py", line 34, in run
return task._orig_run(*args, **kwargs)
[2023-05-26 16:38:29,956: WARNING/ForkPoolWorker-7] File "/root/weblate/weblate/trans/tasks.py", line 397, in auto_translate
auto.process_mt(engines, threshold, source)
[2023-05-26 16:38:29,956: WARNING/ForkPoolWorker-7] File "/root/weblate/weblate/trans/autotranslate.py", line 249, in process_mt
self.post_process()
[2023-05-26 16:38:29,956: WARNING/ForkPoolWorker-7] File "/root/weblate/weblate/trans/autotranslate.py", line 79, in post_process
self.translation.component.update_source_checks()
[2023-05-26 16:38:29,956: WARNING/ForkPoolWorker-7] File "/root/weblate/weblate/trans/models/component.py", line 1842, in update_source_checks
self.log_info("running source checks for %d strings", len(self.updated_sources))
[2023-05-26 16:38:29,956: WARNING/ForkPoolWorker-7] File "/root/weblate/weblate/trans/mixins.py", line 94, in log_info
return LOGGER.info(": ".join((self.full_slug, msg)), *args)

I already tried

  • I've read and searched the documentation.
  • I've searched for similar issues in this repository.

Steps to reproduce the behavior

Use auto translate tool and waiting the translate completed, but I found a error in the weblate-translate.log file. Sometimes the auto translate task not completed, but sometimes can completed.

Expected behavior

No response

Screenshots

No response

Exception traceback

No response

How do you run Weblate?

Docker container

Weblate versions

No response

Weblate deploy checks

No response

Additional context

No response
@weijiang1994 weijiang1994 added the question This is more a question for the support than an issue. label May 26, 2023
@github-actions
Copy link

This issue looks more like a support question than an issue. We strive to answer these reasonably fast, but purchasing the support subscription is not only more responsible and faster for your business but also makes Weblate stronger.

In case your question is already answered, making a donation is the right way to say thank you!

@nijel
Copy link
Member

nijel commented May 29, 2023

What is your logging configuration? Is system running syslog?

@weijiang1994
Copy link
Contributor Author

@nijel Thanks for your response. I use the default configuration below here.

HAVE_SYSLOG = False
if platform.system() != "Windows":
    try:
        handler = SysLogHandler(address="/dev/log", facility=SysLogHandler.LOG_LOCAL2)
        handler.close()
        HAVE_SYSLOG = True
    except OSError:
        HAVE_SYSLOG = False

if DEBUG or not HAVE_SYSLOG:
    DEFAULT_LOG = "console"
else:
    DEFAULT_LOG = "syslog"
DEFAULT_LOGLEVEL = "DEBUG" if DEBUG else "INFO"

LOGGING = {
    "version": 1,
    "disable_existing_loggers": True,
    "filters": {"require_debug_false": {"()": "django.utils.log.RequireDebugFalse"}},
    "formatters": {
        "syslog": {"format": "weblate[%(process)d]: %(levelname)s %(message)s"},
        "simple": {"format": "[%(asctime)s: %(levelname)s/%(process)s] %(message)s"},
        "logfile": {"format": "%(asctime)s %(levelname)s %(message)s"},
        "django.server": {
            "()": "django.utils.log.ServerFormatter",
            "format": "[%(server_time)s] %(message)s",
        },
    },
    "handlers": {
        "mail_admins": {
            "level": "ERROR",
            "filters": ["require_debug_false"],
            "class": "django.utils.log.AdminEmailHandler",
            "include_html": True,
        },
        "console": {
            "level": "DEBUG",
            "class": "logging.StreamHandler",
            "formatter": "simple",
        },
        "django.server": {
            "level": "INFO",
            "class": "logging.StreamHandler",
            "formatter": "django.server",
        },
        "syslog": {
            "level": "DEBUG",
            "class": "logging.handlers.SysLogHandler",
            "formatter": "syslog",
            "address": "/dev/log",
            "facility": SysLogHandler.LOG_LOCAL2,
        },
        # Logging to a file
        # "logfile": {
        #     "level":"DEBUG",
        #     "class":"logging.handlers.RotatingFileHandler",
        #     "filename": "/var/log/weblate/weblate.log",
        #     "maxBytes": 100000,
        #     "backupCount": 3,
        #     "formatter": "logfile",
        # },
    },
    "loggers": {
        "django.request": {
            "handlers": ["mail_admins", DEFAULT_LOG],
            "level": "ERROR",
            "propagate": True,
        },
        "django.server": {
            "handlers": ["django.server"],
            "level": "INFO",
            "propagate": False,
        },
        # Logging database queries
        # "django.db.backends": {
        #     "handlers": [DEFAULT_LOG],
        #     "level": "DEBUG",
        # },
        "weblate": {"handlers": [DEFAULT_LOG], "level": DEFAULT_LOGLEVEL},
        # Logging VCS operations
        "weblate.vcs": {"handlers": [DEFAULT_LOG], "level": DEFAULT_LOGLEVEL},
        # Python Social Auth
        "social": {"handlers": [DEFAULT_LOG], "level": DEFAULT_LOGLEVEL},
        # Django Authentication Using LDAP
        "django_auth_ldap": {"handlers": [DEFAULT_LOG], "level": DEFAULT_LOGLEVEL},
        # SAML IdP
        "djangosaml2idp": {"handlers": [DEFAULT_LOG], "level": DEFAULT_LOGLEVEL},
    },
}

weblate/weblate/settings_example.py

Line 423 in 332d8b1

HAVE_SYSLOG = False

@nijel
Copy link
Member

nijel commented Jun 1, 2023

It seems that syslog was working when Weblate has started, but it is no longer working now. You can disable syslog logging manually by changing DEFAULT_LOG after the autodetection.

@weijiang1994
Copy link
Contributor Author

Mayby I know why this error will occur.

  try:
      handler = SysLogHandler(address="/dev/log", facility=SysLogHandler.LOG_LOCAL2)
      handler.close()
      HAVE_SYSLOG = True
  except OSError:
      HAVE_SYSLOG = False

This code maybe can not detect the syslog is work or syslogd is existed. I execute this code in my docker container, the syslogd is not exist, but this code run well.

import logging
import logging.handlers

logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)

handler = logging.handlers.SysLogHandler('/dev/log')
handler.setFormatter(logging.Formatter('%(asctime)s %(levelname)s %(message)s'))
logger.addHandler(handler)

logger.info('This message will be sent to Syslog on the host machine')

When I execute this code, this error will occur and seem like to above issue's error.
image
Thansk for your help, I know how to solve this problem! ❤️

@nijel
Copy link
Member

nijel commented Jun 1, 2023

Hmm, the detection used to work, but the initialization hides the error since python/cpython@1b038e0 (Python 3.7).

@nijel nijel changed the title autotranslate log error syslog presence detection does not work Jun 1, 2023
@nijel nijel added bug Something is broken. and removed question This is more a question for the support than an issue. labels Jun 1, 2023
@weijiang1994
Copy link
Contributor Author

Yes, My python version is 3.8.10, mayby need to check python version before detect syslog.

@nijel nijel closed this as completed in 89784b4 Jun 1, 2023
@github-actions
Copy link

github-actions bot commented Jun 1, 2023

Thank you for your report; the issue you have reported has just been fixed.

  • In case you see a problem with the fix, please comment on this issue.
  • In case you see a similar problem, please open a separate issue.
  • If you are happy with the outcome, don’t hesitate to support Weblate by making a donation.

@nijel nijel added this to the 4.18 milestone Jun 1, 2023
@nijel nijel self-assigned this Jun 1, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nijel nijel

Labels
bug Something is broken.
Projects
None yet
Milestone
4.18
Development

No branches or pull requests
2 participants
@nijel @weijiang1994