Daily Message sending limit is not reset every day

[AmazingBeu]

On my Modoboa server, the policyd upload limit is not reset every day.

So, my sending limit is lowered each time I send a mail, and never goes back up.

[tonioo]

@AmazingBeu How do you start the policy daemon?

[AmazingBeu]

Im using supervisord as you recommend in the doc:

[program:policyd]
autostart=true
autorestart=true
command=/srv/modoboa/env/bin/python /srv/modoboa/instance/manage.py policy_daemon --host localhost
directory=/srv/modoboa
redirect_stderr=true
user=modoboa
numprocs=1

And I tried in CLI with the following command:

sudo -u modoboa /srv/modoboa/env/bin/python /srv/modoboa/instance/manage.py policy_daemon --debug -v3 --host localhost

And I never had any logs, even with the debug and verbosity options.
BTW I also tried without the host option.

But the port 9999 is listening and postfix is using it without errors in logs.

[tonioo]

@AmazingBeu Logs should be contained in the /var/log/mail.log file, if you configured the logging part in django. Can you check if you see a 'modoboa.policyd' entry in the LOGGING setting?

[AmazingBeu]

@tonioo No, I only have:

LOGGING = {
    'version': 1,
    'formatters': {
        'syslog': {
            'format': '%(name)s: %(levelname)s %(message)s'
        },
    },
    'handlers': {
        'syslog-auth': {
            'class': 'logging.handlers.SysLogHandler',
            'facility': SysLogHandler.LOG_AUTH,
            'formatter': 'syslog'
        },
        'modoboa': {
            'class': 'modoboa.core.loggers.SQLHandler',
        }
    },
    'loggers': {
        'modoboa.auth': {
            'handlers': ['syslog-auth', 'modoboa'],
            'level': 'INFO',
            'propagate': False
        },
        'modoboa.admin': {
            'handlers': ['modoboa'],
            'level': 'INFO',
            'propagate': False
        }
    }
}

[tonioo]

Looks like I forgot to mention this in the documentation... You can get an example here: https://github.com/modoboa/modoboa/blob/master/modoboa/core/commands/templates/settings.py.tpl#L319.

[AmazingBeu]

@tonioo OK I changed all my LOGGING section for the ones you gave me, I restarted supervisor and uwsgi but i have no more logs.

[tonioo]

You mean in the mail.log file?

[AmazingBeu]

Yes, nothing more than before (postfix, dovecot and others)

[tonioo]

@AmazingBeu If logging is configured correctly, you should see a startup message at least. (see https://github.com/modoboa/modoboa/blob/master/modoboa/policyd/management/commands/policy_daemon.py#L50)

[AmazingBeu]

Nothing.

This is my config and other information:

settings.php

"""
Django settings for instance project.

For more information on this file, see
https://docs.djangoproject.com/en/1.6/topics/settings/

For the full list of settings and their values, see
https://docs.djangoproject.com/en/1.6/ref/settings/
"""

# Build paths inside the project like this: os.path.join(BASE_DIR, ...)
import os
from logging.handlers import SysLogHandler



BASE_DIR = os.path.realpath(os.path.dirname(os.path.dirname(__file__)))


# Quick-start development settings - unsuitable for production
# See https://docs.djangoproject.com/en/1.6/howto/deployment/checklist/

# SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY = 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'

# SECURITY WARNING: don't run with debug turned on in production!
DEBUG = False

ALLOWED_HOSTS = [
    'mail.XXXXXXXXXXX.XXX'
]

SITE_ID = 1

# Security settings

X_FRAME_OPTIONS = "SAMEORIGIN"
SESSION_COOKIE_SECURE = True
CSRF_COOKIE_SECURE = True

# Application definition

INSTALLED_APPS = (
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.sites',
    'django.contrib.staticfiles',
    'reversion',
    'ckeditor',
    'ckeditor_uploader',
    'rest_framework',
    'rest_framework.authtoken',

)

# A dedicated place to register Modoboa applications
# Do not delete it.
# Do not change the order.
MODOBOA_APPS = (
    'modoboa',
    'modoboa.core',
    'modoboa.lib',
    'modoboa.admin',
    'modoboa.transport',
    'modoboa.relaydomains',
    'modoboa.limits',
    'modoboa.parameters',
    'modoboa.dnstools',
    # Modoboa extensions here.
    'modoboa_amavis',
    'modoboa_pdfcredentials',
    'modoboa_postfix_autoreply',
    'modoboa_sievefilters',
    'modoboa_webmail',
    'modoboa_contacts',
    'modoboa_radicale',
    'modoboa_dmarc',
    'modoboa.policyd',
    'modoboa.maillog'
)

INSTALLED_APPS += MODOBOA_APPS

AUTH_USER_MODEL = 'core.User'

MIDDLEWARE = (
    'x_forwarded_for.middleware.XForwardedForMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.locale.LocaleMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
    'modoboa.core.middleware.LocalConfigMiddleware',
    'modoboa.lib.middleware.AjaxLoginRedirect',
    'modoboa.lib.middleware.CommonExceptionCatcher',
    'modoboa.lib.middleware.RequestCatcherMiddleware',
)

AUTHENTICATION_BACKENDS = (
    # 'modoboa.lib.authbackends.LDAPBackend',
    # 'modoboa.lib.authbackends.SMTPBackend',
    'django.contrib.auth.backends.ModelBackend',
)

REDIS_HOST = 'localhost'
REDIS_PORT = 6379
REDIS_QUOTA_DB = 0
REDIS_URL = 'redis://{}:{}/{}'.format(REDIS_HOST, REDIS_PORT, REDIS_QUOTA_DB)

# SMTP authentication
# AUTH_SMTP_SERVER_ADDRESS = 'localhost'
# AUTH_SMTP_SERVER_PORT = 25
# AUTH_SMTP_SECURED_MODE = None  # 'ssl' or 'starttls' are accepted


TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': [],
        'APP_DIRS': True,
        'OPTIONS': {
            'context_processors': [
                'django.template.context_processors.debug',
                'django.template.context_processors.request',
                'django.contrib.auth.context_processors.auth',
                'django.template.context_processors.i18n',
                'django.template.context_processors.media',
                'django.template.context_processors.static',
                'django.template.context_processors.tz',
                'django.contrib.messages.context_processors.messages',
                'modoboa.core.context_processors.top_notifications',
            ],
            'debug': False,
        },
    },
]

ROOT_URLCONF = 'instance.urls'

WSGI_APPLICATION = 'instance.wsgi.application'


# Database
# https://docs.djangoproject.com/en/1.6/ref/settings/#databases

DATABASES = {
    
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'modoboa',
        'USER': 'modoboa',
        'PASSWORD': 'XXXXXXXXXXXXXX',
        'HOST': '127.0.0.1',
        'PORT': '',
        'ATOMIC_REQUESTS': True,
        'OPTIONS' : {
            "init_command" : 'SET foreign_key_checks = 0;',
        },
    },

    'amavis': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'amavis',
        'USER': 'amavis',
        'PASSWORD': 'XXXXXXXXXXXXXX',
        'HOST': '127.0.0.1',
        'PORT': '',
        'ATOMIC_REQUESTS': True,
        'OPTIONS' : {
            "init_command" : 'SET foreign_key_checks = 0;',
        },
    },

}

# Internationalization
# https://docs.djangoproject.com/en/1.6/topics/i18n/

LANGUAGE_CODE = 'en-us'

TIME_ZONE = 'Europe/Paris'

USE_I18N = True

USE_L10N = True

USE_TZ = True


# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/1.6/howto/static-files/

STATIC_URL = '/sitestatic/'
STATIC_ROOT = os.path.join(BASE_DIR, 'sitestatic')
STATICFILES_DIRS = (
    '/srv/modoboa/env/lib/python3.7/site-packages/modoboa/bower_components',
)

MEDIA_URL = '/media/'
MEDIA_ROOT = os.path.join(BASE_DIR, 'media')

# Rest framework settings

REST_FRAMEWORK = {
    'DEFAULT_AUTHENTICATION_CLASSES': (
        'rest_framework.authentication.TokenAuthentication',
        'rest_framework.authentication.SessionAuthentication',
    ),
}

# Modoboa settings
#MODOBOA_CUSTOM_LOGO = os.path.join(MEDIA_URL, "custom_logo.png")
DISABLE_DASHBOARD_EXTERNAL_QUERIES = False

#DOVECOT_LOOKUP_PATH = ('/path/to/dovecot', )

MODOBOA_API_URL = 'https://api.modoboa.org/1/'

# Password validation rules

AUTH_PASSWORD_VALIDATORS = [
    {
        'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator',
    },
    {
        'NAME': 'modoboa.core.password_validation.ComplexityValidator',
        'OPTIONS': {
            'upper': 1,
            'lower': 1,
            'digits': 0,
            'specials': 0
        }
    },
]

# CKeditor

CKEDITOR_UPLOAD_PATH = "uploads/"

CKEDITOR_IMAGE_BACKEND = "pillow"

CKEDITOR_RESTRICT_BY_USER = True

CKEDITOR_BROWSE_SHOW_DIRS = True

CKEDITOR_ALLOW_NONIMAGE_FILES = False

CKEDITOR_CONFIGS = {
    'default': {
        'allowedContent': True,
        'toolbar': 'Modoboa',
        'width': None,
        'toolbar_Modoboa': [
            ['Bold', 'Italic', 'Underline'],
            ['JustifyLeft', 'JustifyCenter', 'JustifyRight', 'JustifyBlock'],
            ['BidiLtr', 'BidiRtl', 'Language'],
            ['NumberedList', 'BulletedList', '-', 'Outdent', 'Indent'],
            ['Undo', 'Redo'],
            ['Link', 'Unlink', 'Anchor', '-', 'Smiley'],
            ['TextColor', 'BGColor', '-', 'Source'],
            ['Font', 'FontSize'],
            ['Image', ],
            ['SpellChecker']
        ],
    },
}

# Logging configuration

LOGGING = {
    'version': 1,
    'formatters': {
        'syslog': {
            'format': '%(name)s: %(levelname)s %(message)s'
        },
    },
    'handlers': {
        'mail-admins': {
            'level': 'ERROR',
            'class': 'django.utils.log.AdminEmailHandler',
            'include_html': True
        },
        'syslog-auth': {
            'class': 'logging.handlers.SysLogHandler',
            'facility': SysLogHandler.LOG_AUTH,
            'formatter': 'syslog'
        },
        'syslog-mail': {
            'class': 'logging.handlers.SysLogHandler',
            'facility': SysLogHandler.LOG_MAIL,
            'formatter': 'syslog'
        },
        'modoboa': {
            'class': 'modoboa.core.loggers.SQLHandler',
        }
    },
    'loggers': {
        'django': {
            'handlers': ['mail-admins'],
            'level': 'ERROR',
            'propagate': False
        },
        'modoboa.auth': {
            'handlers': ['syslog-auth', 'modoboa'],
            'level': 'INFO',
            'propagate': False
        },
        'modoboa.admin': {
            'handlers': ['modoboa'],
            'level': 'INFO',
            'propagate': False
        },
        'modoboa.policyd': {
            'handlers': ['syslog-mail'],
            'level': 'INFO',
            'propagate': False
        }
    }
}

SILENCED_SYSTEM_CHECKS = [
    "security.W019",  # modoboa uses iframes to display e-mails
]


# Load settings from extensions

try:
    from modoboa_amavis import settings as modoboa_amavis_settings
    modoboa_amavis_settings.apply(globals())
except AttributeError:
    from modoboa_amavis.settings import *

try:
    from modoboa_contacts import settings as modoboa_contacts_settings
    modoboa_contacts_settings.apply(globals())
except AttributeError:
    from modoboa_contacts.settings import *

try:
    from modoboa_radicale import settings as modoboa_radicale_settings
    modoboa_radicale_settings.apply(globals())
except AttributeError:
    from modoboa_radicale.settings import *  # noqa

$ pip list

aioredis                        1.3.1
aiosmtplib                      1.1.4
asn1crypto                      0.24.0
async-timeout                   3.0.1
Babel                           2.8.0
bcrypt                          3.1.4
caldav                          0.6.2
certifi                         2019.6.16
cffi                            1.12.3
chardet                         3.0.4
Click                           7.0
coreapi                         2.3.3
coreapi-cli                     1.0.6
coreschema                      0.0.4
coverage                        4.5.3
cryptography                    2.7
davclient                       0.2.0
defusedxml                      0.6.0
dj-database-url                 0.5.0
Django                          2.2.12
django-braces                   1.14.0
django-ckeditor                 5.8.0
django-filter                   2.2.0
django-js-asset                 1.2.2
django-phonenumber-field        4.0.0
django-reversion                3.0.5
django-subcommand2              0.1.1
django-webpack-loader           0.6.0
django-xforwardedfor-middleware 2.0
djangorestframework             3.11.0
dnspython                       1.16.0
drf-nested-routers              0.91
feedparser                      5.2.1
future                          0.17.1
futures                         3.1.1
gevent                          1.5.0
greenlet                        0.4.16
hiredis                         1.1.0
html2text                       2018.1.9
icalendar                       4.0.3
idna                            2.8
itypes                          1.1.0
Jinja2                          2.10.1
jsonfield                       2.0.2
LEPL                            5.1.3
lxml                            4.3.4
MarkupSafe                      1.1.1
modoboa                         1.16.1
modoboa-amavis                  1.3.1
modoboa-contacts                0.8.1
modoboa-dmarc                   1.2.0
modoboa-pdfcredentials          1.4.0
modoboa-postfix-autoreply       1.6.0
modoboa-radicale                1.4.3
modoboa-sievefilters            1.5.0
modoboa-stats                   1.5.0
modoboa-webmail                 1.6.0
mysqlclient                     1.4.2.post1
nose                            1.3.7
oath                            1.4.3
ovh                             0.5.0
passlib                         1.7.2
phonenumbers                    8.12.2
Pillow                          6.1.0
pip                             20.1
pkg-resources                   0.0.0
progressbar33                   2.4
py-dateutil                     2.2
pycparser                       2.19
python-dateutil                 2.8.0
pytz                            2019.1
redis                           3.5.3
reportlab                       3.5.23
requests                        2.22.0
requests-file                   1.4.3
reversion                       0.2
rfc6266                         0.0.4
rrdtool                         0.1.15
setuptools                      41.0.1
sievelib                        1.1.1
six                             1.12.0
sqlparse                        0.3.1
tldextract                      2.2.1
toml                            0.10.0
tzlocal                         1.5.1
uritemplate                     3.0.0
urllib3                         1.25.3
vobject                         0.9.6.1
wheel                           0.33.4

Machine informations

• Debian 10
• VM in Proxmox
• 2 CPU
• 8GB
• DB MariaDB
• Softwares:
  • python: 3.7.3
  • postfix: 3.4.14
  • dovecot: 2.3.4.1
  • redis: 5.0.3
  • opendkim: 2.11.0
  • opendmarc: 3.2-6
  • amavis: 2.11.0

[tonioo]

@AmazingBeu Config looks fine. Perhaps an issue with syslog configuration?

[AmazingBeu]

@tonioo Hmm, I don't know, all the other logs seems work perfectly. There is no solution to set logs to stdout ?

[tonioo]

@AmazingBeu you can play with LOGGING settings in order to configure a console output. It shoud be documented in Django's doc.

[AmazingBeu]

@tonioo The problem seems to be solved and I don't know why ahah...

It's possible that the absence of LOGGING causes policyd doesn't reset the counters?

[tonioo]

@AmazingBeu I don't think so...
Strange !

[tonioo]

@AmazingBeu Is it still working?

[AmazingBeu]

@tonioo I'm doing some tests, but each test takes 24h... 😁

[tonioo]

@AmazingBeu Indeed ;)

[tonioo]

@AmazingBeu Any news?

[AmazingBeu]

It's look like works only one time, at the first UTC midnight after restarted Supervisor 🤔
I will confirm in 2 days

[AmazingBeu]

@tonioo Ok that's it. PolicyD only resets the counters once, but why?

[tonioo]

@AmazingBeu I guess it has something to do with rescheduling (see https://github.com/modoboa/modoboa/blob/master/modoboa/policyd/core.py#L277). What python version do you use?

[AmazingBeu]

My virtualenv is based on Python 3.7.3 .

You have all the information here, except for Modoboa which I updated to 1.17 yesterday:
#2043 (comment)

[tonioo]

@AmazingBeu I also use python 3.7 on my own server and I see in logs that the reset_counters coroutine is called every night... I enabled limits on my own account and will check if the counter is reset tomorrow.

[kalimero67]

Sorry boys @AmazingBeu and @tonioo but ...
I have the same problem and I'm waiting for solution.
What is it? Change LOGGING in settings.py? How exactly?

Thanks Jan

[tonioo]

@kalimero67 There is no solution at the moment since I can't reproduce the issue on my own server...

[kalimero67]

Then please help with these fatal questions:

1. Where are message_counters stored? (Redis ???)
2. Where is code for counters reset? (manage.py ???)
3. Who care for code run? (cron ??? code???)
4. Why is cron (see 3) record is not in default installation?

root@mail:/etc/cron.d# nmap localhost
9999/tcp open abyss

sudo -u modoboa /srv/modoboa/env/bin/python /srv/modoboa/instance/manage.py p olicy_daemon --debug -v3 --host 127.0.0.1
File "/usr/lib/python3.7/asyncio/baseevents.py", line 1378, in createserver
% (sa, err.strerror.lower())) from None
OSError: [Errno 98] error while attempting to bind on address ('127.0.0.1', 9999): address already in use
Thanks thank thanks

[AmazingBeu]

FYI, I have the problem with a new installation (Debian 10 and MySQL)

[kalimero67]

If you do NOT find another solution, run (by cron) daily this script:
cat /root/redis.sql | mysql --user=modoboa --password=*** --silent | redis-cli 2>&1

File 'redis.sql':
CONNECT modoboa; SELECT "hset", "messages_count", username, message_limit FROM admin_mailbox AS a JOIN core_user AS c ON a.user_id=c.id WHERE message_limit<>0; SELECT "hset", "messages_count", name, message_limit FROM admin_domain WHERE message_limit<>0; SELECT "hgetall", "messages_count"

[tonioo]

@AmazingBeu I finally reproduced this issue on my own server! I think the way I planify this periodic task is wrong but I don't know what I could use to replace it. A new cron task is the simplest fix but it is one more task... Otherwise I could introduce a asynchronous scheduler like celery or rq to the default Modoboa setup but it creates one new requirement...
What do you think?

[AmazingBeu]

Personally, I would prefer a cron job because it is much simpler (I love the KISS principle 😁)
After that, there might be some advantages to have an asynchronous scheduler, like to get the state of the tasks in Modoboa's web interface, but you might need a lot of work for that...

[alessandromagridotnet]

Hi, same error here. The emails work like a charm except for the daily limit.
Is there any news about this bug? (or any workaround?)

[Mik-Cryo]

Same problem
Any news, or fix we can apply directly ?

And the most important :) : Merci pour tout ton travail sur Modoboa @tonioo

[tonioo]

I think this issue should be fixed with 2.0. There was a sync <> async issue in the process responsible for resetting counters.