deck-chores failes in on_max_instances

[rickardcronholm]

I'm toying around with deck-chores and quite directly ran into an issue.

OS: Ubuntu 18.04
docker: Docker version 18.09.7, build 2d0083d

Log from deck-chores:

Deck Chores 1.0.0 started.
2020-04-02 14:29:09,013|DEBUG   |Config: {'assert_hostname': False, 'client_timeout': 60, 'default_flags': ('image', 'service'), 'docker_host': 'unix://var/run/docker.sock', 'debug': True, 'default_max': 1, 'job_executor_pool_size': 1, 'label_ns': 'deck-chores.', 'logformat': '{asctime}|{levelname:8}|{message}', 'service_identifiers': ('com.docker.compose.project', 'com.docker.compose.service'), 'ssl_version': <_SSLMethod.PROTOCOL_TLS: 2>, 'timezone': 'UTC', 'client': <docker.client.DockerClient object at 0x7f8b3c32b810>}
2020-04-02 14:29:09,031|INFO    |Inspecting running containers.
2020-04-02 14:29:09,038|DEBUG   |Parsing labels: {'org.opencontainers.image.created': '2020-03-27 13:30:03+00:00', 'org.opencontainers.image.description': 'Job scheduler for Docker containers, configured via labels.', 'org.opencontainers.image.documentation': 'https://deck-chores.readthedocs.org/', 'org.opencontainers.image.revision': '3222643d7efa010bfaa9619dc6c245937b9045ed', 'org.opencontainers.image.source': 'https://github.com/funkyfuture/deck-chores', 'org.opencontainers.image.title': 'deck-chores', 'org.opencontainers.image.version': '1.0.0'}
2020-04-02 14:29:09,038|DEBUG   |Considering labels for service id: {}
2020-04-02 14:29:09,038|DEBUG   |Parsed & resolved container flags: image,service
2020-04-02 14:29:09,041|DEBUG   |Considering labels for job definitions: {}
2020-04-02 14:29:09,042|DEBUG   |Job definitions: {}
2020-04-02 14:29:09,045|DEBUG   |Parsing labels: {'deck-chores.echo.command': './script.sh', 'deck-chores.echo.interval': '5s'}
2020-04-02 14:29:09,045|DEBUG   |Considering labels for service id: {}
2020-04-02 14:29:09,048|DEBUG   |Considering labels for job definitions: {'deck-chores.echo.command': './script.sh', 'deck-chores.echo.interval': '5s'}
2020-04-02 14:29:09,048|DEBUG   |Job definitions: {'echo': {'command': './script.sh', 'interval': '5s'}}
2020-04-02 14:29:09,049|DEBUG   |Processing echo
2020-04-02 14:29:09,057|DEBUG   |Normalized definition: {'command': './script.sh', 'name': 'echo', 'user': '', 'environment': {}, 'jitter': None, 'max': 1, 'timezone': 'UTC', 'trigger': (<class 'apscheduler.triggers.interval.IntervalTrigger'>, (0, 0, 0, 0, 5))}
2020-04-02 14:29:09,057|DEBUG   |Adding jobs to container 8a3e0d4cbe9b30c9f5dc3f429a43bbc4471ba38c6dbccfff37ae02d4f5694a84.
2020-04-02 14:29:09,059|INFO    |priceless_mccarthy: Added 'echo' (fbff3edf-7ff9-598f-8c4a-cae07c8633eb).
2020-04-02 14:29:09,063|DEBUG   |Parsing labels: {}
2020-04-02 14:29:09,063|DEBUG   |Considering labels for service id: {}
2020-04-02 14:29:09,066|DEBUG   |Considering labels for job definitions: {}
2020-04-02 14:29:09,066|DEBUG   |Job definitions: {}
2020-04-02 14:29:09,070|DEBUG   |Parsing labels: {'maintainer': 'Sebastian Ramirez <tiangolo@gmail.com>'}
2020-04-02 14:29:09,070|DEBUG   |Considering labels for service id: {}
2020-04-02 14:29:09,075|DEBUG   |Considering labels for job definitions: {}
2020-04-02 14:29:09,075|DEBUG   |Job definitions: {}
2020-04-02 14:29:09,075|DEBUG   |Finished inspection of running containers.
2020-04-02 14:29:09,077|INFO    |Added job "exec_job" to job store "default"
2020-04-02 14:29:09,077|INFO    |Scheduler started
2020-04-02 14:29:09,077|DEBUG   |Looking for jobs to run
2020-04-02 14:29:09,077|INFO    |Listening to events.
2020-04-02 14:29:09,077|DEBUG   |Next wakeup is due at 2020-04-02 14:29:14.057721+00:00 (in 4.979914 seconds)
2020-04-02 14:29:14,058|DEBUG   |Looking for jobs to run
2020-04-02 14:29:14,059|INFO    |priceless_mccarthy: Executing 'echo'.
2020-04-02 14:29:14,059|DEBUG   |Next wakeup is due at 2020-04-02 14:29:19.057721+00:00 (in 4.998639 seconds)
2020-04-02 14:29:14,071|DEBUG   |Daemon event: {'status': 'exec_start: ./script.sh ', 'id': '8a3e0d4cbe9b30c9f5dc3f429a43bbc4471ba38c6dbccfff37ae02d4f5694a84', 'from': 'app2:latest', 'Type': 'container', 'Action': 'exec_start: ./script.sh ', 'Actor': {'ID': '8a3e0d4cbe9b30c9f5dc3f429a43bbc4471ba38c6dbccfff37ae02d4f5694a84', 'Attributes': {'deck-chores.echo.command': './script.sh', 'deck-chores.echo.interval': '5s', 'execID': '65e90163e1636883aed141dfc6f0a730a912c95d076ff78a6928bdca2600370d', 'image': 'app2:latest', 'name': 'priceless_mccarthy'}}, 'scope': 'local', 'time': 1585837754, 'timeNano': 1585837754070516452}
2020-04-02 14:29:19,058|DEBUG   |Looking for jobs to run
2020-04-02 14:29:19,058|WARNING |Execution of job "exec_job (trigger: interval[0:00:05], next run at: 2020-04-02 14:29:19 UTC)" skipped: maximum number of running instances reached (1)
2020-04-02 14:29:19,059|ERROR   |Error notifying listener
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/apscheduler/schedulers/base.py", line 831, in _dispatch_event
    cb(event)
  File "/usr/local/lib/python3.7/site-packages/deck_chores/jobs.py", line 37, in on_max_instances
    definition = job.kwargs
AttributeError: 'dict' object has no attribute 'kwargs'
2020-04-02 14:29:19,059|DEBUG   |Next wakeup is due at 2020-04-02 14:29:24.057721+00:00 (in 4.998914 seconds)
2020-04-02 14:29:24,059|DEBUG   |Looking for jobs to run
2020-04-02 14:29:24,059|WARNING |Execution of job "exec_job (trigger: interval[0:00:05], next run at: 2020-04-02 14:29:24 UTC)" skipped: maximum number of running instances reached (1)
2020-04-02 14:29:24,059|ERROR   |Error notifying listener
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/apscheduler/schedulers/base.py", line 831, in _dispatch_event
    cb(event)
  File "/usr/local/lib/python3.7/site-packages/deck_chores/jobs.py", line 37, in on_max_instances
    definition = job.kwargs
AttributeError: 'dict' object has no attribute 'kwargs'
2020-04-02 14:29:24,059|DEBUG   |Next wakeup is due at 2020-04-02 14:29:29.057721+00:00 (in 4.998511 seconds)
2020-04-02 14:29:29,058|DEBUG   |Looking for jobs to run
2020-04-02 14:29:29,058|WARNING |Execution of job "exec_job (trigger: interval[0:00:05], next run at: 2020-04-02 14:29:29 UTC)" skipped: maximum number of running instances reached (1)
2020-04-02 14:29:29,058|ERROR   |Error notifying listener
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/apscheduler/schedulers/base.py", line 831, in _dispatch_event
    cb(event)
  File "/usr/local/lib/python3.7/site-packages/deck_chores/jobs.py", line 37, in on_max_instances
    definition = job.kwargs
AttributeError: 'dict' object has no attribute 'kwargs'
2020-04-02 14:29:29,058|DEBUG   |Next wakeup is due at 2020-04-02 14:29:34.057721+00:00 (in 4.999146 seconds)

Steps to reproduce:
Dockerfile

COPY ./script.sh /
WORKDIR /
SHELL ["/bin/bash", "-c"]

LABEL   deck-chores.echo.command="./script.sh" \
        deck-chores.echo.interval="5s"

CMD ["ls"]

and the script it runs is simply:

while [ 1 == 1 ]; do echo "app2 says yes"; sleep 10; done

Finally the image built and a container started through
docker run --name app2 app2:latest

[funkyfuture]

have you set JOB_POOL_SIZE deliberately to 1?

[rickardcronholm]

Yes, that's what I'm toying around with. My scenario is that I have a few containers that each allocate a lot of gpu memory such that I can only afford to have one of them running at any given time. What I was trying to see was if that was possible to achieve through deck chores and the job pool size.

…

On Thu, Apr 2, 2020, 19:32 Frank Sachsenheim ***@***.***> wrote: have you set JOB_POOL_SIZE deliberately to 1? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#52 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADYQU5ETXLSD5JHTCXWAEDTRKTD3RANCNFSM4L2QO33A> .

[funkyfuture]

thanks for reporting! you hit a bug in very simple code that is yet hard to test.

the fix will be included in the maintenance release scheduled for early May.

regarding your use-case: i wouldn't limit the pool, but have the jobs check a directory that acts as semaphore in a shared volume. in theory, maybe it doesn't fit your environment or i'm missing something.

[rickardcronholm]

It will probably fit my needs and was my original plan until I noticed the JOB_POOL_SIZE in deck-chores. Would you mind going into detail on why you advice against?

[funkyfuture]

because you restrict other uses than these gpu-hungry containers where failing.