can't resolve joex container, using wrong hostname reference

[whysthatso]

weird issue in my docker compose setup:
it seems that the restserver stores joex node settings in the db, but doesn't check/clean up this table for some reason, see screenshot.

it seems that the reserver only tries so many instances (5, or maybe 6) and then errors out. in the screenshot, my current one is the last, all the others before are previous docker containers and do not exist any longer.

i'm not sure where in the code this is set, and if and where there's a cleanup procedure.

i've removed all the old instances except the current one, then restarted the docker-compose setup, and as you can see the old references stays in there for some reason.

i do manage my docker-compose setup with systemd, so maybe there is an external signal that the code depends on that does not get triggered?

i found this in the logs:

Jan 29 08:25:36 docker-host-1-k18 docker-compose[47158]: docspell-restserver | [ioapp-compute-0] INFO  d.b.ops.ONode - Registering node rest1

but couldn't find a similar entry for the joex instance. i guess for some reason the 'unregister' method is not executed.

i found this reference in the docs and i presume that for some reason the docker compose shutdown doesn't really initiate a graceful shutdown of the joex instance?

[whysthatso]

ah, and of course: removing the old instances did fix my initial problem, and restserver can of course then resolve to the current instance of joex.

[eikek]

Hi @whysthatso thanks for this report.

This problem occurs now with docker, since the joex hostname can change between restarts, as it defaults to the HOST env variable and this seems to change in docker containers frequently. The components (joex and restserver) will insert an entry when started and remove it when stopped. But when being killed (sigkill), this might not happen. It should happen when terminated. AFAIK systemd tries with sigterm and waits a while before it uses sigkill, but I'm not sure. The graceful shutdown for joex (using the request as described) is always better of course. But the main reason is to avoid killing a joex instance in the middle of processing. The request informs joex which stops taking new jobs and stops itself once all jobs are finished.

There is currently no cleanup procedure in place. I wasn't aware about this use case. Despite the fact that a cleanup procedure should be there, I think the setup using the HOST variable is only needed when you want to run multiple instances. I don't think that this is a common case. You can always set the parallelism in the joex config if running on stronger hardware. So running multiple containers makes more sense when they are run on different machines. So I'm currently thinking about to change it back to a constant value to avoid confusion….

Currently you could set app-id (and if you want base-url, too) in the config file to some constant value. This references a single unique joex instance. If you only ever have one instance, that is a better default.

The other point is also interesting: It should actually try all existing entries in this table and not stop after 5 or 6. I have to look into it. Then a startup/housekeeping task needs to run that removes obsolete entries. I'll add this for a next release.

[whysthatso]

i've solve this now by adding a fixed hostname: foobar to the service declaration in the compose file:

  joex:
    hostname: docspell-joex
    image: eikek0/docspell:joex-{{ app.value.version }}
    restart: unless-stopped
    env_file: env
    ports:
      - "7878:7878"
    volumes:
      - ./docspell.conf:/opt/docspell.conf
    depends_on:
      - solr

[eikek]

It should be a hostname, that is actually resolvable by the other components (restserver), otherwise joex cannot be notified for new work. It polls periodically, but it's not as quick:-).

[whysthatso]

as far as i understand docker-composer, the default network that composer sets up when starting the containers will take care of exactly that.