make hostnames explicit

[whysthatso]

this should solve #608

[eikek]

Hi @whysthatso , thank you a lot! I'm in favor for this change. Still want to ping @bjeanes , because he might have a different view?

I would then also revert the change in docspell.conf, wdyt? (there is only the joex url that has a host-based url)

I think it's nice to be able to simply start new joex containers. But tbh, if it's running on the same machine, one can also just set pool-size to some higher value. So I'd think the common case is covered?

For reference, see PR #552

[bjeanes]

If you hardcode the hostname in docker compose, then you can ONLY run one of them.

[whysthatso]

If you hardcode the hostname in docker compose, then you can ONLY run one of them.

indeed, as i am under the impression that this would be the standard case, albeit this should be documented.

[bjeanes]

Still want to ping @bjeanes , because he might have a different view?

I don't mind if this is merged, because I can still do whatever I want in my deployment. But, I think the preferred solution here would be something like tracking "last seen at" for each node and culling nodes that haven't reported in in some period. Could also have it randomly pick a node instead of work down the list (if I correctly understand that this is what is happening).

indeed, as i am under the impression that this would be the standard case, albeit this should be documented.

Perhaps, but being able to elastically scale up workers based on queue size is not possible using just pool-size AIUI, because you'd have to restart the whole joex, which might take time if it's processing a long job.

I would then also revert the change in docspell.conf, wdyt? (there is only the joex url that has a host-based url)

Yes, agreed.

[whysthatso]

i'll try to add something that makes that parameter explicit in the documentation.

[eikek]

I don't mind if this is merged, because I can still do whatever I want in my deployment.

That's what I figured. People usually have their own deployments and different setups etc and probably use their own compose files (or whatever alternative) anyways.

I think we can't and shouldn't support all possible scenarios. Although I can see that this one seems rather like a "best practice". However, I doubt that people use more than one machine most of the time and with one machine, the pool-size option can be used with one container to have multiple job executors. It is true that you need to restart joex to apply this config change. If people have multi machine setups, they know what they do.

But, I think the preferred solution here would be something like tracking "last seen at" for each node and culling nodes that haven't reported in in some period. Could also have it randomly pick a node instead of work down the list (if I correctly understand that this is what is happening).

Absolutely! This will be fixed. It works as you said; the order of the list is left to the db, so not really random. But I want to notify all nodes, not just one. Currently there is nothing in place to cleanup/watch this list, which is a bug actually. I'm going to create a separate issue for this, was first thinking to use this one, but now I find it better to track separately.

I'd say we merge this change for now, until the bug is fixed. Then we can adopt the docker setup again. -?

[bjeanes]

That all seems sensible @eikek! Agreed that most people won't use multiple machines and that ON one machine pool-size is sufficient. I'd just add the caveat that pool-size is appropriate for a static size, but unless it auto-reloads config and I don't know that, it isn't as suitable for dynamic situations. Nonetheless, that's an edge case and I think making it work best for the common case (i.e. @whysthatso's case) should take priority.

[eikek]

Only want to confirm: the config is not auto-reloaded. so pool-size is only appropriate for static sizing. But I like the idea of being able to change pool-size dynamically… might consider this in the future.

@whysthatso Would you mind also adopting docspell.conf? – the joex base-url can now be a constant value. I think this is less confusing when looking at it.

[bjeanes]

Another reason to change docspell.conf is that without changing it, the app-id will be joex-joex:

docspell/docker/docspell.conf

Line 30 in bd32dff

app-id = "joex-"${HOSTNAME}