Share server secret with replicas

[fschulze]

Currently replicas have their own server secret. This isn't an issue yet. I discovered it when I wanted to check a permission for a logged in user in a plugin. The login goes to master and it returns the signed token. That token can't be verified on the replica, because the replica has it's own secret. The question is how to best share it. Either we would need to require an https connection to the master or build our own exchange. We could also add a config option to let the secret be passed via http in cases where you are sure it can't be sniffed.

[aostr]

Requiring https connection seems reasonable.
In the past the devpi-client was sending username and password in clear text. So https is something that users want to have anyway, right?

[fschulze]

@aostr this is not about devpi-client, but solely between a devpi-server master and it's replicas. There it needs to be simple to deploy and self signed https certificates are a hassle and public ones aren't always possible.

[aostr]

Yes, I get it that it's not about the client-server, but master-replica communication.
I am well aware of the hassle with the self-signed SSL certs. :) I think that we were talking about it on the pytest sprint some years ago in Freiburg. You may remember me from the "ver...s dich" phrase.
Anyway, the issue of the SSL certs has been solved on our side. We simply purchased SSL certs from a trusted CA and then everything became green.

If your server is public, then you can go with let's encrypt. If not, then it's probably something like ~$100 per year, but that's a guess.

Cheers,
Andrzej

[fschulze]

@aostr I know who you are 😉 I wasn't aware the SSL certs issue was finally fixed on your end. That's why I thought you may have misunderstood me.

I just had another thought. We could require replicas to have the same secret and use it for authentication between master and replica. That way you would have to copy the secret to the replica out of band and only known replicas can connect to the master.

[EvaSDK]

Sharing secret between master & replicas should be simple enough for people running private devpi instances.

[golyalpha]

If you're talking about docker swarm, then you could probably use docker swarm's secrets. Otherwise you need to figure out a way to get multiple machines to share a directory - NFS might be a start- and put the file-based secret in there.

I personally don't think it should be in the scope of any application to figure out a way to share it's secret. That should always be up to the infrastructure and it's administrator. (In this context, I consider Docker Swarm being the infrastructure, rather than an application)

[RonnyPfannschmidt]

@fschulze fo login transfers, if the master returns a token that the replica either has to ask the server for validation/timeout controll, or return a own derived token,

i beleive there is no need to share security data for logins,

[golyalpha]

@RonnyPfannschmidt If replicas were to return their own derived tokens, you'd have to make sure, that you connect the user to the same replica from then until the token is invalidated. Or you'd also have to have other replicas ask the issuing replica about the token.

[RonnyPfannschmidt]

@golyalpha my understanding based on https://github.com/devpi/devpi/blob/master/doc/adminman/replica.rst is, that replicas don't share urls and fail-over has to be configured manually

without more context i would consider re-login in such cases a acceptable tradeoff

[golyalpha]

@RonnyPfannschmidt To be perfectly fair, making sure at a load balancer (especially if the load balancer is nginx) that a given user is always connected to a given replica until a cookie expires, should be relatively easy.

Something like this should do the trick:

upstream devpi {
  server srv1.example.com;
  server srv2.example.com;
  server srv3.example.com;
}

map $cookie_active_replica $my_upstream {
  default devpi;
  ~^(?P<name>[\w-]+) $name;
}

location / {
  proxy_pass http://$my_upstream;
}

(Taken and slightly modified from this answer: https://serverfault.com/a/303572)

In this case, it would be the replica's job to set the 'active_replica' cookie on the client upon login, in such a way that nginx can connect to it for proxy_pass. Until the cookie is set, requests from that client are load balanced using the logic behind nginx's upstreams.

[fschulze]

With upcoming devpi-server 6.0.0 a master will require the replicas to use the same secret, otherwise the replication endpoint will return 403 Forbidden.

[onelson]

With upcoming devpi-server 6.0.0 a master will require the replicas to use the same secret, otherwise the replication endpoint will return 403 Forbidden.

I wonder if this could be added to the guidance on replication here: https://devpi.net/docs/devpi/devpi/stable/+doc/adminman/replica.html

I have been working my way up through devpi-server upgrades today so we can ultimately start using replicas, but these mystery 403s tripped me up for some time...

[con-f-use]

Tangential question: Is there a way to get the server secret from a running instance, where the secret was random-generated by devpi, i.e. not set by --secretpath or envvar?

[fschulze]

@con-f-use the secret is randomly generated on server start if not set.

[con-f-use]

I know, I'd like to get to that randomly generated secret so I can avoid redeploying/restarting an already in-use instance where the secretpath was not originally set and add a replica. Sorry, if I wasn't clear enough.

[fschulze]

@con-f-use You can create a new one with the devpi-gen-secret script. There is no need to get the in memory one.

[con-f-use]

But then, how do I switch out the old for the new without on the running instance? I'd have to restart it and edit its config, wouldn't I?

[fschulze]

@con-f-use yes, where is the problem with that? The only thing will be existing login tokens will be invalid and users have to login again, but those tokens expire after 10h anyway.