Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrating existing config to version: 2 can cause issues #2852

Closed
chrisdeeming opened this issue Mar 6, 2024 · 9 comments · Fixed by #2850
Closed

Migrating existing config to version: 2 can cause issues #2852

chrisdeeming opened this issue Mar 6, 2024 · 9 comments · Fixed by #2850

Comments

@chrisdeeming
Copy link

Describe the bug

I just updated our smallest production server to Postal v3. Everything went fine, but I wanted to implement the new config format.

I re-wrote the config comparing to the new defaults, taking care to ensure all existing settings were updated to the correct parent key/name.

After doing so, Postal was no longer accepting SMTP connections:

Mar  6 10:01:32 alpha postfix/smtp[8580]: connect to <IP>[<IP>]:25: Connection refused

Telnet was also falling over:

# telnet <IP> 25
Trying <IP>...
telnet: connect to address <IP>: Connection refused

Nothing appeared to be out of the ordinary.

It seemed like the SMTP server was listening (Firewall forwards port 25 to 587):

ruby      3194294 systemd-coredump    5u  IPv4 32470931      0t0  TCP 127.0.0.1:587 (LISTEN)

Luckily I have two servers, one of which I hadn't upgraded yet. Which is when I realised that the exact listening parameters were different on my other Postal server:

ruby      2182364 systemd-coredump    5u  IPv6 31594537      0t0  TCP *:587 (LISTEN)

I had to add:

default_bind_address: '*'

To my config before I could get a response back on port 587.

I also had to explicitly define helo_hostname as without that being defined, the HELO hostname was defaulting to postal.example.com:

220 postal.example.com ESMTP Postal/KMAAQ1VU

I'm not sure why it was defaulting to that versus whichever sensible default that v2 may have been using.

I am about to repeat the process on the other server and I will report back if the same thing happens again, but I will be better placed to fix it more quickly this time and I'll provide any details either way in a subsequent comment along with existing/new config files.

@chrisdeeming
Copy link
Author

Ok second server. Postal upgraded.

We are listening on * port 587:

ruby      3339945 systemd-coredump    5u  IPv6 44482698      0t0  TCP *:587 (LISTEN)

Postal still running version: 1 config.

A new issue though -- getting a handshake failure:

Mar  6 11:40:13 <sender> postfix/smtp[1293399]: 1B6871798B7: to=<me@me.com>, relay=<postal>[<IP>]:587, delay=0.57, delays=0.03/0.03/0.51/0, dsn=4.7.5, status=deferred (Cannot start TLS: handshake failure)

This is new so will proceed with the config change and hope that fixes it

@chrisdeeming
Copy link
Author

chrisdeeming commented Mar 6, 2024

Trying to load certificates from /config/smtp.cert but that is not what is in my config:

smtp-1  | 2024-03-06 12:16:38 +0000 ERROR  Errno::ENOENT: No such file or directory @ rb_sysopen - /config/smtp.cert trace_id=A0GDVGQI component=smtp-server

Config:

  tls_certificate_path: /config/certs/fullchain.pem
  tls_private_key_path: /config/certs/privkey.pem

@chrisdeeming
Copy link
Author

Had to set tls_enabled to false and now I am sending through Postal without issue. So the question remains: why is it not seeing the correct certificate path in my config?

@chrisdeeming
Copy link
Author

I can now confirm my original issue.

Using my original config (albeit with tls disabled now), the SMTP server is listening on *:587:

ruby      3346358 systemd-coredump    5u  IPv6 44517716      0t0  TCP *:587 (LISTEN)

When I update the config to version: 2 (again tls disabled), the SMTP server listens on 127.0.0.1:587:

ruby      3345537 systemd-coredump    5u  IPv4 44512786      0t0  TCP 127.0.0.1:587 (LISTEN)

Note there also seems to be a difference in that with version: 2 config, in that the listen bindings seem to be explicitly IPv4 whereas they are IPv6 when using the original config.

Original (Postal 2.3 config):

general:
  use_ip_pools: true
  maximum_delivery_attempts: 20
  suppression_list_removal_delay: 1
  maximum_hold_expiry_days: 1

web:
  host: server.example.com
  protocol: https

web_server:
  bind_address: 127.0.0.1
  port: 5000

smtp_server:
  port: 587
  tls_enabled: false
  tls_certificate_path: /config/certs/fullchain.pem
  tls_private_key_path: /config/certs/privkey.pem

logging:
  stdout: true

main_db:
  host: 127.0.0.1
  username: root
  password: <password>
  database: postal
  pool_size: 10

message_db:
  host: 127.0.0.1
  username: root
  password: <password>
  prefix: postal

rabbitmq:
  host: 127.0.0.1
  username: postal
  password: <password>
  vhost: postal

dns:
  mx_records:
    - mx.server.example.com
  smtp_server_hostname: server.example.com
  spf_include: spf.server.example.com
  return_path: rp.server.example.com
  route_domain: routes.server.example.com

smtp:
  host: server.example.com
  port: 587
  username: <username>
  password: <password>
  from_name: <From_name>
  from_address: server@example.com

workers:
  quantity: 30
  threads: 12

rails:
  secret_key: <secret_key>

New (Postal v3) config:

version: 2

postal:
  web_hostname: server.example.com
  web_protocol: https
  smtp_hostname: server.example.com
  use_ip_pools: true
  default_maximum_delivery_attempts: 20
  default_suppression_list_automatic_removal_days: 1

smtp_server:
  default_port: 587
  default_bind_address: '*'
  tls_enabled: false
  tls_certificate_path: /config/certs/fullchain.pem
  tls_private_key_path: /config/certs/privkey.pem

logging:
  enabled: true

main_db:
  host: 127.0.0.1
  username: root
  password: <password>
  pool_size: 10

message_db:
  host: 127.0.0.1
  username: root
  password: <password>

dns:
  mx_records:
    - mx.server.example.com
  smtp_server_hostname: server.example.com
  spf_include: spf.server.example.com
  return_path: rp.server.example.com
  route_domain: routes.server.example.com

smtp:
  host: server.example.com
  port: 587
  username: <username>
  password: <password>
  from_name: <From name>
  from_address: server@example.com

rails:
  secret_key: <secret_key>

@chrisdeeming
Copy link
Author

chrisdeeming commented Mar 6, 2024

Okay... another new issue.

Under DNS configuration:

Return Path

WARNING There is a CNAME record at psrp.example.tld but it points to rp.server.example.com which is incorrect. It should point to rp.postal.example.com.

This is optional but we recommend adding this to improve deliverability. You should add a CNAME record at psrp.example.tld to point to the hostname below.

rp.postal.example.com

Where is it getting rp.postal.example.com from? I guess this is because return_path_domain is not defined in the new config? (This is not happening on the original server I upgraded).

@adamcooke
Copy link
Contributor

adamcooke commented Mar 6, 2024

SMTP server not listening on all interfaces

The v2 configuration has a default to listen on 127.0.0.1 so this needs to be changed :: when migrating.

Either we should change this default or document this. I'm leaning towards changing the default.

Having to define dns.helo_hostname

The HELO should use postal.smtp_server_hostname if dns.helo_hostname is not defined.

Return Path

This was renamed from dns.return_path to dns.return_path_domain for consistency.

SSL cert path not found

This is an interesting one. The container sets 4 environment variables which will always override what is in the configuration file. These are all variables which relate to the path where files exist. At present, it is not possible to set these in the config file. To change them, you would need to change your docker-compose.yml file but I do not recommend this as it will cause issues on upgrade.

ENV POSTAL_CONFIG_FILE_PATH=/config/postal.yml
ENV POSTAL_SIGNING_KEY_PATH=/config/signing.key
ENV SMTP_SERVER_TLS_CERTIFICATE_PATH=/config/smtp.cert
ENV SMTP_SERVER_TLS_PRIVATE_KEY_PATH=/config/smtp.key

I think the most useful thing I can do is make a guide on the key changes between v1 and v2 configuration to help with migrations. Broadly the main things are:

  • web.host -> postal.web_hostname
  • web.protocol -> postal.web_protocol
  • web_server.port -> web_server.default_port
  • web_server.bind_address -> web_server.default_bind_address
  • smtp_server.port -> smtp_server.default_port
  • smtp_server.bind_address -> smtp_server.default_bind_address
  • dns.return_path -> dns.return_path_domain
  • dns.smtp_server_hostname -> postal.smtp_server_hostname
  • general.use_ip_pools -> postal.use_ip_pools
  • general.* -> moved to various new names under the postal. namespace
  • smtp_relays -> postal.smtp_relays and now uses an array of strings which should be in the format of smtp://{host}:{port}?ssl_mode={mode}
  • logging.graylog.* -> gelf.*

I'll get this transferred to the docs site shortly.


Did I missing anything?

adamcooke added a commit that referenced this issue Mar 6, 2024
This is the same behaviour as when using v1 configuration. Unlike the
web server which is proxied, most people are going to need this so having
the default remain seems like the easiest path for upgrades.

see #2852
@chrisdeeming
Copy link
Author

chrisdeeming commented Mar 6, 2024

The v2 configuration has a default to listen on 127.0.0.1 so this needs to be changed :: when migrating.
Either we should change this default or document this. I'm leaning towards changing the default.

Yeah I think this will catch fewer people out, hopefully.

The HELO should use postal.smtp_server_hostname if dns.helo_hostname is not defined.

postal.smtp_hostname* 😄

This was renamed from dns.return_path to dns.return_path_domain for consistency.

Yeah looks like this one was my bad and accounts for the inconsistency when I migrated the config the second time.

This is an interesting one.

Does it make sense for now for me to copy the the cert and key to match the env var names?

I'll get this transferred to the docs site shortly.

Great!

Did I missing anything?

Don't think so. Thanks for everything. It was quiet here for a while which made me a little nervous as I was about to move two business critical servers over to Postal but the momentum recently is really impressive and it's very much appreciated. Thank you.

@adamcooke
Copy link
Contributor

Does it make sense for now for me to copy the the cert and key to match the env var names?

Yes

Thanks for everything. It was quiet here for a while which made me a little nervous as I was about to move two business critical servers over to Postal but the momentum recently is really impressive and it's very much appreciated. Thank you.

Not a problem! Quite a few new updates coming soon around authentication and the API too.

@pjv
Copy link

pjv commented Mar 10, 2024

On my Postal 2.3.2 server, I have fail2ban configured on the host to watch the smtp logs and ban repeat evil hitting the server.

I have this set up with the following v1 config:

logging:
  # Specify options for the logging
  stdout: false
  root: /opt/postal/log

(and I also have the directory /opt/postal/log bind mounted via docker into the smtp container)

…which results in postal logs being written to /opt/postal/log on the host making it simple to configure fail2ban to look there.

Is it possible to achieve the same behavior in the v2 configuration (iow, will logging.root work similarly in v2 or is there some equivalent)?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants