Opening this so I don't forget but still need to dig in to the problem to investigate.
Basically I noticed that changes to my config/sys_prod.config were not being included when I ran the release after a rebar3 as prod release. I then checked and found that the sys.config under the prod profile was correct after the rebar3 as prod release but then overwritten when the release is run and sys.config.orig is copied over sys.config.
So it seems something needs to be done when a release is rebuilt to remove sys.config.org?