Rails writes the email. Hailer deals with sending it.
This repository is two things:
hailergem — an ActionMailer adapter, feedback client, and executable launcher. Platform gems bundle the Rust binary. No Ruby AWS SDK or native extension.hailercrate — a standalone Rust mail ship, supervised by Mothership. Sends through SESv2 or Postmark.
flowchart LR
A[ActionMailer] --> B[hailer gem]
B --> C[private Mothership route]
C --> D[hailer binary]
D --> E[SES]
An opt-in durable queued adapter sits alongside the default synchronous
send path ([queued] in hailer.example.toml).
gem "guardship"
gem "shuttlebay"
gem "hailer"Mothership picks up the bay and starts sending mail. Keep using your
mailers, interceptors, observers, deliver_now, and deliver_later -
nothing else about how Rails sends mail changes.
Everything past this point is reference material: protocol internals, TOML knobs, and operator commands for people running or debugging the service, not required reading to use it.
Requires Ruby 4.0+ and Rails 8.1 or newer within Rails 8.x. Rails main (8.2 alpha) is also exercised. Older releases are not supported.
The Railtie registers and selects Hailer automatically. Set the private service URL in the Rails process's environment:
HAILER_URL=http://127.0.0.1:9293/internal/mail/sendNo application initializer is needed. To customize connection settings:
config.action_mailer.hailer_settings = {
endpoint: ENV.fetch("HAILER_URL"),
open_timeout: 5,
read_timeout: 30,
write_timeout: 30
}HAILER_URL is the complete private URL, for example
http://127.0.0.1:9293/internal/mail/send. It is also the default endpoint
when omitted from hailer_settings. There is no default host or fallback to SES.
The app can explicitly override the default, for example
config.action_mailer.delivery_method = :test in its test environment.
The adapter sends rendered MIME and separate envelope recipients, preserving
explicit smtp_envelope_from and smtp_envelope_to overrides. After SES
accepts the message, it sets message.header[:ses_message_id].
Platform gems install the hailer executable, ready for Mothership to launch.
Installing the gem does not start the service or send mail.
The source (ruby) gem has the same Rails client and launcher, but no binary.
For source development or an unsupported platform:
cargo install --path . --locked --features ses,postmark
export HAILER_BINARY=/absolute/path/to/hailerEach provider is a cargo feature, and a build needs at least one. Platform
gems carry both. On glibc Linux the build needs libclang (libclang-dev):
rama generates its libresolv bindings with bindgen.
| Feature | Adds |
|---|---|
ses |
SES sending, SQS feedback, suppression sync, rate discovery, ses-* and sync-suppressions commands (hailer's own SigV4 client, no AWS SDK) |
postmark |
MIME parsing for Postmark's JSON API, postmark-server command |
A binary without the configured provider refuses to start and names the feature to rebuild with.
Run it as a Mothership HTTP bay. Merge this example into your fleet manifest; the bind must stay private and must not be routed on a public listener:
[mothership.bind]
mail = "127.0.0.1:9293"
[[bays.http]]
name = "hailer"
command = "hailer"
workers = 0
threads = 2
routes = [{ bind = "mail", pattern = "^/internal/mail/(send|events|health|ready)$" }]Mothership exports MS_PID, MS_SHIP, and MS_SOCKET_DIR. Hailer binds its
docking Unix socket; it does not run another HTTP server. For containers,
use a private network listener with network policy instead of a loopback address
if Rails runs in a different network namespace.
Give the service, not the Rails adapter, AWS region and credentials. Hailer
looks where the AWS SDKs do, in this order: AWS_ACCESS_KEY_ID /
AWS_SECRET_ACCESS_KEY (+ AWS_SESSION_TOKEN); the shared ~/.aws/credentials
and ~/.aws/config files (AWS_PROFILE, static keys); an EKS web identity
(AWS_WEB_IDENTITY_TOKEN_FILE + AWS_ROLE_ARN); an ECS or EKS Pod Identity
container role; the EC2 instance role over IMDSv2. A profile that needs SSO,
credential_process or role chaining is refused with the key it uses. The
region comes from AWS_REGION, AWS_DEFAULT_REGION, the profile, or IMDS.
AWS_ENDPOINT_URL_SESV2, _SQS and _STS, else AWS_ENDPOINT_URL, point
those calls elsewhere (LocalStack, a test fixture) unless
AWS_IGNORE_CONFIGURED_ENDPOINT_URLS=true.
Its identity needs ses:SendEmail, and the SES account/identities must be
configured for sending.
provider = "postmark"
[postmark]
message_stream = "outbound" # the defaultSet POSTMARK_SERVER_TOKEN in the service's environment, never in
hailer.toml. hailer postmark-server checks the token and names its server.
Postmark's API takes JSON, not raw MIME, so Hailer parses the Rails message
and Postmark rebuilds it. Recipients, subject, text/HTML bodies, attachments
(inline images by cid:), Reply-To, Message-ID and other custom headers
carry over; the MIME structure itself does not. Recipients come from the
envelope, as with SES. An explicit smtp_envelope_from only stands in for a
missing From header: Postmark owns the bounce address.
The SES-only settings (ses_configuration_set, [feedback],
[suppression_sync], rate_limit.discover_from_aws) are refused with
provider = "postmark". Postmark bounces and suppressions are not read yet,
so a queued unknown outcome waits for hailer resolve-message. Postmark
refuses a message (406, suppressed) only when every recipient is inactive in
its suppression list; otherwise it accepts and skips the inactive ones, which
Hailer logs as a warning with the count, never the addresses.
PostgreSQL is the only live backend. Hailer owns a hailer schema,
not Rails tables. Use a separate database/role where practical:
export HAILER_DATABASE_URL='postgres://hailer:password@db/hailer?sslmode=verify-full'
hailer migrateSupply the database CA through the normal PostgreSQL connection options.
Use certificate verification in deployment; a private network is not TLS.
Keep the URL in a secret manager, not TOML or a committed initializer.
Startup also applies embedded, versioned migrations. The bootstrap identity
needs database CREATE and ownership of its schema/tables. Runtime startup
currently runs that same bootstrap/migrator, so provision the service role for
it too; a separate DML-only runtime mode is not implemented.
The runtime never needs Rails schema ownership. Back up the database with your
normal PostgreSQL backup/restore tooling.
Configuration defaults: pool size 5 (range 1–64), shutdown timeout 30 seconds (range 1–300). Every replica shares PostgreSQL; each has its own pool and optional rate budget. Set your supervisor/container stop grace longer than Hailer's shutdown timeout. PostgreSQL 18 is exercised locally and in CI.
flowchart LR
A[SES] --> B[SNS]
B --> C[dedicated SQS queue]
C --> D[Hailer]
D --> E[durable events + suppressions]
Hailer atomically commits each event, deduplication key, and suppression change to PostgreSQL before acknowledging SQS. Failed acknowledgements can be retried without duplicating stored events. Invalid/untrusted messages are left for SQS's dead-letter policy.
- Permanent bounces and complaints suppress recipients.
- Transient/undetermined bounces are recorded without permanent suppression.
- An explicit
not-spamcomplaint does not create a new suppression. - Suppression checks cover the actual To/Cc/Bcc envelope, including overrides.
Matching is case-insensitive and ignores
+aliastags. - Mixed messages keep allowed recipients and their original addresses; MIME is unchanged. If every recipient is suppressed, Hailer returns HTTP 422 without calling SES. A suppression lookup failure fails closed with HTTP 503.
Copy hailer.example.toml, set the real queue/topic, and
pass --config /path/to/hailer.toml or HAILER_CONFIG. Without [feedback],
the consumer is disabled, but previously stored suppressions are still enforced.
HAILER_DATABASE_URL is required, even with feedback disabled.
The old state_path, --state-path, and HAILER_STATE_PATH are not supported.
Replicas may consume the same dedicated queue when they share the same Hailer
database. Commits serialize cursor allocation so a reader cannot skip an earlier
transaction that has not committed yet.
Event history and deduplication have no automatic retention/pruning. Do not delete events casually: event IDs also carry the SQS deduplication contract, and Rails consumers may still need their cursors. Plan retention against consumer lag and redelivery windows before adding a pruning policy.
Hailer also exposes a provider-neutral, nondestructive event feed:
POST /internal/mail/events
Content-Type: application/json
{"after":0,"limit":100}Route it on the same private bind, if application tracking needs it:
routes = [{ bind = "mail", pattern = "^/internal/mail/(send|events)$" }]The gem validates the page and exposes immutable FeedbackEvent objects:
page = Hailer::FeedbackClient.new(endpoint: ENV.fetch("HAILER_URL")).poll(after: 0)
page.events.each { |event| puts [event.event_id, event.kind, event.message_id] }Configure a Rails business-history handler (a callable or reloadable class name):
Rails.application.config.hailer.feedback_handler = "Platform::MailFeedback"Then Hailer::FeedbackConsumer.new.call(after: cursor) calls that handler for
each validated event and returns the next cursor only after all callbacks
succeed. A callback must raise on failure. Persist the cursor and domain
updates in one application database transaction. Retried pages require
idempotent handlers; no hidden poller, scheduling, or checkpoint database is
started by the gem. Reading does not acknowledge or erase events.
The SQS policy must allow only the intended SNS topic, using aws:SourceArn
and account restrictions. Subscribe with RawMessageDelivery=false, restrict
direct sqs:SendMessage, set a dead-letter queue, and monitor its depth.
The SNS topic allowlist in TOML is defense in depth, not signature verification.
Hailer needs sqs:ReceiveMessage and sqs:DeleteMessage on that queue, plus
applicable KMS decrypt permissions if customer-managed encryption is used.
hailer sync-suppressions
# If AWS expires a saved pagination token:
hailer sync-suppressions --restart
# Existing application export, JSON array:
hailer import-suppressions --file suppressions.jsonThe service identity needs ses:ListSuppressedDestinations; STS
GetCallerIdentity identifies the account (AWS does not require an explicit
allow for that operation). The configured region is part of the source identity.
These commands read AWS only; they never mutate the account suppression list.
Missing AWS entries do not unsuppress local entries, and a bounce never
downgrades a complaint. Imports do not fabricate delivery-history events.
Account-level AWS data does not include every possible global SES suppression.
Enable [suppression_sync] as in the example to reconcile on startup and hourly.
The first complete sync gates send readiness. Later errors mark health degraded,
retain committed pages, and retry safe reads with bounded backoff. A token is
stored with its page in one transaction. Replicas coordinate a source-scoped
advisory lock; they cannot advance the same checkpoint concurrently.
An expired token deliberately requires the operator's --restart. Omitting
the section disables scheduled sync, not existing suppression enforcement.
The behavior circuit evaluates pure admission decisions: input validity,
service/storage readiness, suppression, and rate admission. PostgreSQL,
AWS calls, clocks, and token reservations remain owned by the async host.
The SES circuit breaker counts transient/ambiguous failures, not bad addresses
or permanent request rejections. An optional [rate_limit] budgets envelope
recipients through a single shared Postgres-backed token bucket - every
replica draws down the same account-wide budget, not an independent
per-process one. discover_from_aws = true replaces the configured rate
with the SES account's real SendQuota.MaxSendRate, re-checked
periodically; a rate-state read/write failure fails closed (denies
admission) rather than risk exceeding the real AWS quota blind.
Private GET /internal/mail/health returns sanitized component states and
counters. GET /internal/mail/ready returns 503 during initial sync or storage
failure, otherwise 200. A provider circuit can be degraded while readiness
remains true so protected half-open probes can recover it. No credentials,
recipient addresses, message bodies, or pagination tokens appear in these sensors.
They are operational signals, not a guarantee SES will accept a message.
TERM/ctrl-c stops admission and SQS receives, then drains owned requests and the current feedback commit/ack within the configured deadline. Idle links close immediately. Deadline expiry aborts remaining tasks and exits nonzero; an interrupted send can have an unknown outcome.
Background workers (dispatcher, SQS feedback, rate discovery, suppression
reconciliation, rogue watcher) run under an OTP-style supervisor
(supervisor-machines): a worker that panics or exits is restarted on its
own, after 1 s doubling to 30 s while it keeps dying. More than 3 restarts in
60 s, or the docking listener stopping, drains the whole ship and exits
nonzero for Mothership to relaunch.
Security: the route currently has no application-level authentication.
/internal/ is a name, not an access control. Keep the bind private. The client
does not follow redirects or use environment HTTP proxies.
Always on, no configuration: toto@example.com is syntactically a real
address, but RFC 2606 reserves example.com/.net/.org and the
test/example/invalid/localhost TLDs for documentation - no real
mailbox ever lives there. Rejected at the same boundary as a malformed
address (SendRequest::validate), before any guard in the chain runs, for
both the synchronous and queued adapters. Not a policy choice like the
domain blocklist below - there is nothing to configure or an operator could
turn off.
flowchart LR
A[SendRequest] --> B[impossible-address check]
B --> C["domain blocklist<br/>(free, local)"]
C --> D["suppression<br/>(free, local)"]
D --> E["verification<br/>(paid, external)"]
E --> F[SES]
Always active - an empty hailer.domain_blocks table is a no-op, there is
no toggle. A pattern is an exact domain (guerrillamail.com) or a
*.-prefixed suffix glob (*.kp matches that domain and any subdomain).
A small hand-curated list of known disposable-mail providers ships with the
repo (source = 'curated', migration 05_domain_blocks.sql); it only grows
by upgrading Hailer, never fetched from a third party at runtime. Operators
add their own:
hailer ban-domain --pattern "*.kp" --reason "sanctioned TLD"
hailer unban-domain --pattern "*.kp"unban-domain only removes an operator-sourced pattern - a curated one
cannot be removed this way, by design.
flowchart LR
A[SendRequest] --> B["suppression check<br/>(cheap, local)"]
B --> C["address verification<br/>(paid, external)"]
C --> D[SES]
Opt-in pre-send deliverability check, off by default. Enable with
[email_verification] in hailer.toml:
[email_verification]
provider = "emailable"
reject_states = ["undeliverable"] # risky/unknown fail open unless added here
cache_ttl_seconds = 604800
timeout_seconds = 5Set EMAILABLE_API_KEY in the process environment, never in hailer.toml -
the same rule as HAILER_DATABASE_URL and AWS credentials. Suppression
filters first since it is a free local lookup; verification only runs for
recipients that survive it, so a suppressed address never costs a paid
lookup. Verdicts are cached in hailer.address_verifications (a week by
default) - a retried send against the same address reuses the cached verdict
instead of paying for another check.
A verifier outage, timeout, or an unknown/risky result all fail open: this
middleware is never a new reason a real send stops working unless a
deployment explicitly adds those states to reject_states. Adding another
provider (ZeroBounce, NeverBounce, ...) means a new Verifier enum variant in
src/verify.rs; nothing else in the send path changes.
A separate concern from the real-time guard above: pre-verify a whole list before ever trying to send to it, not a per-send check. Emailable's batch API is async (submit, then poll) and only returns per-address results inline for batches up to 1,000 - larger ones get a ZIP/CSV download link this client deliberately does not fetch or parse, to avoid a zip/csv dependency:
hailer verify-batch --file addresses.txt # one address per line, max 1000
hailer batch-status --id BATCH_ID # poll; caches each verdict once completebatch-status writes every returned verdict into hailer.address_verifications
(the same cache the real-time guard reads), so a later send to an
already-screened address reuses the cached verdict instead of paying for
another single-address check. A batch over 1,000 addresses is reported with
its download_file URL but is not cached - split into smaller batches
instead.
verify-batch never submits an address that does not need it: exact
duplicate lines (by the same normalized key the cache itself is keyed by -
USER+alias@x and user@x are the same address), and anything already in
hailer.address_verifications with an unexpired verdict, are dropped before
the batch is ever submitted - never spending a credit on them. Every
exclusion is counted and reported (invalid_lines, duplicate_lines,
already_cached), never silently dropped. hailer emailable-account checks
the Emailable account's remaining credits directly; submit_batch itself
also checks this before submitting and refuses (without ever reaching the
server) if the account has none left or fewer than the batch needs.
Operator-only, standalone (no Mothership, no docking - same tier as
migrate/sync-suppressions), for checking the AWS side of mail delivery
without ever sending anything:
hailer ses-account # quota, reputation, enforcement status
hailer ses-identities --identities commonshare.com,insights.commonshare.comses-identities checks each comma-separated identity independently - one
unverified or nonexistent identity is reported inline (error/code) and
never stops the others from reporting.
An HTTP 200 means SES accepted the message, not that the recipient received
it. A successful response must include a nonblank message_id.
| Failure | Ruby exception |
|---|---|
| HTTP 400 / 422 | Hailer::ValidationError |
| HTTP 502 | Hailer::RejectedError |
| HTTP 429 | Hailer::RateLimitedError |
HTTP 503, code: "outcome_unknown" |
Hailer::AmbiguousDeliveryError |
| HTTP 503 | Hailer::UnavailableError |
| Other non-200 status | Hailer::DeliveryError |
| Invalid/oversized response, missing message ID | Hailer::ProtocolError |
| Connection, TLS, or timeout failure | Hailer::TransportError |
| Missing/invalid endpoint or timeout settings | Hailer::ConfigurationError |
The synchronous adapter does not retry. A lost response can happen after SES accepted the message; retrying may duplicate it. This route does not use the opt-in outbound spool or its submission-key deduplication. Rails jobs can also retry, so configure them deliberately. The SES SDK makes one send attempt. AWS throttling/transient failures return 503, permanent service rejections 502, and disconnects/timeouts/missing message IDs carry the explicit unknown-outcome code. None of these responses authorizes an automatic retry of an ambiguous send. Postmark sends are single-attempt too: throttling, a refused token and 5xx return 503, other refusals 502, and lost or unreadable answers the unknown-outcome code.
Every synchronous send runs on one budget (deadline-machines): Mothership's
request_timeout for the bay, read from MS_BAY_DRAIN_TIMEOUT (60 s when
unset), less one second to write the reply. It starts when the request
arrives; address verification and the provider call draw from what is left. Verification never
spends the last 5 s, and a lookup that cannot finish in what is left counts as
unknown. The provider call's timeout is cut to the remaining budget; with
under 2 s left nothing is sent and the answer is 503 code: "budget_spent"
(Hailer::UnavailableError, safe to retry). Queued dispatch keeps the
provider's own 20 s timeout.
rogue-machines lets an operator make a dependency fail, stall or lie on
purpose, to watch the breaker, the budgets and the Rails side react. Nothing
happens unless [rogue] is configured; replicas with it poll
hailer.rogue_experiments every poll_interval_seconds (default 5).
hailer arm-experiment --target provider --fault raise --percent 10 --expires-in-seconds 300
hailer arm-experiment --target verifier --fault delay --delay-ms 4000 --expires-in-seconds 600
hailer list-experiments
hailer disarm-experiment --target provider # or no --target to disarm everythingOne experiment per target; arming again replaces it. Every experiment expires, at most 24 hours after arming, and the expiry is counted down on each host's monotonic clock from the time PostgreSQL says is left.
| Target | raise |
delay |
garble |
|---|---|---|---|
provider (SES/Postmark) |
unreachable: nothing sent; sync sends answer 503 and count against the breaker, queued dispatch retries | served inside the send timeout; at least as long is an unknown outcome (queued: held_unknown) |
sent, but the id comes back as rogue-<id> |
feedback (SQS) |
receive fails, consumer backs off | receive starts late | bodies replaced and left unacknowledged, so SQS redelivers them |
verifier (Emailable) |
lookup fails open as unknown |
served inside the lookup budget | refused: a garbled verdict would be cached past the experiment |
The delivery contract, Hailer-side feedback processing, domain blocklist,
impossible-address rejection, address verification, the queued/quarantine
pipeline, and SES account diagnostics are all implemented and shipped.
[queued], [domain_intel], [suppression_sync], [feedback], and
[email_verification] ship disabled by default - absent config, not
missing code.
bundle install
bundle exec rake
HAILER_TEST_DATABASE_URL=postgres://test_user:password@127.0.0.1/hailer_test cargo test --locked --features ses,postmark
cargo fmt --all -- --checkThe Ruby suite exercises real local HTTP exchanges and a booted Rails app.
The Rust suite checks the same JSON fixture and the docking exchange with a
fake provider. Feedback tests cover redelivery, acknowledgement ordering,
atomic rollback, restart persistence, imports, and suppression enforcement.
Rust tests require an isolated PostgreSQL server and a test role with CREATEDB;
each test creates and drops its own database. Never point this variable at a
customer or Rails production database. Neither suite sends real mail or requires
AWS credentials.
HAILER_BINARY is an explicit, version-checked source-build override; there is
no download or PATH fallback.
Bundler selects the matching native platform gem automatically; do not set
force_ruby_platform in deployment if you want the bundled executable.