Skip to content
seurosPublic

About

Rails renders the email. Hailer handles SES delivery, feedback, and durable suppressions: a Rails gem backed by a Rust mail ship for Mothership.

Topics

Resources

Stars

5 stars

Watchers

0 watching

Forks

Repository files navigation

Hailer

Rails writes the email. Hailer deals with sending it.

This repository is two things:

  • hailer gem — an ActionMailer adapter, feedback client, and executable launcher. Platform gems bundle the Rust binary. No Ruby AWS SDK or native extension.
  • hailer crate — a standalone Rust mail ship, supervised by Mothership. Sends through SESv2 or Postmark.
flowchart LR
    A[ActionMailer] --> B[hailer gem]
    B --> C[private Mothership route]
    C --> D[hailer binary]
    D --> E[SES]
Loading

An opt-in durable queued adapter sits alongside the default synchronous send path ([queued] in hailer.example.toml).

Install

gem "guardship"
gem "shuttlebay"
gem "hailer"

Mothership picks up the bay and starts sending mail. Keep using your mailers, interceptors, observers, deliver_now, and deliver_later - nothing else about how Rails sends mail changes.

Everything past this point is reference material: protocol internals, TOML knobs, and operator commands for people running or debugging the service, not required reading to use it.

Rails

Requires Ruby 4.0+ and Rails 8.1 or newer within Rails 8.x. Rails main (8.2 alpha) is also exercised. Older releases are not supported.

The Railtie registers and selects Hailer automatically. Set the private service URL in the Rails process's environment:

HAILER_URL=http://127.0.0.1:9293/internal/mail/send

No application initializer is needed. To customize connection settings:

config.action_mailer.hailer_settings = {
  endpoint: ENV.fetch("HAILER_URL"),
  open_timeout: 5,
  read_timeout: 30,
  write_timeout: 30
}

HAILER_URL is the complete private URL, for example http://127.0.0.1:9293/internal/mail/send. It is also the default endpoint when omitted from hailer_settings. There is no default host or fallback to SES. The app can explicitly override the default, for example config.action_mailer.delivery_method = :test in its test environment.

The adapter sends rendered MIME and separate envelope recipients, preserving explicit smtp_envelope_from and smtp_envelope_to overrides. After SES accepts the message, it sets message.header[:ses_message_id].

Rust service

Platform gems install the hailer executable, ready for Mothership to launch. Installing the gem does not start the service or send mail.

The source (ruby) gem has the same Rails client and launcher, but no binary. For source development or an unsupported platform:

cargo install --path . --locked --features ses,postmark
export HAILER_BINARY=/absolute/path/to/hailer

Each provider is a cargo feature, and a build needs at least one. Platform gems carry both. On glibc Linux the build needs libclang (libclang-dev): rama generates its libresolv bindings with bindgen.

Feature Adds
ses SES sending, SQS feedback, suppression sync, rate discovery, ses-* and sync-suppressions commands (hailer's own SigV4 client, no AWS SDK)
postmark MIME parsing for Postmark's JSON API, postmark-server command

A binary without the configured provider refuses to start and names the feature to rebuild with.

Run it as a Mothership HTTP bay. Merge this example into your fleet manifest; the bind must stay private and must not be routed on a public listener:

[mothership.bind]
mail = "127.0.0.1:9293"

[[bays.http]]
name = "hailer"
command = "hailer"
workers = 0
threads = 2
routes = [{ bind = "mail", pattern = "^/internal/mail/(send|events|health|ready)$" }]

Mothership exports MS_PID, MS_SHIP, and MS_SOCKET_DIR. Hailer binds its docking Unix socket; it does not run another HTTP server. For containers, use a private network listener with network policy instead of a loopback address if Rails runs in a different network namespace.

Give the service, not the Rails adapter, AWS region and credentials. Hailer looks where the AWS SDKs do, in this order: AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY (+ AWS_SESSION_TOKEN); the shared ~/.aws/credentials and ~/.aws/config files (AWS_PROFILE, static keys); an EKS web identity (AWS_WEB_IDENTITY_TOKEN_FILE + AWS_ROLE_ARN); an ECS or EKS Pod Identity container role; the EC2 instance role over IMDSv2. A profile that needs SSO, credential_process or role chaining is refused with the key it uses. The region comes from AWS_REGION, AWS_DEFAULT_REGION, the profile, or IMDS. AWS_ENDPOINT_URL_SESV2, _SQS and _STS, else AWS_ENDPOINT_URL, point those calls elsewhere (LocalStack, a test fixture) unless AWS_IGNORE_CONFIGURED_ENDPOINT_URLS=true. Its identity needs ses:SendEmail, and the SES account/identities must be configured for sending.

Postmark

provider = "postmark"

[postmark]
message_stream = "outbound"  # the default

Set POSTMARK_SERVER_TOKEN in the service's environment, never in hailer.toml. hailer postmark-server checks the token and names its server.

Postmark's API takes JSON, not raw MIME, so Hailer parses the Rails message and Postmark rebuilds it. Recipients, subject, text/HTML bodies, attachments (inline images by cid:), Reply-To, Message-ID and other custom headers carry over; the MIME structure itself does not. Recipients come from the envelope, as with SES. An explicit smtp_envelope_from only stands in for a missing From header: Postmark owns the bounce address.

The SES-only settings (ses_configuration_set, [feedback], [suppression_sync], rate_limit.discover_from_aws) are refused with provider = "postmark". Postmark bounces and suppressions are not read yet, so a queued unknown outcome waits for hailer resolve-message. Postmark refuses a message (406, suppressed) only when every recipient is inactive in its suppression list; otherwise it accepts and skips the inactive ones, which Hailer logs as a warning with the count, never the addresses.

PostgreSQL

PostgreSQL is the only live backend. Hailer owns a hailer schema, not Rails tables. Use a separate database/role where practical:

export HAILER_DATABASE_URL='postgres://hailer:password@db/hailer?sslmode=verify-full'
hailer migrate

Supply the database CA through the normal PostgreSQL connection options. Use certificate verification in deployment; a private network is not TLS. Keep the URL in a secret manager, not TOML or a committed initializer. Startup also applies embedded, versioned migrations. The bootstrap identity needs database CREATE and ownership of its schema/tables. Runtime startup currently runs that same bootstrap/migrator, so provision the service role for it too; a separate DML-only runtime mode is not implemented. The runtime never needs Rails schema ownership. Back up the database with your normal PostgreSQL backup/restore tooling.

Configuration defaults: pool size 5 (range 1–64), shutdown timeout 30 seconds (range 1–300). Every replica shares PostgreSQL; each has its own pool and optional rate budget. Set your supervisor/container stop grace longer than Hailer's shutdown timeout. PostgreSQL 18 is exercised locally and in CI.

Bounces, complaints, and suppressions

flowchart LR
    A[SES] --> B[SNS]
    B --> C[dedicated SQS queue]
    C --> D[Hailer]
    D --> E[durable events + suppressions]
Loading

Hailer atomically commits each event, deduplication key, and suppression change to PostgreSQL before acknowledging SQS. Failed acknowledgements can be retried without duplicating stored events. Invalid/untrusted messages are left for SQS's dead-letter policy.

  • Permanent bounces and complaints suppress recipients.
  • Transient/undetermined bounces are recorded without permanent suppression.
  • An explicit not-spam complaint does not create a new suppression.
  • Suppression checks cover the actual To/Cc/Bcc envelope, including overrides. Matching is case-insensitive and ignores +alias tags.
  • Mixed messages keep allowed recipients and their original addresses; MIME is unchanged. If every recipient is suppressed, Hailer returns HTTP 422 without calling SES. A suppression lookup failure fails closed with HTTP 503.

Copy hailer.example.toml, set the real queue/topic, and pass --config /path/to/hailer.toml or HAILER_CONFIG. Without [feedback], the consumer is disabled, but previously stored suppressions are still enforced. HAILER_DATABASE_URL is required, even with feedback disabled. The old state_path, --state-path, and HAILER_STATE_PATH are not supported. Replicas may consume the same dedicated queue when they share the same Hailer database. Commits serialize cursor allocation so a reader cannot skip an earlier transaction that has not committed yet.

Event history and deduplication have no automatic retention/pruning. Do not delete events casually: event IDs also carry the SQS deduplication contract, and Rails consumers may still need their cursors. Plan retention against consumer lag and redelivery windows before adding a pruning policy.

Hailer also exposes a provider-neutral, nondestructive event feed:

POST /internal/mail/events
Content-Type: application/json

{"after":0,"limit":100}

Route it on the same private bind, if application tracking needs it:

routes = [{ bind = "mail", pattern = "^/internal/mail/(send|events)$" }]

The gem validates the page and exposes immutable FeedbackEvent objects:

page = Hailer::FeedbackClient.new(endpoint: ENV.fetch("HAILER_URL")).poll(after: 0)
page.events.each { |event| puts [event.event_id, event.kind, event.message_id] }

Configure a Rails business-history handler (a callable or reloadable class name):

Rails.application.config.hailer.feedback_handler = "Platform::MailFeedback"

Then Hailer::FeedbackConsumer.new.call(after: cursor) calls that handler for each validated event and returns the next cursor only after all callbacks succeed. A callback must raise on failure. Persist the cursor and domain updates in one application database transaction. Retried pages require idempotent handlers; no hidden poller, scheduling, or checkpoint database is started by the gem. Reading does not acknowledge or erase events.

The SQS policy must allow only the intended SNS topic, using aws:SourceArn and account restrictions. Subscribe with RawMessageDelivery=false, restrict direct sqs:SendMessage, set a dead-letter queue, and monitor its depth. The SNS topic allowlist in TOML is defense in depth, not signature verification. Hailer needs sqs:ReceiveMessage and sqs:DeleteMessage on that queue, plus applicable KMS decrypt permissions if customer-managed encryption is used.

Import AWS suppressions

hailer sync-suppressions
# If AWS expires a saved pagination token:
hailer sync-suppressions --restart
# Existing application export, JSON array:
hailer import-suppressions --file suppressions.json

The service identity needs ses:ListSuppressedDestinations; STS GetCallerIdentity identifies the account (AWS does not require an explicit allow for that operation). The configured region is part of the source identity. These commands read AWS only; they never mutate the account suppression list. Missing AWS entries do not unsuppress local entries, and a bounce never downgrades a complaint. Imports do not fabricate delivery-history events. Account-level AWS data does not include every possible global SES suppression.

Enable [suppression_sync] as in the example to reconcile on startup and hourly. The first complete sync gates send readiness. Later errors mark health degraded, retain committed pages, and retry safe reads with bounded backoff. A token is stored with its page in one transaction. Replicas coordinate a source-scoped advisory lock; they cannot advance the same checkpoint concurrently. An expired token deliberately requires the operator's --restart. Omitting the section disables scheduled sync, not existing suppression enforcement.

Control plane

The behavior circuit evaluates pure admission decisions: input validity, service/storage readiness, suppression, and rate admission. PostgreSQL, AWS calls, clocks, and token reservations remain owned by the async host. The SES circuit breaker counts transient/ambiguous failures, not bad addresses or permanent request rejections. An optional [rate_limit] budgets envelope recipients through a single shared Postgres-backed token bucket - every replica draws down the same account-wide budget, not an independent per-process one. discover_from_aws = true replaces the configured rate with the SES account's real SendQuota.MaxSendRate, re-checked periodically; a rate-state read/write failure fails closed (denies admission) rather than risk exceeding the real AWS quota blind.

Private GET /internal/mail/health returns sanitized component states and counters. GET /internal/mail/ready returns 503 during initial sync or storage failure, otherwise 200. A provider circuit can be degraded while readiness remains true so protected half-open probes can recover it. No credentials, recipient addresses, message bodies, or pagination tokens appear in these sensors. They are operational signals, not a guarantee SES will accept a message.

TERM/ctrl-c stops admission and SQS receives, then drains owned requests and the current feedback commit/ack within the configured deadline. Idle links close immediately. Deadline expiry aborts remaining tasks and exits nonzero; an interrupted send can have an unknown outcome.

Background workers (dispatcher, SQS feedback, rate discovery, suppression reconciliation, rogue watcher) run under an OTP-style supervisor (supervisor-machines): a worker that panics or exits is restarted on its own, after 1 s doubling to 30 s while it keeps dying. More than 3 restarts in 60 s, or the docking listener stopping, drains the whole ship and exits nonzero for Mothership to relaunch.

Security: the route currently has no application-level authentication. /internal/ is a name, not an access control. Keep the bind private. The client does not follow redirects or use environment HTTP proxies.

Impossible addresses

Always on, no configuration: toto@example.com is syntactically a real address, but RFC 2606 reserves example.com/.net/.org and the test/example/invalid/localhost TLDs for documentation - no real mailbox ever lives there. Rejected at the same boundary as a malformed address (SendRequest::validate), before any guard in the chain runs, for both the synchronous and queued adapters. Not a policy choice like the domain blocklist below - there is nothing to configure or an operator could turn off.

Domain blocklist

flowchart LR
    A[SendRequest] --> B[impossible-address check]
    B --> C["domain blocklist<br/>(free, local)"]
    C --> D["suppression<br/>(free, local)"]
    D --> E["verification<br/>(paid, external)"]
    E --> F[SES]
Loading

Always active - an empty hailer.domain_blocks table is a no-op, there is no toggle. A pattern is an exact domain (guerrillamail.com) or a *.-prefixed suffix glob (*.kp matches that domain and any subdomain). A small hand-curated list of known disposable-mail providers ships with the repo (source = 'curated', migration 05_domain_blocks.sql); it only grows by upgrading Hailer, never fetched from a third party at runtime. Operators add their own:

hailer ban-domain --pattern "*.kp" --reason "sanctioned TLD"
hailer unban-domain --pattern "*.kp"

unban-domain only removes an operator-sourced pattern - a curated one cannot be removed this way, by design.

Address verification

flowchart LR
    A[SendRequest] --> B["suppression check<br/>(cheap, local)"]
    B --> C["address verification<br/>(paid, external)"]
    C --> D[SES]
Loading

Opt-in pre-send deliverability check, off by default. Enable with [email_verification] in hailer.toml:

[email_verification]
provider = "emailable"
reject_states = ["undeliverable"]  # risky/unknown fail open unless added here
cache_ttl_seconds = 604800
timeout_seconds = 5

Set EMAILABLE_API_KEY in the process environment, never in hailer.toml - the same rule as HAILER_DATABASE_URL and AWS credentials. Suppression filters first since it is a free local lookup; verification only runs for recipients that survive it, so a suppressed address never costs a paid lookup. Verdicts are cached in hailer.address_verifications (a week by default) - a retried send against the same address reuses the cached verdict instead of paying for another check.

A verifier outage, timeout, or an unknown/risky result all fail open: this middleware is never a new reason a real send stops working unless a deployment explicitly adds those states to reject_states. Adding another provider (ZeroBounce, NeverBounce, ...) means a new Verifier enum variant in src/verify.rs; nothing else in the send path changes.

Bulk pre-screening (batch)

A separate concern from the real-time guard above: pre-verify a whole list before ever trying to send to it, not a per-send check. Emailable's batch API is async (submit, then poll) and only returns per-address results inline for batches up to 1,000 - larger ones get a ZIP/CSV download link this client deliberately does not fetch or parse, to avoid a zip/csv dependency:

hailer verify-batch --file addresses.txt   # one address per line, max 1000
hailer batch-status --id BATCH_ID          # poll; caches each verdict once complete

batch-status writes every returned verdict into hailer.address_verifications (the same cache the real-time guard reads), so a later send to an already-screened address reuses the cached verdict instead of paying for another single-address check. A batch over 1,000 addresses is reported with its download_file URL but is not cached - split into smaller batches instead.

verify-batch never submits an address that does not need it: exact duplicate lines (by the same normalized key the cache itself is keyed by - USER+alias@x and user@x are the same address), and anything already in hailer.address_verifications with an unexpired verdict, are dropped before the batch is ever submitted - never spending a credit on them. Every exclusion is counted and reported (invalid_lines, duplicate_lines, already_cached), never silently dropped. hailer emailable-account checks the Emailable account's remaining credits directly; submit_batch itself also checks this before submitting and refuses (without ever reaching the server) if the account has none left or fewer than the batch needs.

SES account diagnostics

Operator-only, standalone (no Mothership, no docking - same tier as migrate/sync-suppressions), for checking the AWS side of mail delivery without ever sending anything:

hailer ses-account                                        # quota, reputation, enforcement status
hailer ses-identities --identities commonshare.com,insights.commonshare.com

ses-identities checks each comma-separated identity independently - one unverified or nonexistent identity is reported inline (error/code) and never stops the others from reporting.

Failure semantics

An HTTP 200 means SES accepted the message, not that the recipient received it. A successful response must include a nonblank message_id.

Failure Ruby exception
HTTP 400 / 422 Hailer::ValidationError
HTTP 502 Hailer::RejectedError
HTTP 429 Hailer::RateLimitedError
HTTP 503, code: "outcome_unknown" Hailer::AmbiguousDeliveryError
HTTP 503 Hailer::UnavailableError
Other non-200 status Hailer::DeliveryError
Invalid/oversized response, missing message ID Hailer::ProtocolError
Connection, TLS, or timeout failure Hailer::TransportError
Missing/invalid endpoint or timeout settings Hailer::ConfigurationError

The synchronous adapter does not retry. A lost response can happen after SES accepted the message; retrying may duplicate it. This route does not use the opt-in outbound spool or its submission-key deduplication. Rails jobs can also retry, so configure them deliberately. The SES SDK makes one send attempt. AWS throttling/transient failures return 503, permanent service rejections 502, and disconnects/timeouts/missing message IDs carry the explicit unknown-outcome code. None of these responses authorizes an automatic retry of an ambiguous send. Postmark sends are single-attempt too: throttling, a refused token and 5xx return 503, other refusals 502, and lost or unreadable answers the unknown-outcome code.

Every synchronous send runs on one budget (deadline-machines): Mothership's request_timeout for the bay, read from MS_BAY_DRAIN_TIMEOUT (60 s when unset), less one second to write the reply. It starts when the request arrives; address verification and the provider call draw from what is left. Verification never spends the last 5 s, and a lookup that cannot finish in what is left counts as unknown. The provider call's timeout is cut to the remaining budget; with under 2 s left nothing is sent and the answer is 503 code: "budget_spent" (Hailer::UnavailableError, safe to retry). Queued dispatch keeps the provider's own 20 s timeout.

Fault injection

rogue-machines lets an operator make a dependency fail, stall or lie on purpose, to watch the breaker, the budgets and the Rails side react. Nothing happens unless [rogue] is configured; replicas with it poll hailer.rogue_experiments every poll_interval_seconds (default 5).

hailer arm-experiment --target provider --fault raise --percent 10 --expires-in-seconds 300
hailer arm-experiment --target verifier --fault delay --delay-ms 4000 --expires-in-seconds 600
hailer list-experiments
hailer disarm-experiment --target provider   # or no --target to disarm everything

One experiment per target; arming again replaces it. Every experiment expires, at most 24 hours after arming, and the expiry is counted down on each host's monotonic clock from the time PostgreSQL says is left.

Target raise delay garble
provider (SES/Postmark) unreachable: nothing sent; sync sends answer 503 and count against the breaker, queued dispatch retries served inside the send timeout; at least as long is an unknown outcome (queued: held_unknown) sent, but the id comes back as rogue-<id>
feedback (SQS) receive fails, consumer backs off receive starts late bodies replaced and left unacknowledged, so SQS redelivers them
verifier (Emailable) lookup fails open as unknown served inside the lookup budget refused: a garbled verdict would be cached past the experiment

Status

The delivery contract, Hailer-side feedback processing, domain blocklist, impossible-address rejection, address verification, the queued/quarantine pipeline, and SES account diagnostics are all implemented and shipped. [queued], [domain_intel], [suppression_sync], [feedback], and [email_verification] ship disabled by default - absent config, not missing code.

Development

bundle install
bundle exec rake
HAILER_TEST_DATABASE_URL=postgres://test_user:password@127.0.0.1/hailer_test cargo test --locked --features ses,postmark
cargo fmt --all -- --check

The Ruby suite exercises real local HTTP exchanges and a booted Rails app. The Rust suite checks the same JSON fixture and the docking exchange with a fake provider. Feedback tests cover redelivery, acknowledgement ordering, atomic rollback, restart persistence, imports, and suppression enforcement. Rust tests require an isolated PostgreSQL server and a test role with CREATEDB; each test creates and drops its own database. Never point this variable at a customer or Rails production database. Neither suite sends real mail or requires AWS credentials.

HAILER_BINARY is an explicit, version-checked source-build override; there is no download or PATH fallback.

Bundler selects the matching native platform gem automatically; do not set force_ruby_platform in deployment if you want the bundled executable.

About

Rails renders the email. Hailer handles SES delivery, feedback, and durable suppressions: a Rails gem backed by a Rust mail ship for Mothership.

Topics

Resources

Stars

5 stars

Watchers

0 watching

Forks

Releases

Packages

Used by

Contributors

Languages