Handle Bad Content-Type Headers RESTfully

[thehabes]

Closes #245
Closes #246
Closes #248
Closes #256

Summary

Requests with wrong or missing Content-Type headers were causing 500 errors and connection resets instead of proper HTTP error responses. This was because express.json() left req.body unparsed, and controllers crashed accessing undefined properties — dropping the TCP connection before any response was sent.

Changes

• app.js — Restricted express.json() to only parse application/json and application/ld+json bodies. This also fixes a latent issue where application/ld+json bodies (core to a JSON-LD API) were never being parsed by the JSON middleware. Removed unnecessary express.urlencoded() middleware (this is a JSON API, not a form-processing app).
• utils.js — Relocated createExpressError() here from controllers/utils.js so it is available to both controllers and REST middleware. Fixed the bug (createExpressError overwrites err.code switch values unconditionally #256) where the switch (err.code) block set statusCode/statusMessage correctly, but unconditional fallback lines immediately overwrote both values. Restructured to set defaults in the initializer, then override for err.code === 11000.
• controllers/utils.js — Removed the old createExpressError() definition and its named export.
• All controllers (bulk.js, crud.js, delete.js, gog.js, history.js, overwrite.js, patchSet.js, patchUnset.js, patchUpdate.js, putUpdate.js, release.js, search.js) — Migrated all createExpressError(err) calls to utils.createExpressError(err). Removed superfluous console.log().
• rest.js — Added three per-route Content-Type validation middlewares applied before controllers:
  • verifyJsonContentType — requires application/json or application/ld+json. Applied to create, update, patch, set, unset, overwrite, query, bulkCreate, and bulkUpdate routes.
  • verifyTextContentType — requires text/plain. Not currently mounted on any route but available for future use.
  • verifyEitherContentType — requires application/json, application/ld+json, or text/plain. Applied to search routes.
  • All three return 415 Unsupported Media Type for missing, empty, multi-value, or unsupported Content-Type headers.
  • Routes that don't receive bodies (GET /id, DELETE /delete, PATCH /release) do not mount any Content-Type middleware.
  • Added 415 case to the messenger error handler.
  • Fixed pre-existing bug in messenger's 403 handler where err.message was used instead of error.message, causing the "Forbidden" text to be silently dropped from the response.
  • Truncated Bearer tokens in 401/403 error messages to token.slice(0, 15) to avoid exposing full credentials in responses.
• Route files (bulkCreate.js, bulkUpdate.js, create.js, overwrite.js, patchSet.js, patchUnset.js, patchUpdate.js, putUpdate.js, query.js, search.js) — Mounted the appropriate Content-Type middleware (rest.verifyJsonContentType or rest.verifyEitherContentType) on each route handler.
• routes/api-routes.js — Spacing fix only.
• routes/static.js — Removed unnecessary express.urlencoded() middleware.
• routes/__tests__/contentType.test.js — 15+ tests covering all three middlewares: valid types, rejected types, case insensitivity, charset parameters, PUT/PATCH methods, and Content-Type smuggling via comma injection.
• routes/__tests__/history.test.js, since.test.js — Increased Jest timeout to 10s to fix intermittent test timeouts.
• routes/__tests__/*.test.js — Removed express.urlencoded() from all test app setups to match production configuration.

In general, to fix missing rate limiting for database-backed route handlers in Express, we should use a rate-limiting middleware (such as express-rate-limit) and apply it to the specific sensitive routes or to the entire router. This middleware should be placed before the handler that performs the database access so that excessive request rates are rejected or delayed before reaching the database.

For this specific file, the best minimally invasive fix is to: (1) import express-rate-limit, (2) define a limiter instance configured for update operations (e.g., a reasonable number of PATCH/POST update requests per IP in a given time window), and (3) add this limiter as middleware to the router.route('/') chain before auth.checkJwt / rest.jsonContent / controller.patchUpdate. This ensures both the .patch and .post handlers share the same rate limiting, covering all the alert variants without changing existing business logic or response semantics. Concretely, inside routes/patchUpdate.js, add a RateLimit import near the other imports, define a patchUpdateLimiter (e.g., 100 requests per 15 minutes per IP, or a stricter value as desired), and include patchUpdateLimiter as the first middleware in the .patch and .post routes. No changes to the controller or auth logic are needed.

In general, you should protect expensive HTTP handlers (especially those that perform bulk database operations) by applying a rate-limiting middleware. For an Express-based router, a common approach is to use express-rate-limit and attach a limiter to the specific route or to the router as a whole. This ensures that regardless of how heavy controller.bulkCreate is, any single client can only invoke it a limited number of times in a defined time window.

For this specific file (routes/bulkCreate.js), the minimal, non-breaking fix is:

• Import express-rate-limit (CommonJS-style require is the idiomatic form for that library, but mixing import and require in the same file is legal in Node when using transpilation or appropriate settings, and we must not alter existing imports).
• Create a limiter instance configured for this bulk-create endpoint (for example, allowing a modest number of bulk requests per 15 minutes).
• Insert this limiter into the middleware chain of the POST route before controller.bulkCreate, so that requests exceeding the limit are rejected before hitting the database logic.
• Do not alter the behavior of other routes or middlewares.

Concretely:

• At the top of routes/bulkCreate.js, add const RateLimit = require('express-rate-limit') and define const bulkCreateLimiter = RateLimit({ ... }).
• Update line 11 so that the .post(...) call becomes .post(auth.checkJwt, rest.jsonContent, bulkCreateLimiter, controller.bulkCreate).

In general, to fix missing rate limiting you should introduce a rate-limiting middleware (e.g., using express-rate-limit) and apply it to routes that perform expensive operations such as database writes. The middleware should be placed before the heavy controller in the middleware chain so that excessive requests are rejected early.

For this specific route, the best minimal change is to add a rate limiter middleware function in routes/bulkUpdate.js and insert it into the .put(...) chain before controller.bulkUpdate. We will:

• Import express-rate-limit at the top of this file.
• Define a limiter instance (e.g., bulkUpdateLimiter) with a reasonable window and max setting, local to this router.
• Update line 11 to include bulkUpdateLimiter between rest.jsonContent and controller.bulkUpdate.

This preserves existing authentication and JSON processing, while adding rate limiting immediately before the database-heavy handler.

Concretely:

• At the top of routes/bulkUpdate.js, add import rateLimit from 'express-rate-limit'.
• Define a const bulkUpdateLimiter = rateLimit({...}) just after the imports, with sensible defaults (e.g., 100 requests per 15 minutes from a single IP).
• Change line 11 from .put(auth.checkJwt, rest.jsonContent, controller.bulkUpdate) to .put(auth.checkJwt, rest.jsonContent, bulkUpdateLimiter, controller.bulkUpdate).

No other behavior of the route changes, aside from rate limiting.

---

In general, the fix is to ensure that requests reaching the expensive controller.create database operation are subject to a rate limit. In an Express environment, this is typically done by adding a middleware like express-rate-limit that tracks requests per IP (or per user) over a time window and rejects excess requests.

In this file, the least invasive fix that preserves existing behavior is:

• Import express-rate-limit.
• Create a rate limiter instance configured for this specific route (e.g., allowing some number of create operations per IP per time window).
• Insert this limiter into the middleware chain before controller.create on the POST handler.

Concretely:

• At the top of routes/create.js, add an import rateLimit from 'express-rate-limit'.
• After creating the router (after const router = express.Router()), define a createRateLimiter with appropriate settings (e.g., a certain windowMs, max, and a simple JSON or text response when the limit is exceeded).
• Modify line 10 so that the middleware sequence is .post(auth.checkJwt, rest.jsonContent, createRateLimiter, controller.create).

No existing behavior for non-limited requests changes: authenticated, well-formed JSON POSTs will still reach controller.create, just with the added protection that very frequent requests will be rejected.

In general, fix this by adding an Express-compatible rate-limiting middleware in front of the expensive handler so that repeated requests from the same client are capped over a time window. The standard way is to use a well-known library such as express-rate-limit and apply a limiter either to this specific route or the router as a whole.

For this specific file (routes/overwrite.js), the minimal change that preserves existing behavior is:

1. Import express-rate-limit.
2. Define a limiter instance configured for this overwrite endpoint (for example, a modest limit like 60 requests per 15 minutes per IP, which you can later tune).
3. Insert this limiter into the middleware chain for PUT /, between auth.checkJwt and rest.jsonContent (or at the start). This ensures:
   • Authentication still happens as before.
   • The handler controller.overwrite runs only if the client is within the allowed request budget.
   • No other routes are affected.

Concretely:

• At the top of routes/overwrite.js, add import rateLimit from 'express-rate-limit'.
• After creating the router (after line 3), define const overwriteLimiter = rateLimit({ ... }).
• Update line 10 to:
  .put(auth.checkJwt, overwriteLimiter, rest.jsonContent, controller.overwrite)

In general, the problem is fixed by applying a rate-limiting middleware to routes that perform expensive operations (like database access). In an Express app, a common solution is to use the express-rate-limit package to limit the number of requests per IP over a defined time window, and to attach that limiter middleware to the vulnerable route or router.

For this specific file (routes/patchSet.js), the best minimally invasive fix is:

• Import express-rate-limit at the top of the file.
• Define a limiter instance configured for a sensible rate (e.g., 100 requests per 15 minutes per IP; you can later tune this).
• Apply this limiter to the router so that all HTTP methods on / that reach controller.patchSet are rate-limited.
  • Since all routes in this file are on the same router, adding router.use(limiter) after creating the router will ensure that .patch, .post, and .all on this router are all rate-limited, without changing existing behavior of the handlers themselves.

Concretely:

• At the top of routes/patchSet.js, add import rateLimit from 'express-rate-limit'.
• Immediately after const router = express.Router(), define const patchSetLimiter = rateLimit({ windowMs: 15 * 60 * 1000, max: 100 }) (or similar configuration).
• Then add router.use(patchSetLimiter) so the limiter is applied to all routes within this router.

No changes are needed to controller.patchSet or to the existing route chaining; we’re only inserting middleware in front of them.

In general, to fix missing rate limiting on an Express route that performs database access, you introduce a rate-limiting middleware (e.g., via express-rate-limit) and apply it either globally or specifically to the sensitive route(s). The middleware should be placed before the expensive handler in the middleware chain so that abusive traffic is rejected before reaching the database.

For this specific code, the least invasive and clearest fix is:

• Import a known rate-limiting library, such as express-rate-limit, at the top of routes/putUpdate.js without altering existing imports.
• Define a limiter configured for this update route only (e.g., a reasonable per-IP limit over a time window, such as 100 requests per 15 minutes). This preserves existing functionality while adding protection.
• Insert the limiter into the route definition before controller.putUpdate, so the middleware order becomes: auth.checkJwt, rest.jsonContent, limiter, controller.putUpdate.
• Leave the .all(...) handler and other existing behavior unchanged.

Concretely:

• In routes/putUpdate.js, add import rateLimit from 'express-rate-limit' near the top.

• Add something like:

  const updateLimiter = rateLimit({
      windowMs: 15 * 60 * 1000,
      max: 100,
  })

  after the imports and before using router.

• Update line 10 to:

  .put(auth.checkJwt, rest.jsonContent, updateLimiter, controller.putUpdate)

No other files need to be changed to address the alert variants, since all variants refer to the same route handler.

In general, the issue is fixed by ensuring that any route that triggers database access is protected by a rate-limiting middleware. In Express, the standard approach is to use a library such as express-rate-limit and apply a limiter either globally (via app.use) or to specific routes (by inserting the limiter in the route’s middleware chain). This prevents a client from issuing unlimited requests in a given time window.

For this specific file, the best fix with minimal behavioral change is to import express-rate-limit, define a rate limiter instance appropriate for query operations, and insert that limiter into the POST / route before rest.jsonContent and controller.query. This keeps the existing controller and JSON parsing logic untouched and only adds a protective wrapper. Concretely, in routes/query.js, we will:

• Add import rateLimit from 'express-rate-limit' alongside the other imports.
• Define a const queryLimiter = rateLimit({ ... }) with conservative defaults (e.g., 100 requests per 15 minutes per IP) near the top of the file.
• Update the route definition on lines 7–8 so that .post(rest.jsonContent, controller.query) becomes .post(queryLimiter, rest.jsonContent, controller.query).

No additional helper methods are required beyond the limiter instance, and no existing imports need to be modified.

In general, to fix missing rate limiting on Express routes that hit the database, we should introduce a rate-limiting middleware and apply it to the relevant routes before the expensive handlers. The common approach is to use a well-known library such as express-rate-limit, configure appropriate thresholds (e.g., N requests per window), and attach the limiter either to all routes in the router or only to specific sensitive endpoints.

For this file, the least invasive and clearest fix is:

• Import express-rate-limit.
• Create one or more limiter instances configured for search endpoints.
• Apply the limiter as middleware on the routes that call controller.searchAsWords and controller.searchAsPhrase, placing it before rest.eitherContent so it executes first and can reject excessive traffic before any body parsing or DB access.
• Keep existing behavior and messages intact for valid and invalid methods.

Concretely, within routes/search.js:

1. Add import rateLimit from 'express-rate-limit' (ES module form to match existing imports).

2. Define a searchLimiter constant, e.g.:

   const searchLimiter = rateLimit({
       windowMs: 15 * 60 * 1000,
       max: 100,
       standardHeaders: true,
       legacyHeaders: false,
   })

   This uses standard, sensible defaults while not altering any existing functionality of the search handlers themselves.

3. Attach searchLimiter to both router.route('/') and router.route('/phrase'), before rest.eitherContent:

   router.route('/')
       .post(searchLimiter, rest.eitherContent, controller.searchAsWords)
       ...

   and similarly for /phrase.

No other files need to be changed, and the core functional behavior of the handlers remains the same except they will now be rate-limited.

To fix this, add an Express rate-limiting middleware and apply it to the search routes so that requests hitting controller.searchAsPhrase (and ideally all search routes) are throttled. We should use a well-known package such as express-rate-limit rather than building custom logic. To avoid changing existing functionality, the limiter should sit before the existing middlewares (rest.eitherContent, controller methods) and simply block or delay excessive requests; normal traffic should behave exactly as before.

Concretely in routes/search.js:

1. Add an import for express-rate-limit near the other imports.
2. Define a limiter instance configured for a reasonable window and maximum, e.g. 100 requests per 15 minutes per IP (matching the example in the background).
3. Apply this limiter to the relevant routes. To cover both word and phrase search (and address both alert variants), apply it to the base '/' route and the /phrase route, by inserting it as a middleware argument before rest.eitherContent.

No existing imports need changing; we just add a new one. No other files are modified.