Product Guide: Developer: Backend

Backend Architecture

Our architecture is built in Go using the default net/http package and httprouter, a high-performance router for net/http. We use Negroni for middleware, which allows us to modify the request and responses for all handlers, say, for example, to block potential CSRF attacks. On the data end, rather than relying on an ORM we make manual requests to the database using the pq driver for Postgres and the Squirrel SQL generator. (Data validation largely occurs at the database level, rather than at the application server.) For authentication, we manually forked and modified an existing CAS library for our needs.

In general, our architecture favors composition of small, stable packages rather than any monolithic framework.

The server code is organized loosely in the MVC pattern, without the views, since the frontend takes care of the view, so we can just render out the data as JSON. More applicably, every (non-helper) function in the folder represents the handler for a path that is dispatched by the router as necessary.

database.sql       a dump of the database schema, used to track the database schema to the code that expects it
emails.go          hooks for sending emails
helpers.go         useful heleprs for controllers
listings.go        CRUD operations and starring for listings
main.go            entrypoint, includes initializers and middleware
photos.go          http handler for uploading photos
router.go          route to controller mappings
savedsearches.go   CRUD operations for watched searches
seeks.go           CRUD operations for seeks
users.go           user login, logout, and login check

Each non-helpers file in server has a corresponding file in server/models that handles making requests to the database and web services, with the exception of photos.go, where the upload logic was simple enough as not to warrant a model.

CRUD Handlers

Each of the main datatypes have some subset of the create-read-update-delete operation handlers, as defined and named below. All handlers should return HTTP-compliant response status codes, and further descriptions are available.

• ReadDatatypes: read multiple of the datatype, potentially filtered by query parameters
• ReadDatatype: reads the specified datum
• CreateDatatype: creates the specified datum, and return it
• UpdateDatatype: updates the specified datum
• DeleteDatatype: deletes the specified datum

Custom Handlers

We try to avoid custom handlers and push for resourceful routing whenever possible to reduce the number of design decisions we need to make. Here are important custom handlers that handle key functionality:

CreatePhoto in photos.go receives a photo, resizes it (using the imaging library), uploads to S3, and returns the URL of the uploaded photo. Note that it stores the user that uploaded the photo in the object name in S3, for accountability reasons.

ContactListing and ContactSeek in emails.go sends a contact email to the owner of a listing or seek, and notifies the user of this email. The template is set using Sendgrid transactional templates.

UpdateListingStar in listings.go sets whether the user favorited a listing. This is implemented by adding or removing a row to the listings-stars gerund table.

GetCurrentUser in users.go returns with the profile of the current user. Creates the user if they do not yet exist. (This is secure because we are using CAS as our source of truth, not the user.)

RedirectUser in users.go does one of two things: - If logged in, redirects back to the given redirect URL (sanitizing for validity and correct host origin). - Otherwise, redirects to CAS login with the proper parameters.

LogoutUser in users.go does one of two things: - If logged out, redirects to the root of the page. - Otherwise, redirects to CAS logout with the proper parameters.

Middleware

All middleware is defined in server/main.go. Below is a brief summary of what each middleware does:

• casMiddleware: enables CAS authentication in handlers
• sentryMiddleware: reports any handler panics to Sentry
• logMiddleware: logs http requests to the console
• corsMiddleware: sends CORS headers as appropriate
• csrfMiddleware: blocks potential cross-site request forgery, as outlined below in "Security"

Search Alerting

After any user creates or updates a listing, we dispatch a goroutine to check whether the listing matches any watches. (This is performant because goroutines are non-blocking, so this check happens asynchronously, and allows us to spend more time on checking without slowing down the user.) The code that does this is available in server/models/savedsearches.go in the checkNewListing function. Essentially, we check by writing a SQL query that is the inverse of what we used to write the query in the first place--so if the user, for example, searches for a minimum expiration date, we query for watches whose maximum expiration dates are less than the listing's. For every match, we send an email to the owner of the saved search alerting of the match.

Semantic Indexing

Like for search alerting, we dispatch a goroutine on listing and seek creation (we'll call them "posts" from now on as a general term) and update to index the post's keywords. This process is found in server/models/search.go, with indexListing as the entrypoint.

1. Get the post title and description, and concatenate them (we'll call this, as an egregious abuse of NLP terminology, the "corpus")
2. Split the corpus into a set of words and drop all stop words
3. For each word, add the word's synonyms from Princeton's WordNet database (this database is stored in memory, from a forked version of a Go package called wnram)
4. Deduplicate the words, and add them to the listing's keywords field

When searching, semantic matches are very simple--we simply add a query that checks whether the set search terms in the query and the keywords in the site intersect at all. This is about a 10ms operation in practice.