Thus far we’ve addressed modular design and API design concerns from a high-level perspective, but avoided plunging into the deep end of implementation details. In contrast, this chapter is devoted to advice and concrete actions we can take to improve the quality of our component implementations. We’ll discuss complexity, ways to remediate it, the perils of state, and how to better leverage data structures.
Every piece of code we write is a source of internal complexity, with the potential to become a large pain point for our codebase as a whole. That said, most bits of code are relatively harmless when compared to the entire corpus of our codebase, and trying to proof our code against complexity is a sure way of increasing complexity for no observable benefit. The question is, then, how do we identify the small problems before they grow into a serious threat to the maintainability of our project?
Making a conscious effort to track pieces of code that we haven’t changed or interacted with in a while, and identifying if they’re simple enough to understand can help us determine whether refactoring may be in order. We could perhaps set a rule whereby team members should watch out for garden paths in the codebase and fix them as they are making changes in the same functional area as the affected code. When we track complexity methodically, often, and across the entire team that’s responsible for a codebase, we can expect to see many small but cumulative gains in our battle against complexity.
In JavaScript, deep nesting is one of the clearest signs of complexity. Understanding the code at any given nesting level involves understanding how the flow arrives there, the state at every level in scope, how the flow might break out of the level, and which other flows might lead to the same level. Granted, we don’t always need to keep all this derived information in our memory. The problem is that, when we do, we might have to spend quite a few minutes reading and understanding the code, deriving such information, and otherwise not fixing the bug or implementing the feature that we had set out to resolve in the first place.
Nesting is the underlying source of complexity in patterns such as "Callback Hell", or "Promise Hell", where callbacks are nested on top of one another. The complexity has little to do with spacing, although when taken to the extreme it does make code harder to read. Instead, the complexity exists at the seams, where we need to fully understand the context in order to go deep into the callback chain and make fixes or improvements. An insidious variant of callback hell is the one where we have logic in every nesting level. This variant is coincidentally the one we can observe most often in real applications: we rarely have callbacks as depicted in the bit of code below, partly because it’s immediately obvious that something is wrong. We should probably either change the API so that we get everything we need at once, or we could leverage a small library that takes care of the flow while eliminating the deep nesting we’d otherwise have in our own code.
getProducts(products => {
getProductPrices(products, prices => {
getProductDetails({ products, prices }, details => {
// …
})
})
})When we have synchronous logic intermixed with asynchronous callbacks, things get more challenging. The problem here is, almost always, a coupling of concerns. When a program has a series of nested callbacks that also include logic in between, it can be a sign that we’re mixing flow control concerns with business concerns. In other words, our program would be in a better place if we kept the flow separate from the business logic. By splitting the code that purely determines the flow from the rest, we can better isolate our logic into its individual components. The flow, in turn, also becomes more clear because it’s now spelled out in plain sight instead of interleaved with business concerns.
Suppose that each nesting level in a series of callbacks contains about 50 lines of code. Each function in the series needs to reference zero, one, or more variables in its parent scope. If it needs zero references to its immediate parent scope, we can safely move it up to the same scope as its parent. We can repeat this process until the function is in the highest possible level where it can, given the variables it has to reference. When functions reference at least one variable from the parent scope, we could opt to leave them unchanged or to pass those references as parameters so that we can keep on decoupling the functions.
As we move logic into their own functions and flatten the callback chain, we’ll be left with the bare flow of operations being separate from the operations themselves. Libraries like contra can help manage the flow itself while user code worries about business logic.
As a module becomes larger, it also gets easier to mistakenly collapse distinct features together by interleaving their code in such a way that it is hard to reuse each feature independently, debug and maintain them, or otherwise extricate the features from one another.
For example, if we have a feature for notifying subscribers and a feature to send notifications, we could strive to keep the features apart by clearly defining how notifications can be constructed and handed off to a different service which then sends those notifications. That way, subscriber notifications can be sent through the notification service, but given the clear separation we won’t be letting subscriber-specific notions to get in the way of sending other kinds of notifications to our customers.
One way of reducing the risk of entanglement would be to design features upfront, being particularly on the lookout about concerns that could be componentized or otherwise clearly delineated. By doing a little work before sitting down to write code, we might avert the risks of tight coupling.
Being alert when reading old code can also be key in identifying what was previously a well-contained module that evolved to cover a broad range of concerns. We can then, over time, break apart these concerns into individual modules or better-isolated functions so that each concern is easier to maintain and understand separately.
Instead of trying to build a large feature all at once, it could come in handy to build it from the inside out, keeping each stage of the process in functions that live at the same level instead of being deeply nested. Doing this methodically will lead to better decoupling, as we’ll move away from monolithic structures and towards a more modular approach, where functions have smaller scopes and take what they need in the form of parameters.
When we’d have to repeat ourselves by passing a lot of scope variables as function parameters just to avoid nested functions, a light degree of nesting is desirable to avoid this repetition. In key functional boundaries where our concerns go from "gather model details" to "render HTML page" to "print HTML page to PDF", nesting will invariably lead to coupling and less reusability, which is why repeating ourselves a little bit may be warranted in these cases.
Conventions are useful because they allow for better self-direction amongst developers, without causing lagoons of inconsistency to spread across our codebase as fate would have it were we to allow a team of developers too much freedom without sound design direction and conventions that dictate how different portions of an application should be shaped. A large number of conventions might hinder productivity, especially if some of our conventions appeared to work as if by magic.
When it comes to conventions, frameworks are a special case. Frameworks are packed to the brim with conventions and best practices. Some of them live in the library and tooling ecosystem around the framework, while many live in the shape our code takes when we rely on said framework. Upon adopting a framework, we’re buying into its conventions and practices. Most modern JavaScript frameworks offer ways of breaking our application into small chunks, regardless of whether the framework is for the client or server.
Express has middleware and routes, Angular has directives, services, and controllers, React has components, and so on and so forth. These conventions and abstractions are tremendously helpful to keep complexity in check while building an application. As our components grow larger, regardless of the abstraction or framework of choice, things will get more complicated. At this moment we usually can refactor our code into smaller components that are then wrapped with larger ones, preserving separation of concerns and keeping complexity on a short leash.
Eventually, we’ll come into requirements that don’t exactly fit the mold proposed by our framework of choice. Generally, this means the required functionality belongs on a separate layer. For example, Express in Node.js is a framework concerned with handling HTTP requests and serving responses. If one of our API endpoints needs to result in an email being sent, we could embed email-sending logic in the controller for that API endpoint. However, if an API endpoint controller is already concerned with, say, publishing blog posts, then it would be hardly right to embed email-sending logic on that same controller, since it’s a different concern entirely. Instead, what we could do is create a subscribers service component, with functionality such as subscribe which adds a subscriber after verifying their email, and notify which takes care of sending the emails. Taking this idea further still, perhaps most of the work in subscribers.notify should occur via yet another service component called emails, which takes care of properly configuring our email sending capability, and also has functionality to turn would-be emails into plain console.log statements for quick access to the contents of the emails during debug sessions.
Having clearly defined layers is paramount to the design of effective and maintainable applications once we’re past the prototyping stages. Layers can be made up of components which follow the conventions proposed by the frameworks we use, or they can be self-imposed like the service layer we discussed in the previous paragraph. Using layers, and as long as we favor function parameters over scope for context-passing, we can introduce horizontal scaling by placing several orthogonal components alongside each other, without letting them run into each others' concerns.
Code is ever-evolving, and we’ll almost invariably end up with large projects that are not always the easiest to maintain. While we’ll reserve the following couple of sections for practical recommendations to reduce complexity at an architectural level, this section focuses on reducing complexity in portions of an application that are already complex.
Complex code is predominantly shorter than it should be, and often deceitfully so. An expression that might have involved 5 to 10 short lines of code usually ends up being represented in 1 or 2 clever lines of code. The problem with clever code is that we need to expend time and energy to read it whenever it’s intent is not clear on our mind, which is only the case when we first write said code or right after spending considerable time analyzing it.
One of the underlying issues that can be identified when reading complex code is that it uses few variables. In the dawn of programming, memory resources were scarce and thus programmers had to optimize allocation and this often meant reusing variables and using fewer of them. In modern systems, we don’t have the need to treat memory as a sacred, precious, and limited resource. Instead, we can focus on making programs readable to both our future selves and fellow developers.
Readability is better served by an abundance of properly named variables or functions than by sparsity. Consider the following example, part of a larger routine, where a program ensures that the user is currently logged in with a valid session, and otherwise redirects them to a login page.
if (
auth !== undefined &&
auth.token !== undefined &&
auth.expires > Date.now()
) {
// we have a valid token that hasn't expired yet
return
}As the routine becomes larger, we collect if statements with non-obvious or complicated clauses, such as the reason why we’re checking auth has a token value if we’re not doing anything with it here. The solution is usually to add a comment explaining the reason why this check exists. In this case, the comment tells us this is a valid token that hasn’t expired. We could turn that comment into code, and simplify the if statement in the process, by creating a small function that breaks down the conditional, as shown next.
function hasValidToken(auth) {
if (auth === undefined || auth.token === undefined) {
return false
}
const hasNotExpiredYet = auth.expires > Date.now()
return hasNotExpiredYet
}We can now turn our if statement plus comment into a function call, as shown in the following bit of code. Certainly, the totality of our refactored code is a bit longer, but now it’s self-descriptive. Code that describes what it does in the process of doing it doesn’t require as many comments, and that’s important because comments can become easily outdated. Moreover, we’ve extracted the long conditional in the if statement to a function, which keeps us more focused while parsing the codebase. If every condition or task was inline, we’d have to understand everything in order to understand how a program works. When we offload tasks and conditions to other functions, we’re letting the reader know they can trust hasValidToken to check for validity of the auth object, and the conditional becomes a lot easier to digest.
if (hasValidToken(auth)) {
return
}We could’ve used more variables without creating a function, inlining the computation of hasValidToken right before the if check. A crucial difference between the function-based refactor and the inlining solution is that we used a short-circuiting return statement to preemptively bail when we already knew the token was invalid[1], however we can’t use return statements to bail from the snippet of code that computes hasValidToken in the following piece of code without coupling its computation to knowledge about what the routine should return for failure cases. As a result, our only options are tightly coupling the inline subroutine to its containing function, or using a logical or ternary operator in the intermediate steps of the inlined computation.
const hasToken = auth !== undefined && auth.token !== undefined
const hasValidToken = hasToken && auth.expires > Date.now()
if (hasValidToken) {
return
}Both of these options have their downsides. If we couple the return statements with the parent function, we’ll need to be careful if we want to replicate the logic elsewhere, as the return statements and possibly their logic will have to adapt as well. If we decide to use ternary operators as a way of short-circuiting, we’ll end up with logic that might be as complex as what we originally had in the if statement.
Using a function not only avoids these two problems thanks to the ability to return intermediate results, but also defers reasoning about its contents until we actually need to understand how tokens are checked for validity.
While moving conditionals to a function might sound like a trivial task, this approach is at the heart of modular design. It is by composing small bits of complexity using several additive functions that we can build large applications that are less straining to read. A large pool of mostly trivial functions can add up to a veritable codebase where each bit of code is relatively isolated and easy to understand, provided we trust functions do what their name says they do. In this vein, it is of utmost importance to think long and deep about the name of every function, every variable, and every package, directory, or data structure we conceive.
When used deliberately and extensively, early returns — sometimes referred to as guard clauses or short-circuits — can be unparalleled when it comes to making an application as readable as possible. Let’s explore this concept in further detail.
When we have a long branch inside a conditional statement, chances are we’re doing something wrong. Pieces of code like the following are commonplace in real world applications, with a long success case branch taking up significant amounts of code while having several else branches sprinkled near the end that would log an error, throw, return, or otherwise perform a failure handling action.
if (response) {
if (!response.errors) {
// … use `response`
} else {
return false
}
} else {
return false
}In the example, we’re optimizing readability for the success case, while the failure handling is relegated to the very end of our piece of code. There’s several problems with this approach. For one, we have to indulge in unnecessary nesting of every success condition, or otherwise put them all in a huge conditional statement. While it’s rather easy to understand the success case, things can get tricky when we’re trying to debug programs like this, as we need to keep the conditionals in our head the whole time we’re reading the program.
A better alternative is to flip the conditionals, placing all failure handling statements near the top. While counterintuitive at first, this approach has several benefits. It reduces nesting and eliminates else branches while promoting failure handling to the top of our code, and this has the added benefit that we’ll become more aware of error handling and naturally gravitate towards thinking about the failure cases first, a great trait to have when doing application development, where forgetting to handle a failure case might result in an inconsistent experience for end users with a hard-to-trace error on top.
if (!response) {
return false
}
if (response.errors) {
return false
}
// … use `response`This early exit approach is often referred to as guard clauses, and one of their biggest benefits is that we can learn all the failure cases upon reading the first few lines of a function or piece of code. We’re not limited to return statements: we could throw errors in a promise-based context or in an async function, and in callback chaining contexts we might opt for a done(error) callback followed by a return statement.
Another benefit of guard clauses is almost implicit: given that they’re placed near the top of a function, we have quick access to its parameters, we can better understand how the function validates its inputs, and we can more effectively decide we need to add new guard clauses to improve validation rules.
Guard clauses don’t tell the reader everything they need to know that might go wrong when calling a function, but they give them a peek into expected immediate failure cases. Other things that might go wrong lie in the implementation details of the function. Perhaps we use a different service or library to fulfill the bulk of our function’s task, and that service or library comes with its own set of nested guard clauses and potential failure cases that will bubble up all the way to our own function’s outcome.
Writing straightforward code is not all that different from writing other straightforward texts. Texts are often arranged in paragraphs, which are somewhat comparable with functions: we can consider their input to be the reader’s knowledge and everything else they’ve read so far in the text, and the output can be what the reader gets out of the paragraph.
Within a book chapter or any other piece of long-form text, paragraphs are organized in a sequential manner, allowing the reader time to digest each paragraph before they jump onto the next. The logical sequence is very much intentional: without a coherent sequencing, it would be nearly impossible to make sense of a text. Thus, writers optimize for making sure concepts are introduced before they’re discussed, providing context to the reader.
Function expressions such as the one in the next snippet won’t be assigned to the variable binding until the line containing the assignment is evaluated. Until then, the variable binding exists in the scope, thanks to hoisting, but it is undefined until the assignment statement is evaluated.
double(6) // TypeError: double is not a function
var double = function(x) {
return x * 2
}Furthermore, if we’re dealing with a let or const binding, then TDZ semantics produce an error if we reference the binding at all before the variable declaration statement is reached.
double(6) // TypeError: double is not defined
const double = function(x) {
return x * 2
}Function declarations like the one in the following snippet, in contrast, are hoisted to the top of the scope. This means we can reference them anywhere in our code.
double(6) // 12
function double(x) {
return x * 2
}Now, we mentioned texts are written sequentially, and how the writers avoids surprises by presenting concepts before discussing them. Establishing a context in a program is a different endeavor, however. If we have a module that has the goal of rendering a chart with user engagement statistics, the top of the function should address things the reader already knows, namely the high-level flow for what the rendering function is meant to do: analyze the data, construct some data views and model that data into something we can feed into a visualization library that then renders the desired chart.
What we have to avoid is jumping directly into unimportant functions such as a data point label formatter, or the specifics of the data modelling. By keeping only the high-level flow near the top, and the specifics towards the end, complex functionality can be designed in such a way that the reader experiences a zoomed out overview of the functionality at first, and as they read the code they uncover the details of how this chart was implemented.
In a concrete sense, this means we should present functions in a codebase in the order that they’ll be read by the consumer (a first-in first-out queue), and not in the execution order (a last-in first-out stack). While computers do as they’re told and dig ever deeper into the flow, executing the most deeply nested routines before jumping out of a series of subroutines and executing the next line, this is an unfruitful way for humans to read a codebase, given we’re ill-suited to keeping all that state in our heads.
Perhaps a more specific analogy for this kind of spiraling approach can be found in newspaper articles, where the author typically offers a title that describes an event at the highest possible level, and then follows up with a lead paragraph that summarizes what happened, again at a high-level. The body of the article starts also at a high-level, carefully avoiding to spook the reader with too many details. It is only midway through the article that we’ll start finding details about the event which, aided by the context set forth in the beginning of the article, can give us a complete picture of what transpired.
Given the stack-based nature of programming, it’s not that easy to naturally approach programs as if they were newspaper articles. We can, however, defer execution of implementation details to other functions or subroutines, and thanks to hoisting, we can place those subroutines after their higher level counterparts. In doing so, we’re organizing our programs in a way that invites readers in, shows them a few high-level hints, and then gradually unveils the spooky details of how a feature is implemented.
Deliberate, pyramidal structures where we deal with higher level concerns near the top and switch to more specific problems as we go deeper into the inner workings of a system works wonders in keeping complexity on a tight leash. Such structures are particularly powerful because they break up complex items into their own individual units near the flat bottom of the system, avoiding a complicated interweaving of concerns that are fuzzied together, becoming undistinguishable from one another over time.
Pushing anything that gets in the way of the current flow to the bottom of a function is an effective way of streamlining readability. As an example, consider the case where we have a non-trivial mapper inline in the heart of a function. In the following code snippet we’re mapping the users into user models, as we often need to do when preparing JSON responses for API calls.
function getUserModels(done) {
findUsers((err, users) => {
if (err) {
done(err)
return
}
const models = users.map(user => {
const { name, email } = user
const model = { name, email }
if (user.type.includes('admin')) {
model.admin = true
}
return model
})
done(null, models)
})
}Now compare that code to the following bit of code, where we extracted the mapping function and shoved it out of the way. Given the mapping function doesn’t need any of the scope from getUserModels, we can pull it out of that scope entirely, without the need to place toUserModel at the bottom of the getUserModels function. This means we can now also reuse toUserModel in other routines, we don’t have to wonder whether the function actually depends on any of the containing scope’s context anymore, and getUserModels is now focused on the higher level flow where we find users, map them to their models, and return them.
function getUserModels(done) {
findUsers((err, users) => {
if (err) {
done(err)
return
}
const models = users.map(toUserModel)
done(null, models)
})
}
function toUserModel(user) {
const { name, email } = user
const model = { name, email }
if (user.type.includes('admin')) {
model.admin = true
}
return model
}Furthermore, if there were additional work to be done between the mapping and the callback, it could also be moved into another small function that wouldn’t get in the way of our higher level getUserModels function.
A similar case occurs when we have a variable that gets defined based on a condition, as shown in the next snippet. Bits of code like this can distract the reader away from the core purpose of a function, to the point where they’re often ignored or glossed over.
// …
let website = null
if (user.details) {
website = user.details.website
} else if (user.website) {
website = user.website
}
// …It’s best to refactor this kind of assignments into a function, like the one shown next. Note how we’ve included a user parameter so that we can push the function out of the scope chain where we’ve originally defined the user object, and at the same time went from a let binding to a const binding. When reading this piece of code later down the line, the benefit of const is that we’ll know the binding won’t change, as opposed to let with which we can’t be certain bindings won’t change over time, adding to the pile of things the reader should be watching out for when trying to understand the algorithm.
// …
const website = getUserWebsite(user)
// …
function getUserWebsite(user) {
if (user.details) {
return user.details.website
}
if (user.website) {
return user.website
}
return null
}Regardless of your flavor of choice when it comes to variable binding, bits of code that select some slice of application state are best shoved away from the relevant logic that will use this selected state to perform some action. This way, we’re not distracted with concerns about how state is selected, instead of being focused on the action that our application logic is trying to carry out.
When we want to name an aspect of a routine without adding a comment, we could create a function to host that functionality. Doing so doesn’t just give a name to what the algorithm is doing, but it also allows us to push that code out of the way, leaving behind only the high-level description of what’s going to happen.
Codebases with asynchronous code flows often fall into the so-called "callback hell", where each callback creates a new level of indentation, making code harder and harder to read as we approach the deep end of the asynchronous flow chain.
a(function () {
b(function () {
c(function () {
d(function () {
console.log('hi!')
})
})
})
})The foremost problem with this kind of structure is scope inheritance. In the deepest callback, passed to the d function, we’ve inherited the combined scopes of all the parent callbacks. As functions become larger, and more variables are bound into each of these scopes, it becomes ever more challenging to understand one of the callbacks in isolation from its parents.
This kind of coupling can be reverted by naming the callbacks and placing them all in the same nesting level. Named functions may be reused in other parts of our component, or exported to be used elsewhere. In the following example we’ve eliminated up to 3 levels of unnecessary nesting, and by eliminating nesting we’ve made the scope for each function more explicit.
a(a1)
function a1() {
b(b1)
}
function b1() {
c(c1)
}
function c1() {
d(d1)
}
function d1() {
console.log('hi!')
}When we do need some of the variables that existed in the parent scope, we can explicitly pass them on to the next callback in the chain. The following example passes an arrow function to d, as opposed to passing the d1 callback directly. When executed, the arrow function ends up calling d1 anyway, but now it has the additional parameters we needed. These parameters can come from anywhere, and we can do this throughout the chain, while keeping it all in the same indentation level.
a(a1)
function a1() {
b(b1)
}
function b1() {
c(c1)
}
function c1() {
d(() => d1('hi!'))
}
function d1(salute) {
console.log(salute) // <- 'hi!'
}Now, this could also be resolved using a library such as async, which simplifies the flattened chaining process by establishing patterns. The async.series method accepts an array of task functions. When called, the first task is executed and async waits until the next callback is invoked, before jumping onto the next task. When all tasks have been executed, or an error arises in one of the tasks, the completion callback in the second argument passed to async.series is executed. In the following illustrative example, each of the 3 tasks is executed in series, one at a time, waiting a second before each task signals its own completion. Lastly, the 'done!' message is printed to the console.
async.series([
next => setTimeout(() => next(), 1000),
next => setTimeout(() => next(), 1000),
next => setTimeout(() => next(), 1000)
], err => console.log(err ? 'failed!' : 'done!'))Libraries like async come with several ways of mixing and matching asynchronous code flows, in series or concurrent, allowing us to pass variables between callbacks without having to nest together entire asynchronous flows.
Naturally, callbacks aren’t the only asynchronous flow pattern that might end up in hell. Promises can end up in this state just as easily, as shown in the next contrived snippet.
Promise.resolve(1).then(() =>
Promise.resolve(2).then(() =>
Promise.resolve(3).then(() =>
Promise.resolve(4).then(value => {
console.log(value) // <- 4
})
)
)
)A similar piece of code that wouldn’t be affected by the nesting problem is shown next. Here, we’re taking advantage that promises behave in a tree-like manner, where we don’t necessarily need to attach reactions onto the last promise, and instead we can return those promises so that the chaining can always occur at the top level, allowing us to avoid any and all scope inheritance.
Promise.resolve(1)
.then(() => Promise.resolve(2))
.then(() => Promise.resolve(3))
.then(() => Promise.resolve(4))
.then(value => {
console.log(value) // <- 4
})Similarly, using async functions can turn what was previously a promise-based flow and turn it into something that can be mapped to our own mental model of how the program’s execution flows. The following bit of code is similar to the last snippet we looked at, but using async/await instead.
async function main() {
await Promise.resolve(1)
await Promise.resolve(2)
await Promise.resolve(3)
const value = await Promise.resolve(4)
console.log(value) // <- 4
}We’ve already discussed at length why creating abstractions isn’t always the best way of reducing complexity in an application. Abstractions can be particularly damaging when created too early: at the time we might not have enough information about the shape and requirements for other components that we might want to hide behind the abstraction layer, and over time we might end up aggressively shaping components only so that they fit the abstraction, which could have been avoided by not settling for an abstraction too early.
When we do avoid creating abstractions prematurely, we’ll start noticing functions that have an uncanny resemblance to the shape of similar functions: maybe the flow is identical, maybe the output is similar, or maybe all that really changes is we’re accessing an attribute named href in one case and an attribute named src in another case.
Consider the case of an HTML crawler which needs to pull out snippets of an HTML page and reuse them later in a different context. Among other things, this crawler needs to take relative resource locators like /weekly and resolve them to absolute endpoints like https://ponyfoo.com/weekly, depending on the origin where the resource was crawled from. This way, the HTML snippets can then be repurposed on other mediums such as on a different origin or a PDF file, without breaking the end-user experience.
The following piece of code takes a piece of HTML and transforms a[href] and img[src] into absolute endpoints using the $ jQuery-like DOM utility library.
function absolutizeHtml(html, origin) {
const $dom = $(html)
$dom.find('a[href]').each(function () {
const $element = $(this)
const href = $element.attr('href')
const absolute = absolutize(href, origin)
$element.attr('href', absolute)
})
$dom.find('img[src]').each(function () {
const $element = $(this)
const src = $element.attr('src')
const absolute = absolutize(src, origin)
$element.attr('src', absolute)
})
return $dom.html()
}As the small function it is, it’d be perfectly acceptable to keep absolutizeHtml as-is. However, if we later decide to add iframe[src], script[src], and link[href] to the list of attributes that might contain endpoints we want to transform, we’ll probably want to avoid having five copies of the same routine, as that’s more likely to be confusing and result in changes being made to one of them without being mirrored in the other cases, increasing complexity.
The following bit of code keeps all attributes we want to transform in an array, and abstracts the repeated bit of code so that it’s reused for every tag and attribute.
const attributes = [
['a', 'href'],
['img', 'src'],
['iframe', 'src'],
['script', 'src'],
['link', 'href']
]
function absolutizeHtml(html, origin) {
const $dom = $(html)
attributes.forEach(absolutizeAttribute)
return $dom.html()
function absolutizeAttribute([ tag, property ]) {
$dom.find(`${ tag }[${ property }]`).each(function () {
const $element = $(this)
const value = $element.attr(property)
const absolute = absolutize(value, origin)
$element.attr(property, absolute)
})
}
}A similar situation occurs when we have a concurrent flow that remains more or less constant across a number of different functions, in which case we might want to consider keeping the flow in its own function, and passing a callback for the actual processing logic that is different in each case.
In other cases, we might notice how there’s a few different components that all need the same piece of functionality. Commenting features often fall in this case, where different components like user profiles, projects, or artifacts, might need the ability to receive, show, edit, and delete comments. This case can be interesting because it’s the business requirement is not always identified upfront, and we might embed the child feature into the parent component before realizing it’d be useful to extract the feature so that it can be reused in other parent components. While this sounds obvious in hindsight, it’s not always clear when we’ll need to reuse some functionality somewhere else, and keeping every aspect of functionality isolated just in case we need to reuse them can be costly in terms of time and development effort.
More often than not, however, abstractions can end up complicating matters. It might be that the trade-off isn’t worth it because the code becomes much harder to read, or maybe because the underlying code isn’t mature enough yet, or we don’t know what special requirements we may end up with for other objects adopting similar functionality, meaning we’re not comfortable creating an abstraction that could lead to unforeseen problems in the future.
Whenever we are uncertain about whether an abstraction is up to muster, it pays to go back to the original piece of code we had before introducing the abstraction, and comparing the two pieces. Is the new piece easier to understand, modify, and consume? Would that still be the case as a newcomer? Try and consider how the outcome to those questions would change if you hadn’t looked at this code in a while. Ask your co-workers for their opinion, too; given they haven’t seen that code yet and they may end up having to consume it, they’re great candidates to help decide which approach is better.
Consider breaking what would otherwise inevitably be a single large function into smaller functions. These may be organized by splitting functionality by steps, by different aspects of the same task, always relying on guard clauses to do all of our error checking up front, ensuring that state is constrained by what we allow it to be at each point in time.
The overall structure of your typical function should begin with guard clauses, making sure the input we receive is what we expect: enforcing required parameters, their correct data types, correct data ranges, and so on. If these inputs are malformed, we should bail immediately, ensuring we don’t work with inputs we’re unprepared to deal with, and ensuring the consumer gets an error message that explains the root reason why they’re not getting the results they expect, as opposed to a message that might involve debugging work, such as undefined is not a function caused by trying to call an input that was supposed to be a function but wasn’t — or was supposed to result in our routine finding a function, but didn’t.
Once we know the inputs are well-formed, data processing can begin. We’ll transform the inputs, map them to the output we want to produce, and return that output. Here we have the opportunity to break apart the function into several pieces. Each aspect of the transformation of inputs into output is potentially its own function. The way of reducing complexity in a function is not by collapsing hundreds of lines of code into tens of complicated lines of code. Instead, we can move each of these long pieces of code into individual functions that only deal with one aspect of the data. Those functions can then also be hoisted out of our function and onto its parent scope, showing that there wasn’t a reason why a particular aspect of transforming the inputs had to be coupled to the entire function doing the transformation.
Each aspect of a transformation operation can be analyzed and moved into its own function. The smaller function may take a few of the inputs in the larger function, or perhaps some of the intermediate values that were produced in the larger function. It can then conduct its own input sanitization, and be broken apart even further. The process of identifying aspects of an operation that can be recursively compartimentalized and moved into their own functions is highly effective because it allows for dauntingly large functions to be broken into simpler pieces that aren’t as daunting to refactor.
At first, we can identify the 3 or 4 largest aspects of a function, and break those apart. The first part might involve filtering out the parts of the input we’re not interested in, the second might involve mapping that into something else, and the third part might involve merging all of the data together. Once we’ve identified each aspect of the function we might break those into their own functions, with their own inputs and output. Subsequently, we can do this for each of those smaller functions.
We can keep doing this for as long as there’s opportunity for the functions to be simplified. As discussed in the previous section, it’s valuable to take a step back after each of these refactors, and evaluate whether the end result is indeed simpler and easier to work with than what we had before it was refactored.
Entropy can be defined as a lack of order or predictability. The more entropy there is in a system, the more disordered and unpredictable the system becomes. Program state is a lot like entropy. Whether we’re discussing global application state, user session state, or a particular component instance’s state for a given user session, each bit of state we introduce to an application creates a new dimension to take into account when trying to understand the flow of a program, how it came to the state it’s currently at, or how the current state dictates and helps predict the flow moving forward.
In this section, we’ll discuss ways of eliminating and containing state, as well as immutability. First off, let’s discuss what constitutes current state.
The problem with state is that, as an application grows, its state tree inevitably grows with it, and for this reason large applications are hopelessly complex. We shall highlight that this complexity exists in the whole, but not necessarily in individual pieces. This is why breaking an application into ever smaller components might reduce local complexity even when it increases overall complexity. That is to say, breaking a single large function into a dozen small functions might make the overall application more complex, — as there would be ten times as many pieces — but it also makes the individual aspects of the previously-large function that are now covered by each small function simpler when we’re focused on them, as thus easier to maintain individual pieces of a large, complicated system, without requiring a complete or even vast understanding of the system as a whole.
At its heart, state is mutable. Even if the variable bindings themselves are immutable, as we’ll consider in section 4.3.1, the complete picture is mutable. A function might return a different object every time, and we may even make that object immutable so that the object itself doesn’t change either, but anything that consumes the function receives a different object each time. Different objects mean different references, meaning the state as a whole mutates.
Consider a game of chess, where each of two players starts with 16 pieces, each deterministically assigned a position on a checkerboard. The initial state is always the same. As each player inputs their actions, moving and trading pieces, the system state mutates. A few moves into the game, there is a good chance we’ll be facing a game state we haven’t ever experienced before. Computer program state is a lot like a game of chess, except there’s more nuance in the way of user input, and an infinitude of possible board positions and state permutations.
In the world of web development, a human decides to open a new tab in their favorite web browser and they then google for "cat in a pickle gifs". The browser allocates a new process through a system call to the operating system, which shifts some bits around on the physical hardware that lies inside the human’s computer. Before the HTTP request hits the network, we need to hit DNS servers, engaging in the elaborate process of casting google.com into an IP address. The browser then checks whether there’s a ServiceWorker installed, and assuming there isn’t one the request finally takes the default route of querying Google’s servers for the phrase “cat in a pickle gifs”.
Naturally, Google receives this request at one of the front-end edges of its public network, in charge of balancing the load and routing requests to healthy back-end services. The query goes through a variety of analyzers that attempt to break it down to its semantic roots, stripping the query down to its essential keywords in an attempt to better match relevant results.
The search engine figures out the 10 most relevant results for “cat pickle gif” out of billions of pages in its index – which was of course primed by a different system that’s also part of the whole – and at the same time, Google pulls down a highly targeted piece of relevant advertisement about cat gifs that matches what they believe is the demographic the human making the query belongs to, thanks to a sophisticated ad network that figures out whether the user is authenticated with Google through an HTTP header session cookie and the search results page starts being constructed and streamed to the human, who now appears impatient and fidgety.
As the first few bits of HTML being streaming down the wire, the search engine produces its results and hands them back to the front-end servers, which includes it in the HTML stream that’s sent back to the human. The web browser has been working hard at this too, parsing the incomplete pieces of HTML that have been streaming down the wire as best it could, even daring to launch other admirably and equally-mind-boggling requests for HTTP resources presumed to be JavaScript, CSS, font, and image files as the HTML continues to stream down the wire. The first few chunks of HTML are converted into a DOM tree, and the browser would finally be able to begin rendering bits and pieces of the page on the screen, weren’t it for the pending, equally-mind-boggling CSS and font requests.
As the CSS stylesheets and fonts are transmitted, the browser begins modeling the CSSOM and getting a more complete picture of how to turn the HTML and CSS plain text chunks provided by Google servers into a graphical representation that the human finds pleasant. Browser extensions get a chance to meddle with the content, removing the highly targeted piece of relevant advertisement about cat gifs before I even realize Google hoped I wouldn’t block ads this time around.
A few seconds have passed by since I first decided to search for cat in a pickle gifs. Needless to say, thousands of others brought similarly inane requests. To the same systems. During this time.
Not only does this example demonstrate the marvelous machinery and infrastructure that fuels even our most flippant daily computing experiences, but it also illustrates how abundantly hopeless it is to make sense of a system as a whole, let alone its comprehensive state at any given point in time. After all, where do we draw the boundaries? Within the code we wrote? The code that powers our customer’s computers? Their hardware? The code that powers our servers? Its hardware? The internet as a whole? The power grid?
We’ve established that the overall state of a system has little to do with our ability to comprehend parts of that same system. Our focus in reducing state-based entropy must then lie in the individual aspects of the system. It’s for this reason that breaking apart large pieces of code is so effective. We’re reducing the amount of state local to each given aspect of the system, and that’s the kind of state that’s worth taking care of, since it’s what we can keep in our heads and make sense of.
Whenever there’s persistence involved, there’s going to be a discrepancy between ephemeral state and realized state. In the case of a web application, we could define ephemeral state as any user input that hasn’t resulted in state being persisted yet, as might be the case of an unsaved user preference that might be lost unless persisted. We can say realized state is the state that has been persisted, and that different programs might have different strategies on how to convert ephemeral state into realized state. A web application might adopt an Offline-First pattern where ephemeral state is automatically synchronized to an IndexedDB database in the browser, and eventually realized by updating the state persisted on a back-end system. When the Offline-First page is reloaded, unrealized state may be pushed to the back-end or discarded.
Incidental state can occur when we have a piece of data that’s used in several parts of an application, and which is derived from other pieces of data. When the original piece of data is updated, it wouldn’t be hard to inadvertently leave the derived pieces of data in their current state, making them stale when compared to the updated original pieces of data. As an example, consider a piece of user input in Markdown and the HTML representation derived from that piece of Markdown. If the piece of Markdown is updated but the previously compiled pieces of HTML are not, then different parts of the system might display different bits of HTML out of what was apparently the same single Markdown source.
When we persist derived state, we’re putting the original and the derived data at risk of falling out of sync. This isn’t the case just when dealing with persistence layers, but can also occur in a few other scenarios as well. When dealing with caching layers, their content may become stale because the underlying original piece of content is updated but we forget to invalidate pieces of content derived from the updated data. Database denormalization is another common occurrence of this problem, whereby creating derived state can result in synchronization problems and stale byproducts of the original data.
This lack of synchronization is often observed in discussion forum software, where user profiles are denormalized into comment objects in an effort to save a database roundtrip. When users update later update their profile, however, their old comments preserve an stale avatar, signature, or display name. To avoid this kind of issue, we should always consider recomputing derived state from its roots. Even though doing so won’t always be possible, performant, or even practical, encouraging this kind of thinking across a development team will, if anything, increase awareness about the subtle intricacies of denormalized state.
As long as we’re aware of the risks of data denormalization, we can then indulge in it. A parallel could be drawn to the case of performance optimization, where we should be aware of how attempting to optimize a program basing off of microbenchmarks in stead of data-driven optimization will most likely result in wasted developer time. Furthermore, just like with caches and other intermediate representations of data, performance optimization can lead to bugs and code that’s ultimately harder to maintain, which is why neither should be embarked upon lightly, unless there’s a business case where performance is hurting the bottom line.
State is inevitable. As we discussed in section 4.3.1, though, the full picture hardly affects our ability to maintain small parts of that state tree. In the local case — each of the interrelated but ultimately separate pieces of code we work with in our day to day — all that matters are the inputs we receive and the outputs we produce. That said, generating a large amount of output where we could instead emit a single piece of information is undesirable.
When all intermediate state is contained inside a component instead of being leaked to others, we’re reducing the friction in interacting with our component or function. The more we condense state into its smallest possible representation for output purposes, the better contained our functions will become. Incidentally, we’re making the interface easier to consume. Since there’s less state to draw from, there’s fewer ways of consuming that state. This reduces the amount of possible use cases, but by favoring composability over serving every possible need, we’re making each piece of functionality, when evaluated on its own, simpler.
One other case where we may incidentally increase complexity is whenever we modify the property values of an input. This type of operation should be made extremely explicit, as to not be confused, and avoided where possible. If we assume functions to be defined as the equation between the inputs we receive and the outputs we produce, then the side-effects are ill-advised. Mutations on the input within the body of a function is one example of side-effects, which can be a source of bugs and confusion, particularly due to the difficulty in tracking down the source for these mutations.
It is not uncommon to observe functions that modify an input parameter and then return that parameter. This is often the case with Array#map callbacks, where the developer wants to change a property or two on each object in a list, but also to preserve the original objects as the elements in the collection, as shown in the following example.
movies.map(movie => {
movie.profit = movie.gross - movie.budget
return movie
})In these cases it might be best to avoid using Array#map altogether, using Array#forEach or for..of instead, as shown next.
for (const movie of movies) {
movie.profit = movie.gross - movie.budget
}Neither Array#forEach nor for..of allow for chaining, assuming you wanted to filter the movies by a criteria such as "profit is greater than $15M": they’re pure loops that don’t produce any output. This is a good problem to have, however, because it explicitly separates data mutations at the movie item level, where we’re adding a profit property to each item in movies; from transformations at the movies level, where we want to produce an entirely new collection consisting only of expensive movies.
for (const movie of movies) {
movie.profit = movie.amount * movie.unitCost
}
const successfulMovies = movies.filter(
movie => movie.profit > 15
)Relying on immutability would be an alternative that doesn’t involve pure loops nor does resort to breakage-prone side-effects.
The following example takes advantage of the object spread operator to copy every property of movie into a new object, and then adds a profit property to it. Here we’re creating a new collection, made up of new movie objects.
const movieModels = movies.map(movie => ({
...movie,
profit: movie.amount * movie.unitCost
}))
const successfulMovies = movieModels.filter(
movie => movie.profit > 15
)Thanks to us making fresh copies of the objects we’re working with, we’ve preserved the movies collection. If we now assume that movies was an input to our function, we could say that modifying any movie in that collection would’ve made our function impure, since it’d have the side-effect of unexpectedly altering the input.
By introducing immutability, we’ve kept the function pure. That means that its output only depends on its inputs, and that we don’t create any side-effects such as changing the inputs themselves. This in turn guarantees that the function is idempotent, where calling a function repeatedly with the same input always produces the same result, given the output depends solely on the inputs and there are no side-effects. In contrast, the idempotence property would’ve been brought into question if we had tainted the input by adding a profit field to every movie.
Large amounts of intermediate state or logic which permutates data into different shapes, back and forth, may be a signal that we’ve picked poor representations of our data. When the right data structures are identified, we’ll notice there’s a lot less transformation, mapping, and looping involved into getting inputs to become the outputs we need to produce. In section 4.4 we’ll dive deeper into data structures.
Data structures can make or break an application, as design decisions around data structures govern how those structures will be accessed. Consider the following piece of code, where we have a list of blog posts.
[{
slug: 'understanding-javascript-async-await',
title: 'Understanding JavaScript’s async await',
contents: '…'
}, {
slug: 'pattern-matching-in-ecmascript',
title: 'Pattern Matching in ECMAScript',
contents: '…'
}, …]An array-based list is great whenever we need to sort the list or map its objects into a different representation, such as HTML. It’s not so great at other things, such as finding individual elements to use, update, or remove. Arrays also make it harder to preserve uniqueness, such as if we wanted to ensure the slug field was unique across all blog posts. In these cases, we could opt for an object map based approach, as the one shown next.
{
'understanding-javascript-async-await': {
slug: 'understanding-javascript-async-await',
title: 'Understanding JavaScript’s async await',
contents: '…'
},
'pattern-matching-in-ecmascript': {
slug: 'pattern-matching-in-ecmascript',
title: 'Pattern Matching in ECMAScript',
contents: '…'
},
…
}Using Map we could create a similar structure and benefit from the native Map API as well.
new Map([
['understanding-javascript-async-await', {
slug: 'understanding-javascript-async-await',
title: 'Understanding JavaScript’s async await',
contents: '…'
}],
['pattern-matching-in-ecmascript', {
slug: 'pattern-matching-in-ecmascript',
title: 'Pattern Matching in ECMAScript',
contents: '…'
}],
…
])The data structure we pick constrains and determines the shape our API can take. Complex programs are often, in no small part, the end result of combining poor data structures with new or unforeseen requirements that don’t exactly fit in well with those structures. It’s usually well worth it to transform data into something that’s amenable to the task at hand, so that the algorithm is simplified by making the data easier to consume.
Now, we can’t possibly foresee all scenarios when coming up with the data structure we’ll use at first, but what we can do is create intermediate representations of the same underlying data using new structures that do fit the new requirements. We can then leverage these structures, which were optimized for the new requirements, when writing code to fulfill those requirements. The alternative, resorting to the original data structure when writing new code that doesn’t quite fit with it, will invariably result in logic that has to work around the limitations of the existing data structure, and as a result we’ll end up with less than ideal code, that might take some effort understanding and updating.
When we take the road of adapting data structures to the changing needs of our programs, we’ll find that writing programs in such a data-driven way is better than relying on logic alone to drive their behaviors. When the data lends itself to the algorithms that work with it, our programs become straightforward: the logic focuses on the business problem being solved while the data is focused on avoiding an interleaving of data transformations within the program logic itself. By making a hard separation between data or its representations and the logic that acts upon it, we’re keeping different concerns separate. When we differentiate the two, data is data and logic stays logic.
Keeping data strictly separate from methods that modify or access said data structures can help reduce complexity. When data is not cluttered with functionality, it becomes detached from it and thus easier to read, understand, and serialize. At the same time, the logic that was previously tied to our data can now be used when accessing different bits of data that share some trait with it.
As an example, the following piece of code shows a piece of data that’s encumbered by the logic which works with it. Whenever we want to leverage the methods of Value, we’ll have to box our input in this class, and if we later want to unbox the output, we’ll need to cast it with a custom-built valueOf method or similar.
class Value {
constructor(value) {
this.state = value
}
add(value) {
this.state += value
return this
}
multiply(value) {
this.state *= value
return this
}
valueOf() {
return this.state
}
}
console.log(+new Value(5).add(3).multiply(2)) // <- 16Consider now, in contrast, the following piece of code. Here we have a couple of functions that purely compute addition and multiplication of their inputs, which are idempotent, and which can be used without boxing inputs into instances of Value, making the code more transparent to the reader. The idempotence aspect is of great benefit, because it makes the code more digestible: whenever we add 3 to 5 we know the output will be 8, whereas whenever we add 3 to the current state we only know that Value will increment its state by 3.
function add(current, value) {
return current + value
}
function multiply(current, value) {
return current * value
}
console.log(multiply(add(5, 3), 2)) // <- 16Taking this concept beyond basic mathematics, we can begin to see how this decoupling of form and function, or state and logic, can be increasingly beneficial. It’s easier to serialize plain data over the wire, keep it consistent across different environments, and make it interoperable regardless of the logic, than if we tightly coupled data and the logic around it.
Functions are, to a certain degree, hopelessly coupled to the data they receive as inputs: in order for the function to work as expected, the data it receives must satisfy its contract for that piece of input. Within the bounds of a function’s proper execution, the data must have a certain shape, traits, or adhere to whatever restrictions the function has in place. These restrictions may be somewhat lax (e.g. "must have a toString method"), highly specific (e.g. "must be a function that accepts 3 arguments and returns a decimal number between 0 and 1"), or anywhere in between. A simple interface is usually highly restrictive (e.g. accepting only a boolean value). Meanwhile, it’s not uncommon for loose interfaces to become burdened by their own flexibility, leading to complex implementations that attempt to accommodate many different shapes and sizes of the same input parameter.
We should aim to keep logic restrictive and only as flexible as deemed necessary by business requirements. When an interface starts out being restrictive we can always slowly open it up later as new use cases and requirements arise, but by starting out with a small use case we’re able to grow the interface into something that’s naturally better fit to handle specific, real-world use cases.
Data, on the other hand, should be transformed to fit elegant interfaces, rather than trying to fit the same data structure into every function. Doing so would result in frustration similar to how a rushed abstraction layer that doesn’t lend itself to being effortlessly consumed to leverage the implementations underlying it. These transformations should be kept separate from the data itself, as to ensure reusability of each intermediate representation of the data on its own.
Should a data structure — or code that leverages said data structure — require changes, the ripple effects can be devastating when the relevant logic is sprinkled all across the codebase. Consequently, when this happens, we need to update code from all over, making a point of not missing any occurrences, updating and fixing test cases as we go, and testing some more to certify that the updates haven’t broken down our application logic, all in one fell swoop.
For this reason, we should strive to keep code that deals with a particular data structure contained in as few modules as possible. For instance, if we have a BlogPost database model, it probably makes sense to start out having all the logic regarding a BlogPost in a single file. In that file, we could expose an API allowing consumers to create, publish, edit, delete, update, search, or share blog posts. As the functionality around blog posts grows, we might opt for spreading the logic into multiple colocated files: one might deal with search, parsing raw end-user queries for tags and terms that are then passed to Elasticsearch or some other search engine; another might deal with sharing, exposing an API to share articles via email or through different social media platforms; and so on.
Splitting logic into a few files under the same directory helps us prevent an explosion of functionality that mostly just has a data structure in common, bringing together code that’s closely related in terms of functionality.
The alternative, placing logic related to a particular aspect of our application such as blog posts directly in the components where it’s needed, will cause trouble if left unchecked. Doing so might be beneficial in terms of short-term productivity, but longer-term we need to worry about coupling logic, strictly related to blog posts in this case, together with entirely different concerns. At the same time, if we sprinkle a bulk of the logic across several unrelated components, we become at risk of missing critical aspects of functionality when making large-scale updates to the codebase, and because of this we might end up making the wrong assumptions, or mistakes that only become evident much further down the line.
It’s acceptable to start out placing logic directly where it’s needed at first, when it’s unclear whether the functionality will grow or how much. Once this initial exploratory period elapses, and it becomes clear the functionality is here to stay and more might be to come, it’s advisable that we isolate the functionality for the reasons stated above. Later, as the functionality grows in size and in concerns that need to be addressed, we can componentize each aspect into different modules that are still grouped together logically in the file system, making it easy to take all of interrelated concerns into account when need be.
Now that we have broken down the essentials of module design and how to delineate interfaces, as well as how to lockdown, isolate, and drive down complexity in our internal implementations, we’re ready to start discussing JavaScript-specific language features and an assortment of patterns that we can benefit from.
false when the token isn’t present.