Hiring software engineers is hard due to:
- Competitive.
- Writing good software is a skill that requires competencies across multiple disciplines and abstractions.
- An effective engineer is close quarters work, even on fully distributed teams.
Software doesn't fail because it's buggy, software fails because the people who should want it do not find it attractive or compelling enough to use it → hiring software engineers is complicated.
A given piece of software succeeds or fails is based on the engineer's ability to:
- Empathize with the customer.
- Mentor others.
- Learn from past experience.
This is a book primarily for software engineers who want:
- Make the collaboration between their group and Recruiting/HR as painless and straight forward as possible.
- Tasked with building a pipeline for their teams.
- Pitch in interviewing candidates.
- Decide whether or not to recommend hiring other technical people.
Bureaucracy should be avoid while cultivating engineering teams and building the processes.
Good process:
- Empower people → escape having to learn everything via trial and error and gives them a framework to think about how they do what they do.
- Not to eliminate one-on-one collaboration and communication between coworkers.
- Not babysit employees who can't be trusted to do the right thing.
- Not to avoid taking responsibility for negative outcomes or abusive coworkers.
- Start with trust.
- Start with the assumption that everyone involved in the hiring process wants to make fair decisions
- Help them make the best decision possible.
- Clearly communicate what the standards are and then providing specific guidance on how candidates that meet those standards will present in an interview.
Bureaucracy:
- Create hiring pipelines to eliminate all bias or stop the occasional bad hire.
- No trust on your colleagues.
Arranged roughly in the order that a pipeline runs:
- Starts with standards of what the organization hires for defined upfront → core competencies → every candidate that interviews for a role in engineering regardless of the exact position or experience level will be evaluated against those standards.
- Review resumes.
- Screen prospects for suitability.
- Interview candidates.
- Make a hiring decision.
Preparation items:
- Competency definitions.
- Written position descriptions.
- A question pool with scoring rubrics.
- What is legal or illegal to ask a candidate during an interview. → assume you have knowledge.
- Craft offers or compensation schemes.
- Recruiting → only includes small section on some thoughts about sourcing, particularly when it comes to diversity as a practice.
- Craft job announcements, allocate headcount or anything like that. → just slight touches on career ladders and position descriptions at various points.
Early stage organizations don't get very far with a standardized pipeline as they do not have well-formed identities for the questions: who are we? and what do we hire for?. They only need answers for whatever will make enough money to survive and whoever is willing to agree to work for us. Only few people do interviews → It's better to to encourage them to share notes on candidates after interviews are completed and try to keep each other in sync.
When to formalize the pipeline:
- Hiring needs: How many open positions do we have?
- Product market fit: How likely are we to experience a major pivot that might radically shift our core competencies and hiring needs?
- Potential interviewers: Are candidates vetted by a small group of people or do people just sit down with the CEO?
- Referral rate: Are we mainly looking to recruit our friends and former colleagues or are we thinking about posting to job boards and recruiting?
If you're not really hiring that much → don't need to invest the time and energy needed to build a pipeline.
Once you start bringing people you do not know in for interviews or asking multiple people to assess the same candidate → adding structure and process will pay dividends.
Summary:
- Standardize the hiring process in stages, evolving the pipeline as the organization grows and matures.
Definition: skills and characteristics critical for success that are common to every engineer who works at the organization → high level categories like engineering, design or product → about the specific culture of the organization itself.
Purpose: Mitigate unconscious bias.
Different organizations can have different identities and internal priorities. Some types of organizations:
- Learning organization.
- Triple-bottom-line mission driven.
- Customer always right.
- Obsessive about security.
- Remote first.
- Have a set schedule of agile ceremonies.
For example:
- Heavily regulated industry organization: value structured specifications that mesh well with compliance frameworks.
- Consumer business: value plain language tutorials and walk throughs that can be repurposed for the customer.
People usually:
- Find an environment conducive to success.
- Need a work schedule that fits with our lives, colleagues we can establish trust with, tasks that are interesting to us, the right tools for the job and a path clear of blockers.
Competencies are skills, not tasks → hunt for different work activities that have a common foundation.
For example: Need to handle petabytes of data → Experience in scaling technology was a big deal. Originally, it is believed to be applicable to backend and system administrator/operations types, not frontend engineers and machine learning teams. In fact, scaling affects everything:
- Machine learning: The approaches change at different data sizes that certain solutions become efficient and cost effective that weren't with less data → is a scaling challenge.
- Frontend: compile or minify your assets? the trade-offs of using a CDN? Use robust framework with lots of functions out of the box or pared down to just what it needs?
Definition: a group of individuals covered by one set of competencies.
For example:
- Engineering, design, and product → Smaller disciplines: Information Technology (IT) and security break off from engineering.
- Operations or system administration → Automation increase writing code needs smaller disciplines: DevOps/SRE.
Things that should be asked when thinking about breaking off a new discipline and writing competencies:
- Existing members of that would-be discipline can support their own hiring pipeline?
- Do they have enough people who can interview?
- Can they develop their own pool of questions?
- Are they doing enough hiring to justify the extra work setting all of that up?
If hiring for that group only happens once a quarter, interviewers will gradually drift back to their own unconscious biases.
If the company is in a place where it would benefit from hiring people who have deep expertise in one specific area and there's enough work for those experts to only work in that area → break that group off into its own discipline.
Definition: a set of work tasks that reflect the role being applied for → use the same competencies as everyone else in the discipline but they develop and maintain their own questions in the question pool.
For example:
- Frontend engineering
- SRE
- Data
- Management
Tip: Every competency should have questions that reflect good general engineering principles and questions that test how the competency might look for various specializations i.e. for a mid-size engineering team, this usually means two or three questions that are specific to a specialization.
Competencies should be defined in levels of experience or proficiency.
The easiest break down: Junior, Midcareer, Senior. You can go further but on this point less could be more.
Levels need to answer the questions:
- What is the difference between a candidate on a lower level and an unqualified one?
- Does levels overlap too much?
- Does it avoid the situation - a little junior for this role but worth investing in?
Defining competencies, disciplines, specializations and levels in order to mitigate bias is about:
- not eliminating what makes an organization special and unique.
- articulating that uniqueness so that the individuals can stay in sync with one another.
Good competencies construct an image of the type of software engineer that will be successful at the organization.
The hardest hiring decisions are never the bad candidates but are what kill a pipeline → Candidates seem nice and the interviewer feels bad about disappointing them leads to excuses are made to advance them round after round → Set boundaries (disciplines and specializations) up front, making it less likely that unqualified candidates will slip through.
Less is also more when it comes to the number of competencies → The upper bound on the number of competencies is defined by how many interviews you want one interview loop to have.
The competitive market for good software engineers helps keep our worst tendencies in check here → The longer the interview cycle takes, the more likely candidates will drop out in the middle of it.
Having a specific pool of interview questions minimizes difference between individual interviewers and makes the interview easier on both the person conducting it and the person being interviewed → The more competencies, the more questions needed in the pool → The less value in a standard set of questions everyone asks.
For most organizations, three to four core competencies are usually fine.
Tips:
- Don't search the internet for lists of competencies used at other companies and just copy them → when taken out of context they become empty platitudes that confuse interviewers.
- Build them from the ground up by looking at what people who actually do the job do every day.
- Don't pick some characteristics that sound good only → might end up with competencies that are divorced from reality and functionally useless.
- Allow the competencies to be general but still communicate a clear vision of what it means for a candidate to be a good fit with detailed quotes from actual experiences at the organization.
Three stages of competency development:
- The initial discovery work where we collect anecdotes about people who are valuable at the organization and what makes them that way.
- Distill that data down to common themes.
- Take those themes and decide what that characteristic looks like across our experience levels.
Typically, it takes two to three weeks doing all the stages.
High touch version: Group of engineers directly participating in this stage of competency development. Low touch version: Engineers don't participate directly into the process, only use indirect methods such as survey, individual interviews, etc.
Purpose of discovery: Get as much data as possible about what an engineer does on the job at the organization.
The best people to talk to about what engineers do at the organization are engineers at the organization, not the bosses those engineers report up to - the engineering managers.
The focus question for group: “What are the most important tasks you do every day?“ → If there seems to be a disconnect between what people think is important (what they think management wants to hear) and what they actually prioritize, we can ask “what do people who are successful here do?” instead.
Should perform this exercise with colleagues at an organization that you knew pretty well → It is easier to read the room and figure out exactly how to phrase my research question to get the information
In a perfect world, we can proceed with:
- Put an hour session on everyone's calendar.
- Get small groups of four or five people in a room together.
- Have them jot down their work responsibilities on Post-It notes or a whiteboard.
Tips:
- Allow people to think about the question from their personal experience.
- Compare their answers to their colleagues → often leads to interesting discussions and even more data.
- Smaller groups will encourage more people to contribute and produce more data.
- Don't lose the duplication when similar answers are already up on the board → help group things by similarity and give the top priority to the largest, tightest clusters.
- Be mindful of engineering diversity -> include some junior people, some people who have worked at the organization for only a year, some people from each specialization, etc. not just those with seniority, those with the most experience, or those with leadership roles.
Ask people to answer the research question: What are the most important tasks you do every day? asynchronously over Slack, email, or another medium.
Tips:
- Compare your responses to the org chart.
- If important voices are not included follow up with them directly.
- Don't assume they don't want to help, they just procrastinate sometimes.
When burned out on surveys, it might be better to reach out to people directly and chat with them one-on-one → usually take 15 minutes or less.
Tips:
- Take lots of notes.
- Make sure the person you're chatting with can see those notes.
The lower touch the data collection, the more vulnerable you are to missing critical parts of the engineering experience.
Tips:
- Make your work files transparent and viewable to all of engineering.
- Ask people to leave comments.
Affinity mapping: Once you have your data on what engineers are actually doing, you can start grouping those tasks into general competencies.
Process:
- For group brainstorming → end the session by asking people to move the Post-It's around until like is grouped with like. 🤔
- For a lower touch discovery exercise → do the rough groupings yourself and then send them out to engineering for feedback before finalizing them.
- Once you've given each grouping a pithy name → have your draft competencies.
- If you have more than the total number of competencies you want to move forward → rank them by order of importance and take the top groups/consolidated competencies into one whenever possible.
- Don't add new competencies with no connection to the data → they should not be aspirational and should reflect the work reality.
- If there's a characteristic you want to hire for that's not reflected in your competencies → figure out how to build that characteristic into your organization first, not by hiring new members (when new people come in, they will most likely adjust to the same incentives that have shaped the existing culture and lose whatever unique characteristics you were hoping they would introduce).
- Close this phase by drafting a short paragraph (maybe two or three sentences) for each competency and posting it publicly for engineering to comment on → the transparency will make people feel better about adopting the competencies as a decision making tool once they are finalized.
The final stage: map each competency across the experience level the organization currently hires for.
Can answer the question “How does a candidate at this level demonstrate this competency?”
For example:
| Junior role | Mid-career role | Senior role |
|---|---|---|
| Fast learners | Able to work autonomously without a senior team member holding their hand or telling them what to do | Force multiplier |
| Curious | Makes other engineers better engineers with in-depth experience | |
| Flexible | Good mentors and leaders |
For example:
| Junior role | Mid-career role | Senior role |
|---|---|---|
| Okay with lacking a clear style + taking feedback well | Look for evidence of a connection between usage and structure | Look for setting standards ability + make them happen through proactive mentorship and collaboration |
| Show signs of experimenting many different types of programming (object oriented, functional, strongly/static/dynamic typed, etc) | Help others navigate their code by writing good documentation | Make it easy for everyone else to “fall into the pit of success” with tooling |
| Use comments + writing tests for testing code functionality + illustrating the intention of functions | Don't shame people into arbitrary style preferences |
Tips:
- Connect the competency to our expectations of each other within the team.
- Don't ask "how does this person rank against all software engineers in the world?"
- Do ask "what role do we need this candidate to play on the team and do they fit?"
Be careful not to let your levels be influenced by those external optics → focus on the role they're supposed to be playing on the team is clearly defined and whether or not they're fulfilling those requirements.
Small organizations often has some issues:
- Title inflation issue: junior engineers, engineers and senior engineers → junior engineers, “senior engineers”, and principal engineers.
- Put everyone on the same level officially and have people take on different roles informally.
Example of a full set of levels for a large engineering organization:
| Junior engineers | Engineers | Senior engineers | Staff engineers |
|---|---|---|---|
| Investment, good juniors will grow into great talent bought early (and therefore cheaper) but will also help mid-career engineers level up to senior engineers by helping them learn how to mentor others. | The workhorse of the team. Engineers should spend almost all of their time working on code and building as much deep expertise as possible. | Still building deep expertise, but in charge of the long term health of what the team owns. How will this scale? How do we manage dependencies? When does optimization make sense? How much test coverage do we need? What best practices do we subscribe to? | Extending the senior engineer role to cross multiple teams, ensuring consistency between them. |
Core competencies can be a resource for engineers to plan their careers.
Competencies are about comparing groups of people, not the tasks an individual person does → Do not use competencies in performance reviews.
Performance reviews should be about that specific person's work → Using them during reviews tends to turn them into checklists that dehumanize engineers and lower their performance.
However when there's a promotion to a new role → competencies can help compare and contrast the suitability of each candidate.
Can sometimes use competencies as a reference in the day-to-day mentoring of engineers that report up to you → the full matrix paints a pretty nice picture of what can be expected from engineers at different stages and what advancing looks like → particularly helpful with engineers in the early stages of their careers.
For example:
- Show engineers that one level up from where they are is not about running meetings or mentoring others.
- It's more about becoming proficient in multiple languages, or working on systems that cross teams, or being on call.
Using competencies here takes some management skill as people sometimes interpret it super literally and miss the point entirely → take the time to talk it out with them.
For example:
- A competency: Representing us out in the wider engineering community → Many engineers thinks that they couldn't get promoted if they were introverts because in order to be more senior they needed to give conference talks or write blog posts → Explain that submitting pull requests to open source projects we depend on was also representing us in the community.
It is necessary to review and revise various parts of the engineering hiring pipeline to ensure it still reflects reality → Normally, redo competencies every three to four years → If the organization is growing super fast then might redo them after two years → Some other situations for early redo can be:
If your organization has switched products, mission or industries → it makes sense to quickly review competencies.
When ready to break a specialization off into its own discipline → not only need to write competencies for that discipline → review existing competencies.
As more and more teams are created → break the ladder into more levels to capture and formalize the cross functional coordination of staff and principal engineers → need to be reflected in the core competencies.
Highlights on core competencies:
- Are the characteristics that make engineers successful in their discipline → are general and common across roles and specializations.
- Should focus on what engineers actually do regularly in their role → aspirational competencies are useless.
- Should be mapped to the career ladder → any engineer can tell the difference between a too junior candidate and a unqualified/undesirable candidate.
- Are about comparing groups of people → are less useful when evaluating for individuals.
It is useful to not just specify what the goals of the process are but also what it will look like when the process has gone wrong.
Antipatterns → empower the individual employees to say "things aren't working here." → when they should bypass the process + adapt on the fly → when the process may be overdue for a rethink.
Process is not about preventing our people from doing wrong, it's more about helping them achieve their intention to do right → ignoring process to do right should not be punished → good process helps everyone understand when such overrides are justified → people will tend to hide the fact that they had to overrule a bad/unrealistic process to get a good result + denying leadership.
Defining antipatterns doesn't tell your team to ignore process → it tells them to let you know when they are going to ignore process because it isn't working → also help prevent people from unfairly throwing out process.
Antipatterns at the outcome scope address what a bad result would look like.
For example:
- Interviewing takes up more than 20% of any one engineer's weekly work time.
- High rate of rejection of candidates referred by existing employees.
- Diversity levels dropping by more than 50% after the loop.
- Negative candidate accounts (especially in public forums: Glassdoor, Twitter, etc).
- Offer rejection rises above 65%.
Insights:
- The best people to assess a candidate for a position are the people who already do the work.
- The more time they spend interviewing the less they do the job we're hiring for and interviews are pretty intense and burn people out fast → Overburdening interviewers is a clear sign we need to train more interviewers or rethink something about how we organize and schedule the interviewing loop.
- Rejecting a lot of candidates is a red flag → our competencies do not reflect reality, or that interviewers are adding extra unwritten and unspoken standards on top of our formal competencies.
- Tracking diversity levels → standardizing the pipeline is to minimize bias → i.e. the ratio of women to men applying is 3:4 → doesn't mean that women are less likely to be qualified than men.
- The last two antipatterns reflect how critical candidate experience is to the pipeline → Should aim for everyone to feel good about the interview experience → avoid to let people feel like they've been treated unfairly or to be embarrassed by their performance → first things to tell candidate that interviews are bi-directional.
Questions can be used:
- What is it okay for a candidate in a given position to be weak on?
- What mistakes are acceptable?
For example:
- A candidate for an engineering role needs to know how to scale a webservice → Does it matter if they can only talk about it in terms of an AWS cloud? Do we want them to demonstrate experience with many different vendors?
Set an upper bound on how hard interviews are allowed to be. Some common issues own engineers usually make:
- Tend to want to make interviews harder and harder to pass + not to apply these new criteria evenly.
- Overestimate what they would be able to answer themselves if they didn't have all the background information an interviewer does.
- Be more flexible and forgiving with people they like.
- Be more strict with people that don't fit their mental model of a good engineer.
Deal breakers should be used sparingly + be more about EQ issues than technical knowledge or experience.
No-hire criteria standard example:
- Candidates that speed through answers.
- Don't engage in discussion or debate.
- Display a lot of entitlement.
Give interviewers permission to reject a candidate if they feel uncomfortable with them Even Linus Torvalds had to eventually admit that his attitude hurt the technology he had built his career around.
Include in documentation outlining the whole pipeline + training materials as appropriate → post where they are called out specifically and read as critical factors in a successful pipeline rather than footnotes.
Can be mixed as into scoring rubrics and competencies instead of labeling as antipatterns specifically as it is a part of shaping the whole picture.
Antipatterns are more successful when they're defined up front + writing outcomes be as specific as possible. (ground them in numbers)
→ Three years after rolling it out, Medium has since retired Snowflake. 😅
People tend to assume that decisions calculated with points and scores are more objective and therefore more fair than those involving more descriptive methods → only advocate for using numbers in places where numbers naturally exist.
Example: interviews should not take up more than 20% of any one engineer's work → 20% of a 40-hours week is 8 hours → When you factor in prep time and writing feedback 8 hours is roughly four interviews a week. None of those things are abstractions, those numbers already exist.
The Snowflake model from Medium: Has a total of 16 competencies, 4 of pure technical: mobile, web client, foundations, server and the rest are soft skills, evangelism, project and time management.
→ Why is building a diverse pipeline worth two points? Why is building complex, reusable architectures worth four points? → it is alarming seeing how many competencies Medium attempt to maintain and how little information there was about trade-offs or prioritizations between them.
- If zero out all the competency scores and set all the technical competencies to the maximum value → become a staff engineer.
- If zero out all the competencies and set Project Management, Career Development and (team) Wellbeing to the highest possible values → be considered a staff engineer. → A successful staff engineer under the Snowflake model is either a technical savant who struggles to collaborate with others or the equivalent of a guidance counselor who does no technical work at all but keeps the trains running on time.
Something to keep in mind: The harder it is to understand the process, the more inconsistent and bias decisions will be.
- Define what outcomes will be seen as failures and help the engineers running the pipeline know when and where to exercise their discretion.
- Keep standards from shifting by clearly articulating what our expectations are.
- Avoid overspecification. Having too many variables, especially if they're abstract in nature, will trap the engineers running the pipeline into situations where they must override the process in order to get good results.
Frustrations:
- The positions may get lost in the noise of the internet.
- Flooded by resumes.
Position descriptions are what guides the rest of the interview process internally - is a reference that the interviewers use to determine what questions in the pools are the most appropriate and at what level of seniority candidates should demonstrate.
Well designed pipeline:
- Modular.
- Provides enough options to custom fit the assessment method and loop schedule to the specific position.
- Limits variance enough to standardize the experience and the quality of the outcomes hopefully.
→ List the competencies followed by a small paragraph describing what set of skills and experiences that competency included.
This document should include:
- Position metadata
- Role description
- Minimum qualifications across competencies
- Interview loop plan
Includes information like:
- who the position reports up to.
- what team the position is for.
- the expected level of the role.
- which recruiter is assigned.
- links to the job posting on whatever the organization considers the most authoritative source.
Tips:
- build a template for type of role.
- reuse that template to create position descriptions for specific positions available.
Prefer descriptions that reflect what success would look like after we hire someone:
- A quick sentence or two about what a successful hire will have done within their first 30 days, their first quarter and their first year.
- Including what the organization needs to provide in order for a good candidate to accomplish those things is also valuable. (need access to leadership in other sections of the organization? need access to production?)
2 ways:
-
1st way: decide on a level for the role and evaluate all competencies for the discipline at that level e.g. qualified candidate for a junior engineering role would satisfy all the hire criteria listed under the junior level for every competency.
-
2nd way: use competencies to create positions is to specify different minimum levels for each competency i.e. define a good candidate as someone who should be particularly strong in some competencies but can be weaker in others. → Useful for specialists and roles that have cross functional obligations but is more complicated than the 1st way.
United States Digital Service (USDS) used Unfamiliar, Familiar, Experienced, and Master instead of No Hire, Hire, Strong Hire → the hiring authority and mandate was to recruit experts to act as advisors and consultants to the highest levels of government. Determining what it meant to be a junior engineer at USDS had no value → competencies needed to be structured in a way that allowed us to set a minimum qualification bar that was the same for a broad range of people.
Discourage organizations from pulling competencies from multiple disciplines to create positions as it might introduce bias.
- specifies how many interviews are in the loop.
- what competencies those interviews assess.
- possibly which interviewers will do them.
→ Finding times that work for both the candidate and the interviewers is the hardest part of running a loop.
Good pipelines balance:
- The hiring manager's need to establish a team identity that matches their goals, expectations.
- Leadership style with the benefits of talent potentially fitting in multiple places.
Recruiters are not the best people to determine whether or not a Fullstack Engineer and a Senior Software Engineer are the same thing → Engineering leadership should be monitoring discrepancies in how positions are described and what their titles.
The best candidate sourcing strategy is one that focuses on relationships:
- getting existing employees to refer candidates.
- proactively building relationships with candidates discovered via events, browsing LinkedIn.
- dialogue with users and open source contributors.
Good recruiters develop a strategy in which leads are sourced and cultivated with regular contact e.g. every sixth months without fail a recruiter from Amazon, Facebook or Google reaches out to me to see and pitches sample open roles.
a good salesperson told me once that he doesn't really expect to get a yes to a pitch until the third meeting → success is about how high the conversion rate of the funnel is.
-
1st level - content marketing: gets potential hires exposed to who you are and what you do. → Possible content with recruiting value includes blog posts, conference talks, building the personal brands of the organization's experts, sponsored research, open sourcing tools, and engaging plain language documentation
-
2nd level - in person events: is about inviting them into the conversation. → If you have an engineer going to a tech conference to give a talk, send a recruiter with them. Or better yet, look into sponsoring a booth and send a team of engineers and recruiters. Schedule small, closed door roundtables for potential candidates with most experienced and highest profile engineers in the company.
-
last part of the funnel - reaching out and pitching a few open roles to encourage the candidate to apply.
Female engineers are really good at recruiting female engineers. People of color (PoC) are really good at recruiting other people of color. → If you don't have women or PoC working as engineers then you need to recruit proxies. That is, you need to reach out and build relationships with people in that community who would be willing to vouch for you → Proxies can be meetup organizers, bloggers, conference speakers, other entrepreneurs → It works as long as you treat them with respect (pay them properly and reasonably).
If you do have engineers that will play an active role in recruiting then recruiting needs to circle back and provide feedback on outcomes.
Referrals–all referrals–should be treated like VIPs, even by junior employees.
Organization needs to support growth for “They're junior but with so much potential” situation where interviewers like candidates and want hire them anyway.
When trying to decide Build or Buy the following questions can be asked:
- What is our turnaround time for this type of work?
- What failure cases are common with this type of work, do we have the proper systems and processes in place to recover?
- Who do we expect to mentor this hire? Do those people have the bandwidth to do so or do we need them focused elsewhere?
- Is our engineering brand built around our expertise here?
Build:
- Less experienced engineers also tend to start on older systems where things have been stable and predictable for a while.
- If you've built your engineering brand around the idea that you invented a technique, approach or set of tools, you really should be growing your talent in those areas.
Buy:
- If the people who are supposed to mentor them are under water with current responsibilities. _ If you're hiring junior people to save money and still expecting them to turn out product at the same pace as people with twice the experience.
Big success tip: keeping in touch with candidates who my team interviewed at one time for a role where:
- they just weren't a good fit.
- don't have the right experience.
- they turned you down for another opportunity.
- they couldn't move.
- they had a timing issue.
- they were inched out by another candidate who did a little bit better on interviews
What to do:
- Leave a note in their candidate profile and keep an eye on them.
- Do periodic reviews once or twice a year of candidates that we liked but just couldn't make work.
- Have links to their Twitter, Github, Stack Overflow accounts in the system plus their blog or personal site if they have one.
When you do reach out to old candidates, acknowledge their past experience and make it personal. For example:
- You interviewed with us two years ago. We liked you a lot but thought it wasn't a perfect fit. Since then I see you've been doing ABC at Company XYZ, which sounds like a great experience. Can we grab coffee or set up time to chat for 15 minutes? We have a role opening up that we think you'll be good for.
Engineers need to get into the habit of skimming the resume pile, plucking out particularly good candidates or demoting particularly bad ones. How much time they should spend is the key → the less time they invest → the more reliant the pipeline on HR → the slower the pipeline moves → the less accurate. (15 or 30 minutes with resume reviews should be a good threshold)
Hiring managers should devote a small percentage of their work week–perhaps 15%∼20% to resume reviews for open positions on their teams → can also be adhoc if needed.
The goal of reviews is to make an educated guess on whether they could fit the competencies.
Comparing descriptions of what we expect a successful hire to do to and what the resume says the candidate has done is much more useful.
A good rule of thumb for the reviewer: Is this candidate worth the hour it will take to interview them?
The other benefit of using the position description for reviews rather than the competencies: Reviews done by engineering and HR will use the same criteria.
The process changes for hundreds or thousands of incoming resumes was actually a real problem for the government or at marquee tech companies like Google and Amazon.
The government's worst possible solution: capped the number of potential applications → go live with the caveat that only the first two hundred resumes would be accepted.
A little more elegant option: keeping positions open for a short period of time → missed great candidates that comes late.
The last and perhaps most positive impact on morale option: help recruiters deepen their technical literacy → Brown bags and guest speakers are good ways to familiarize recruiters with different types of engineering → Send a team of people and encourage recruiters to occasionally leave the booth in technical conferences and sit in on a talk or two.
Pedigree is a poor indicator of talent. I've dealt with many engineers who didn't have CS degrees from good schools who were fantastic and many engineers who spent years at the top tech companies but gave shockingly bad answers when interviewed.
Age is another troublesome bias often revealed in the body of the resume.
The best way to anonymize resumes (resource intensive): have a team of people who go through the resumes and remove the relevant information before the reviewer gets them.
- A job posting is ultimately written using data from the position description, but they are not the same thing. Job postings are external facing, position descriptions are internal facing.
- Position descriptions tie everything together: how to use the competencies for this role, how many interviews there will be, what our expectations are for a successful hire.
- Competencies can be applied to positions a couple of different ways depending on whether the role is general engineering or cross functional.
- Recruiting is best when it's run like a sales team with the expectation of building a funnel.
- The recruiting funnel should develop content,create opportunities for in person interactions and establish an engineering brand to attract top tier candidates
- Both recruiting and resume screens are more successful when engineering is involved, but time commitments need to be managed carefully.
- Since resumes describe tasks and experiences, not competencies it is easier to judge them by the position description instead of the competencies.
Asking the same questions in the same order over and over again with no freedom to deviate lowers the quality of the interviews and more importantly, the candidate experience.
Interviewing is stressful even for interviewers:
- the candidate will give an answer they don't understand.
- they'll make an error and the candidate will point it out.
- the candidate will ask a clarifying question that will stump the interviewer.
- the interview will run long.
All the questions should follow this template: • Question • Background • Lead-in for juniors • Potential follow-ups • Scoring rubric
It's the question as the candidate will be exposed to it. If we're going to use visuals or code samples, those are included here.
It is made up of short descriptive paragraphs explaining what the goals and context of the question are for the interviewer → fact should be recorded and provide a link to useful documentation or postmortems for the question → can sketch out a few potential pathways a candidate could take in answering the question.
Sometimes junior candidates need a little warm up → write down a series of questions to ask before the main question to get them think in the right direction.
2 benefits of using the same questions for all experience levels:
- write fewer questions!
- if the conclusion you came to about the candidate's experience level from reading their resume was wrong for any reason, you can salvage the interview.
A quick list of follow-up questions depending on how the candidate tries to answer the main question.
Define what a good answer to the question is and what a bad answer to the question is across all experience levels.
Good interviewers can rescue bad questions. Bad interviewers can still inflict blunt force trauma with good questions.
Live coding exercises are very controversial.
Write code during a live coding exercise frequently bears no resemblance to how you will write code as an employee → real engineers Google things.
When you consider the real world experience, try to close the delta for live coding exercises with the following questions:
- What tools will they have to use on the job?
- What language should it be written in?
- Do there need to be tests?
- Do they need to use version control?
- Are they writing an application from scratch or are they given an existing repository?
- How is development represented in the product management system?
- How do they research possible solutions?
The interviewer pair programming with the candidate issues:
- The interviewer won't want to give too much away to the candidate → won't be collaborating sincerely.
- The candidate won't want to delegate work or give up control of the problem. → Pairing interviews can feel a bit more like interrogations.
For pairing approach, consider the following things:
- use test driven development to give the interviewer something useful to do that doesn't interfere with the candidate's process.
- have the interviewer write the code as the candidate gives instructions, explains the approach and looks for bugs.
Sketching out the implementation of an algorithm on a whiteboard in the candidate's programming language of choice issues:
- Favor new graduates and people who study for them.
- Assume that knowledge of various algorithms is both necessary to be a successful software engineer and fundamental. → Are out of fashion now because for much of the industry those things are not true.
“Pub trivia” questions: short snappy questions that focus on granular details about specific technologies.
Examples:
- How do you reverse a string in JavaScript?
- What is the difference between the GNU implementation of base64 and BSD's implementation of base64?
- What does the slash mean in an IP address?
Notices:
- works only when the knowledge is common.
- should ask yourself if you want to prioritize memorizing or problem solving → nothing much candidates can do if they don't remember.
Give the candidate a hypothetical piece of software in detail and ask them to describe various scenarios to debug it, scale it or maintain it.
Useful point: There are a number of correct approaches → approaches the candidate sees and chooses to pursue reveals a lot about the depth of their experience.
Notices:
- Interviewers feel pressure to know more than the candidate at all times.
- No one right answer to create a feeling of security.
Present the candidate with a piece of software someone else has written and ask them to conduct a code review.
Tips:
- Do not show our code repositories or even pull requests on open source repos.
- Make sure code written specifically for interviewing is feature-lean, low on the level of abstraction, and has a mix of potential improvements as low hanging fruit. → Code reviews is prefered over live coding → tests a situation that is closer to the day-to-day job than just writing a simple function from scratch.
Notice:
- It is misleading that candidates inevitably assume that there are a certain number of errors in the code and they will lose points for the ones they miss → The thoroughness displayed during an interview won't always carry over to the job.
Send the candidate home with an assignment to complete at their own pace.
Pros:
- Work well when embrace the opportunity to consider more than pure technical skill.
Tips:
- look at time management.
- how they collaborate with other potential team members.
- whether they know how to work remotely.
- how they design and iterate.
- how they prioritize requirements.
- be proactive about respecting the candidate's time.
- all nighters candidate aren't necessary good → two candidates competing for the same job. If you don't pick the one who devoted too much time, will they come away from the experience thinking they were treated fairly?
Check for process than pure technical outputs:
- How do you invest your time?
- What kind of workplace boundaries do you have?
- Can you connect the assignment to user needs?
- How do you evaluate potential technical solutions?
A good rubric explains in detail how candidates might demonstrate the core competencies through both their answer and their approach.
A bad answer is not simply not knowing the right answer.
Should hire:
- A candidate doesn't know the answer but asks great follow up questions and talks themselves through figuring things out.
Shouldn't hire:
- Someone who knew the right answer but couldn't explain it.
- Someone who was dismissive or talked down to you.
Junior candidates
Characteristics:
- Good ones are high growth → ask questions, the speculate and demonstrate great deductive reasoning skills.
- Great ones will show some level of autonomy → display confidence in their ability to execute on their answers.
Shouldn't hire:
- Get overwhelmed and lost in the interview.
- Are afraid to ask questions (don't want to look dumb).
Mid-career engineers
Characteristics:
- Good ones can be trusted to execute without hand holding → draw from their experience when answering.
- Great ones demonstrate awareness of users needs or business case in their answers → demonstrate the fact that they understand the trade-offs that govern architecture.
Shouldn't hire:
- Understand only the theory and can't provide real life examples.
- Repeat things regarded as truisms in the industry without being able to justify them.
Senior engineers
Characteristics:
- Good ones understand that the right technology is all about context.
- Great ones are force multipliers, their experience with multiple approaches to accomplishing the same thing making other engineers better engineers → are active in transferring knowledge by mentoring other engineers → may have developed rare and highly desirable skills.
Shouldn't hire:
- Want to focus on their own work, without contributing to the larger picture either in setting internal standards or discussing the value add of different approaches.
- Are afraid to ask questions (don't want to look dumb).
The criteria for a Strong Hire on each level roughly match the criteria for a Hire on the level above it. The criteria for a No Hire match the criteria for Hire on the level below it.
One option is just a straightforward Pass/Fail split:
- keeps things simple.
- use for questions where there isn't much room for discussion.
For No case, a no is a no, strong or weak shouldn't matter → prefer to roll the Strong No Hire criteria into the No Hire criteria.
Short Question Rubric: How could you calculate the airspeed velocity of an unladen swallow?
| No Hire | Hire | Strong Hire |
|---|---|---|
| Candidate confuses velocity and acceleration. May not realize that “swallow” represents several distinct species of bird and that more information is required. Doesn't ask questions in general. | Can outline the basic factors that influence velocity– What is the size of the bird? How many times can it flap its wings per second? Do we want to consider headwinds or tailwinds? Arrives at a reasonable conclusion given those determinations. May not know the term Strouhal number. | Candidate understands that this is a Monty Python reference ;) |
Long Question Rubric: Given the task of building a real time chat application, what are the most important things to get right from the start?
Junior
| No Hire | Hire | Strong Hire |
|---|---|---|
| Struggles with language selection, possibly distracted by the “real time” requirement. Decides to build in a technology unfamiliar to them and fails to give clear and specific answers. | Can use what they know to develop a workable plan even if that means their application is doing things like polling. Takes feedback well. | Familiar with and can speak to the differences between WebRTC, websockets, push -vs- polling. Might gravitate a bit towards buzzwords but can demonstrate comprehension when challenged. |
Mid-Career
| No Hire | Hire | Strong Hire |
|---|---|---|
| Gets super into the weeds right away with WebRTC -vs- websockets -vs- other solutions and yet cannot communicate the tradeoffs of different approaches, especially in regard to burdening the client or the server. | Moves beyond application design and considers other infrastructure such as monitoring, data storage, encrypted connections, etc. Might connect these questions to security and privacy requirements and ask probing questions. | Focuses on use case. Is this a customer service app for 1:1 communication or is this a chat room with many users? Can explain how various requirements will change technology selection. |
Senior
| No Hire | Hire | Strong Hire |
|---|---|---|
| Views the challenge as a purely technical question. Doesn't consider whether messages should be long lived, for example. Whether the company should be able to read everything. Can spec out a perfectly reasonable technical implementation but doesn't or won't consider the context. | Asks for more information about what our goals are before making any suggestions. Can describe scaling challenges for each potential approach and the possible mitigation strategies. May be able to provide real world examples. Highlights the fact that “real time” is vague and asks for more detail. | Considers “real time” in terms of impact on the user. May go as far as to discuss a SLO for it. Will ask about mobile usage -vs- desktop usage and can explain potential challenges, especially in regard to low connectivity. |
Questions test core competencies and core competencies are more about skills than they are about knowledge → no matter what format you choose you need to be able to explain what you really know from a right answer.
For example: The interviewer asked a pub trivia question during that short period where the information was top of mind → I got a job offer as the interviewer assumed that knowing one command meant I knew a whole operating system well, but I didn't. I just knew that one command.
Issue: When the interviewer has heard the same answers too many times, candidates only mention key words then the interviewer jump ahead and assume the candidate understands what the right answer is.
Tips:
- keep a transcript becomes really useful.
- go back later and figure out whether the candidate gave a complete answer, or they mentioned some component of the right answer out of context.
Do not ask the candidate to answer questions about a problem you are actively attempting to solve → cannot evaluate the candidate's answers objectively because you don't actually know what the solution is yourself.
The goal is to have a pool of questions that can cover every interview in the loop with enough of a mix to handle any specializations → interviewers can be as prepared as possible.
Tips:
- feel free to ask whatever follow up questions come to mind and keep the conversation feeling like a discussion.
- discourage interviewers from using questions that are not documented.
- sit down with interviewers who will not give up a set of preferred questions and just document them and put those questions into the pool → everyone can use them.
The number of questions you need in a pool is determined by:
- how many competencies you have,
- how many interviews you need to run to cover those competencies
- how many specializations you are hiring for.
Tips:
- prefer each question assess one competency only.
- watch out for is the same question assessing different competencies each time its asked.
- when questions test one competency, never ask the candidate a question that another interviewer has asked them already.
- write interview templates (a quick outline of what's going to happen in the interview and how much time we assume each section should take) for each interview in the loop.
Example Interview Template
Technical Interview 1 Competency: Coding Introductions (5 mins) Question 1: Refactoring Problem (20 mins) Question 2: Testing Strategy (20 mins) Q&A (10 mins)
Different Template Formats
Tips:
- list all the questions that will test the competencies appropriate for that interview.
- let the interviewer pick which ones will work the best.
There are three combinations that fit a one hour interview block:
- Two long discussion questions.
- Four short 10 min questions.
- Two short questions and a long question.
Two ways of handling specialization in an interview:
- adjust an existing general engineering question with a theme of the specialization, for example, any system design question can be tweaked slightly to focus on security or operations.
- write questions specifically for that specialization to be asked alongside more general engineering questions.
Notice:
- Don't allow a candidate to get through an interview loop without ever asking them about a specialization they are claiming to have.
It's important to remind them that interviewers are not the one being assessed → if an expert candidate says something they cannot verify, they should write it down and research it later.
Focus on why the organization wants to hire experts in the first place:
- want the expert to transfer that knowledge to other people.
- want a resource other engineers can consult.
- want a mentor to level up their engineers.
- want someone who can build a new team around their expert skill set.
Tips:
- assess whether the answer is correct (with additional research if needed).
- assess whether they communicated their answer clearly.
- put people on part-time contracts just to interview for very specific talent.
First stage: put questions in front of interviewers and ask for feedback → grab two or three engineers actively interviewing for the organization → talking about what they think good answers and bad answers to the question (20 mins).
Second stage: using the question and refining it → good follow up questions may only come up when you see a candidate struggling to understand the question → may be things about the question that are misleading or confusing when two or three candidates make the same mistake.
Tips:
- running a brand new question short and including at least one more established question e.g. if the question is intended to take 20 minutes, cap the time you spend on it at 10 minutes so that you don't run out of time.
- making the question itself more specific, e.g. asking “what do you think about the security of this system?” instead of “what do you think about this system?”
- assuring the candidate that they have given you enough information when time has expired.
- ask brand new questions at the very end of the interview → first impressions are so important
when to retire questions:
- they show up on a website like Glassdoor.
- our needs have evolved.
- interviewers are going on autopilot when asking them.
- a good question will last about a year. (even less when the organization does a lot of interviewing or is considered particularly prestigious)
- A standardized question pool minimizes the burden of interviewing on the interviewers. They need only pick a few questions to ask, read the documentation and compare the candidate's answers to the criteria in the scoring rubric. The pool isn't about controlling the interviewing, but giving the interviewer what they need to succeed.
- Every question in the pool should be documented, even the short ones. Documentation is more than just the question and scoring rubric. It should also include background information and suggested follow ups that help the interviewer adjust to different candidates and unexpected responses in the moment.
- There are many different types of interview questions. My favorite are system design questions that focus on informal discussion and create more of a conversation between the interviewer and candidate. My least favorite are “pub trivia” questions that test random facts about engineering that may or may not be a reflection of actual skills.
- Questions should test competencies. Preferably, one question tests one competency so that question can only be used in one interview in the loop. This makes it impossible for a candidate to get asked the same question twice by accident.
- Interview templates can help set expectations with interviewers and increase their confidence, but I would recommend stopping short of giving interviewers scripts to follow. Conversations create better experiences for candidates and require more freedom for the interviewer.
- You can use the same questions for juniors as well as seniors. You can also successfully interview an expert who knows more than you do. Both situations require setting clear expectations with the interviewer about what a good answer and bad answer sound like. Those expectations should be written out in the question documentation.
The series of interviews a candidate must do before you make a hiring decision and they receive an offer is call the interview loop.
Pros and cons of some interview scenarios:
- invite the candidate to an on-site and do them all back to back → can be emotionally and mentally draining for candidates → speed up the process substantially, which will keep the organization competitive.
- stretch their loops out over several weeks.
- conducting a basic technical interview remotely before inviting a candidate to a longer session → make sure your interviewers have the tools they need and are trained to use them → if giving technical interviews over the phone, a collaborative doc like a Google doc or a hack pad can work as a virtual white board in a pinch → Share the link before the interview starts over email.
It makes sense for the first contact to be a short chat to establish any deal breakers, particularly if the organization has any specific hiring limitations like citizenship or visa status, or travel requirements.
Focus on disqualifying factors:
- the candidate can legally work for the organization.
- the terms of the future offer are acceptable. (salary requirements, relocation, etc.)
Sometimes can bypass this step in the beginning and move it to the end, closer to offer.
An informal chat with the manager the candidate would report up to if hired → selling the candidate on the job and getting them excited.
Things to look for:
- attitude and perspective.
- Why did the candidate apply?
- What are their expectations about the job?
- what does the candidate need to be successful?
- their past work experience.
Focus on:
- if the candidate is a good fit.
- establishing rapport between myself and the candidate → having a good relationship with your manager is a huge part of being happy and productive at work.
There are two models:
- The first is breadth -vs- depth: One interview will explore a candidate's broad technical skills, the other interviews will go deep on specific topics → a little difficult to keep on track as interviews are related → need to specify which questions in the pool are designed for breadth interviews and which for depth. (the shorter form questions will be breadth questions and the longer, more discussion-based questions will explore depth)
- The second is testing one or two competencies per interview → allows each interview to be conducted in isolation
Sometimes called “culture fit” interviews and they're designed to identify and eliminate candidates that would be bad teammates.
The best decisions on EQ criteria are made by looking at trends across interviews → an attitude problem is usually evident to every person who talks to them or from different interviewers running different sessions.
Founder/Leadership Screens
This type of interview is strongly advise against because:
- it creates situations where the interview loop says ‘yes' but the senior leader says ‘no' → then what to do? (especially when the Founder/Leadership does not properly performed the interview with no notes, asked no technical questions, etc.)
- these interviews tend to be unstructured, with no attempt to create a transcript or even detailed notes.
Leadership screens reflect a need to control the process to ensure that the people interviewing don't make the wrong decision, but the goal of our process is not to prevent the wrong decision but to empower people to make the right decisions.
If the pipeline is making more than the occasional mistake → reevaluate and ultimately fix the pipeline.
Two exceptions:
- when the organization is very small and any incoming employees would be expected to interact daily with the senior leader (usually a co-founder).
- when the interview with leadership is designed not as a screen but to close the deal.
Cross-Team/Cross-Functional Interviews
Are interviews with members of other teams or other disciplines.
May choose questions or steer the conversation towards issues that help or hurt collaboration between their team and the candidate's team.
Skills Assessments
Test the candidate with a skill assessment that is essentially a test conducted via an online platform → are probably more useful for junior roles and the hiring manager should be able to to waive candidates out of the requirement as needed.
Pros:
- easy to roll out.
- be a great filter for pipelines that have lots of unqualified candidates.
Cons:
- annoy experienced candidates.
- penalize creative out-of-the-box answers.
Need to make good use of your interviewers' time by not scheduling interviews with candidates who will never make it to offer.
Every interview loop will have breakpoints or gates where no further interviews will be scheduled until the candidate passes.
Efficient pipelines have 1 or 2 breakpoints.
Inefficient pipelines are long, with the candidate needing to pass each round of interviewing before advancing to the next.
Potential breakpoints:
- The recruiting screen.
- The hiring manager screen.
- The breadth technical interviews.
Hiring decisions ought to be made using the repeated themes about candidates from multiple interviews administered by different people.
When you can put candidates at ease and get them to enjoy the experience of talking about technology with another engineer → will get a more accurate assessment of their skills and ensure they feel like their time was well spent.
Potential negative effects:
- asking them to put other offers in jeopardy by insisting they wait to hear from you.
- the longer the loop, the more of the candidate's time and energy is being invested → candidates may get attached to the image of themselves working for you.
Be mindful of how much you are asking the candidate to invest in between each breakpoint in the loop.
When long interviews happen:
- interviews built around take home assignments.
- the number of interviews and the gap between interviews → most candidates prefer fewer and smaller breaks.
Consider:
- which interviewers are in the best position to test the candidate's technical skills.
- who will inspire or impress the candidate.
- who will they enjoy talking to?
- who are they likely to click with or get excited about?
Advice: questions should reflect work you actually do, challenges you actually have, and also what you love about your job. (focus on positive points)
When organizations have distinct teams that report up to individual managers → have the engineers on the team with the open position do most of the interviewing.
If the organization is large–a Google or Facebook size → have one pipeline that qualifies everyone and place the candidates with a specific role later on → avoid creating a special class of engineers who interview. (especially when promotions and raises are being evaluated without interview work).
Solos → conventional Pairs (author preferred choice) → can have very different impressions of the same conversation and get twice the insight → one person can focus on talking with the candidate, while the other interviewer takes detailed notes. Panels → formal organizations sometimes prefer → get more opinions out of one conversation but is tricky to coordinate who on the panel gets to speak when, or budget enough time for all follow ups + candidate it must feel a bit like arguing a case in front of the Supreme Court.
The loop benefits from reflecting what a hire needs to do to be successful if employed → most loops for interviewing engineers will be populated with other software engineers → may get better results with certain positions and levels if you mix things up.
People Who Look Different
Can tell a lot about a candidate by the assumptions they make and how that informs their behavior when part of the interview team is an underrepresented group. This works best in pairs.
Juniors Interviewing Seniors
The work of a senior engineer is much more about leadership and mentorship than pure technical ability → can show you whether the candidate can do the job in the first place.
Cross Functional
Software requires several different types of expertise and successful engineers need to be listen to and solicit advice from their peers in Product, Design and Security → will give you the best data possible on whether a person claiming to be collaborative actually is.
Need to figure out how to actually interview the candidate together.
The Grand Inquisitor
One interviewer does all the talking, while the second interviewer observes and takes notes of the conversation.
Good for: use this format when training new interviewers → they shadow me during an interview once or twice → when they're ready I shadow them and they play the role of grand inquisitor.
Round Robin
For each interviewer to pick a question from the template that they want to ask and simply take turns.
Good for:
- transitioning new interviewers from the shadowing stage to regular interviews.
- a pair of people have never interviewed together before and do not know each other's style very well.
Primary/Secondary
One interviewing takes the lead on a question and the second interviewer jumps in from time to time with follow-ups.
Notices:
- requires interviewers to think on their feet and play off each other well.
- follow up questions can take the conversation in strange directions.
It gets easier when you remember that the candidate is the one being tested, not the interviewer.
If the candidate pulls the conversation into a field or topic you don't understand → move it back by politely redirecting them e.g. “Actually this question is really more about security...”.
If a candidate asks you a clarifying question you don't know the answer to → parry by asking them to explain what the possibilities are and what would change in each situation or totally okay to say “I don't know” or “I'm not sure.”
Remote issues:
- potential technical failure that may disrupt the interview.
- requires the interviewer to have thought through how best to interact with the candidate if code needs to be written or files shared.
In-person issues:
- maintaining eye contact makes it more difficult to keep detailed notes → having a partner can help a lot here.
- logistical burden is higher → have to find a space to give the interview.
Remote interviews are more favored because:
- most technical jobs these days require at least a little work using the same conferencing and collaboration tools needed in the interview.
- can also assess how comfortable and familiar the candidate is with these essentials.
If the interview is remote → shoot the candidate a quick “Do we need to reschedule” email after five minutes (15 if an in-person interview.)
If they haven't responded by the twenty minute mark → alert recruiting and let them figure out what the next steps are.
Letting candidates wander a bit is okay → learn a lot about them from observing how they interpret a question and the logical progression of their answers → will naturally gravitate towards the areas where they have the most experience (useful information).
Digressions: some candidates will word vomit buzzwords and terminology before answering a different question or bringing up an off-topic story → a strong signal that they don't know how to answer the question and aren't willing to work through it → don't have to accept every single answer if you feel it doesn't give you the information you need about the candidate's skills.
Can be dispatched by asking the candidate to try to step through it and come up with the answer themselves, or by just saying “I don't know.” e.g. questions about compensation are best left to HR or Recruiting.
- The loop is a series of interviews that inform the hiring committee.
- There are many different types of interviews that can make up the loop, some of them technical, some of them focusing on soft skills. Not all interviews in the loop are run by engineering, but the results of each interview should all have roughly the same weight in the final result.
- The interview process should include planned breakpoints where poor interview performance can keep a candidate from moving on and wasting valuable engineering time.
- Other than those breakpoints, the loops should consider interview results as a package and look for repeated patterns and themes across interviews.
- Interviewers should understand that they control the scope and direction of the interview. If the candidate moves into areas the interviewer feels uncomfortable with, the interviewer can interrupt, redirect or just move on to other questions.
Introduce engineers to the interview process and give them the space to practice working within that system.
Interviewers should understand how to identify and talk about the hiring theory of the candidate.
The hiring theory:
- is what will make this specific person successful in this specific role.
- is specific to the candidate, not the role.
- tells everyone what the organization expects the candidates to do if hired and how they should do it.
A hiring theory are the answers to the following questions:
- Who do we think this candidate is from our interactions so far? What is their experience? What are they looking for in their position?
- What position are we considering them for and why are they a good fit?
- What other places could they add value?
Pros:
- allows interviewers to choose the questions that are most likely to reveal the most useful information.
- forces the organization to articulate how they are going to enable success.
- avoids hire great people that they cannot or will not do their jobs.
- in the scenarios where that candidate could work out and scenarios where that candidate could prove disastrous, a hiring theory helps tell the interviewer which of those scenario the organization is capable of supporting and which are unlikely to be viable conditions.
- eliminate some of their imagined scenarios and hopefully settle on a clear recommendation.
Case studies:
- Case 1: when you run a hiring pipeline you will sometimes find great candidates that have applied for the wrong role → the organization may choose to move them over to the process for that role, but it would be better to invite the other hiring manager and their team to collaborate on the loop in order to keep the experience smooth and efficient → minimizes the risk that great candidates will get lost in between team silos with a good hiring theory.
- Case 2: a candidate was going to be a great hire because they had all these amazing and innovative ideas → innovative ideas were wonderful, but engineering's role in our product development cycle was to determine technical feasibility and implementation → Was our hiring theory that we were going to create a new role for this person so that they were involved in the conversation earlier? How did we expect this hire to inject those innovative ideas?
Recommendations:
- focus on using the candidate effectively than the candidate's pure skills.
- start with phrases like “if we're willing to train them” or “Would be great for a future leadership role as we grow.”
A prebrief is a fifteen minute conversation where the hiring manager and the recruiter:
- explain the hiring theory.
- answer any questions about the candidate's background and performance during screens.
- give the interviewers a little time to strategize about who will ask what.
People who participate in a pipeline do not have equal access → get everyone in the loop on roughly the same page.
Are a great confidence booster for newer interviewers → go into your part of the interview knowing exactly what the rest of the loop is expecting from you.
This includes:
- the ins and outs of the process.
- the mechanics of any tooling used to coordinate interviews and collect feedback.
- general interviewing practice.
Don't:
- do one big training that covers everything with everyone.
- recording a training and dropping it in the interviewer's lap to review.
- indoctrinating interviewers into a process.
Do:
- more one-on-one interactions an interviewer can get during training → easier it is to absorb what you need them to do and how to do it.
- have the rules around discrimination → what you can and cannot ask a candidate are worth getting right.
- making the interviewer feel comfortable.
When run a training, I break things down into three sections:
- An overview of the loop:
- logistics are often first and foremost on people's minds.
- encourage interviewer to see the full picture.
- how their interview contributes to the overall result.
- what other factors are considered.
- how determinations are made.
- Debunking misconceptions about interviewing in general:
- good interviews should not be as hard as possible to keep high standards.
- the interview is not a test, it is an assessment.
- Scenarios and frequent problems:
- calling out the nightmare scenarios directly and talking through what to do.
→ make sure that the deck and any relevant assets are available to the interviewers to consult later on.
When shadowing, the new interviewer should spends most of the time to:
- watch.
- listen.
- take their own notes (up to speed with the candidate's answers).
Main interviewer should:
- kick things over to the shadow at the end for any follow up questions.
- don't discourage shadows from playing a more active role.
- don't penalize them from wanting to just observe either.
- when new interviewers seem uncomfortable after shadowing → pair with them a few more times + assign them sections or questions in the interview to conduct.
The last step in training is for the new interviewer to take the lead conducting an interview.
Main interviewer should:
- play primary for part of the interview.
- just back the new interviewer up as secondary.
- don't put them into rotation until they have enough experience to draw on in order to adapt on the fly.
Has two components:
- a factual accounting of the exchange between the interviewer and candidate.
- an analysis of the significance of those exchanges.
Do:
- keep a transcript of the interview.
- detailed notes are the next best thing.
Don't:
- just provide your opinion and leave it at that.
The analysis portion of the feedback should be:
- an opening paragraph summary of the conclusions and the justification.
- in-line in parenthesis as part of my transcript → useful when the candidate's answer to the question is wrong or need to condense rambling, vague answers.
It depends because:
- scores tend to not match the written explanation from the interviewer.
- people give a low passing score when their feedback is full of reasons → the right score is actual a failing one.
- people give glowing reviews and downgrade their scores → avoid looking like an easy interviewer to impress.
- when interviewers see scores from previous rounds before writing their own feedback → they most likely gravitate towards confirming those scores. → Eventually someone has to decide whether the decision is being made based on the score or based on the feedback.
When must attach numbers, then keep it simple.
Only really need four possible conclusions from an interview:
- I believe so strongly in this candidate I will fight for them.
- I would hire this candidate but can be convinced otherwise.
- I would not hire this candidate but can be convinced otherwise.
- I believe strongly that this candidate is not a good fit for us.
Tips:
- An interview should never end with a maybe.
- Rather than asking them to make a hiring recommendation, ask them to rate how excited they are about this candidate coming to work with us.
- Every candidate should have a hiring theory which defines what the organization is expecting from them and what they see in the candidate's experience that speaks to that.
- Training interviewers is not a one and done situation. The best training happens in stages, gradually increasing the new interviewer's role until they are comfortable conducting interviews.
- Don't just explain how to interview, talk interviewers in training through their fears, give them advice, remove as many of the unknowns as possible
- After formal training, give new interviewers an opportunity to shadow and pair them with more experienced interviewers for the first couple sessions. Nothing will prepare them like live candidate experience.
- More is more with feedback. Interviewers need to accurately report both the facts of the conversation that happened in the interview and their assessment of the answer so that the hiring committee can make the best decision possible.
Sometimes the weight of sunk costs that have accrued over the weeks of investing time and energy into candidates can increase the pressure of this final stage → the impact of the hiring decision will outlast the fatigue of the hiring process.
Possible choices on Who gets to make the hiring decision:
- a single person. (presumably the hiring manager)
- a group of people acting a hiring committee. (preferred by the author)
Reasons:
- the new hire comes into a new job discover that only the manager wanted him or thought he would be valuable.
- easier to fire someone who isn't working out → locks people into denial, delaying the process of removing the bad teammate as the hiring manager doesn't want to admit they made a mistake.
Getting the interviewers in the loop together to discuss their thoughts.
Include the hiring manager who may moderate the discussion or just receive the committee's recommendation.
The staff the committee with people who did not conduct the interview.
Made up of senior leaders, perhaps directors or senior managers of the engineering organization.
Best for: when the organization has reached a scale where running a pipeline for each position separately is inefficient.
Don't:
- don't vote during the hiring committee → this put people under pressure to agree with their boss - the Hiring Manager.
- are not inappropriate people to vote if the Hiring Manager just run the initial screen.
Do:
- moderate the discussion to ensure that everyone who participated gets to speak their mind.
- run thought exercises to help people clarify their feelings when they are hedging or indecisive.
- cast the tie breaking vote if needed.
- let team to make the decision about who does or does not qualify.
Do:
- presumably people have had time to read each other's feedback → If there are No Hire recommendations, then likely start the conversation there.
- give people a forum to speak.
- follow up with the interviewers to get their feedback and feelings clarified if they are not clear enough for a non-loop-based committee to make sense of.
Maybes are noes in hiring.
A good way to know is to ask - “If we issued an offer and this person turned us down, would we be disappointed?” → If no, then you're hiring a maybe.
Might be unfair for a team to decide for another team where priorities might be different → get the interviewers to focus on whether they would want the candidate on their team.
A candidate who is a good fit for only one team is unlikely to be successful.
Sometimes, there are both strong pros and cons to a candidate → the hiring committee is split between the people feeling the benefits outweigh the disadvantages and the people feeling the opposite → need to tease apart the distance between the yes votes and the no votes.
Probabilistic decision making can also be described as making decisions with bets.
How it works:
- ask the hiring committee to briefly brainstorm possible outcomes from hiring (or not hiring) this candidate.
- use the hiring theory as a reference in this exercise and discuss about:
- What did we think this candidate's value would be when we started the process?
- What potential outcomes can we foresee now?
- sometimes restrict the conversation to positive outcomes or negative outcomes only in order to keep the discussion focused.
- ask everyone on the committee to place a bet on each outcome once having list of potential outcomes. (can bet imaginary money, time).
- for time bet, the direct consequences of a bad hire are that they inevitably make more work for people and great hires save time. (Great hires get higher bets, bad hires get a small number of hours)
- discuss the new hire in terms of probabilities with:
- What outcomes are we the most confident about?
Bets work better than discussions because they establish a reference point with value → people know how much two hours of their time is worth; people know how much $100 is worth → connect that to real impact on their lives.
Rate a candidate 1-5 or 1-10 doesn't give them a clear picture of how much a four is worth vs a five.
- Hiring decisions are improved when it isn't one person alone who contributes to the decision.
- Committees can include interviewers, but they also can include people not involved in the loop at all.
- As an organization scales up hiring it makes sense for the pipeline to reuse resources for multiple positions. The hiring committee is one option.
- Maybes are noes. Every new hire deserves to come into a new role with colleagues that are excited to work with them.
- In situations where the committee is split between seemingly equal strong opinions it works well to get committee members to talk about probabilities and confidence levels. Ultimately candidates should receive offers not because they are smart or nice people, but because the organization believes they will be successful.
Provide a roadmap of how hiring changes as an organization scales → are subject to economies of scale, larger operations open up options that would cripple a smaller firm.
There's no switch that gets flipped when you hire engineer number 10, 50, 100 or 500.
If you're on the lower end of the range for your stage → study both the goals for your stage and the one below.
If you're on the higher end of the range → study the goals one above.
Problems are likely to start developing a bit earlier and linger longer than sometimes we would like.
Focus on sales pitch over process → a lot to gain from shaping interview questions around what is exciting and attractive about your work.
Tips:
- Keep it simple.
- Shouldn't have more than three levels (junior, mid-career, senior).
- Competencies and rubrics can stick with No-Hire/Hire/Strong-Hire breakdowns → write a few competencies but don't spend too much time on them if you expect to reach 70 engineers within a year or two as they will change fast.
- Competencies are more useful when focus on how we do things at work.
- Early employees have a profound impact on company culture → use competencies to answer critical questions like:
- Do we hire people we can grow?
- Do we incubate cutting edge research?
- Does design dictate engineering challenges or do we expect designers to polish engineering's output?
- Get everyone on the same page around what it means to be an engineer at your organization.
"Telling me your organization has no politics is like telling me you live in a place without weather: you do, you just don't notice it." - Mikey Dickerson of USDS
At small organizations collaboration is high and built on personal relationships and trust.
At large organization that breaks down because not everybody in the organization can know each other.
The politics operating behind the scenes depends on whether or not you have the personal connections to get stuff done → If you do, everything seems smooth and efficient → If you don't, everything seems helplessly cliquey and political.
→ This is a good time to prep for that transition by starting to develop levels → it's easier for people to self-organize → getting stricter about what it means to be a “senior” engineer.
Right attitude is more important than the right job title → it's okay for a mid-career engineer calls themselves a senior if they are making an honest effort to both fulfill the responsibilities of a mid-career engineer and grow into a senior one → it's not okay for mid-career engineers who interfere with the work of others in a desperate attempt to prove their seniority or who use their insecurity to bully others → don't be afraid to lose one or two early employees.
Do:
- figure out what roles are empty.
- start thinking about management.
- learn to hold a vision of the future in your mind while rolling out a more modest implementation.
- right scale your process and redo it earlier.
Don't:
- roll out a process that no one actually uses.
At this point, people actually want to have a boss when that means they have someone to build rapport with and help them get things done.
Managers become important for three reasons:
- People are going to get lost in the cracks if no one is assigned to check in on them and their work.
- At this stage, other divisions is built up to support and mature the actual business → negotiations between the technical and non-technical sides become a real thing → without a management engineering, no one to represent for people's interests → allows the business side to asking multiple engineers the same question → damages the long term sustainability of the technology and frustrates engineering.
- Startups are crazy and the people in the very early days tend to bond like crazy → at this point, membership into that club of early employees is closed → new hires will never truly get all the inside jokes or form the relationships with the same intensity → good managers act as an anchor point, making they feel like they belong.
Enough work for a whole class of employees to do nothing but management → the coordination required to keep a large organization moving in a positive direction can easily fill up a 40 hour work week.
Do:
- some of that coordination can be offloaded to staff and principal engineers.
- require a split of the career ladder to a management track and non-management track.
- have clearly defined antipatterns.
- set the expectation that both competencies and questions need regular refreshes.
- think critically about value add when making changes.
Consider either:
- treat engineering managers as a separate discipline with a full set of competencies.
- use the same competencies but develop position descriptions that specify a different ratio of technical skills to soft skills. → depends on what your competencies for engineering look like when reach this point.
A mature process pays huge dividends but rolling out changes will be harder.
Splitting off new disciplines will either cultivate greater expertise or create opportunities for ambitious career builders to plant a flag in the ground at the organization's expense.
The biggest challenge: the incentives around hiring, career growth, and product development have diverged.
Once a specialization is broken off into a discipline → can hire or develop deep expertise.
Engineers collaborating and working with other engineers specialized in the same fashion get stronger, better, and more valuable.
Have the resources to build mini engineering organizations for each product or service → do that and get backed into matrix structures → Does an engineer report up to a manager in their discipline or a manager for the product they are building?
The critical challenge from the hiring perspective: determine when things should be standardized across engineering and when they should be standardized within an individual discipline or team: → running one pipeline that ensures candidates are minimally qualified + can fit with multiple teams → then develop an allocation process that matches them with an available slot.
Lose a bit candidate experience as people want to work on problems that are interesting to them, not just any old open position at a big company.
The pipeline development goals of mega organizations are the same as micro organizations: what sales pitch to the candidate does the interview create? → for mega organization the pitch needs to inspire the engineer to look at challenges independent of the veneer of the product sitting in front of it e.g. when your role is designing the product and the output of these tasks will be reused and iterated on by other internal groups → get candidates excited about that so that they no longer think about things in terms of what team they're on.
Remember, talking to candidates should always be fun. 😁