talk-notes.md

Devops

• The war for definition
• The resistance to a definition
• The filling of the vacuum
• Devops comes from the lived experience of the practitioner
• Making it, in all cases, unique to the practitioner
• We had a lot of overlapping experiences though
• Relationship to failure
• High trust, high interaction relationships with each other
• A penchant for automation
• A tendency towards self reliance
• Obsession with metrics and graphs
• A focus on revenue and availability

Devops and the struggle for definition

• I think the struggle to define devops comes from this nature
• You can look at an individual, and see the 'school' they came from
• If you look at the 10 people I worked with longest, they way they 'do devops' is very, very similar to mine
• Its not suprise - our experiences are very similar
• The further you go from there, the more small differences there are
• In general, the macro trend holds - you can see us all under one umbrella, with many small differences

Kung fu is like that

• Gōngfu (功夫)
• 功 (gōng) meaning "work", "achievement", or "merit",
• 夫 (fū) which is alternately treated as being a word for "man" or as a particle or nominal suffix with diverse meanings (the same character is used to write both)
• Originally, it did not mean just to practice martial arts
• It referred to the process of ones training - the refinement of body and mind, the learning and perfection of your skills
• The excellence achieved through long practice in ones skills Source - http://en.wikipedia.org/wiki/Kung_fu_%28term%29

Many styles of Wushu

• Hundreds of different schools
• They vary widely on what they emphasize
• But the broad methods are the same
• Basics - the fundamental internal/external foundation
• Forms - a series of predermined behaviors that can combine to a continuous way of acting
• Applications - using your Kung fu in the field, and adapting the basics and forms to your situational awareness
• When you see someone doing it, and you are broadly ignorant, you know its different than boxing
• When you are someone who knows it well, you can tell the differences right away
• Sound familiar? Source - http://en.wikipedia.org/wiki/Kung_fu_%28term%29

We need to understand Devops like we understand Kung fu

• There is a deep well of practical philosophy
• There is a broad base of experience, unique to each individual
• We are recognizably similar
• There is a way to teach people how to do it
• It starts with the basics, and everyone must master them
• It moves to Forms - predetermined behaviors that can be combined into continuous sets of actions
• It ends in Application - actually doing your Kung fu in the field, using what you have learned
• TMTOWTDI is true and honest and important
• However - if you don't know how to do it, you could probably skip the 15+ year journey of difficulty
• And practice the basics, learn the forms, and apply them in your life

We can and often do argue about what it means

• Those arguments might change what overall makes a form
• But they don't change its nature
• We have a common language and pallete for expression of our ideas

This is my Devops Kung fu

• I call it "Chef Style"
• It comes from my experience
• As a systems administrator
• As a software developer
• As an entrepreneur
• As an executive
• As a human being dedicating my adult life to the internet
• As a father
• It draws on many important influences
• Mark Burgess
• John Allspaw
• Christopher Brown
• Tom Thomas
• Nathen Haneysmith
• Jesse Robbins
• Barry Crist
• Mitch Hill
• Alex Ethier
• Paul Edelhertz
• Larry Wall
• Yukihiro Matsumoto
• Theo Schlossnagle
• Jeff Hackert
• Lao Tzu
• Thich Naht Hanh
• Jesus Christ
• Jez Humble
• Ben Rockwood
• Jesse Leach (Killswitch Engage)
• Ronnie James Dio
• Sammy Hagar
• Johnny Cash
• This talk is my foundational document of that Kung fu
• Once you have watched it, you can join my school
• We can develop the basics, our forms, and help on application
• Talib Kweli explained to me that artists amplify the movement they see in the world
• Marvin Gaye didn't write What's Goin On and
• then
• the movement happened
• The movement happened, and Marvin had to respond to, and amplify it, with his art
• That is my intent here - I will use my platform to raise up the voices of those who are doing the work
• And we can then leverage it as a platform to help each other
• And to further our aims of having our workplace be a better place

I know nothing about Wushu

• I took Tae Kwon Do for a month when I was a kid
• It's not even Chinese, its Korean - that's where it ends
• But - my Devops Kung fu is strong
• So is my Comic Book Kung fu
• So I will now use exclusively Comic Book Martial Arts references

Chef Style DevOps

Breakdown of a DevOps Style

• A definition of DevOps
• The principles we aspire to (How you recognize DevOps when you see it)
• Wushu calls this the 'basics', but it involves a lot of pushups, and there isn't a lot of pushups in devops
• The forms we use - the actual things we practice, that try and re-enforce and bolster our principles
• The application - the use of those forms in our actual situation
• These four things are the evolutionary arc of your practice - in the beginning, you learn from others
• As you grow, you will use the forms others have used to learn about the deeper meaning and philosophy
• The more you apply them in practice, the less reliant on the forms you will be - it all just becomes the natural flow
• 'Do not be timid about the scope or the influence - we are re-inventing the way we run businesses'
• Leading you, and anyone else, to be where those of us with already strong Kung Fu are - uniquely suited to our task

Definition of DevOps

• A cultural and professional movement, focused on how we build and operate high velocity organizations, born from the experiences of the it's practitioners.
• Cultural and professional - cultural like Heavy Metal, professional like Vocalist, Lead Guitar, Rythm Guitar, Drums, and Keyboards
• Focused on how we build and operate - it encompasses the details of how we build things, and the details of how we operate them
• High velocity organizations - because we aren't very good at operating slow moving organizations; we were built to go fast
• Born from the experiences of it's practitioners - many of whom were the web innovators - because they were the first ones that had to go fast
• It is a Job Description (and can be a useful signal of intent)

Who cares about it?

• Everyone involved in building and operating a high velocity organization
• Including, but not limited to
• CEOs and Executives (Professor X)
• Managers (Cyclops)
• Systems Administrators (Beast)
• Software Developers (Storm)
• Network Administrators (Kitty Pryde)
• Security Professionals (Wolverine)
• Sales Reps (Cannonball)
• Product Managers (Forge)
• All of these roles (and more) should practice our Kung fu
• The application may be very different, but the basics and forms are the same
• In this way, we are not generalists - we are highly connected specialists

Principles of Chef Style DevOps Kung fu

(It's not about whether you are good at it - it's about the soul of our kung fu) (It's your intent)

Design for Safety, Contentment, Knowledge, and Freedom - take from the good at ops deck

People -> Product -> Company

• Both the producer and the consumer
• When you do the right thing for the people, the right things happen for the product and the company

Lean is good

(Waste Reduction/Pull)
• Muda
• Pull
• Kaizen
• Kaikaku
• Small Batch + Experimentation
• Lean product development

Failure is a learning opportunity/a status quo event/not an anomaly

Ubiquitous Workflow Automation that makes the easy path flow with our principles

• Conway's law

Diversity is good, both in thought, opinion, experience, and background

• Voices, stupdity, similarity
• You want harmony, thinking in different ways, bringing their skills to bear
• Good D&D parties are diverse

Forms of Chef Style DevOps Kung Fu

Define your purpose

• Need a sense of clearly defined purpose thats bigger than the task at hand
• Sources of engagement
• Gallup Q12
• Write it down publicly
• Always refer back to it in every internal communication
• Shoot for one sentence
• Include the people you are trying to help
• Include your product
• Include the change the people you are helping will see "The most enduring and transformative companies use Chef to become fast, efficient, and innovative software driven organizations" (Contentment, Knowledge, Freedom) (People over Product over Company) (Lean Product Development)

Define your beliefs

• What, in your environment, bring about good outcomes
• Write them down publicly, and refer back to them on every internal communication
• Remember the principles of DevOps
• Mix in the things that are unique to your industry and problem (Contentment, Knowledge, Freedom) (People over Product over Company) (Lean Product Development)

Focus on empowered teams

• Empowerment
• Permission to act
• Context to make good decisions
• Access to help, guidance, and more context
• Find leaders who
• Care about your purpose
• Align with your beliefs
• Have leadership as a talent
• Holding of context and caring for your subordinates
• Be available for context and caring at any time - up to senior leaders, down to individuals

Form diverse bonds (wu-tang financial - diversify your bonds)

• Take people to lunch, or have meetings, outside your specialty
• Ask them what they do, and try and understand
• You need to talk to:
• Legal (Ask them what the role of a lawyer is, and how you can help make any interactions you have smooth)
• Finance (Ask them what the role of a finance is, what key metrics they watch, and how you can help)
• Sales (Ask them what their quota is, what do they struggle with at customers, and how can you help them make their number)
• Marketing (Ask them what message is resonating, why, and how they see the market evolving)
• Business Development (Ask them what partners are moving the needle, how they think about those arrangements, and whats on the horizon)
• Software Development (Ask them what they are building, and why it matters, and what they are excited about in the future)
• Systems Administrators (Ask them about availability, what do they think is the biggest limit to your organizational efficiency)
• Security Professionals (Ask them how they evaluate your business and technology in terms of risk, how you can help)
• Product (Ask them what they think the gaps in your product are, and to talk about the last 5 customers they've met)

Build consensus for important decisions

• Before you try and do a major decision, make sure you know the answer going in
• Have a written down plan, and why you think its important
• Talk to all the key stakeholders, preferably 1:1
• Get their criticism and feedback
• Update your plan, incorporating thier feedback and criticsim (Make It Theirs)
• Present plan to the group
• Get it done

Make products that have strong value propositions

• Product/Pain Relievers/Gain Creators
• Map to Customer Jobs, Customer Pains, and Customer Gains
• Pain Killers not vitamins
• Make something people LOVE
• Needs not wants (customer wants a feature, customer(s) need features)
• How to identify the opportunities
• Call out customer pain through engagement
• Interview customers to learn about their problems
• Solve the pain you experience yourself
• Write down your ideas for products What is it? (product) Who is it for? (target customers) What problem does it solves and why does it matter? (problem & pain) How does it work? (pain relievers & gain creators) What is unique about it? (uniqueness) What is the value proposition? (value) How do we make money? (cost & revenue)

Build a roadmap

• Lead with vision
• Align with customer feedback
• Balance innovation with customer needs
• Group them into themes
• Define the outcomes for the themes
• Distill those into features, and validate with customers
• Iterate on features according to vision and feedback
• Put it on a grid
• Focus on themes
• The power of the parenthetical
• Keep it alive through iteration
• Your THEMES should hold - your OUTCOMES might hold, and your FEATURES should shift all the time

Always have delighters

(The Creating Delight slide)

Features are iterative, not incremental

Mona lisa slide
• You don't have debt if nobody is using it
• Don't mistake "doing it right" for doing it "incrementally"

Manage Risk

• You are building for high velocity
• So small batches, with near term hypothesis
• Shipping the smallest and simplest thing possible to help Get closer to our objectives Get feedback learn validate De-risk
• Experiments and validation with customers
• You introduce near-term volatility to gain decreased long-term failure
• Stylistic difference - there wasn't a way to be declarative about stylistic choices
• There is no reason you have to act drunk when doing drunk boxing

Don't worry about scaling until you know it works

(Early stage risk)

Plan for execution

• Theory is fun for yak shaving
• Reality is complex
• Execution is all that matters

Always Demo

• Demo every week
• Invite everyone
• Record it and post it
• Gives you a celebration all the time
• Helps you focus on ITERATIVE vs INCREMENTAL (hard to demo one leg)
• Teams that aren't demo-ing are probably being INCREMENTAL

Choose languages and tools that fit the job

• We are all polyglots
• Learning new languages and tools is one of the great joys of our industry
• Be willing to choose langauges and tools that fit
• Small batch + iterative development protects you from risk

Use source code control

• You must have lightweight feature branches

Have a bug database, and triage your bugs

• Triage and prioritize the bugs
• Be nice to your customers
• Do you have a way to track their requests

A Feature can be implemented from its definition

• A feature needs to be complete enough that anyone can build it

Continuous Integration

• Always integrate branches to master
• They should be short lived, iterative branches
• This does not mean just automated testing - it means integrating to master and testing frequently
• You are going to fix the build when it goes red.

Four-eyes minimum

• No changes happen to the system with a single pair of eyes
• Code review
• Acceptance

Write tests as you go

• Write one test at a time
• Write the tests that help avoid future problems
• Test from 4 different angles
• Unit test (a single function)
• Integration tests (multiple classes/units)
• Functional tests (user-oriented, high level, full stack)
• Smoke tests (quickly determine if the system is "working")

One artifact

• Use the same artifact that you test when you deploy

Continuous Delivery

• Always be able to ship
• Have the role of who ships be clearly demarcated

Have a single workflow for change

• The way change moves through your organization is fixed
• Designed to re-enforce the flow and principles you hold
• Flexible at the level of execution
• Local->Change->Review->Verify->Accept->Build->Union->Rehearsal->Delivered
• Avoid using words like
• Production
• Staging
• QA
• Those are activities and locations all rolled in to one
• They happen ALL OVER The workflow

Infrastructure is code, and all code goes through the same workflow

• Applications are code
• Infrastructure is code
• All code goes through the same workflow

Focus on Availability

• Availability = uptime/(uptime+downtime)
• Focus your efforts on reducing mean time to diagnose and mean time to repair
• Failure is inevitable, its how you detect and react that matter most

Collect metrics

• From the operating system, network, applications, and your process
• High resolution matters
• Have as few systems as possible

Plan for capacity

• Identify key metrics
• Put them on a graph
• Set a limit
• Plot a trend line
• Expand your time horizon

Only alert on what is actionable

• Get the attention of the right humans
• As few alerts as possible
• Routed to the people who can take action
• Start with the "is it up" alert (see also smoke testing)
• Never create an alert that isn't actionable!

Practice incident response

• OODA
• Orient is the step that matters most - rob pike/ken thompson
• The First Responder is the default Incident Commander
• Decides what to do next
• Coordinates resources
• Can hand off command
• Communicates status
• Not about rank
• There is only ONE Incident Commander.

Incidents lead to Post Mortems

• Invoke the space: we are here to learn, not to blame
• Describe the incident
• Establish the timeline
• Identify contributing factors
• Describe customer impact
• Describe remediation tasks for the root cause
• Describe improvement tasks for response process

Use scalable systems design

• Autonomous actors
• Responsible for their correct operation
• Making progress toward that goal
• Making clear promises to other actors
• Having those actors evaluate the quality of those promises

The sense of humility

• The sense of code ownership across the product
• Break down the one person silo
• Code review, cross training, skills development

Software for simplicity, extensibility, re-use

• Conway's Law "Interpersonal variability, that is the capability and behavior differences between programmers using the same language, tends to account for more differences between programs than a change of the programming language."
• Make a component as simple as it can be, but no more
• Design for re-use when you can
• CERN and the LXC - one framework for four experiemnts
• The number is 3 - you don't need a hundred re-use points
• Can you get 1 less than the numebr of actors
• Make extensible system
• There is a loop that gets you here
• Complicated
• Break down
• Simple

The technique is the safe

• When you use your core principles, and you use your discernment, you are in the right place
• The process by which we do things is what matters
• It is training and the situation and experience that brings you to the right value

Application of Chef Style

Breathe

• Lets all take a minute and breathe
• Cause that was a lot
• But we're not talking about a small thing - we're talking about changing the way we build and run our businesses
• And it's the work of your entire career - so you've got some time to work on it

Application

• The next step is application - actually doing all of these things in your organization
• Given a long enough application, you will develop your own style - what works, what doesn't, what vocabularty you choose

Where do you start?

• While we can't - and wouldn't presume - to tell you how to do it forever
• We can tell you how to start tying all these things together, and gain your first experiences with what it feels like

The Project

Step 0 -

Step 1 - Assemble the stakeholders

You have the things

When the moment comes, the way you have trained

• Your discernment
• Your experience
• That's what the real active action is

• You talk about moving risk and complexity to the left and that is where it is in the kata (or tai chi form) as well. The most frequently needed/used moves are at the front, the ornate or specialized moves at the end or on the right.