High Performance Organizations Reading List
Most of these books, web pages, and videos are about how to design better organizations. Some are about how to be more effective individuals within the organizations we currently have.
Slack: Getting Past Burnout, Busywork, and the Myth of Total Efficiency
There is a tradeoff between efficiency (meeting previous well-defined needs with minimal effort) versus effectiveness (meeting newly emerging needs with flexibility and responsiveness through organizational learning). If you optimize only for efficiency in meeting previous needs from past opportunities, you will by neccessity eliminate your organizations's capacity to respond effectively to future needs from newly emerging opportunities. This ability to learn and grow as an organization requires "slack" time. Middle management has a vital role to play in organizational adaptability -- but only if they are not over-scheduled.
Peopleware: Productive Projects and Teams
by Tom DeMarco and Tim Lister
"Few books in computing have had as profound an influence on software management as Peopleware. The unique insight of this longtime best seller is that the major issues of software development are human, not technical. They’re not easy issues; but solve them, and you’ll maximize your chances of success."
Managing the Unmanageable: Rules, Tools, and Insights for Managing Software People and Teams
by Mickey W. Mantle and Ron Lichty
"practical hands-on advice about management software people and projects"
Dialogue Mapping: Building Shared Understanding of Wicked Problems
"In contrast to the use of agendas and restrictive structures, dialogue mapping is a facilitation technique that allows the intelligence and learning of the group to emerge naturally. Each participant can see how their comments contribute (or don't) to the coherence and order of the group's thinking. The first full-length book to bring dialogue mapping to a wider audience, Dialogue Mapping provides an exciting new conceptual framework that will change the way readers view projects and project management."
The book explains how we can visualize discussions on complex topics using the IBIS notation (Questions, Ideas, Pros & Cons) which provides just enough structure to aid a group's short-term memory without getting in the way. What might be arguments over the best way to proceed become collaborations in constructing a dialogue map exploring all the possibilities and their evaluations.
More on Dialog Mapping can be found at the author's website here:
"Dialogue mapping has the same intention as facilitation: to help the group members hold an effective conversation on a complex topic. By “effective” we mean a conversation that both accomplished the objectives and built higher levels of shared understanding, respect, alignment, and transparency. But dialogue mapping uses two tools that are relatively new to the conference room. The first is to capture key elements of the conversation in a shared display. This could be whiteboards or flipcharts, but more often these days it's a computer projector. Shared display means that what is projected in the display is being crafted by the group actively. People's comments are somehow reflected in the display. We're not talking about PowerPoint here!! Sometimes referred to as interactive visual modeling, shared display requires that there be someone driving the computer who has the skills and intention of adding value to the group's interaction and creating group memory of the group's thinking and learning. The second aspect of dialogue mapping that is new and different is the use of a simple conversational grammar called IBIS, Issue Based Information System. IBIS represents the moves in a conversation as Questions, Ideas (possible answers to the Question), and Arguments (pros and cons to the ideas). The power of IBIS is its emphasis on questions. In an IBIS diagram new questions arise to clarify assumptions, challenge arguments, shift the context, and explore the deeper implications of ideas. Dialogue mapping requires that the mapper be so fluent in IBIS that they can translate everyday meeting-speak (e.g. “Why are we talking about this?”, “That's not the issue!”, etc) on the fly into IBIS and write or type it into the shared display for the group to see and validate. The pinnacle of fluency in IBIS is being able hear the hidden questions behind participants' comments. ... [Compendium is free software useful for doing IBIS diagrams.]"
On "Wicked Problems"
"A wicked problem is one for which each attempt to create a solution changes the understanding of the problem. Wicked problems cannot be solved in a traditional linear fashion, because the problem definition evolves as new possible solutions are considered and/or implemented. The term was originally coined by Horst Rittel. Wicked problems always occur in a social context -- the wickedness of the problem reflects the diversity among the stakeholders in the problem. Most projects in organizations -- and virtually all technology-related projects these days -- are about wicked problems. Indeed, it is the social complexity of these problems, not their technical complexity, that overwhelms most current problem solving and project management approaches."
And also by Jeff Conklin, as essentially the first chapter of his book above:
"Wicked Problems and Social Complexity"
"“Some problems are so complex that you have to be highly intelligent and well informed just to be undecided about them.” (Laurence J. Peter) ... If we step back and take a systemic view, we can see that the issue is not whose fault the mess is – the issue is our collective failure to recognize the recurring and inevitable dynamics of the mess. If we take a systemic view, we turn away from blame and away from easy technical fixes, and to look in the social domain – in building capacity to collaborate effectively on wicked problems. ... The process of Dialogue Mapping is a powerful approach for addressing the problem of fragmentation, as it allows a diverse group of people to generate coherence around wicked problems. This group coherence is a necessary step toward addressing fragmentation, yet it is neither a silver bullet nor a cureall. Given the complex nature of organizations, it is not sufficient for a single team or even multiple teams to achieve coherence; the organization as a whole needs to become a knowledge organization, and gain a kind of ‘literacy’ or ‘fluency’ in the language of coherence: distinctions, tools, methods, and practices for crafting shared understanding and shared commitment. Dialogue Mapping is a first step toward that kind of literacy."
Constructing Knowledge Art: An Experiential Perspective on Crafting Participatory Representations
by Al Selvin and Simon Buckingham Shum (who created the Compendium software)
"This book is about how people (we refer to them as practitioners) can help guide participants in creating representations of issues or ideas, such as collaborative diagrams, especially in the context of Participatory Design (PD). At its best, such representations can reach a very high level of expressiveness and usefulness, an ideal we refer to as Knowledge Art. Achieving that level requires effective engagement, often aided by facilitators or other practitioners. Most PD research focuses on tools and methods, or on participant experience. The next source of advantage is to better illuminate the role of practitioners -- the people working with participants, tools, and methods in service of a project's larger goals. Just like participants, practitioners experience challenges, interactions, and setbacks, and come up with creative ways to address them while maintaining their stance of service to participants and stakeholders. Our research interest is in understanding what moves and choices practitioners make that either help or hinder participants' engagement with representations. We present a theoretical framework that looks at these choices from the experiential perspectives of narrative, aesthetics, ethics, sensemaking and improvisation and apply it to five diverse case studies of actual practice."
This is a broader exploration of dialog mapping and similar participatory technologies from an advanced facilitator's perspective. Most people would probably want to read Jeff Conklin's "how to" book on Dialogue Mapping first, and then move onto this one once they are ready to grow further as a facilitator of group work.
Working with Stories in Your Community or Organization: Participatory Narrative Inquiry
by Cynthia F. Kurtz
"Working with Stories is a textbook that can help you use participatory narrative inquiry (PNI) in your community or organization. PNI methods can help you discover insights, catch emerging trends, make decisions, generate ideas, resolve conflicts, and connect people."
Conventional surveys of your customers or employees typically only give you a shallow understanding of what is going on. Engaging people to tell their personal experiences using your product or services or working for your organization can give you deeper actionable insights into how to improve your processes. These techniques can also help improve how people communicate with each other to form a supportive community.
The companion FOSS NarraFirma software leads people through the steps outlined in the 700 page Working with Stories textbook:
(Disclaimer: Paul Fernhout, author of this README, is married to Cynthia and co-wrote this software.)
The Happiness Track: How to Apply the Science of Happiness to Accelerate Your Success
"In The Happiness Track, Emma Seppala, the science director of the Center for Compassion and Altruism Research and Education at Stanford University and director of the Yale College Emotional Intelligence Project, explains that our inability to achieve sustainable fulfillment is tied to common but outdated notions about success. We are taught that getting ahead means doing everything that’s thrown at us (and then some) with razor-sharp focus and iron discipline; that success depends on our drive and talents; and that achievement cannot happen without stress. The Happiness Track demolishes these counter-productive theories. Drawing on the latest findings from the fields of cognitive psychology and neuroscience—research on happiness, resilience, willpower, compassion, positive stress, creativity, mindfulness—Seppala shows that finding happiness and fulfillment may, in fact, be the most productive thing we can do to thrive professionally. Filled with practical advice on how to apply these scientific findings to our daily lives, The Happiness Track is a life-changing guide to fast tracking our success and creating the anxiety-free life we want."
Carol Dweck's research on a growth mindset
"Mindset is a simple idea discovered by world-renowned Stanford University psychologist Carol Dweck in decades of research on achievement and success—a simple idea that makes all the difference. Teaching a growth mindset creates motivation and productivity in the worlds of business, education, and sports."
"Mindset: The New Psychology of Success"
"After decades of research, world-renowned Stanford University psychologist Carol S. Dweck, Ph.D., discovered a simple but groundbreaking idea: the power of mindset. In this brilliant book, she shows how success in school, work, sports, the arts, and almost every area of human endeavor can be dramatically influenced by how we think about our talents and abilities. People with a fixed mindset—those who believe that abilities are fixed—are less likely to flourish than those with a growth mindset—those who believe that abilities can be developed. Mindset reveals how great parents, teachers, managers, and athletes can put this idea to use to foster outstanding accomplishment."
Grit: The Power of Passion and Perseverance
by Angela Duckworth
"shows anyone striving to succeed -- be it parents, students, educators, athletes, or business people -- that the secret to outstanding achievement is not talent but a special blend of passion and persistence she calls “grit.”"
How to Have a Good Day: Harness the Power of Behavioral Science to Transform Your Working Life
by Caroline Webb
"In How to Have a Good Day, economist and former McKinsey partner Caroline Webb shows readers how to use recent findings from behavioral economics, psychology, and neuroscience to transform our approach to everyday working life."
Punished by Rewards: The Trouble with Gold Stars, Incentive Plans, A's, Praise, and Other Bribes
"Our basic strategy for raising children, teaching students, and managing workers can be summarized in six words: Do this and you’ll get that. We dangle goodies (from candy bars to sales commissions) in front of people in much the same way that we train the family pet. In this groundbreaking book, Alfie Kohn shows that while manipulating people with incentives seems to work in the short run, it is a strategy that ultimately fails and even does lasting harm. Our workplaces and classrooms will continue to decline, he argues, until we begin to question our reliance on a theory of motivation derived from laboratory animals. Drawing from hundreds of studies, Kohn demonstrates that people actually do inferior work when they are enticed with money, grades, or other incentives. Programs that use rewards to change people’s behavior are similarly ineffective over the long run. Promising goodies to children for good behavior can never produce anything more than temporary obedience. In fact, the more we use artificial inducements to motivate people, the more they lose interest in what we’re bribing them to do. Rewards turn play into work, and work into drudgery."
This Wikipedia page lists a variety of theories of motivation to compare and contrast with Kohn's thesis:
Drive: The surprising truth about what motivates us
People in knowledge-intensive organizations are motivated mainly in situations where they have autonomy in how they do their work, have opportunities learn and increase their mastery, and have a sense of shared purpose as part of a community.
While performance on physical labor tends to be increased by external rewards, mental labor involving creativity tends to suffer if done for external rewards.
To Sell is Human
by Dan Pink
It may be no suprise that about 10% of workers are directly working in "sales". But what may be suprising is that on average the rest of us spend about 40% of our work day trying to influence others to devote resources to some endeavor or other sort of change (roughly in a bimodal distribution of about 20% of time for most people and about 80% for super-sellers). The book explains why and how "non-sales selling" became an essential part of most people's work with recent changes to our economy -- driven by the spread of the internet and a rise in enterprenuership. The book also explains how to "sell" ideas in better ways.
The Stupidity Paradox: The Power and Pitfalls of Functional Stupidity at Work Paperback
by Mats Alvesson and André Spicer
Suggests that most organizations hire smart and creative people and then don't allow them to be smart or creative due to organizational imperatives.
A summary of the key points can be found in this essay by the author:
"Stupefied: How organisations enshrine collective stupidity and employees are rewarded for checking their brains at the office door"
Pros and cons of open plan office space
"58% of High-Performance Employees Say They Need More Quiet Work Spaces"
One insightful comment there by Maxo-Texas:
"Seriously, it's a complex subject because:
- Different kinds of projects require different environments.
- Different kinds of people require different environments.
- Different kinds of work require different environments.
- And unfortunately offices are associated with status so some people who don't care about quiet are going to require an office if others are getting offices."
The Difference: How the Power of Diversity Creates Better Groups, Firms, Schools, and Societies
"In this landmark book, Scott Page redefines the way we understand ourselves in relation to one another. The Difference is about how we think in groups--and how our collective wisdom exceeds the sum of its parts. Why can teams of people find better solutions than brilliant individuals working alone? And why are the best group decisions and predictions those that draw upon the very qualities that make each of us unique? The answers lie in diversity--not what we look like outside, but what we look like within, our distinct tools and abilities. The Difference reveals that progress and innovation may depend less on lone thinkers with enormous IQs than on diverse people working together and capitalizing on their individuality. Page shows how groups that display a range of perspectives outperform groups of like-minded experts."
On overcoming a natural disconnect between executives and high-performance employees
"Know the soul of the high-performance employee, then we can build the 10,000 things"
by William Belk
"Well-being and retention can only come through understanding. One of the issues to plague organizations of all sizes is a natural disconnect between executives and high-performance employees (HPEs). This disconnect affects culture, innovation and everything in-between."
Have Fun at Work
by W. L. Livingston
Much of the book is about what the individual engineer must do to succeed despite the organization. Advocates for "Skunk Works" for vital projects.
by Jeff Schmidt
"In this riveting book about the world of professional work, Jeff Schmidt demonstrates that the workplace is a battleground for the very identity of the individual, as is graduate school, where professionals are trained. He shows that professional work is inherently political, and that professionals are hired to subordinate their own vision and maintain strict "ideological discipline." The hidden root of much career dissatisfaction, argues Schmidt, is the professional's lack of control over the political component of his or her creative work. Many professionals set out to make a contribution to society and add meaning to their lives. Yet our system of professional education and employment abusively inculcates an acceptance of politically subordinate roles in which professionals typically do not make a significant difference, undermining the creative potential of individuals, organizations and even democracy. Schmidt details the battle one must fight to be an independent thinker and to pursue one's own social vision in today's corporate society. He shows how an honest reassessment of what it really means to be a professional employee can be remarkably liberating. After reading this brutally frank book, no one who works for a living will ever think the same way about his or her job."
by Michael Phillips (a founder of MasterCard)
"An inspirational guide to ethical business practice explains how to create and manage a small business that emphasizes openness, personal integrity, and community involvement as the keys to success."
While the book is mainly about small business, it can also apply to smaller units within bigger businesses.
Toward High-Performance Organizations: A Strategic Role for Groupware [and Bootstrapping]
by Douglas Engelbart (inventor of the mouse and creator of "The Mother of All Demos")
Engelbart suggests that healthy successful communities involve a co-evolution of tools, knowledge, culture, processes, and more -- and that we should be building learning organizations based on this principle.
A Group is Its Own Worst Enemy
by Clay Shirky
More and more companies revolve around both internal and external social software. Similar to one of Englebart's point, Shirky argues social software needs to be update-able to a group's emerging needs and that an online group and its software are deeply intertwined. Also, a key point is that hosting social software is more like having rental tenants (with rights) than having "users" for a desktop application. These ideas can help both in designing and selecting social software.
Meshworks, Hierarchies, and Interfaces
by Manuel De Landa
Much of the essay is about AI design and is probably not of interest to most people, but a key insightful paragraph at the end -- about the need for an appropriate balance of both meshwork and hierarchy in all systems -- applies generally to all organizations:
"To make things worse, the solution to this is not simply to begin adding meshwork components to the mix. Indeed, one must resist the temptation to make hierarchies into villains and meshworks into heroes, not only because, as I said, they are constantly turning into one another, but because in real life we find only mixtures and hybrids, and the properties of these cannot be established through theory alone but demand concrete experimentation. Certain standardizations, say, of electric outlet designs or of data-structures traveling through the Internet, may actually turn out to promote heterogenization at another level, in terms of the appliances that may be designed around the standard outlet, or of the services that a common data-structure may make possible. On the other hand, the mere presence of increased heterogeneity is no guarantee that a better state for society has been achieved. After all, the territory occupied by former Yugoslavia is more heterogeneous now than it was ten years ago, but the lack of uniformity at one level simply hides an increase of homogeneity at the level of the warring ethnic communities. But even if we managed to promote not only heterogeneity, but diversity articulated into a meshwork, that still would not be a perfect solution. After all, meshworks grow by drift and they may drift to places where we do not want to go. The goal-directedness of hierarchies is the kind of property that we may desire to keep at least for certain institutions. Hence, demonizing centralization and glorifying decentralization as the solution to all our problems would be wrong. An open and experimental attitude towards the question of different hybrids and mixtures is what the complexity of reality itself seems to call for. To paraphrase Deleuze and Guattari, never believe that a meshwork will suffice to save us."
Birth of the Chaordic Age
by Dee Hock (founder of Visa)
"In Birth of the Chaordic Age, Dee Hock argues that traditional organizational forms can no longer work because organizations have become too complex. Hock advocates a new organizational form that he calls "chaordic", or simultaneously chaotic and orderly. He credits the worldwide success of VISA with its chaordic structure -- it is owned by its member banks which both compete with each other for customers and must cooperate by honoring one another's transactions across borders and currencies. The book shows how these same chaordic concepts are now being put into practice in a broad range of business, social, community, and government organizations."
Hock's book can be seen as taking De Landa's general point above and applying it more formally to organizational design.
Functional Medicine as an analogy for organizational health and also a way to keep employees healthy
Example: Dr. Mark Hyman's work
"Functional Medicine is the future of conventional medicine -- available now. It seeks to identify and address the root causes of disease, and views the body as one integrated system, not a collection of independent organs divided up by medical specialties. It treats the whole system, not just the symptoms. Functional medicine addresses the underlying causes of disease, using a systems-oriented approach and engaging both patient and practitioner in a therapeutic partnership."
There are many other advocates for functional medicine like Dr. Andrew Weil, Dr. Joel Fuhrman, Dr. Dean Ornish, and so on. Dr. Hyman is just one of the most famous from having helped reverse President Bill Clinton's heart disease.
Although, for contrast, others disagree with Dr. Hyman and Functional Medicine:
See also what other healthy companies are doing:
"It’s common sense that happy people make for more productive and innovative employees. Yet 42 percent of workers have left a job due to a stressful environment, and another 35 percent have considered changing jobs due to stress, according to a 2014 Monster.com survey of 6,700 people. ... These 44 businesses go above and beyond, proving that a workplace that considers its employees' health, happiness, and work/life balance vital to its own success isn’t so far-fetched."
A book by a CEO of Whole Foods, a company created to help people be healthier through good nutrition:
The Whole Foods Diet: The Lifesaving Plan for Health and Longevity by John Mackey, Alona Pulde, Matthew Lederman
THE WHOLE FOODS DIET simplifies the huge body of science, research, and advice that is available today and reveals the undeniable consensus: a whole foods, plant-based diet is the optimum diet for health and longevity.
And see what individuals like Joe Cross are doing -- as well as others joining a movement arising from a funny movie he made on his own health transformation by improving his diet and exercising in the sunshine:
And also Blue Zones on "Reverse Engineering Longevity":
To answer the question [of why some people live longer than others], we teamed up with National Geographic to find the world’s longest-lived people and study them. We knew most of the answers lied within their lifestyle and environment. (The Danish Twin Study established that only about 20% of how long the average person lives is determined by genes.) Then we worked with a team of demographers to find pockets of people around the world with the highest life expectancy, or with the highest proportions of people who reach age 100. ... We then assembled a team of medical researchers, anthropologists, demographers, and epidemiologists to search for evidence-based common denominators among all places. We found nine:
- Move Naturally ...
- Purpose ...
- Down Shift ...
- 80% Rule ...
- Plant Slant ...
- Wine @ 5 ...
- Belong ...
- Loved Ones First ...
- Right Tribe ...
The Upward Spiral: Using Neuroscience to Reverse the Course of Depression, One Small Change at a Time
"Depression can feel like a downward spiral, pulling you into a vortex of sadness, fatigue, and apathy. In The Upward Spiral, neuroscientist Alex Korb demystifies the intricate brain processes that cause depression and offers a practical and effective approach to getting better. Based on the latest research in neuroscience, this book provides dozens of straightforward tips you can do every day to rewire your brain and create an upward spiral towards a happier, healthier life."
See also Phil Hickey:
"Depression Is Not An Illness: It is an Adaptive Mechanism" http://behaviorismandmentalhealth.com/2009/07/28/depression/
Depression or despondency is not as acute a sensation as pain. It is more generalized and it signals – not imminent tissue damage – but problems of a more general nature. In order to feel good, the following eight factors must be present in our lives.
– good nutrition
– fresh air
– sunshine (in moderation)
– physical activity
– purposeful activity with regular experiences of success
– good relationships
– adequate and regular sleep
– ability to avoid destructive social entanglements, while remaining receptive to positive encounters
When any of these factors are missing, or are present to only a slight degree, we begin to feel despondent or depressed. When many of these factors are missing to a large degree, we become very depressed.
Vitamin D deficiency in particular for lack of sunlight is an occupational hazard of modern indoor work (made worse by an increasingly indoor lifestyle in front of screens). See for example: http://www.grassrootshealth.net/
Why We Sleep: Unlocking the Power of Sleep and Dreams
by Matthew Walker
Adequate sleep (eight hours) is essential for learning, creativity, health, ethics, and safety in the workplace.
"Within the brain, sleep enriches a diversity of functions, including our ability to learn, memorize, and make logical decisions. It recalibrates our emotions, restocks our immune system, fine-tunes our metabolism, and regulates our appetite. Dreaming creates a virtual reality space in which the brain melds past and present knowledge, inspiring creativity. In this “compelling and utterly convincing” (The Sunday Times) book, preeminent neuroscientist and sleep expert Matthew Walker provides a revolutionary exploration of sleep, examining how it affects every aspect of our physical and mental well-being. Charting the most cutting-edge scientific breakthroughs, and marshalling his decades of research and clinical practice, Walker explains how we can harness sleep to improve learning, mood and energy levels, regulate hormones, prevent cancer, Alzheimer’s and diabetes, slow the effects of aging, and increase longevity. He also provides actionable steps towards getting a better night’s sleep every night."
Software Development Specific:
Mythical Man Month: Essays on Software Engineering
by Frederick P. Brooks Jr.
"Few books on software project management have been as influential and timeless as The Mythical Man-Month. With a blend of software engineering facts and thought-provoking opinions, Fred Brooks offers insight for anyone managing complex projects. These essays draw from his experience as project manager for the IBM System/360 computer family and then for OS/360, its massive software system. Now, 20 years after the initial publication of his book, Brooks has revisited his original ideas and added new thoughts and advice, both for readers already familiar with his work and for readers discovering it for the first time."
Khan Academy Engineering Management Reading List
"Must reads for all KA engineering managers. And anyone interested in eng management."
Khan Academy especially seems to like Michael Lopp's Rands blog "Rands in Repose" and his book (the next item).
Managing Humans: Biting and Humorous Tales of a Software Engineering Manager
"Read hilarious stories with serious lessons that Michael Lopp extracts from his varied and sometimes bizarre experiences as a manager at Apple, Pinterest, Palantir, Netscape, Symantec, Slack, and Borland. Many of the stories first appeared in primitive form in Lopp’s perennially popular blog, Rands in Repose. The Third Edition of Managing Humans contains a whole new season of episodes from the ongoing saga of Lopp's adventures in Silicon Valley, together with classic episodes remastered for high fidelity and freshness. Whether you're an aspiring manager, a current manager, or just wondering what the heck a manager does all day, there is a story in this book that will speak to you―and help you survive and prosper amid the general craziness of dysfunctional bright people caught up in the chase of riches and power. Scattered in repose among these manic misfits are managers, an even stranger breed of people who, through a mystical organizational ritual, have been given power over the futures and the bank accounts of many others. Lopp's straight-from-the-hip style is unlike that of any other writer on management and leadership. He pulls no punches and tells stories he probably shouldn't. But they are magically instructive and yield Lopp’s trenchant insights on leadership that cut to the heart of the matter―whether it's dealing with your boss, handling a slacker, hiring top guns, or seeing a knotty project through to completion. Writing code is easy. Managing humans is not. You need a book to help you do it, and this is it."
Why "Software is Hard"
"What makes the virtual world so much more brittle and unpredictable than the physical world? Why can't we build software the way we build bridges? ... The difference is that the overruns on a physical construction project are bounded. You never get to the point where you have to hammer in a nail and discover that the nail will take an estimated six months of research and development, with a high level of uncertainty. But software is fractal in complexity. If you're doing top-down design, you produce a specification that stops at some level of granularity. And you always risk discovering, come implementation time, that the module or class that was the lowest level of your specification hides untold worlds of complexity that will take as much development effort as you'd budgeted for the rest of the project combined. The only way to avoid that is to have your design go all the way down to specifying individual lines of code, in which case you aren't designing at all, you're just programming. ... But the nature of software is that the problems are always different. You never have to solve the exact problem that someone's solved before, because if software already existed that solved your need, you wouldn't have to write it. Writing software is expensive. Copying software is cheap. Scott Rosenberg coins this as Rosenberg's Law: Software is easy to make, except when you want it to do something new. The corollary is, The only software that's worth making is software that does something new."
To (over) generalize on why software estimates so often go wrong: when you build a bridge, the model you use to estimate from is 1% of the effort. When you build software, building the "model" itself is 99% of the effort.
Every layer of abstraction costs something
A famous aphorism of David Wheeler is "All problems in computer science can be solved by another level of indirection". This is often deliberately misquoted with "abstraction" substituted for "indirection". It is also sometimes misattributed to Butler Lampson. Kevlin Henney's corollary to this is, "...except for the problem of too many layers of indirection."
This is why some duplication in software code is often a benefit because consolidating code as DRY (Do Not Repeat Yourself) principle suggests can require more though to understand and can constrain future development in unexpected ways. One way to deal with lots of almost redundant code is to have better tools for managing it.
Related (but the threshold may depend on the context):
Decreasing Cognitive Load
by Leo Horie (author of the Mithril.js vdom library)
From the conclusion there:
We can make complex systems less complex by creating consistent patterns, and documenting these patterns effectively.
Code grows and rots, so it's important to plan ahead.
I once read a theory that developers hit walls of complexity every time they increase the size of their codebases by an order of magnitude (i.e. the idea is that a junior developer might find it hard to expand their simple procedural programs past 3000 lines, and that developers hit another wall of complexity at 30,000 lines, and again at 300,000 and so on)
My own theory is that the these walls appear when the volume of complexity of a codebase exceeds the volume of complexity solved by the libraries it uses. For example, jQuery is undoubtedly useful when dealing with browser quirks, but once an application grows over a few thousand lines of code, unstructured jQuery code simply becomes too difficult to maintain, and you start needing the discipline of a framework to organize code. But when you're at tens of thousands of lines of code, you start to run out of entity types to CRUD, and your application growth starts to build on top of existing concepts. This is when you need the mental shift from being a library consumer to being a reusable component author, but with a focus on the interacting parts within the application (as opposed to generic one-glove-fit-all open source libraries).
The Virtues of Negative Code
"McIlroy is attributed the quote "The real hero of programming is the one who writes negative code," where the meaning of negative code is taken to be similar to the famous Apple developer team anecdote (i.e., when a change in a program source makes the amount of lines of code decrease ('negative' code), while its overall quality, readability or speed improves)."
Referenced there is this part of QuickDraw's history on "How do you measure programmer productivity?"
"When the Lisa team was pushing to finalize their software in 1982, project managers started requiring programmers to submit weekly forms reporting on the number of lines of code they had written. Bill Atkinson thought that was silly. For the week in which he had rewritten QuickDraw’s region calculation routines to be six times faster and 2000 lines shorter, he put “-2000″ on the form. After a few more weeks the managers stopped asking him to fill out the form, and he gladly complied."
That QuickDraw story is another example of why organizations should be careful about what they measure because they likely will get more of it. Most employees will try to deliver what is formally requested -- with some rare exceptions like Bill Atkinson who will deliver what is informally needed even despite the formal process and related incentives.
Premature optimization is the root of all evil
"In Donald Knuth's paper "Structured Programming With Goto Statements", he wrote: "Programmers waste enormous amounts of time thinking about, or worrying about, the speed of noncritical parts of their programs, and these attempts at efficiency actually have a strong negative impact when debugging and maintenance are considered. We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%.""
A Conceptual Framework for System Fault Tolerance
by Walter L. Heimerdinger and Charles B. Weinstock
"A major problem in transitioning fault tolerance practices to the practitioner community is a lack of a common view of what fault tolerance is, and how it can help in the design of reliable computer systems. This document takes a step towards making fault tolerance more understandable by proposing a conceptual framework. The framework provides a consistent vocabulary for fault tolerance concepts, discusses how systems fail, describes commonly used mechanisms for making systems fault tolerant, and provides some rules for developing fault tolerant systems."
Systems development life cycle
"The systems development life cycle (SDLC), also referred to as the application development life-cycle, is a term used in systems engineering, information systems and software engineering to describe a process for planning, creating, testing, and deploying an information system. The systems development life-cycle concept applies to a range of hardware and software configurations, as a system can be composed of hardware only, software only, or a combination of both. ... Computer systems are complex and often (especially with the recent rise of service-oriented architecture) link multiple traditional systems potentially supplied by different software vendors. To manage this level of complexity, a number of SDLC models or methodologies have been created, such as "waterfall"; "spiral"; "Agile software development"; "rapid prototyping"; "incremental"; and "synchronize and stabilize". SDLC can be described along a spectrum of agile to iterative to sequential. Agile methodologies, such as XP and Scrum, focus on lightweight processes which allow for rapid changes (without necessarily following the pattern of SDLC approach) along the development cycle. Iterative methodologies, such as Rational Unified Process and dynamic systems development method, focus on limited project scope and expanding or improving products by multiple iterations. Sequential or big-design-up-front (BDUF) models, such as waterfall, focus on complete and correct planning to guide large projects and risks to successful and predictable results. Other models, such as anamorphic development, tend to focus on a form of development that is guided by project scope and adaptive iterations of feature development."
Key point: there are a lot of possible ways of organizing software development activities depending on circumstances.
The Computer Revolution Hasn't Happened Yet, OOPSLA 1997
"Here's one we haven't faced up to much yet, that, now we have to construct this stuff and soon we'll be required to grow it. It's very easy, for instance, to grow a baby six inches. They do it about ten times in their life and you never have to take it down for maintenance. But if you try and grow a 747, you are faced with an unbelievable problem, because it's in this simple-minded mechanical world, in which the only object has been to make the artifact in the first place. Not to fix it. Not to change it. Not to let it live for a hundred years. So let me ask a question. I won't take names. But how many people here still use a language that essentially forces you -- and the development system forces you to develop outside of the language; compile and reload, and go, even if it's fast, like Virtual Café. How many here still do that? Let's just see. Come on. Admit it! We can have a Texas tent beating later. That cannot possibly be other than a dead end for building complex systems, where much of the building of complex systems is in part going to go into trying to understand what the possibilities for interoperability is with things that already exist."
Agile is Dead (Long Live Agility)
by Dave Thomas (one of the authors of the Agile Manifesto)
He outlines some basic problems with the Agile industry (including selling fear). He says we need to distinguish between the implementation (Agile/Scrum/Lean/Kanban/etc) and the specification (Agility). He says "No rules are universal (except for this one)" and that "all rules are contextual". He espouses holding close to the value of "Agility" involving figuring out where you are, taking a small step towards your goal, re-evaluating and adjusting your understanding based on what you learned, and then iterating. He suggests choosing between alternatives delivering similar short-term value based on which keeps more options open to make future change easier -- outlining Dave's Rule of Design: "A good design is easier to change than a bad design." He calls for courage at the individual, team, and company levels to know you are going to make mistakes in order to find out what needs to be done -- and to work hard to make sure those mistakes small and correctable.
A text version:
"The word “agile” has been subverted to the point where it is effectively meaningless, and what passes for an agile community seems to be largely an arena for consultants and vendors to hawk services and products. So I think it is time to retire the word “Agile.”"
On management approaches without deadlines
"Why Deadlines Need to Drop Dead"
by Eric Elliott
"Deadlines are incredibly destructive to team productivity and morale. The #1 challenge software developers face is unrealistic expectations. Can we motivate & push without arbitrary deadlines? You can lead teams without deadlines. You can still deliver code quickly. You can still ship in time for Christmas. But you’ll be dealing with a different management framework. One that puts the deadline pressure on product managers rather than developers. ... The point here is that instead of trying to build out the CEO’s ideal vision of the product you’re bringing to market, you build out a beautiful stepping stone on the way towards that vision. In place of arbitrary deadlines, you deliver consistent, steady progress."
Advocates for marketing creating each hype cycle for what is already finished and tested.
Why do some developers at strong companies like Google consider Agile development to be nonsense?
Where Agile works, where it does not...
Key point: "Agile is nothing more than focusing on reducing the cost of uncertainty and change. To understand if that's a worthwhile focus, you need to know both your product and the market. Internally a company will likely have many different departments and some should definitely use Agile (R&D is a great example) and some should definitely not (Accounting). As products/services mature, they should transition from Agile to Lean to more formal methodologies such as TQM, PRINCE2, ISO9000, and so on (interesting that there's no catch-all term for the latter)."
Why SCRUM Backlogs Lead to Bad Product Decisions
"At Blossom we’ve completely replaced our product backlog with a free form Google Docs file. It reads more like a marketing document than a traditional backlog. We call it our product strategy roadmap. In this file we’ve defined a few simple things on top... What are our customers hiring our product for? What kind of super powers can we give our customers? Then the body of the document consists of high level (strategic) marketing-esque announcements of (future) product improvements. Similar to what you would read in a blog post when a company like Facebook, Apple or Twitter releases a new major feature. .... Then every time space on our product board frees up we take a look at our roadmap document and pick something from it to pull into our process. This might be a full ‘announcement’ at once or more often just a subset of an announcement that we then work on step by step. The main advantage here is that we have a strategic document that’s clear and concise but at the same time very easy to change. It is very high level and mainly there to provide context for what a great execution of the strategy might be. We don’t start with specifying implementation details until we’ve actually started to work on an item. This saves a lot of time and resources that we used to spend on doing backlog estimates, re-prioritizations and groomings in the past."
Frankenbuilds; if Agile is so good, why are our Products so bad?
by Gabrielle Benefield
Key point: "Focus on outcomes, not outputs" Suggests "Never ask your users what they want. Never ask your developers what they think their users want. ... Get out of your chair and find out what they need." Once you understand your users and their needs based on real data, then you can evaluate what features will have the greatest benefits to them based on the severity of the problem or the size of the opportunity. Points out how the typical way of prioritizing product backlogs in Agile projects does not reflect this. Suggests superflous features can make it harder for users to succeed in using a product and may also require more ongoing maintenance costs. Brings up the issue of what to measure and when.
One comment there by Mark Proffitt: "Gabrielle is correct. Delivering customer value is the #1 item on the Agile Manifesto. I was one of the contributors to the creation of Agile when I was at Apple in 1992. The whole reason for the Agile concept was requirements were incorrect, or incomplete, or changed after we started developing. We could already build anything nearly perfectly. [emphasis added] More quickly delivering working code was a side effect of Agile iterations, it was not the purpose. What kept happening is we would build it then customers would reject it because it didn't do what they wanted so we had to start all over again. The only reason for iterations is to check to make sure you are building the right thing, what customers actually want. Additionally, at the time we realized that software would not sit on a single computer, it would be spread over many clients and servers. You could not build software as a single monolithic item. It had to be flexible and allow pieces to change and be added later. Building that type of software needed totally different methods from the Waterfall method."
Flaws In Scrum And Agile
by Giles Bowkett
"When the Agile Manifesto was written, waterfall development was king. Agile deposed waterfall from its status as the dominant software development paradigm. Waterfall revolved around writing tedious documents, and face-to-face conversation was indeed superior to that. In 2001, Subversion was still new, Git did not exist, Skype didn't exist either, and the closest contemporary equivalent to GitHub was SourceForge. "Face-to-face conversation," in the Agile Manifesto, was "the most efficient and effective method of conveying information" because the other methods were terrible. But when you're reviewing a pull request, a GitHub diff usually beats face-to-face conversation. And this is not a fluke. Remote work requires a ton of writing, and that's one of the best things about it. How many times have you sat down to email somebody a question, and found that in the process of writing the email, the answer became obvious to you? This step is built in to the workflow of every remote team. Likewise, have you ever had to tell a co-worker the same thing twice? Technical work involves lots of tradeoffs, and people don't always remember the intricacies of every debate. But you can refer to a conversation on GitHub three years later and review every detail with perfect clarity. In a distributed company, the written word becomes much more influential than the spoken word."
Why “Agile” and especially Scrum are terrible
A bit over-the-top and certainly one-sided -- but makes some good points about Agile/Scrum in practice as opposed to theory. The blog post is notable for having over 1000 comments with agreements and disagreements.
Some other rebuttals to Michael O. Church's essay:
Continuous Delivery vs. Traditional Agile
by Kief Morris
"In working with development teams at organizations which are adopting Continuous Delivery, I have found there can be friction over practices that many developers have come to consider as the right way for Agile teams to work. I believe the root of conflicts between what I’ve come to think of as traditional agile and CD is the approach to making software “ready for release”."
GitFlow Considered Harmful
by Adam Ruka (full-stack web developer working for Amazon)
And a comment he references by dasil003 in favor of rebasing:
Key points are that:
- GitFlow is needlessly complex
- Tags can be used to greater benefit
- Merges rather than rebases create spaghetti history that is hard to understand when determign where a problem was introduced
- It is bad Q/A policy to create untested new merge commits in master rather than always fast-forward master to (presumably) tested code in a branch (explained in detail in a comment by Marco Ciaschini in the follow up)
The Pragmatic Programmer: From Journeyman to Master
by Andrew Hunt and David Thomas
Some key ideas are outlined in this interview with the authors:
One of them is "Software Development as Gardening":
Bill Venners: In your book, The Pragmatic Programmer, you say, "Rather than construction, programming is more like gardening." I really like your gardening metaphor for software development. Can you elaborate on it?
Andy Hunt: There is a persistent notion in a lot of literature that software development should be like engineering. First, an architect draws up some great plans. Then you get a flood of warm bodies to come in and fill the chairs, bang out all the code, and you're done. A lot of people still feel that way; I saw an interview in the last six months of a big outsourcing house in India where this was how they felt. They paint a picture of constructing software like buildings. The high talent architects do the design. The coders do the constructing. The tenants move in, and everyone lives happily ever after. We don't think that's very realistic. It doesn't work that way with software.
We paint a different picture. Instead of that very neat and orderly procession, which doesn't happen even in the real world with buildings, software is much more like gardening. You do plan. You plan that you're going to make a plot this big. You're going to prepare the soil. You bring in a landscape person who says to put the big plants in the back and short ones in the front. You've got a great plan, a whole design.
But when you plant the bulbs and the seeds, what happens? The garden doesn't quite come up the way you drew the picture. This plant gets a lot bigger than you thought it would. You've got to prune it. You've got to split it. You've got to move it around the garden. This big plant in the back died. You've got to dig it up and throw it into the compost pile. These colors ended up not looking like they did on the package. They don't look good next to each other. You've got to transplant this one over to the other side of the garden.
Dave Thomas: Also, with a garden, there's a constant assumption of maintenance. Everybody says, I want a low-maintenance garden, but the reality is a garden is something that you're always interacting with to improve or even just keep the same. Although I know there's building maintenance, you typically don't change the shape of a building. It just sits there. We want people to view software as being far more organic, far more malleable, and something that you have to be prepared to interact with to improve all the time.
Another idea from there is on moving beyond "Stratification of Developer Jobs":
What we should be doing is creating an environment in which people get to use all their skills. Maybe as time goes on, people move through the skill set. So today, you're 80% coder, 20% designer. On the next project, you're 70% coder and 30% designer. You're moving up a little bit each time, rather than suddenly discovering a letter on your desk that says, "Today you are a designer."