Agencies should require government contractors to provide complete (and real-time) version control and collaborative history

[benbalter]

Rather than requiring a single snapshot in time of the software project at the conclusion of a procurement, as part of the final deliverable agencies should require software contractors to provide the project's entire version control and collaborative history, ideally in real time.

How we got here

The most commonly used arrows in the federal IT procurement quiver are CTRL-C and CTRL-V. Rightly not wanting to reinvent the wheel with each contract, the first step of most procurements is to find an analogous prior award and swap out the inapplicable terms. Need a mobile app? Take the web design contract you awarded a few months back and swap out Objective-C for PHP and uptime for cross-platform. Boom. Contract. No need to rewrite potentially shared clauses like the contractor needing to supply their own laptops or the need to establish a test environment.

The problem comes when you rewind this practice across thirty years of technology change. Where a contract once called for 5 1/4" floppies in triplicate (because what if it gets demagnetized — these things happen), it was replaced with 3 1/2" floppies, then by CD-ROMs, and today that role is filled by zip files and email. And just as before, whereas when version 1.1 came out, you'd stick the 1.0 floppies on the shelf and accept the new floppies labeled 1.1, today, the agencies receives subsequent iterations of its software via incrementally named zip files — agency-website-v1-1.zip. Seriously, this is how we build the world's most technologically advanced nation.

Deliverables outside the beltway

Over those same thirty years, while contracting officers swapped out mediums word-for-word in deliverable clause of government contracts, the software industry fundamental reimagined the way it developed software, a change that isn't necessarily possible through a piecemeal adaptation of prior clauses.

Today, you'd be hard pressed to find a startup, small business, or respectable corporation that doesn't version day-to-day changes to their software projects via a version control systems, and likewise expect their external contractors to do the same. And I suspect government contractors, for the most part, have also adopted these industry-standard best practices, as I'd be shocked if large beltway firms would be able to deliver the scale of software they deliver without some form of tracking who made what change when along the way. It'd be tantamount to a construction contractor ignoring the existence of power tools.

When government only gets half the code

Despite this shift, when the government asks for code, it still often asks for emailed zip files. Depending on the contract terms, the contractor may not get paid if they deliver the code in any other form. So government contractors take the software, maintained in a version control system, rich with context and pedigree, and simply gives the government a neutered snapshot of its current state, a tiny fraction of the work developed under the contract.

The value of version control extends beyond being able to CTRL-Z a given change. Modern software development tools let you drill down to a code's history, line-by-line, to see not only the code as it exists today, but how and why it got to be the way it is. It's institutional knowledge, stored and presented in its immediately useful form. It's who to ask when there's a bug, where to look for documentation, and what other parts of the codebase that particular line touches. It's context, and without it, you're setting yourself up for failure and duplicated efforts.

Version control as a hedge against vendor lock in

It used to be that vendor lock in meant a particular format or platform. In many ways the procurement process prefers large, generalist contracting firms that supply everything from tanks and aircraft carriers to websites and mobile apps because the government knows they're going to be around in a decade (and thus can support the government's purchase).

But with open source, when done right, that threat's a moot point. Ruby is Ruby. WordPress is WordPress. Given shared frameworks, shared design patterns, and shared practices, if properly documented, swapping contractor "A" for contractor "B" becomes trivial because both are working from the same shared playbook. The secret sauce is now in their approach to difficult challenges, not in the code itself or the additional context they withhold.

Forcing contractors to show their work

Forcing a government contractor to show their work not only gives the agency insight into the development process, which allows it to gain a better understanding of the code's strengths and weaknesses (think Carfax for code), but also ensures the contractor doesn't wrongfully hold a monopoly on the associated organizational knowledge.

Version control forces knowledge transfer in a way no human-curated documentation can ever provide, and once in the agency's hands can subsequently be shared with other contractors and open source developers alike, providing a broader picture, and ensuring the government, contractors, and public are all aligned in making the code the best it can be. After all, our code deserves it.

[mgifford]

Nice! Thanks for highlighting these issues. Hadn't even thought that contractors would do this, but of course....

[jhourcle]

I've seen quite a few version control systems where authentication information (logins & passwords) were committed. With a single release, it's possible to go through and scrub this information. Releasing the whole version control system is such a huge headache that you're basically giving people an easy out on why not to release the code, as it'd be a security risk.

I work for an agency where something like that would likely qualify as a 'security incident' ... which can lead to men w/ guns visiting and seizing servers.

(insert disclaimer here about these being personal comments, and not that the of the agency I work for, blah blah blah).

[konklone]

For what it's worth, the policy does not cover code already developed:

This policy does not require that existing custom-developed source code created by third party developers or vendors for the Federal Government be retroactively made available for Government-wide reuse or as OSS;

And it recommends that agencies develop code in the open from the beginning, which is one helpful way to lower the likelihood of passwords ending up in the repository (or for them to at least be caught early, before it's in production).

[jhourcle]

@konklone:

Define 'already developed'. Does it mean code that has already been completed, or code that has already been started?

Because this could also get us into granularity issues -- if I'm only looking at the project as a whole, well, that won't be completed for years even if we might have a few updates before then. (the updates were just part of the development process ... a 'release candidate' or something like that ... but we don't consider it to be 'complete'. We can be like Google Mail, and stay in beta for 10+ years).

If you consider additions & patches on top of stuff that's already been released to be part of that one system (that's not covered), then you don't have the burden of dealing with the quagmire that I already mentioned in #101 , but you exempt that project from compliance in perpetuity.

(standard disclaimer, blah blah, not my agency's comments, etc.)

[ctubbsii]

However one defines "existing...code", it'd be nice if the policy encouraged existing code to follow that route (when appropriate and when resources allow), even if it's not required.

[konklone]

Define 'already developed'. Does it mean code that has already been completed, or code that has already been started?

I suppose that's up to OMB to define.