Browse files


  • Loading branch information...
tlevine committed Aug 21, 2013
1 parent f6a7f95 commit 7cd5aba98d1c1fb43b2c68e62b6f24abd0f33f10
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@@ -0,0 +1,264 @@
+<!DOCTYPE html>
+<!--[if lt IE 7]> <html class="no-js lt-ie9 lt-ie8 lt-ie7"> <![endif]-->
+<!--[if IE 7]> <html class="no-js lt-ie9 lt-ie8"> <![endif]-->
+<!--[if IE 8]> <html class="no-js lt-ie9"> <![endif]-->
+<!--[if gt IE 8]><!-->
+<html class='no-js'>
+ <!--<![endif]-->
+ <head>
+ <meta charset='utf-8'>
+ <title>Open by default</title>
+ <meta content='How can software encourage that its data be open by default?' name='description'>
+ <meta content='Thomas Levine' name='author'>
+ <link href='http://domain/humans.txt' rel='author' type='text/plain'>
+ <meta content='nanoc 3.6.4' name='generator'>
+ <meta content='width=device-width' name='viewport'>
+ <meta content='summary' name='twitter:card'>
+ <meta content='@thomaslevine' name='twitter:site'>
+ <meta content='The first of Sunlight Foundation&#x0027;s 32 Open Data Policy Guidelines is to "Set The Default To Open".' name='twitter:title'>
+ <meta content='This guideline is not just for policy; it applies to software too!' name='twitter:description'>
+ <meta content='@thomaslevine' name='twitter:creator'>
+ <meta content='!/open-by-default/screenshot.png' name='twitter:image:src'>
+ <meta content='' name='twitter:domain'>
+ <meta content='' name='twitter:app:name:iphone'>
+ <meta content='' name='twitter:app:name:ipad'>
+ <meta content='' name='twitter:app:name:googleplay'>
+ <meta content='' name='twitter:app:url:iphone'>
+ <meta content='' name='twitter:app:url:ipad'>
+ <meta content='' name='twitter:app:url:googleplay'>
+ <meta content='' name='twitter:app:id:iphone'>
+ <meta content='' name='twitter:app:id:ipad'>
+ <meta content='' name='twitter:app:id:googleplay'>
+ <meta content='!/open-by-default/' property='og:url'>
+ <meta content='' property='og:site_name'>
+ <meta content='How can software encourage that its data be open by default?' property='og:description'>
+ <meta content='Open by default' property='og:title'>
+ <meta content='!/open-by-default/screenshot.png' property='og:image'>
+ <link href='/favicon.ico' rel='icon' type='image/x-icon'>
+ <link href='/!/feed.xml' rel='alternate' title='Thomas Levine' type='application/atom+xml'>
+ <link href=',400,700' rel='stylesheet' type='text/css'>
+ <link href='/css/style-cb653401acb.css' rel='stylesheet'>
+ <script src='' type='text/javascript'></script>
+ <script src='/js/modernizr-cb42306a279.js'></script>
+ </head>
+ <body>
+ <!--[if lt IE 7 ]>
+ <p class='chromeframe'>
+ You are using an <strong>outdated</strong> browser.
+ Please <a href="">upgrade your browser</a> or
+ <a href="">activate Google Chrome Frame</a>
+ to improve your experience.
+ </p>
+ <![endif]-->
+ <div id='wrapper'>
+ <div id='container'>
+ <nav>
+ <ul class='nobullet'>
+ <li class='link'>
+ <a href='/'>
+ <div>~</div>
+ </a>
+ </li>
+ <li class='link'>
+ <a href='/!/'>
+ <div>!</div>
+ </a>
+ </li>
+ <li class='link'>
+ <a href='/!/about/'>
+ <div>?</div>
+ </a>
+ </li>
+ </ul>
+ </nav>
+ <header class='title-card'>
+ <hgroup>
+ <h1>
+ Open by default
+ </h1>
+ <p>
+ </p>
+ </hgroup>
+ <div class='date'>
+ </div>
+ </header>
+ <div id='article-wrapper'>
+ <article>
+ <p>The first of Sunlight Foundation’s 32
+ <a href="">Open Data Policy Guidelines</a>
+ is to “Set The Default To Open”.</p>
+ <blockquote>
+ <p>Most public records systems, including the Freedom of Information Act itself, are systems of reactive disclosure – meaning that a question has to be asked before an answer given; public information requested, before it is disclosed.</p>
+ <p>Proactive disclosure is the opposite. Proactive disclosure is the release of public information – online and in open formats (see Provisions 8 and 9) – before it is asked for. This is no simple task, but, in a way, it’s what all “open data” is aiming to accomplish. Setting the default to open means that the government and parties acting on its behalf will make public information available proactively and that they’ll put that information within reach of the public (online), with low to no barriers for its reuse and consumption. Open formats may help us maximize on the value we can extract from certain kinds of public data today, but to ensure that data publishing is sustained and, in fact, made easier over time, we need to reset the default for how data is released and disclosed.</p>
+ <p>Setting the default to open is about living up to the potential of our information, about looking at comprehensive information management, and making determinations that fall in the public interest. It’s about purely practical government improvements, too, and taking steps that not only keep government systems up to date, but ensure that we have the foresight to survive changes in technology that we can’t predict.</p>
+ <p>Usually, for information to be defined as public, important restrictions have already been applied. Therefore, policy language can be used to outline that “all public data and information must be considered open and accessible.” Whether listed as part of a statement of intent (as Austin, Texas does; a concept explored more in Provision 21), as direction to a new oversight authority (Provision 22), or as the underlying aim of new data guidance (Provision 20), openness by default is a critical tool in crafting open data policies that are both ambitious and sustainable.</p>
+ </blockquote>
+ <p>After discovering something on Socrata data portals, I remarked that
+ software can encourage this practice of making data open by default.</p>
+ <h2 id="types-of-visualizations-on-socrata-portals">Types of visualizations on Socrata portals</h2>
+ <p>I previously <a href="/!/socrata-summary">downloaded</a> metadata about all of
+ the datasets on all of the Socrata portals, and I continue to find
+ interesting things in these data. Let’s look at the different types
+ of visualizations (“<a href="/!/socrata-genealogies#term-view">views</a>”) on the portals.</p>
+ <p><img src="figure/not_boring.png" alt="" class="wide" /></p>
+ <p>(I excluded tables and external links from the above plot.)</p>
+ <p>I was somewhat surprised to see forms and calendars in the portals.
+ I’ve <a href="/!/open-calendars">previously</a> written about why I think Socrata calendars are cool,
+ so now I’m just going to talk about forms.</p>
+ <h3 id="popularity-of-forms">Popularity of forms</h3>
+ <p>Much of the goal of these portals is to open up existing government data, but
+ <a href="">forms</a> provide a way for citizens to create data.
+ lets you enter data. A bunch of people have implemented them, but none seems to get accessed much.</p>
+ <p><img src="figure/form_use_3.png" alt="Form use by portal" class="wide" /></p>
+ <p>I’m gonna remove to make that easier to read.</p>
+ <p><img src="figure/form_use_4.png" alt="Form use by portal, excluding" class="wide" /></p>
+ <h3 id="cool-forms">Cool Forms</h3>
+ <p>I hadn’t seen <a href=""></a> before.
+ It belongs to the <a href="">National Oceanic and Atmospheric Administration Fisheries Service</a>,
+ which apparently used <a href="">a Socrata form</a> to power a
+ <a href="">policy comments website</a>.</p>
+ <p>New York made a form for <a href="">feedback on the portal</a>.</p>
+ <p>World Bank Open Finances made a
+ <a href="">form</a>
+ that populates an
+ <a href="">open data events calendar</a>.</p>
+ <h2 id="relevance-to-software">Relevance to software</h2>
+ <p>The three examples of Socrata forms show us how we can turn user input on a website into
+ open data automatically. Using a Socrata form to compose a dataset is quite inconvenient,
+ unreliable, limited, and other bad things, but I see this as a nice example of how software
+ can encourage that data be open by default. I previously
+ <a href="!/socrata-calendars#opening-data-at-their-sources">hinted</a> at this,
+ but now I have two specific ideas as to how software can encourage that data be open by default.</p>
+ <h3 id="standard-formats">1. Standard formats</h3>
+ <p>If you run any sort of involved website, you are probably already storing data in some
+ reasonably standard way, and you probably could send it to a data portal somewhat easily.</p>
+ <h4 id="opening-user-entered-application-data-on-your-websites-database">Opening user-entered application data on your website’s database</h4>
+ <p>One advantage of the Socrata form approach is that the data go automatically into a
+ reasonably standard format (a Socrata dataset). It happens that most websites work this
+ way, except that the standard format is something like MySQL.</p>
+ <p>A notable difference is that database software generally doesn’t concern itself as
+ strongly with opening the data. Many websites have HTTP APIs, but few will give out
+ direct access to their databases. And even if they did this, it wouldn’t provide the
+ various cataloging and format conversion features that people expect of data portals.
+ This is why we make data portals that import from these databases and provide all the
+ fancy features.</p>
+ <p>If you have a website that stores information in a standard database (like MySQL) and
+ you separate the private information from the public information, you already can quite
+ safely and easily have it sent to a data portal.</p>
+ <p>If you are making a new website and care about open data, try to choose a common
+ database for which integrations will already exist.</p>
+ <h4 id="storing-user-entered-application-data-directly-in-a-data-portal">Storing user-entered application data directly in a data portal</h4>
+ <p>If you have a simple website, maybe you don’t have to run your own database
+ and write your own web APIs. You could store the data directly in the data portal
+ and query it from the data portal. If this is powerful enough for you, it
+ simplifies your database management, and it naturally makes your data open by default.</p>
+ <h4 id="opening-data-from-some-other-software">Opening data from some other software</h4>
+ <p>Every time you save something in a computer program, you are creating some sort
+ of data, just like when you fill out a form on an open data portal.
+ If you have purchased a software service, you might not have access to the
+ underlying database, but you can still send it to a data portal.</p>
+ <p>When a lot of people use services like these, the services’ protocols naturally
+ become standard, so it becomes worthwhile to write tools that pull data from these
+ services into some standard place like a data portal. Using a standard service
+ with lots of users and integrations should make it easier for you to get the data
+ into a data portal.</p>
+ <h3 id="explicit-separation-between-public-and-private-data">2. Explicit separation between public and private data</h3>
+ <p>With a questionnaire, you might be able to just say that all of the responses are
+ private or that all are public. With other datasets, you might be able to say that
+ certain fields are private and others are public; in a database of employees, name
+ and salary can be public, but Social Security number can’t.</p>
+ <p>Things aren’t always this simple. With something like project management software,
+ some records/documents should be private and others should be public. Many of the
+ entries in project management software are probably safe for public disclosure,
+ but there might be some private information; for example, I’ve put passwords inside
+ calendar entries and issue tracker tickets.</p>
+ <p>Project management software, email clients, calendars, web browsers and image
+ editors all contain rich data that can help people understand how government
+ and other organizations work, so we should find ways of separating the public
+ information and opening that. Software can help with this.</p>
+ <p>Separate public information and private information from the beginning, and it
+ should be easier to open the data that is behind all of these applications.
+ The user interface can expose the separation between public and private and
+ encourage that information public by default.</p>
+ <h2 id="things-to-think-about">Things to think about</h2>
+ <p>Think about what programs you and others are already using, especially if you
+ don’t think of them as data programs, and think about how you can open the data in these programs.
+ A program’s data will be easy to open if the program already stores its data in
+ a standard format on the internet and it clearly separates public data from
+ private data.</p>
+ <p>Also think about how we can make software that follows the policy guideline of
+ open data by default. I’ve proposed that clear separations between public and
+ private data is part of this and that standard storage methods is another, but
+ there are surely other relevant features.</p>
+ </article>
+ </div>
+ <div id='pagination'>
+ <div class='base-little-card'>
+ <a href="!/open-by-default/">View source</a>
+ <a href="">Discuss</a>
+ </div>
+ </div>
+ </div>
+ </div>
+ <div id='feedback'>
+ <strong>
+ Tom requests your feedback.
+ </strong>
+ <p>
+ I can never decide what to write;
+ tell me what you like,
+ and my decisions will be easier.
+ (Contact information is <a href="/" target="_blank" >here</a>.)
+ </p>
+ <a class='close' href='javascript:$("#feedback").fadeOut()'>
+ Close
+ </a>
+ </div>
+ <script src='/js/application-cb286d6f677.js'></script>
+ <!-- Piwik -->
+ <script type="text/javascript">
+ var pkBaseURL = (("https:" == document.location.protocol) ? "" : "");
+ document.write(unescape("%3Cscript src='" + pkBaseURL + "piwik.js' type='text/javascript'%3E%3C/script%3E"));
+ </script><script type="text/javascript">
+ try {
+ var piwikTracker = Piwik.getTracker(pkBaseURL + "piwik.php", 2);
+ piwikTracker.trackPageView();
+ piwikTracker.enableLinkTracking();
+ } catch( err ) {}
+ </script><noscript><p><img src="" style="border:0" alt="Piwik tracking image" /></p></noscript>
+ <!-- End Piwik Tracking Code -->
+ </body>
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Oops, something went wrong.

0 comments on commit 7cd5aba

Please sign in to comment.