first-pass.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3.6.3 (default, Nov 13 2017, 08:48:07) \n",
      "[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)]\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "print(sys.version)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import gensim"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I'm going to work with the Courtlistener Supreme Court opinions dataset, found at [https://www.courtlistener.com/api/bulk-data/opinions/scotus.tar.gz](https://www.courtlistener.com/api/bulk-data/opinions/scotus.tar.gz) --- they're a nonprofit without a lot of bandwidth, so leave a good half an hour to download it even on a fast connection --- which has been previously downloaded into supcourt and then unzipped into a bunch of individual case jsons (there are a lot of them)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you look at the data, you'll see that each JSON has a \"html\" field with a html-ized version of the opinion, as well as a \"plain_text\" field... but that the latter is sometimes empty.  So in order to extract texts, we'll try and get it out of plain_text first; if there's no plain text, we'll use beautiful soup to extract from html (or from several other alternate html fields)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from bs4 import BeautifulSoup\n",
    "def extract_text(html):\n",
    "    soup = BeautifulSoup(html, \"lxml\")\n",
    "    for crap in soup([\"script\", \"style\", \"meta\"]):\n",
    "        crap.extract()\n",
    "    return soup.get_text()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The Rule of Law in the Real WorldThe Rule of Law in the Real WorldPaul Gowder\"Paul Gowder's masterpiece articulates a new vision of the rule of the law that protects the disempowered and marginalized, and that demands that the nation-state rationalize its coercive power. He relentlessly attacks irrational social, economic and political hierarchies, particularly those that give continued vitality to racial inequality in the US today. As such his rule of law is firmly rooted in notions of human rights, and looks askance at soaring inequality in the US. This rule of law protects real people in need of real protection, rather than serving as just another rhetorical instrument in the arsenal of the wealthy and powerful. It also serves, incidentally, as a bedrock foundation for economic development and a tool that can help avoid financial disruptions like that seen in 2008.\"Steven A. Ramirez, author of Lawless Capitalism\"The Rule of Law in the Real World explores and connects legal philosophy, conceptual and normative, as well as historical interpretation and social sciences, policy analysis, and advocacy. It makes insightful and distinctive contributions in each domain. Moreover, it brings them into mutually informing connection. Standard inmates of the disciplines rarely make (or even imagine) such connections, even when their subject - the rule of law - bears the same name. Gowder's achievement is thus far more than a sum of its exemplary parts. It is an all too rare combination of theoretical cogency, normative sophistication and ambition, empirical attentiveness, and real world engagement and pay-off.\"Martin Krygier, Gordon Samuels Professor of Law and Social Theory, and Co-Director, Network for Interdisciplinary Studies of Law, University of New South WalesAbout Graphs Supplemental Materials References and Links BuyAbout the BookThe Rule of Law in the Real World gives a comprehensive new theory of the political and legal ideal known as \"the rule of law\". The rule of law should be understood as the coordinated collective control of power, and matters because it constitutes a morally important kind of social equality. The book develops a case for how those properties should be taken into account in social scientific attempts to measure the rule of law as well as policy efforts to promote it.This highly interdisciplinary work begins with analytic political philosophy, developing a distinctively egalitarian account of the rule of law and its moral value. It then moves to history and political science. It shows that states with more egalitarian legal systems are likely to do better at sustaining the rule of law than states with inegalitarian systems. It also shows that the self-understanding of participants in political conflict in real-world historical rule of law states---including democratic Athens and seventeenth-century England---understood the rule of law in egalitarian terms.It then turns to the development enterprise. The book offers a critique of existing empirical measurement strategies for the rule of law as well as an alternative rule of law measure. The new measure is rooted in the philosophical basis of the concept, behaves much as existing theoretical claims about the rule of law would predict (e.g., is highly correlated with economic development: see plots below), and is more readily interpretable than existing measures.In addition, the book explains how the egalitarian account of the rule of law can support interventions in the debate among rule of law development professionals and scholars between advocates of \"bottom-up\" and \"top-down\" development strategies. Finally, The Rule of Law in the Real World calls for the developed states to pay closer attention to promoting and maintaing the rule of law in their own political communities. In particular, it argues that policing in the U.S.---especially the systematic discrimination against African-Americans---is a dire failure of the rule of law.Recently reviewed in Ethics.Paul Gowder is a professor of law and, by courtesy, of political science and philosophy, at the University of Iowa researching constitutional law, ethics, normative and conceptual jurisprudence, political theory, and quantitative legal studies. His areas of speciality (other than the rule of law) include Classical Athenian law, contemporary liberal political philosophy, game theory, predictive modeling, and computational law. He holds a J.D. from Harvard University and a Ph.D. in political science from Stanford University. Before joining academia, he worked as a public interest litigator practicing poverty and civil rights law in Oregon and Virginia.Map of Rule of Law Scores by Country See the Book on Cambridge University PressDarker colors are higher rule of law scores. White countries have no scores.About Graphs Supplemental Materials References and Links BuyWhat the Rule of Law Looks LikeMultidimensionalHistogramGlobe with ScoresProperty RightsPolitical PluralismElectoral ProcessDemocracy in 3DGDP Per CapitaIndividual LibertyAbout Graphs Supplemental Materials References and Links BuySupplemental MaterialsDescriptionDownloadDataset: Rule of law scores with correlates per charts aboverol-scores.csvDataset: Simulation data reported in chapter eight (equal law and stability)simul_ds.csv (31.6mb)Code: Simulation code in R (computationally intensive)simulation.RCode: Plots in chapter nine of the book, plus displayed on this website, in Rplots.RTable: Text of survey items used to construct rule of law measure, from World Justice Project 2012 data (actual data per state available only from WJP)WJPitems.pdfNew analysis: A comparison between the rule of law scores from the book and the first principal component of the World Justice Project's eight-factor measurerlrw-wjp.htmlNew analysis and presentation:Further explorations and comparisons with other datasets, plus slides based on this analysis + book as a whole, presented to the ABA Rule of Law Initiative on June 8, 2016aba/Presentation: A slideshow prepared for a 35-minute presentation on The Rule of Law in the Real World (reformatted for the web)slides/About Graphs Supplemental Materials References and Links BuyExternal References and LinksRelated Work by Paul GowderDescriptionBook ReferenceLinkPaper offering a preliminary version of the core normative/conceptual account of the rule of lawPreliminary version of claims developed in chapters one and twoThe Rule of Law and EqualityPaper offering a preliminary version of the account of general law and social equalityPreliminary version of claims developed in chapters two and threeEqual Law in an Unequal WorldPaper offering a preliminary version of the account of the rule of law in Classical AthensPreliminary version of claims developed in chapters five and sixDemocracy, Solidarity and the Rule of Law: Lessons from AthensPaper describing some of the methodological presuppositions of the theory of the rule of lawDefends some of the base-level philosophical claims that ground the book as a wholeInstitutional Values, or How to Say What Democracy isSpin-off paper about how the egalitarian theory of the rule of law can shed light on political corruptionReferenced in chapter oneInstitutional Corruption and the Rule of LawBrief commentary further discussing the relationship between the rule of law and democracyReferenced in chapter eightThe Rule of Law Against Sovereign Immunity in a Democratic StateAnalysis of Plato's Crito in terms of the strength topos of chapter fiveReferenced in chapter fiveWhat the Laws Demand of Socrates---and of UsExternal Data: Rule of Law and Correlates Links referenced in the bookExternal Datasets and Rule of Law MeasuresDescriptionBook ReferenceLinkWorld Justice Project Rule of Law Surveys and Results2012 survey is source for data in chapter nine and this websiteThe World Justice ProjectNardulli, Peyton & Bajjalieh, Conceptualizing and Measuring Rule of Law ConstructsAlternative unidimensional rule of law measure, discussed in chapter nineNardulli, Peyton & Bajjalieh (2013)United Nations Rule of Law Indicators: Implementation Guide and Project ToolsFramework for alternative rule of law measure, discussed in chapter nineU.N. Rule of Law GuideWorld Bank: Worldwide Governance IndicatorsAlternative rule of law measure, discussed in introduction and chapter nineWorldwide Governance IndicatorsFreedom HouseSupplies electoral process, political pluralism, and personal autonomy measures used in validation of chapter nine rule of law indicatorsFreedom HousePolity Project DatasetAlternative source of measures of characteristics related to liberal democracyPolity IVHeritage Foundation Index of Economic FreedomSource of property rights measure used in validation of rule of law measure in chapter nineHeritageTransparency InternationalAdditional data on corruption, relevant to rule of law measurement (considered, but ultimately not used in chapter nine measure)Transparency International: ResearchBertelsmann Stiftung’s Transformation IndexAlternative measures of rule of law and related concepts.BTI IndexComparative Constitutions ProjectAdditional data on de jure content of constitutions across the world (considered, but ultimately not used in chapter nine measure).CCP DataSelected Links Referenced in the BookDescriptionBook ReferenceLinkWorld Bank: Rule of Law Index SpecificationIntroduction, chapter nineWGI: Rule of Law (pdf)BBC World News: Why does Singapore Top So Many Tables?Chapter fourBBC on SingaporeHuffington Post: Saudi Arabia Partying: Elite, Boozy And SecretChapter fourHuffpo on Saudi ArabiaPerseus Digital LibrarySource for a number of English translations referenced in chapters five and sixPerseusTa-Nehisi Coates, \"Nonviolence as Compliance,\" The Atlantic, 4/27/15Chapter eightCoates, 2015Hasso Plattner Institute of Design, Stanford University, \"Use Our Methods\"Chapter nineD-School MethodsParsons Design for Social Innovation and Sustainability Lab at the New SchoolChapter nineDESIS LabDanny Vinik, “An 18-Year-Old Baltimore Rioter Faces a Higher Bail Than the Cop Accused of Murdering Freddie Gray,” New Republic, 5/2/2015ConclusionVinik NROFrank Stoltze, “’Rough Rides’ in Baltimore police cars are ‘Screen Tests’ in LA,” K.P.C.C. radio, 5/2/2015ConclusionStoltze, \"Screen Tests\"Bill Keller, “David Simon on Baltimore’s Anguish,” The Marshall Project, 4/29/2015ConclusionKeller/SimonLarry Celona and Bob Fredericks, “City Housing Puts Workers in Bright Vests in Fear of NYPD Shootings,” New York Post, 5/25/2015ConclusionCelona/FredericksTimothy Williams, “San Francisco Police Officers to Be Dismissed Over Racist Texts,” New York Times, 4/3/2015ConclusionNYT: Racist TextsAleksander Chan, “The Horrible, Bigoted Texts Traded Among San Francisco Police Officers,” Gawker, 3/3/2015ConclusionGawker: Racist TextsRadley Balko, “How Municipalities in St. Louis County, Mo., Profit from Poverty,” The Washington Post, 9/3/2014ConclusionBalko on FergusonCourt Record Reflecting St. Louis County Arrest Warrant StatisticsConclusionMO Table 95Spencer Ackerman, “The Disappeared: Chicago Police Detain Americans at Abuse-Laden ‘Black Site’,” The Guardian, 2/24/2015ConclusionGuardian on Chicago Black SiteAbout Graphs Supplemental Materials References and Links BuyBuy the Book\n",
      "Buy from Cambridge University Press     Buy from Amazon     Buy from Apple iBooks storeThanks for tools and resources used in the creation of this site: Book cover image created by Vero Rose SmithInteractive plots coded in R using Rstudio and enhanced + served by PlotlyWebsite based on the Skeleton CSS framework and written with Atomthe rule of law in the real world, by paul gowder | book published by cambridge university press, 2016 | website by authorTweet\n"
     ]
    }
   ],
   "source": [
    "# let's do an experiment and make sure this works right\n",
    "import requests\n",
    "response = requests.get(\"http://rulelaw.net\")\n",
    "print(extract_text(response.text))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Actually, this is ugly.  After the data finishes downloading, if cases come out this ugly too, I might break out jsoup or some other parser that does a better job getting text out of these things."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import glob\n",
    "jsons = list(glob.glob(\"supcourt/*.json\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "63967"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(jsons)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "with open(jsons[0]) as j:\n",
    "    example = json.load(j)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'resource_uri': 'http://www.courtlistener.com/api/rest/v3/opinions/100000/', 'absolute_url': '/opinion/100000/morrisdale-coal-co-v-united-states/', 'cluster': 'http://www.courtlistener.com/api/rest/v3/clusters/100000/', 'author': 'http://www.courtlistener.com/api/rest/v3/people/1501/', 'joined_by': [], 'author_str': '', 'per_curiam': False, 'date_created': '2010-04-28T16:47:22Z', 'date_modified': '2017-03-24T04:07:08.420443Z', 'type': '010combined', 'sha1': 'f966678c479af550803b000aecea9d1f16897a6a', 'page_count': None, 'download_url': None, 'local_path': None, 'plain_text': '', 'html': '<p class=\"case_cite\">259 U.S. 188</p>\\n    <p class=\"case_cite\">42 S.Ct. 481</p>\\n    <p class=\"case_cite\">66 L.Ed. 892</p>\\n    <p class=\"parties\">MORRISDALE COAL CO.<br>v.<br>UNITED STATES.</p>\\n    <p class=\"docket\">No. 65.</p>\\n    <p class=\"date\">Argued Jan. 6-9, 1922.</p>\\n    <p class=\"date\">Decided May 29, 1922.</p>\\n    <div class=\"prelims\">\\n      <p class=\"indent\">Messrs. Gibbs L. Baker and Karl Knox Gartner, both of Washington, D. C., for appellant.</p>\\n      <p class=\"indent\">Mr. Assistant Attorney General Riter, for the United States.</p>\\n      <p class=\"indent\">Mr. Justice HOLMES delivered the opinion of the Court.</p>\\n    </div>\\n    <div class=\"num\" id=\"p1\">\\n      <span class=\"num\">1</span>\\n      <p class=\"indent\">This is an appeal from a judgment of the Court of Claims dismissing the appellant\\'s petition upon demurrer. The petition alleges that the claimant had outstanding contracts calling for more than the actual production of its mines for the months of June and following through November, 1918, at a price of $4.50 per gross ton; that the Fuel Administration appointed by the President during the war \\'requisitioned and compelled petitioner to divert 12,823.29 tons of coal\\' during the period mentioned; that the price received for this coal was $3.304 per gross ton, and that the claimant thereby suffered a loss of $15,337.37, for which loss it asks judgment against the United States.</p>\\n    </div>\\n    <div class=\"num\" id=\"p2\">\\n      <span class=\"num\">2</span>\\n      <p class=\"indent\">The petition does not allege or mean that the United States took the coal to its own use. The meaning attributed to it by the claimant is merely that the Fuel Administration fixed the price on coal of this quality at $3.304 per gross ton and issued orders from time to time directing coal to such employments as best would promote the prosecution of the war. The Fuel Administration acted under a delegation from the President of the power conferred upon him by the Act of August 10, 1917, c. 53, &#167; 25, 40 Stat. 276, 284 (Comp. St. 1918, Comp. St. Ann. Supp. 1919, &#167; 3115 1/8 q), to fix the price of coal and to regulate distribution of it among dealers and consumers; the price so fixed not to invalidate contracts previously made in good faith in which prices are fixed. 40 Stat. 286. The claimant does not argue that this section provides compensation for obedience to orders made in pursuance of the same; it agrees, and rightly, that its remedy, if any, is under section 145 of the Judicial Code (Comp. St. &#167; 1136), giving the Court of Claims jurisdiction of claims upon any contract, express or implied, with the Government. It contends that upon the facts stated a contract on the part of the Government must be implied, both from the statute and by virtue of the Fifth Amendment on the ground that its property was taken for public use.</p>\\n    </div>\\n    <div class=\"num\" id=\"p3\">\\n      <span class=\"num\">3</span>\\n      <p class=\"indent\">We see no ground for the claim. The claimant in consequence of the regulation mentioned sold some of its coal to other parties at a less price than what otherwise it would have got. That is all. It now seeks to hold the Government answerable for making a rule that it saw fit to obey. Whether the rule was valid or void no such consequence follows. Making the rule was not a taking and no lawmaking power promises by implication to make good losses that may be incurred by obedience to its commands. If the law requires a party to give up property to a third person without adequate compensation the remedy is, if necessary, to refuse to obey it, not to sue the lawmaker. The statute provides remedies against the Government in other cases, but the claimant argues that this case does not fall within them, and it did not follow the steps prescribed for them. The petition does not even allege that the price the claimant got was not a fair one but only that if the Government had not issued the regulation it would have got more under its contract. Considerably more than that is needed before a promise of indemnity from the Government can be implied. See American Smelting &amp; Refining Co. v. United States, 259 U. S. 75, 42 Sup. Ct. 420, 66 L. Ed. &#8212;&#8212;, May 15, 1922.</p>\\n    </div>\\n    <div class=\"num\" id=\"p4\">\\n      <span class=\"num\">4</span>\\n      <p class=\"indent\">Judgment affirmed.</p>\\n    </div>\\n    ', 'html_lawbox': '<div>\\n<center><b>259 U.S. 188 (1922)</b></center>\\n<center><h1>MORRISDALE COAL COMPANY<br>\\nv.<br>\\nUNITED STATES.</h1></center>\\n<center>No. 65.</center>\\n<center><p><b>Supreme Court of United States.</b></p></center>\\n<center>Argued January 6, 9, 1922.</center>\\n<center>Decided May 29, 1922.</center>\\nAPPEAL FROM THE COURT OF CLAIMS.\\n<p><span class=\"star-pagination\">*189</span> <i>Mr. Gibbs L. Baker,</i> with whom <i>Mr. Karl Knox Gartner</i> was on the brief, for appellant.</p>\\n<p><i>Mr. Assistant Attorney General Riter,</i> with whom <i>Mr. Solicitor General Beck</i> and <i>Mr. Charles S. Lawrence</i> were on the brief, for the United States.</p>\\n<p>MR. JUSTICE HOLMES delivered the opinion of the court.</p>\\n<p>This is an appeal from a judgment of the Court of Claims dismissing the appellant\\'s petition upon demurrer. The petition alleges that the claimant had outstanding contracts calling for more than the actual production of its mines for the months of June and following through November, 1918, at a price of $4.50 per gross ton; that the Fuel Administration appointed by the President during the war \"requisitioned and compelled petitioner to divert 12,823.89 tons of coal\" during the period mentioned; that the price received for this coal was $3.304 per gross ton, and that the claimant thereby suffered a loss of $15,337.37, for which loss it asks judgment against the United States.</p>\\n<p>The petition does not allege or mean that the United States took the coal to its own use. The meaning attributed to it by the claimant is merely that the Fuel Administration fixed the price on coal of this quality at $3.304 per gross ton and issued orders from time to time directing coal to such employments as best would promote the prosecution of the war. The Fuel Administration acted under a delegation from the President of the power conferred upon him by the Act of August 10, 1917, c. 53, § 25, 40 Stat. 276, 284, to fix the price of coal and to regulate distribution <span class=\"star-pagination\">*190</span> of it among dealers and consumers; the price so fixed not to invalidate contracts previously made in good faith in which prices are fixed. 40 Stat. 286. The claimant does not argue that this section provides compensation for obedience to orders made in pursuance of the same; it agrees, and rightly, that its remedy, if any, is under § 145 of the Judicial Code giving the Court of Claims jurisdiction of claims upon any contract, express or implied, with the Government. It contends that upon the facts stated a contract on the part of the Government must be implied, both from the statute and by virtue of the Fifth Amendment on the ground that its property was taken for public use.</p>\\n<p>We see no ground for the claim. The claimant in consequence of the regulation mentioned sold some of its coal to other parties at a less price than what otherwise it would have got. That is all. It now seeks to hold the Government answerable for making a rule that it saw fit to obey. Whether the rule was valid or void no such consequence follows. Making the rule was not a taking and no lawmaking power promises by implication to make good losses that may be incurred by obedience to its commands. If the law requires a party to give up property to a third person without adequate compensation the remedy is, if necessary, to refuse to obey it, not to sue the lawmaker. The statute provides remedies against the Government in other cases, but the claimant argues that this case does not fall within them, and it did not follow the steps prescribed for them. The petition does not even allege that the price the claimant got was not a fair one but only that if the Government had not issued the regulation it would have got more under its contract. Considerably more than that is needed before a promise of indemnity from the Government can be implied. See <i>American Smelting &amp; Refining Co.</i> v. <i>United States, ante,</i> 75.</p>\\n<p><i>Judgment affirmed.</i></p>\\n</div>', 'html_columbia': None, 'html_with_citations': '<div>\\n<center><b><span class=\"citation no-link\"><span class=\"volume\">259</span> <span class=\"reporter\">U.S.</span> <span class=\"page\">188</span></span> (1922)</b></center>\\n<center><h1>MORRISDALE COAL COMPANY<br>\\nv.<br>\\nUNITED STATES.</h1></center>\\n<center>No. 65.</center>\\n<center><p><b>Supreme Court of United States.</b></p></center>\\n<center>Argued January 6, 9, 1922.</center>\\n<center>Decided May 29, 1922.</center>\\nAPPEAL FROM THE COURT OF CLAIMS.\\n<p><span class=\"star-pagination\">*189</span> <i>Mr. Gibbs L. Baker,</i> with whom <i>Mr. Karl Knox Gartner</i> was on the brief, for appellant.</p>\\n<p><i>Mr. Assistant Attorney General Riter,</i> with whom <i>Mr. Solicitor General Beck</i> and <i>Mr. Charles S. Lawrence</i> were on the brief, for the United States.</p>\\n<p>MR. JUSTICE HOLMES delivered the opinion of the court.</p>\\n<p>This is an appeal from a judgment of the Court of Claims dismissing the appellant\\'s petition upon demurrer. The petition alleges that the claimant had outstanding contracts calling for more than the actual production of its mines for the months of June and following through November, 1918, at a price of $4.50 per gross ton; that the Fuel Administration appointed by the President during the war \"requisitioned and compelled petitioner to divert 12,823.89 tons of coal\" during the period mentioned; that the price received for this coal was $3.304 per gross ton, and that the claimant thereby suffered a loss of $15,337.37, for which loss it asks judgment against the United States.</p>\\n<p>The petition does not allege or mean that the United States took the coal to its own use. The meaning attributed to it by the claimant is merely that the Fuel Administration fixed the price on coal of this quality at $3.304 per gross ton and issued orders from time to time directing coal to such employments as best would promote the prosecution of the war. The Fuel Administration acted under a delegation from the President of the power conferred upon him by the Act of August 10, 1917, c. 53, § 25, 40 Stat. 276, 284, to fix the price of coal and to regulate distribution <span class=\"star-pagination\">*190</span> of it among dealers and consumers; the price so fixed not to invalidate contracts previously made in good faith in which prices are fixed. 40 Stat. 286. The claimant does not argue that this section provides compensation for obedience to orders made in pursuance of the same; it agrees, and rightly, that its remedy, if any, is under § 145 of the Judicial Code giving the Court of Claims jurisdiction of claims upon any contract, express or implied, with the Government. It contends that upon the facts stated a contract on the part of the Government must be implied, both from the statute and by virtue of the Fifth Amendment on the ground that its property was taken for public use.</p>\\n<p>We see no ground for the claim. The claimant in consequence of the regulation mentioned sold some of its coal to other parties at a less price than what otherwise it would have got. That is all. It now seeks to hold the Government answerable for making a rule that it saw fit to obey. Whether the rule was valid or void no such consequence follows. Making the rule was not a taking and no lawmaking power promises by implication to make good losses that may be incurred by obedience to its commands. If the law requires a party to give up property to a third person without adequate compensation the remedy is, if necessary, to refuse to obey it, not to sue the lawmaker. The statute provides remedies against the Government in other cases, but the claimant argues that this case does not fall within them, and it did not follow the steps prescribed for them. The petition does not even allege that the price the claimant got was not a fair one but only that if the Government had not issued the regulation it would have got more under its contract. Considerably more than that is needed before a promise of indemnity from the Government can be implied. See <i>American Smelting &amp; Refining Co.</i> v. <i>United States, ante,</i> 75.</p>\\n<p><i>Judgment affirmed.</i></p>\\n</div>', 'extracted_by_ocr': False, 'opinions_cited': []}\n"
     ]
    }
   ],
   "source": [
    "print(example)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dict_keys(['resource_uri', 'absolute_url', 'cluster', 'author', 'joined_by', 'author_str', 'per_curiam', 'date_created', 'date_modified', 'type', 'sha1', 'page_count', 'download_url', 'local_path', 'plain_text', 'html', 'html_lawbox', 'html_columbia', 'html_with_citations', 'extracted_by_ocr', 'opinions_cited'])\n"
     ]
    }
   ],
   "source": [
    "print(example.keys())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "print(example[\"plain_text\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "259 U.S. 188\n",
      "42 S.Ct. 481\n",
      "66 L.Ed. 892\n",
      "MORRISDALE COAL CO.v.UNITED STATES.\n",
      "No. 65.\n",
      "Argued Jan. 6-9, 1922.\n",
      "Decided May 29, 1922.\n",
      "\n",
      "Messrs. Gibbs L. Baker and Karl Knox Gartner, both of Washington, D. C., for appellant.\n",
      "Mr. Assistant Attorney General Riter, for the United States.\n",
      "Mr. Justice HOLMES delivered the opinion of the Court.\n",
      "\n",
      "\n",
      "1\n",
      "This is an appeal from a judgment of the Court of Claims dismissing the appellant's petition upon demurrer. The petition alleges that the claimant had outstanding contracts calling for more than the actual production of its mines for the months of June and following through November, 1918, at a price of $4.50 per gross ton; that the Fuel Administration appointed by the President during the war 'requisitioned and compelled petitioner to divert 12,823.29 tons of coal' during the period mentioned; that the price received for this coal was $3.304 per gross ton, and that the claimant thereby suffered a loss of $15,337.37, for which loss it asks judgment against the United States.\n",
      "\n",
      "\n",
      "2\n",
      "The petition does not allege or mean that the United States took the coal to its own use. The meaning attributed to it by the claimant is merely that the Fuel Administration fixed the price on coal of this quality at $3.304 per gross ton and issued orders from time to time directing coal to such employments as best would promote the prosecution of the war. The Fuel Administration acted under a delegation from the President of the power conferred upon him by the Act of August 10, 1917, c. 53, § 25, 40 Stat. 276, 284 (Comp. St. 1918, Comp. St. Ann. Supp. 1919, § 3115 1/8 q), to fix the price of coal and to regulate distribution of it among dealers and consumers; the price so fixed not to invalidate contracts previously made in good faith in which prices are fixed. 40 Stat. 286. The claimant does not argue that this section provides compensation for obedience to orders made in pursuance of the same; it agrees, and rightly, that its remedy, if any, is under section 145 of the Judicial Code (Comp. St. § 1136), giving the Court of Claims jurisdiction of claims upon any contract, express or implied, with the Government. It contends that upon the facts stated a contract on the part of the Government must be implied, both from the statute and by virtue of the Fifth Amendment on the ground that its property was taken for public use.\n",
      "\n",
      "\n",
      "3\n",
      "We see no ground for the claim. The claimant in consequence of the regulation mentioned sold some of its coal to other parties at a less price than what otherwise it would have got. That is all. It now seeks to hold the Government answerable for making a rule that it saw fit to obey. Whether the rule was valid or void no such consequence follows. Making the rule was not a taking and no lawmaking power promises by implication to make good losses that may be incurred by obedience to its commands. If the law requires a party to give up property to a third person without adequate compensation the remedy is, if necessary, to refuse to obey it, not to sue the lawmaker. The statute provides remedies against the Government in other cases, but the claimant argues that this case does not fall within them, and it did not follow the steps prescribed for them. The petition does not even allege that the price the claimant got was not a fair one but only that if the Government had not issued the regulation it would have got more under its contract. Considerably more than that is needed before a promise of indemnity from the Government can be implied. See American Smelting & Refining Co. v. United States, 259 U. S. 75, 42 Sup. Ct. 420, 66 L. Ed. ——, May 15, 1922.\n",
      "\n",
      "\n",
      "4\n",
      "Judgment affirmed.\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(extract_text(example[\"html\"]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import re\n",
    "example_text = extract_text(example[\"html\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "test_despace = re.sub('\\s+', \" \", example_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "259 U.S. 188 42 S.Ct. 481 66 L.Ed. 892 MORRISDALE COAL CO.v.UNITED STATES. No. 65. Argued Jan. 6-9, 1922. Decided May 29, 1922. Messrs. Gibbs L. Baker and Karl Knox Gartner, both of Washington, D. C., for appellant. Mr. Assistant Attorney General Riter, for the United States. Mr. Justice HOLMES delivered the opinion of the Court. 1 This is an appeal from a judgment of the Court of Claims dismissing the appellant's petition upon demurrer. The petition alleges that the claimant had outstanding contracts calling for more than the actual production of its mines for the months of June and following through November, 1918, at a price of $4.50 per gross ton; that the Fuel Administration appointed by the President during the war 'requisitioned and compelled petitioner to divert 12,823.29 tons of coal' during the period mentioned; that the price received for this coal was $3.304 per gross ton, and that the claimant thereby suffered a loss of $15,337.37, for which loss it asks judgment against the United States. 2 The petition does not allege or mean that the United States took the coal to its own use. The meaning attributed to it by the claimant is merely that the Fuel Administration fixed the price on coal of this quality at $3.304 per gross ton and issued orders from time to time directing coal to such employments as best would promote the prosecution of the war. The Fuel Administration acted under a delegation from the President of the power conferred upon him by the Act of August 10, 1917, c. 53, § 25, 40 Stat. 276, 284 (Comp. St. 1918, Comp. St. Ann. Supp. 1919, § 3115 1/8 q), to fix the price of coal and to regulate distribution of it among dealers and consumers; the price so fixed not to invalidate contracts previously made in good faith in which prices are fixed. 40 Stat. 286. The claimant does not argue that this section provides compensation for obedience to orders made in pursuance of the same; it agrees, and rightly, that its remedy, if any, is under section 145 of the Judicial Code (Comp. St. § 1136), giving the Court of Claims jurisdiction of claims upon any contract, express or implied, with the Government. It contends that upon the facts stated a contract on the part of the Government must be implied, both from the statute and by virtue of the Fifth Amendment on the ground that its property was taken for public use. 3 We see no ground for the claim. The claimant in consequence of the regulation mentioned sold some of its coal to other parties at a less price than what otherwise it would have got. That is all. It now seeks to hold the Government answerable for making a rule that it saw fit to obey. Whether the rule was valid or void no such consequence follows. Making the rule was not a taking and no lawmaking power promises by implication to make good losses that may be incurred by obedience to its commands. If the law requires a party to give up property to a third person without adequate compensation the remedy is, if necessary, to refuse to obey it, not to sue the lawmaker. The statute provides remedies against the Government in other cases, but the claimant argues that this case does not fall within them, and it did not follow the steps prescribed for them. The petition does not even allege that the price the claimant got was not a fair one but only that if the Government had not issued the regulation it would have got more under its contract. Considerably more than that is needed before a promise of indemnity from the Government can be implied. See American Smelting & Refining Co. v. United States, 259 U. S. 75, 42 Sup. Ct. 420, 66 L. Ed. ——, May 15, 1922. 4 Judgment affirmed. \n"
     ]
    }
   ],
   "source": [
    "print(test_despace)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"259 U.S. 188 42 S.Ct. 481 66 L.Ed. 892 MORRISDALE COAL CO.v.UNITED STATES. No. 65. Argued Jan. 6-9, 1922. Decided May 29, 1922. Messrs. Gibbs L. Baker and Karl Knox Gartner, both of Washington, D. C., for appellant. Mr. Assistant Attorney General Riter, for the United States. Mr. Justice HOLMES delivered the opinion of the Court. 1 This is an appeal from a judgment of the Court of Claims dismissing the appellant's petition upon demurrer. The petition alleges that the claimant had outstanding contracts calling for more than the actual production of its mines for the months of June and following through November, 1918, at a price of $4.50 per gross ton; that the Fuel Administration appointed by the President during the war 'requisitioned and compelled petitioner to divert 12,823.29 tons of coal' during the period mentioned; that the price received for this coal was $3.304 per gross ton, and that the claimant thereby suffered a loss of $15,337.37, for which loss it asks judgment against the United States. 2 The petition does not allege or mean that the United States took the coal to its own use. The meaning attributed to it by the claimant is merely that the Fuel Administration fixed the price on coal of this quality at $3.304 per gross ton and issued orders from time to time directing coal to such employments as best would promote the prosecution of the war. The Fuel Administration acted under a delegation from the President of the power conferred upon him by the Act of August 10, 1917, c. 53,  25, 40 Stat. 276, 284 (Comp. St. 1918, Comp. St. Ann. Supp. 1919,  3115 1/8 q), to fix the price of coal and to regulate distribution of it among dealers and consumers; the price so fixed not to invalidate contracts previously made in good faith in which prices are fixed. 40 Stat. 286. The claimant does not argue that this section provides compensation for obedience to orders made in pursuance of the same; it agrees, and rightly, that its remedy, if any, is under section 145 of the Judicial Code (Comp. St.  1136), giving the Court of Claims jurisdiction of claims upon any contract, express or implied, with the Government. It contends that upon the facts stated a contract on the part of the Government must be implied, both from the statute and by virtue of the Fifth Amendment on the ground that its property was taken for public use. 3 We see no ground for the claim. The claimant in consequence of the regulation mentioned sold some of its coal to other parties at a less price than what otherwise it would have got. That is all. It now seeks to hold the Government answerable for making a rule that it saw fit to obey. Whether the rule was valid or void no such consequence follows. Making the rule was not a taking and no lawmaking power promises by implication to make good losses that may be incurred by obedience to its commands. If the law requires a party to give up property to a third person without adequate compensation the remedy is, if necessary, to refuse to obey it, not to sue the lawmaker. The statute provides remedies against the Government in other cases, but the claimant argues that this case does not fall within them, and it did not follow the steps prescribed for them. The petition does not even allege that the price the claimant got was not a fair one but only that if the Government had not issued the regulation it would have got more under its contract. Considerably more than that is needed before a promise of indemnity from the Government can be implied. See American Smelting & Refining Co. v. United States, 259 U. S. 75, 42 Sup. Ct. 420, 66 L. Ed. , May 15, 1922. 4 Judgment affirmed. \""
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_despace.encode(\"ascii\", errors=\"ignore\").decode(\"ascii\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import string\n",
    "transdict = {ord(x): \" \" for x in string.punctuation + string.digits}\n",
    "def squish_spaces(text):\n",
    "    return re.sub('\\s+', \" \", text)\n",
    "\n",
    "def asciify(text):\n",
    "    return text.encode(\"ascii\", errors=\"ignore\").decode(\"ascii\")\n",
    "\n",
    "def remove_nonletter(text):\n",
    "    return text.translate(transdict)\n",
    "\n",
    "def cleanup(text):\n",
    "    return squish_spaces(remove_nonletter(asciify(text))).lower()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " u s s ct l ed morrisdale coal co v united states no argued jan decided may messrs gibbs l baker and karl knox gartner both of washington d c for appellant mr assistant attorney general riter for the united states mr justice holmes delivered the opinion of the court this is an appeal from a judgment of the court of claims dismissing the appellant s petition upon demurrer the petition alleges that the claimant had outstanding contracts calling for more than the actual production of its mines for the months of june and following through november at a price of per gross ton that the fuel administration appointed by the president during the war requisitioned and compelled petitioner to divert tons of coal during the period mentioned that the price received for this coal was per gross ton and that the claimant thereby suffered a loss of for which loss it asks judgment against the united states the petition does not allege or mean that the united states took the coal to its own use the meaning attributed to it by the claimant is merely that the fuel administration fixed the price on coal of this quality at per gross ton and issued orders from time to time directing coal to such employments as best would promote the prosecution of the war the fuel administration acted under a delegation from the president of the power conferred upon him by the act of august c stat comp st comp st ann supp q to fix the price of coal and to regulate distribution of it among dealers and consumers the price so fixed not to invalidate contracts previously made in good faith in which prices are fixed stat the claimant does not argue that this section provides compensation for obedience to orders made in pursuance of the same it agrees and rightly that its remedy if any is under section of the judicial code comp st giving the court of claims jurisdiction of claims upon any contract express or implied with the government it contends that upon the facts stated a contract on the part of the government must be implied both from the statute and by virtue of the fifth amendment on the ground that its property was taken for public use we see no ground for the claim the claimant in consequence of the regulation mentioned sold some of its coal to other parties at a less price than what otherwise it would have got that is all it now seeks to hold the government answerable for making a rule that it saw fit to obey whether the rule was valid or void no such consequence follows making the rule was not a taking and no lawmaking power promises by implication to make good losses that may be incurred by obedience to its commands if the law requires a party to give up property to a third person without adequate compensation the remedy is if necessary to refuse to obey it not to sue the lawmaker the statute provides remedies against the government in other cases but the claimant argues that this case does not fall within them and it did not follow the steps prescribed for them the petition does not even allege that the price the claimant got was not a fair one but only that if the government had not issued the regulation it would have got more under its contract considerably more than that is needed before a promise of indemnity from the government can be implied see american smelting refining co v united states u s sup ct l ed may judgment affirmed \n"
     ]
    }
   ],
   "source": [
    "print(cleanup(example_text))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def extract_clean_text(casedict):\n",
    "    if casedict[\"plain_text\"]:\n",
    "        text = casedict[\"plain_text\"]\n",
    "    else:\n",
    "        text = extract_text(casedict[\"html\"])\n",
    "    return cleanup(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " u s s ct l ed pine hill coal co inc v united states no argued jan decided may mr henry s drinker jr of philadephia pa for appellant argument of counsel from pages intentionally omitted mr assistant attorney general riter for the united states mr justice holmes delivered the opinion of the court this case like morrisdale coal co v united states u s sup ct l ed is a claim based upon the action of the fuel administration under the act of august c stat comp st comp st ann supp q fixing prices for coal the allegations and arguments however are different the transactions of the claimant from and including september through january are set forth in detail they embrace large sales at government prices and smaller sales at other than those prices it is alleged that the prices fixed for the claimant s coal were unjust and unreasonable and did not afford just compensation and that as a result of keeping to them as the claimant did the receipts were actually less than the cost of production on these facts the petition sets up a contract of indemnity on the part of the united states arising out of the language to be quoted from section it was dismissed on demurrer by the court of claims the paragraph of section that is relied upon follows paragraphs giving authority to the president personally or through the federal trade commission to fix the price of coal and coke to regulate the method of distribution among dealers and consumers during the war and if a producer or dealer neglects to conform to such prices or regulations etc to take over the plant and business paying a just compensation the paragraph in question reads that if the prices so fixed or if in the case of the taking over or requisitioning of the mines or business of any such producer or dealer the compensation therefor as determined by the provisions of this act be not satisfactory to the person or persons entitled to receive the same such person shall be paid seventy five per centum of the amount so determined and shall be entitled to sue the united states to recover such further sum as added to said seventy five per centum will make up such amount as will be just compensation in the manner provided by section twenty four paragraph twenty and section one hundred and forty five of the judicial code the latter section of the judicial code comp st is the one that gives jurisdiction to the court of claims and the former that which gives a limited concurrent jurisdiction to the district courts it is obvious that the words as they stand cannot be applied to sales by producers to third persons for it would be absurd to suppose that the united states undertook to pay not only such additional sum as might be awarded but also the last per centum of the price as fixed leaving the buyer to retain that amount the claimant admits this but insists that however read the pragraph cannot be followed without correction it argues that the opening words if the price so fixed necessarily apply to prices in general as fixed by the power just given in the section therefore it says there should be interpolated in the provision that the seller shall be paid per centum the words the prices so fixed or and in like manner that the provision for recovery should read that he shall recover such sum as added to the said prices or said per centum will be just it points out that while seeking to stimulate production in aid of the war the government could not fix very high prices without arousing householders and manufacturers or very low ones without endangering the supply and incurring the charge of confiscation it is said that the natural way out of the difficulty was for the government to guarantee a just return and that by so doing it avoided doubts as to the constitutionality of the statute there is offered a critical and refined scrutiny of the history of the amendment that introduced the claim the argument is that the section that became section when originally offered as an amendment clearly provided for payment in all cases that a modification was introduced for payment of only per centum upon takings by the united states but that it was not intended to change the general scope of the relief other makeweights are thrown in to which we think it unnecessary to advert it is a delicate business to base speculations about the purposes or construction of a statute upon the vicissitudes of its passage here we have as against the arguments of the claimant the fundamental and necessarily governing consideration that rightly prevailed below a liability in any case is not to be imposed upon a government without clear words but liability for a regulation for the consequences of a law on the part of the legislating power is most unusual and where as here the liability would mount to great sums only the plainest language could warrant a court in taking it to be imposed the general words the prices so fixed taken by themselves no doubt would include prices to private purchasers but the specific provision as to paying per centum prevails over them on the usual principles of construction and excludes a reference to any prices except those paid by the government it is said that those prices are provided for elsewhere but the claimant s argument presses the consideration that the law had to be hastily passed and unnecessary reduplication is far more easy to admit than an enormous charge upon the united states that can be fastened upon it only by inserting into a statute words that are not there judgment affirmed \n"
     ]
    }
   ],
   "source": [
    "with open(jsons[1]) as j:\n",
    "    print(extract_clean_text(json.load(j)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from os import path\n",
    "# let's make some texts now.\n",
    "for jfile in jsons:\n",
    "    with open(jfile) as j:\n",
    "        text = extract_clean_text(json.load(j))\n",
    "    filename = path.split(jfile)[-1]\n",
    "    outfile = \"texts/\" + filename.partition(\".\")[0] + \".txt\"\n",
    "    with open(outfile, \"w\") as o:\n",
    "        o.write(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "texts = list(glob.glob(\"texts/*.txt\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "63967"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(texts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " u s s ct l ed morrisdale coal co v united states no argued jan decided may messrs gibbs l baker and karl knox gartner both of washington d c for appellant mr assistant attorney general riter for the united states mr justice holmes delivered the opinion of the court this is an appeal from a judgment of the court of claims dismissing the appellant s petition upon demurrer the petition alleges that the claimant had outstanding contracts calling for more than the actual production of its mines for the months of june and following through november at a price of per gross ton that the fuel administration appointed by the president during the war requisitioned and compelled petitioner to divert tons of coal during the period mentioned that the price received for this coal was per gross ton and that the claimant thereby suffered a loss of for which loss it asks judgment against the united states the petition does not allege or mean that the united states took the coal to its own use the meaning attributed to it by the claimant is merely that the fuel administration fixed the price on coal of this quality at per gross ton and issued orders from time to time directing coal to such employments as best would promote the prosecution of the war the fuel administration acted under a delegation from the president of the power conferred upon him by the act of august c stat comp st comp st ann supp q to fix the price of coal and to regulate distribution of it among dealers and consumers the price so fixed not to invalidate contracts previously made in good faith in which prices are fixed stat the claimant does not argue that this section provides compensation for obedience to orders made in pursuance of the same it agrees and rightly that its remedy if any is under section of the judicial code comp st giving the court of claims jurisdiction of claims upon any contract express or implied with the government it contends that upon the facts stated a contract on the part of the government must be implied both from the statute and by virtue of the fifth amendment on the ground that its property was taken for public use we see no ground for the claim the claimant in consequence of the regulation mentioned sold some of its coal to other parties at a less price than what otherwise it would have got that is all it now seeks to hold the government answerable for making a rule that it saw fit to obey whether the rule was valid or void no such consequence follows making the rule was not a taking and no lawmaking power promises by implication to make good losses that may be incurred by obedience to its commands if the law requires a party to give up property to a third person without adequate compensation the remedy is if necessary to refuse to obey it not to sue the lawmaker the statute provides remedies against the government in other cases but the claimant argues that this case does not fall within them and it did not follow the steps prescribed for them the petition does not even allege that the price the claimant got was not a fair one but only that if the government had not issued the regulation it would have got more under its contract considerably more than that is needed before a promise of indemnity from the government can be implied see american smelting refining co v united states u s sup ct l ed may judgment affirmed \n"
     ]
    }
   ],
   "source": [
    "with open(texts[0]) as t:\n",
    "    print(t.read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok, let's do this and see what happens.  First I'm going to create a massive in-memory list.  This is going to be kind of a shitshow because it's huge, and gensim is supposed to work nicely with streaming data from disk, but I don't have time right now in this experiment to figure out how to do that.  So let's see if I can do this in memory without blowing up my machine."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def make_row(path):\n",
    "    with open(path) as p:\n",
    "        return p.read().split()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['u', 's', 's', 'ct', 'l', 'ed', 'morrisdale', 'coal', 'co', 'v', 'united', 'states', 'no', 'argued', 'jan', 'decided', 'may', 'messrs', 'gibbs', 'l', 'baker', 'and', 'karl', 'knox', 'gartner', 'both', 'of', 'washington', 'd', 'c', 'for', 'appellant', 'mr', 'assistant', 'attorney', 'general', 'riter', 'for', 'the', 'united', 'states', 'mr', 'justice', 'holmes', 'delivered', 'the', 'opinion', 'of', 'the', 'court', 'this', 'is', 'an', 'appeal', 'from', 'a', 'judgment', 'of', 'the', 'court', 'of', 'claims', 'dismissing', 'the', 'appellant', 's', 'petition', 'upon', 'demurrer', 'the', 'petition', 'alleges', 'that', 'the', 'claimant', 'had', 'outstanding', 'contracts', 'calling', 'for', 'more', 'than', 'the', 'actual', 'production', 'of', 'its', 'mines', 'for', 'the', 'months', 'of', 'june', 'and', 'following', 'through', 'november', 'at', 'a', 'price', 'of', 'per', 'gross', 'ton', 'that', 'the', 'fuel', 'administration', 'appointed', 'by', 'the', 'president', 'during', 'the', 'war', 'requisitioned', 'and', 'compelled', 'petitioner', 'to', 'divert', 'tons', 'of', 'coal', 'during', 'the', 'period', 'mentioned', 'that', 'the', 'price', 'received', 'for', 'this', 'coal', 'was', 'per', 'gross', 'ton', 'and', 'that', 'the', 'claimant', 'thereby', 'suffered', 'a', 'loss', 'of', 'for', 'which', 'loss', 'it', 'asks', 'judgment', 'against', 'the', 'united', 'states', 'the', 'petition', 'does', 'not', 'allege', 'or', 'mean', 'that', 'the', 'united', 'states', 'took', 'the', 'coal', 'to', 'its', 'own', 'use', 'the', 'meaning', 'attributed', 'to', 'it', 'by', 'the', 'claimant', 'is', 'merely', 'that', 'the', 'fuel', 'administration', 'fixed', 'the', 'price', 'on', 'coal', 'of', 'this', 'quality', 'at', 'per', 'gross', 'ton', 'and', 'issued', 'orders', 'from', 'time', 'to', 'time', 'directing', 'coal', 'to', 'such', 'employments', 'as', 'best', 'would', 'promote', 'the', 'prosecution', 'of', 'the', 'war', 'the', 'fuel', 'administration', 'acted', 'under', 'a', 'delegation', 'from', 'the', 'president', 'of', 'the', 'power', 'conferred', 'upon', 'him', 'by', 'the', 'act', 'of', 'august', 'c', 'stat', 'comp', 'st', 'comp', 'st', 'ann', 'supp', 'q', 'to', 'fix', 'the', 'price', 'of', 'coal', 'and', 'to', 'regulate', 'distribution', 'of', 'it', 'among', 'dealers', 'and', 'consumers', 'the', 'price', 'so', 'fixed', 'not', 'to', 'invalidate', 'contracts', 'previously', 'made', 'in', 'good', 'faith', 'in', 'which', 'prices', 'are', 'fixed', 'stat', 'the', 'claimant', 'does', 'not', 'argue', 'that', 'this', 'section', 'provides', 'compensation', 'for', 'obedience', 'to', 'orders', 'made', 'in', 'pursuance', 'of', 'the', 'same', 'it', 'agrees', 'and', 'rightly', 'that', 'its', 'remedy', 'if', 'any', 'is', 'under', 'section', 'of', 'the', 'judicial', 'code', 'comp', 'st', 'giving', 'the', 'court', 'of', 'claims', 'jurisdiction', 'of', 'claims', 'upon', 'any', 'contract', 'express', 'or', 'implied', 'with', 'the', 'government', 'it', 'contends', 'that', 'upon', 'the', 'facts', 'stated', 'a', 'contract', 'on', 'the', 'part', 'of', 'the', 'government', 'must', 'be', 'implied', 'both', 'from', 'the', 'statute', 'and', 'by', 'virtue', 'of', 'the', 'fifth', 'amendment', 'on', 'the', 'ground', 'that', 'its', 'property', 'was', 'taken', 'for', 'public', 'use', 'we', 'see', 'no', 'ground', 'for', 'the', 'claim', 'the', 'claimant', 'in', 'consequence', 'of', 'the', 'regulation', 'mentioned', 'sold', 'some', 'of', 'its', 'coal', 'to', 'other', 'parties', 'at', 'a', 'less', 'price', 'than', 'what', 'otherwise', 'it', 'would', 'have', 'got', 'that', 'is', 'all', 'it', 'now', 'seeks', 'to', 'hold', 'the', 'government', 'answerable', 'for', 'making', 'a', 'rule', 'that', 'it', 'saw', 'fit', 'to', 'obey', 'whether', 'the', 'rule', 'was', 'valid', 'or', 'void', 'no', 'such', 'consequence', 'follows', 'making', 'the', 'rule', 'was', 'not', 'a', 'taking', 'and', 'no', 'lawmaking', 'power', 'promises', 'by', 'implication', 'to', 'make', 'good', 'losses', 'that', 'may', 'be', 'incurred', 'by', 'obedience', 'to', 'its', 'commands', 'if', 'the', 'law', 'requires', 'a', 'party', 'to', 'give', 'up', 'property', 'to', 'a', 'third', 'person', 'without', 'adequate', 'compensation', 'the', 'remedy', 'is', 'if', 'necessary', 'to', 'refuse', 'to', 'obey', 'it', 'not', 'to', 'sue', 'the', 'lawmaker', 'the', 'statute', 'provides', 'remedies', 'against', 'the', 'government', 'in', 'other', 'cases', 'but', 'the', 'claimant', 'argues', 'that', 'this', 'case', 'does', 'not', 'fall', 'within', 'them', 'and', 'it', 'did', 'not', 'follow', 'the', 'steps', 'prescribed', 'for', 'them', 'the', 'petition', 'does', 'not', 'even', 'allege', 'that', 'the', 'price', 'the', 'claimant', 'got', 'was', 'not', 'a', 'fair', 'one', 'but', 'only', 'that', 'if', 'the', 'government', 'had', 'not', 'issued', 'the', 'regulation', 'it', 'would', 'have', 'got', 'more', 'under', 'its', 'contract', 'considerably', 'more', 'than', 'that', 'is', 'needed', 'before', 'a', 'promise', 'of', 'indemnity', 'from', 'the', 'government', 'can', 'be', 'implied', 'see', 'american', 'smelting', 'refining', 'co', 'v', 'united', 'states', 'u', 's', 'sup', 'ct', 'l', 'ed', 'may', 'judgment', 'affirmed']\n"
     ]
    }
   ],
   "source": [
    "print(make_row(texts[0]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "corpus = [make_row(x) for x in texts]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# operating off the basic tutorial here: \n",
    "# https://towardsdatascience.com/word-embedding-with-word2vec-and-fasttext-a209c1d3e12c\n",
    "# here goes nothin...\n",
    "from gensim.models import FastText\n",
    "lawvec_first_try = FastText(corpus, size=100, window=5, min_count=10, workers=4,sg=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# god only knows when this will finish, but I'd like to persist it when it does.\n",
    "# https://radimrehurek.com/gensim/models/keyedvectors.html\n",
    "word_vectors = lawvec_first_try.wv\n",
    "word_vectors.save(\"lawvec_first_try_vectors\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('reversed', 0.8952404260635376),\n",
       " ('affirming', 0.8114692568778992),\n",
       " ('vacated', 0.805119514465332),\n",
       " ('firmed', 0.7917196750640869),\n",
       " ('affir', 0.7770520448684692),\n",
       " ('dismissed', 0.773088812828064),\n",
       " ('judgment', 0.7602195739746094),\n",
       " ('reverser', 0.7573016881942749),\n",
       " ('appealss', 0.7538301944732666),\n",
       " ('reverses', 0.7528746128082275)]"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# it trained!  And only took like an hour! And there are two files, which are about 100 megs each holy wow.\n",
    "# let's see if it works.  \n",
    "word_vectors.most_similar(\"affirmed\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('agreement', 0.8729448318481445),\n",
       " ('ontract', 0.8552714586257935),\n",
       " ('contrac', 0.855073869228363),\n",
       " ('contracts', 0.8496118783950806),\n",
       " ('contracta', 0.8409367203712463),\n",
       " ('contractthe', 0.8360118269920349),\n",
       " ('contraction', 0.8354951739311218),\n",
       " ('contractually', 0.825141191482544),\n",
       " ('contractual', 0.8237504959106445),\n",
       " ('subcontract', 0.7991645932197571)]"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "word_vectors.most_similar(\"contract\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('ongress', 0.9122786521911621),\n",
       " ('congressnot', 0.8799211978912354),\n",
       " ('congressand', 0.8607625365257263),\n",
       " ('congres', 0.8511303663253784),\n",
       " ('congresso', 0.8462387323379517),\n",
       " ('congressthat', 0.8452374935150146),\n",
       " ('congressin', 0.8421956300735474),\n",
       " ('congresses', 0.8274695873260498),\n",
       " ('congresss', 0.8192774057388306),\n",
       " ('congressthe', 0.8118948936462402)]"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "word_vectors.most_similar(\"congress\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('egislation', 0.9143565893173218),\n",
       " ('legislationthe', 0.8688306212425232),\n",
       " ('legislating', 0.8595491647720337),\n",
       " ('legislated', 0.8259429931640625),\n",
       " ('legislat', 0.8166864514350891),\n",
       " ('legislates', 0.7770982980728149),\n",
       " ('legislate', 0.7757251858711243),\n",
       " ('islation', 0.7734143733978271),\n",
       " ('enactments', 0.7717022895812988),\n",
       " ('congress', 0.7686550617218018)]"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "word_vectors.most_similar(\"legislation\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('neversink', 0.6836391687393188),\n",
       " ('reversing', 0.6747123599052429),\n",
       " ('lowers', 0.6567318439483643),\n",
       " ('sucrose', 0.6534938812255859),\n",
       " ('inclining', 0.6525051593780518),\n",
       " ('circumference', 0.6448646783828735),\n",
       " ('former', 0.644787073135376),\n",
       " ('thus', 0.6428877115249634),\n",
       " ('fact', 0.642458975315094),\n",
       " ('thse', 0.6418173313140869)]"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "word_vectors.most_similar(positive=['judgment', 'below', \"lower\"], negative=['affirmed'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('tip', 0.1563124656677246),\n",
       " ('ussf', 0.11941806972026825),\n",
       " ('lockup', 0.08927061408758163),\n",
       " ('enveloping', 0.08607660979032516),\n",
       " ('purposeto', 0.07854565978050232),\n",
       " ('effort', 0.07158321887254715),\n",
       " ('handcuffs', 0.0702473372220993),\n",
       " ('liabil', 0.0661613866686821),\n",
       " ('overdahl', 0.06313156336545944),\n",
       " ('noncustodial', 0.061929717659950256)]"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "word_vectors.most_similar(positive=['judgment'], negative=['affirmed', 'affirm'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok, now I'd like to do the following: \n",
    "\n",
    "1.  Refine the cleanup function a bit (I'd like to keep stuff like section symbols and periods for abbreviations, I'll probably get rid of the ascii-ification and move this into a separate file)\n",
    "2.  Download a lot more data (cases first, other stuff later)\n",
    "3.  Figure out how to stream data into the fasttext fit.\n",
    "4.  Tune the fasttext fit a bit---maybe a narrower window, and lower vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from extract_text import get_cleaned_text # local"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'text' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-87-164580901381>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;32mwith\u001b[0m \u001b[0mopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mjsons\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mj\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m     \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mget_cleaned_text\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mjson\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mload\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mj\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m~/code/lawvec/extract_text.py\u001b[0m in \u001b[0;36mget_cleaned_text\u001b[0;34m(opinion_from_courtlistener)\u001b[0m\n\u001b[1;32m     33\u001b[0m     \u001b[0;32melif\u001b[0m \u001b[0mopinion_from_courtlistener\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'html_with_citations'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     34\u001b[0m         \u001b[0mtext\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mextract_text\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mopinion_from_courtlistener\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"html_with_citations\"\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 35\u001b[0;31m         \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"got citations\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     36\u001b[0m     \u001b[0;32melif\u001b[0m \u001b[0mopinion_from_courtlistener\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'extracted_by_ocr'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     37\u001b[0m         \u001b[0mtext\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mopinion_from_courtlistener\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'extracted_by_ocr'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/code/lawvec/extract_text.py\u001b[0m in \u001b[0;36mclean_text\u001b[0;34m(dirty_case)\u001b[0m\n\u001b[1;32m     14\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mclean_text\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdirty_case\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     15\u001b[0m     \u001b[0mdepunctuated\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mdirty_case\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtranslate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtransdict\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 16\u001b[0;31m     \u001b[0mdesentenced\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mre\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msub\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'\\.\\s+'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\" \"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdepunctuated\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;31m# idea is to try to keep most abbreviation periods (often not followed by spaces, esp. in legal text) and ditch sentence periods\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     17\u001b[0m     \u001b[0;31m# I'm also assuming there's no unicode punctuation other than section symbols, paragraph symbols, etc.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     18\u001b[0m     \u001b[0;32mreturn\u001b[0m \u001b[0mre\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msub\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'\\s+'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\" \"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdesentenced\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlower\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mNameError\u001b[0m: name 'text' is not defined"
     ]
    }
   ],
   "source": [
    "with open(jsons[1]) as j:\n",
    "    print(get_cleaned_text(json.load(j)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'dreload' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-88-75fdefb6f754>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mdreload\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mextract_text\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m: name 'dreload' is not defined"
     ]
    }
   ],
   "source": [
    "dreload(extract_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'reload' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-89-6089d73bc5fc>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;31m# ffs, I corrected this error and I can't get the fucking module to reload.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0;31m# aah, here we are.  https://support.enthought.com/hc/en-us/articles/204469240-Jupyter-IPython-After-editing-a-module-changes-are-not-effective-without-kernel-restart\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m \u001b[0mreload\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mextract_text\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m: name 'reload' is not defined"
     ]
    }
   ],
   "source": [
    "# ffs, I corrected this error and I can't get the fucking module to reload.\n",
    "# aah, here we are.  https://support.enthought.com/hc/en-us/articles/204469240-Jupyter-IPython-After-editing-a-module-changes-are-not-effective-without-kernel-restart \n",
    "reload(extract_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "TypeError",
     "evalue": "reload() argument must be a module",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-90-df5c2fa18b32>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mimportlib\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mimportlib\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mreload\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mextract_text\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m~/.local/share/virtualenvs/lawvec-yM8VjmAh/lib/python3.6/importlib/__init__.py\u001b[0m in \u001b[0;36mreload\u001b[0;34m(module)\u001b[0m\n\u001b[1;32m    137\u001b[0m     \"\"\"\n\u001b[1;32m    138\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mmodule\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodule\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtypes\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mModuleType\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 139\u001b[0;31m         \u001b[0;32mraise\u001b[0m \u001b[0mTypeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"reload() argument must be a module\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    140\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    141\u001b[0m         \u001b[0mname\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmodule\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__spec__\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mTypeError\u001b[0m: reload() argument must be a module"
     ]
    }
   ],
   "source": [
    "import importlib\n",
    "importlib.reload(extract_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# oh goodie, this doesn't work with local code without all the module ceremony.  For christ's sake, \n",
    "# I'm seriously just going to rename the local file.\n",
    "from get_clean_text import get_cleaned_text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " u.s s.ct l.ed pine hill coal co inc v.united states no argued jan decided may mr henry s drinker jr of philadephia pa for appellant argument of counsel from pages intentionally omitted mr assistant attorney general riter for the united states mr justice holmes delivered the opinion of the court this case like morrisdale coal co v united states u s sup ct l ed —— is a claim based upon the action of the fuel administration under the act of august c § stat comp st comp st ann supp § q fixing prices for coal the allegations and arguments however are different the transactions of the claimant from and including september through january are set forth in detail they embrace large sales at government prices and smaller sales at other than those prices it is alleged that the prices fixed for the claimant s coal were unjust and unreasonable and did not afford just compensation and that as a result of keeping to them as the claimant did the receipts were actually less than the cost of production on these facts the petition sets up a contract of indemnity on the part of the united states arising out of the language to be quoted from section it was dismissed on demurrer by the court of claims the paragraph of section that is relied upon follows paragraphs giving authority to the president personally or through the federal trade commission to fix the price of coal and coke to regulate the method of distribution among dealers and consumers during the war and if a producer or dealer neglects to conform to such prices or regulations etc to take over the plant and business paying a just compensation the paragraph in question reads that if the prices so fixed or if in the case of the taking over or requisitioning of the mines or business of any such producer or dealer the compensation therefor as determined by the provisions of this act be not satisfactory to the person or persons entitled to receive the same such person shall be paid seventy five per centum of the amount so determined and shall be entitled to sue the united states to recover such further sum as added to said seventy five per centum will make up such amount as will be just compensation in the manner provided by section twenty four paragraph twenty and section one hundred and forty five of the judicial code the latter section of the judicial code comp st § is the one that gives jurisdiction to the court of claims and the former that which gives a limited concurrent jurisdiction to the district courts it is obvious that the words as they stand cannot be applied to sales by producers to third persons for it would be absurd to suppose that the united states undertook to pay not only such additional sum as might be awarded but also the last per centum of the price as fixed leaving the buyer to retain that amount the claimant admits this but insists that however read the pragraph cannot be followed without correction it argues that the opening words if the price so fixed necessarily apply to prices in general as fixed by the power just given in the section therefore it says there should be interpolated in the provision that the seller shall be paid per centum the words the prices so fixed or and in like manner that the provision for recovery should read that he shall recover such sum as added to the said prices or said per centum will be just it points out that while seeking to stimulate production in aid of the war the government could not fix very high prices without arousing householders and manufacturers or very low ones without endangering the supply and incurring the charge of confiscation it is said that the natural way out of the difficulty was for the government to guarantee a just return and that by so doing it avoided doubts as to the constitutionality of the statute there is offered a critical and refined scrutiny of the history of the amendment that introduced the claim the argument is that the section that became section when originally offered as an amendment clearly provided for payment in all cases that a modification was introduced for payment of only per centum upon takings by the united states but that it was not intended to change the general scope of the relief other makeweights are thrown in to which we think it unnecessary to advert it is a delicate business to base speculations about the purposes or construction of a statute upon the vicissitudes of its passage here we have as against the arguments of the claimant the fundamental and necessarily governing consideration that rightly prevailed below a liability in any case is not to be imposed upon a government without clear words but liability for a regulation for the consequences of a law on the part of the legislating power is most unusual and where as here the liability would mount to great sums only the plainest language could warrant a court in taking it to be imposed the general words the prices so fixed taken by themselves no doubt would include prices to private purchasers but the specific provision as to paying per centum prevails over them on the usual principles of construction and excludes a reference to any prices except those paid by the government it is said that those prices are provided for elsewhere but the claimant s argument presses the consideration that the law had to be hastily passed and unnecessary reduplication is far more easy to admit than an enormous charge upon the united states that can be fastened upon it only by inserting into a statute words that are not there judgment affirmed \n"
     ]
    }
   ],
   "source": [
    "with open(jsons[1]) as j:\n",
    "    print(get_cleaned_text(json.load(j)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " u.s s.ct l.ed morrisdale coal co.v.united states no argued jan decided may messrs gibbs l baker and karl knox gartner both of washington d c for appellant mr assistant attorney general riter for the united states mr justice holmes delivered the opinion of the court this is an appeal from a judgment of the court of claims dismissing the appellant s petition upon demurrer the petition alleges that the claimant had outstanding contracts calling for more than the actual production of its mines for the months of june and following through november at a price of per gross ton that the fuel administration appointed by the president during the war requisitioned and compelled petitioner to divert tons of coal during the period mentioned that the price received for this coal was per gross ton and that the claimant thereby suffered a loss of for which loss it asks judgment against the united states the petition does not allege or mean that the united states took the coal to its own use the meaning attributed to it by the claimant is merely that the fuel administration fixed the price on coal of this quality at per gross ton and issued orders from time to time directing coal to such employments as best would promote the prosecution of the war the fuel administration acted under a delegation from the president of the power conferred upon him by the act of august c § stat comp st comp st ann supp § q to fix the price of coal and to regulate distribution of it among dealers and consumers the price so fixed not to invalidate contracts previously made in good faith in which prices are fixed stat the claimant does not argue that this section provides compensation for obedience to orders made in pursuance of the same it agrees and rightly that its remedy if any is under section of the judicial code comp st § giving the court of claims jurisdiction of claims upon any contract express or implied with the government it contends that upon the facts stated a contract on the part of the government must be implied both from the statute and by virtue of the fifth amendment on the ground that its property was taken for public use we see no ground for the claim the claimant in consequence of the regulation mentioned sold some of its coal to other parties at a less price than what otherwise it would have got that is all it now seeks to hold the government answerable for making a rule that it saw fit to obey whether the rule was valid or void no such consequence follows making the rule was not a taking and no lawmaking power promises by implication to make good losses that may be incurred by obedience to its commands if the law requires a party to give up property to a third person without adequate compensation the remedy is if necessary to refuse to obey it not to sue the lawmaker the statute provides remedies against the government in other cases but the claimant argues that this case does not fall within them and it did not follow the steps prescribed for them the petition does not even allege that the price the claimant got was not a fair one but only that if the government had not issued the regulation it would have got more under its contract considerably more than that is needed before a promise of indemnity from the government can be implied see american smelting refining co v united states u s sup ct l ed —— may judgment affirmed \n",
      " u.s s.ct l.ed pine hill coal co inc v.united states no argued jan decided may mr henry s drinker jr of philadephia pa for appellant argument of counsel from pages intentionally omitted mr assistant attorney general riter for the united states mr justice holmes delivered the opinion of the court this case like morrisdale coal co v united states u s sup ct l ed —— is a claim based upon the action of the fuel administration under the act of august c § stat comp st comp st ann supp § q fixing prices for coal the allegations and arguments however are different the transactions of the claimant from and including september through january are set forth in detail they embrace large sales at government prices and smaller sales at other than those prices it is alleged that the prices fixed for the claimant s coal were unjust and unreasonable and did not afford just compensation and that as a result of keeping to them as the claimant did the receipts were actually less than the cost of production on these facts the petition sets up a contract of indemnity on the part of the united states arising out of the language to be quoted from section it was dismissed on demurrer by the court of claims the paragraph of section that is relied upon follows paragraphs giving authority to the president personally or through the federal trade commission to fix the price of coal and coke to regulate the method of distribution among dealers and consumers during the war and if a producer or dealer neglects to conform to such prices or regulations etc to take over the plant and business paying a just compensation the paragraph in question reads that if the prices so fixed or if in the case of the taking over or requisitioning of the mines or business of any such producer or dealer the compensation therefor as determined by the provisions of this act be not satisfactory to the person or persons entitled to receive the same such person shall be paid seventy five per centum of the amount so determined and shall be entitled to sue the united states to recover such further sum as added to said seventy five per centum will make up such amount as will be just compensation in the manner provided by section twenty four paragraph twenty and section one hundred and forty five of the judicial code the latter section of the judicial code comp st § is the one that gives jurisdiction to the court of claims and the former that which gives a limited concurrent jurisdiction to the district courts it is obvious that the words as they stand cannot be applied to sales by producers to third persons for it would be absurd to suppose that the united states undertook to pay not only such additional sum as might be awarded but also the last per centum of the price as fixed leaving the buyer to retain that amount the claimant admits this but insists that however read the pragraph cannot be followed without correction it argues that the opening words if the price so fixed necessarily apply to prices in general as fixed by the power just given in the section therefore it says there should be interpolated in the provision that the seller shall be paid per centum the words the prices so fixed or and in like manner that the provision for recovery should read that he shall recover such sum as added to the said prices or said per centum will be just it points out that while seeking to stimulate production in aid of the war the government could not fix very high prices without arousing householders and manufacturers or very low ones without endangering the supply and incurring the charge of confiscation it is said that the natural way out of the difficulty was for the government to guarantee a just return and that by so doing it avoided doubts as to the constitutionality of the statute there is offered a critical and refined scrutiny of the history of the amendment that introduced the claim the argument is that the section that became section when originally offered as an amendment clearly provided for payment in all cases that a modification was introduced for payment of only per centum upon takings by the united states but that it was not intended to change the general scope of the relief other makeweights are thrown in to which we think it unnecessary to advert it is a delicate business to base speculations about the purposes or construction of a statute upon the vicissitudes of its passage here we have as against the arguments of the claimant the fundamental and necessarily governing consideration that rightly prevailed below a liability in any case is not to be imposed upon a government without clear words but liability for a regulation for the consequences of a law on the part of the legislating power is most unusual and where as here the liability would mount to great sums only the plainest language could warrant a court in taking it to be imposed the general words the prices so fixed taken by themselves no doubt would include prices to private purchasers but the specific provision as to paying per centum prevails over them on the usual principles of construction and excludes a reference to any prices except those paid by the government it is said that those prices are provided for elsewhere but the claimant s argument presses the consideration that the law had to be hastily passed and unnecessary reduplication is far more easy to admit than an enormous charge upon the united states that can be fastened upon it only by inserting into a statute words that are not there judgment affirmed \n",
      " u.s s.ct l.ed santa fe pac r co.v.payne secretary of the interior two cases nos argued january decided may mr francis w clements of washington d c for appellant mr assistant attorney general riter for appellee mr justice holmes delivered the opinion of the court these are two bills in equity brought in respect of different parcels of land but seeking the same kind of relief against the secretary of the interior and raising the same question of law the facts are simple under the land grant to its predecessor the atlantic pacific railroad company the santa fe pacific railroad company was the owner of coal lands in new mexico by the act of april c § stat the first named road and its successors may when requested by the secretary of the interior so to do relinquish or deed as may be proper to the united states any sections of their land grant in new mexico any portion of which was and had been occupied by a settler as a homestead for not less than years and shall then be entitled to select in lieu thereof and to have patented other sections of vacant public land of equal uqality in said territory as may be agreed upon with the secretary of the interior under this act at the request of the secretary of the interior the railroad company relinquished specified tracts of coal land and on may selected other tracts also of coal land after the selections questions were raised as to the value of the selected lands and ultimately after some years the selections were rejected on the ground of the greater value of the latter lands as shown by investigations since the choice thereupon the railroad company brought these bills to enjoin the secretary of the interior from cancelling its selections and from taking further action except to issue patents to the company for the selected lands the bills were dismissed on motion by the courts below app d c fed the government argues that there was no jurisdiction over the bills because the question whether the lands selected were of the same quality as those relinquished rested wholly in the judgment of the secretary but the position of the railroad company is that the secretary went beyond the powers conferred upon him by the statute when he took into account facts not known at the time of the selection and we are of opinion that the company is entitled to bring that question into court we are of opinion also that the company s position is right at first sight the words of the statute entitling the company to have patented other sections of equal quality as may be agreed upon with the secretary of the interior might be taken to sustain the decision below but upon consideration they seem to us not to have that effect the moment that lands were relinquished at the request of the secretary a contract was made and the government was bound to convey to the company such vacant lands within the territory as the company should select provided only that they were of equal quality in theory of law the obligation was immediate when the selection was made if it complied with the condition it is true that the secretary had to be satisfied upon that point but his discretion was not arbitrary it went only to the quality of the lands if as chief justice shaw put it a piepoudre court could have been summoned and the matter determined forthwith the secretary would have been bound to act on the facts as they then appeared and could not have elected to wait for better days at that time may the only relevant classification in the statutes we believe was of coal lands within miles of a railroad valued at not less than per acre and those more than miles from one valued at not less than per acre rev stats § comp st § the department through the geological survey had classified further and had valued the products in all the lands concerned at not less than per acre these were all the elements for decision when the selection was made and if the secretary had been required to proceed at once as the statute evidently contemplated that he would section he would have been bound to agree to the company s choice indeed in the first case he did agree to it and did not attempt to revoke his decision until more than two years later on the ground of subsequently discovered facts it is established in the parallel cases of payne v central pacific ry co u s sup ct l ed payne v new mexico u s sup ct l ed and wyoming v united states u s sup ct l ed that the validity of the selection must be determined according to the conditions existing at the time when it was made these decisions were later than that in the court below and show without the need of further argument that the decrees must be reversed decrees reversed \n",
      " u.s s.ct l.ed federal base ball club of baltimore inc v.national league of professional base ball clubs et al no argued april decided may messrs charles a douglas of washington d c wm l marbury of baltimore md and william l rawls hugh h obear jo v morgan and charles s douglas all of washington d c and l edwin goldman of baltimore md for plaintiff in error argument of counsel from pages intentionally omitted messrs george wharton pepper of philadelphia pa and benjamin s minor of washington d c for defendants in error mr justice holmes delivered the opinion of the court this is a suit for threefold damages brought by the plaintiff in error under the anti trust acts of july c § stat comp st § and of october c § stat comp st § d the defendants are the national league of professional base ball clubs and the american league of professional base ball clubs unincorporated associations composed respectively of groups of eight incorporated base ball clubs joined as defendants the presidents of the two leagues and a third person constituting what is known as the national commission having considerable powers in carrying out an agreement between the two leagues and three other persons having powers in the federal league of professional base ball clubs the relation of which to this case will be explained it is alleged that these defendants conspired to monopolize the base ball business the means adopted being set forth with a detail which in the view that we take it is unnecessary to repeat the plaintiff is a base ball club incorporated in maryland and with seven other corporations was a member of the federal league of professional base ball players a corporation under the laws of indiana that attempted to compete with the combined defendants it alleges that the defendants destroyed the federal league by buying up some of the constituent clubs and in one way or another inducing all those clubs except the plaintiff to leave their league and that the three persons connected with the federal league and named as defendants one of them being the president of the league took part in the conspiracy great damage to the plaintiff is alleged the plaintiff obtained a verdict for in the supreme court and a judgment for treble the amount was entered but the court of appeals after an elaborate discussion held that the defendants were not within the sherman act the appellee the plaintiff elected to stand on the record in order to bring the case to this court at once and thereupon judgment was ordered for the defendants national league of professional baseball clubs v federal baseball club of baltimore fed app d c it is not argued that the plaintiff waived any rights by its course thomsen v cayser u s sup ct l ed ann cas d the decision of the court of appeals went to the root of the case and if correct makes it unnecessary to consider other serious difficulties in the way of the plaintiff s recovery a summary statement of the nature of the business involved will be enough to present the point the clubs composing the leagues are in different cities and for the most part in different states the end of the elaborate organizations and sub organizations that are described in the pleadings and evidence is that these clubs shall play against one another in public exhibitions for money one or the other club crossing a state line in order to make the meeting possible when as the result of these contests one club has won the pennant of its league and another club has won the pennant of the other league there is a final competition for the world s championship between these two of course the scheme requires coustantly repeated travelling on the part of the clubs which is provided for controlled and disciplined by the organizations and this it is said means commerce among the states but we are of opinion that the court of appeals was right the business is giving exhibitions of base ball which are purely state affairs it is true that in order to attain for these exhibitions the great popularity that they have achieved competitions must be arranged between clubs from different cities and states but the fact that in order to give the exhibitions the leagues must induce free persons to cross state lines and must arrange and pay for their doing so is not enough to change the character of the business according to the distinction insisted upon in hooper v california u s sup ct l ed the transport is a mere incident not the essential thing that to which it is incident the exhibition although made for money would not be called trade of commerce in the commonly accepted use of those words as it is put by defendant personal effort not related to production is not a subject of commerce that which in its consummation is not commerce does not become commerce among the states because the transportation that we have mentioned takes place to repeat the illustrations given by the court below a firm of lawyers sending out a member to argue a case or the chautauqua lecture bureau sending out lecturers does not engage in such commerce because the lawyer or lecturer goes to another state if we are right the plaintiff s business is to be described in the same way and the restrictions by contract that prevented the plaintiff from getting players to break their bargains and the other conduct charged against the defendants were not an interference with commerce among the states judgment affirmed \n",
      " u.s s.ct l.ed mutual life ins co of new yorkv.liebing no argued april decided may messrs wm marshall bullitt of louisville ky frederick d mckenney of washington d c john h holliday of st louis mo and frederick l allen of new york city for plaintiff in error argument of counsel from pages intentionally omitted mr james j o donohoe of st louis mo for defendant in error mr justice holmes delivered the opinion of the court this is a suit to recover upon a policy insuring the life of one blees issued to him and subsequently assigned by him to his wife now mrs liebing the plaintiff defendant in error the contract was made on september by the defendant the plaintiff in error in missouri by a delivery of the policy to blees in macon missouri where he lived three annual premiums were paid after the fourth was due within the time allowed blees and his wife signed an application for a loan of and sent it with the policy to the defendant s agency at st louis by which it was forwarded to new york the application followed the terms of the policy which agreed that after it had been in force three years the company would lend amounts within the cash surrender value upon certain conditions the policy being assigned as security following these terms the application deducted from the cash to be received the fourth annual premium and an adjustment of interest leaving the balance to be paid the loan was to be for one year and the application authorized the company upon default to cancel the policy and apply the customary cash surrender consideration to the payment of the loan the application was approved in new york and a check for to the order of mr and mrs blees with a receipt for the fourth premium was sent from new york to the company s manager in st louis and by him forwarded to a local agent who delivered the documents to blees the check was endorsed and paid a year later when repayment was due it was not made thereupon on december the company cancelled the policy and applied the surrender value to the loan which was of equal amount leaving a deficit of interest blees died on september and upon inquiry from mrs blees the company notified her of what had been done its action had been in accordance with the terms of its contract and the law of new york but some years later mrs blees now mrs liebing brought the present action relying upon the revised statutes of missouri § set forth and considered in new york life insurance co v dodge u s sup ct l ed ann cas e and after a previous decision the other way she recovered by the final judgment of the supreme court of the state s w the missouri statute provided that such policies as the present after three annual payments should not become void for non payment of premiums but that three fourths of the net value of the policy after deducting certain liabilities should be taken as a premium for temporary insurance for the full amount written in the policy it is not disputed that if this statute governs the case the plaintiff stood as having a policy for the original amount at the death of mr blees in new york life insurance co v dodge u s sup ct l ed ann cas e it was held that when the later transaction was consummated in new york missouri could not prohibit a citizen within her borders from executing it but if the later contract was made in missouri then by the present and earlier decisions notwithstanding any contrary agreement the statute does govern the case see u s sup ct l ed ann cas e the policy now sued upon contained a positive promise to make the loan if asked whereas in the one last mentioned it might be held that some discretion was reserved to the company for here the language is the company will loan amounts within the limits of the cash surrender value etc whereas there it was cash loans can be obtained on this distinction the missouri court seems to have held that as soon as the application was delivered to a representative of the company in missouri the offer in the policy was accepted and the new contract complete and therefore subject to missouri law if however the application should be regarded as only an offer the effective acceptance of it did not take place until the check was delivered to blees which again was in missouri where he lived in whichever way regarded the facts lead to the same conclusion and although the circumstances may present some temptation to seek a different one by ingenuity the constitution and the first principles of legal thinking allow the law of the place where a contract is made to determine the validity and the consequences of the act judgment affirmed \n",
      " u.s s.ct l.ed united statesv.southern pac co no reargued april and decided may syllabus from pages intentionally omitted messrs edward f mcclennen of boston mass and solicitor general beck of washington d c for the united states arguments of counsel from pages intentionally omitted mr garrett w mcenerney and mr joseph p blair with whom wm f herrin was on the briefs for appellees arguments of counsel from pages intentionally omitted mr perry d trafford and mr james gore king filed a brief on hehalf of the union now central union trust company of new york appellee mr justice day delivered the opinion of the court the united states on february filed its bill in the district court of the united states for the district of utah against the southern pacific company the central pacific railway company the union trust company of new york and the directors and officials of the southern pacific company the charge of the petition is that the defendants restrain or attempt to monopolize and do monopolize trade and commerce in violation of the act of july known as the sherman act comp st §§ and have also violated the provisions of the act of congress of july stat the prayer of the petition is that the lines of the southern pacific company and those of the central pacific railway company be decreed to constitute competitive systems and that the ownership acquired by the southern pacific company of all or a controlling interest in the capital stock of the central pacific railway company and its lease control and operation of the lines thereof be declared violative of the sherman act that the southern pacific company be required to dispose of such capital stock and cancel and relinquish its lease control management and operation thereof and that the control of the central pacific railway company by the southern pacific be decreed to be in violation of the act of congress of july entitled an act to aid in the construction of a railroad and telegraph line from the missouri river to the pacific ocean and to secure to the government the use of the same for postal military and other purposes and also violative of the act supplemental to the act of the act of july stat and the act of june stat comp st § the government miantaining that the effect of such acts is to require the central pacific to maintain physical connection with the union pacific to make a through line to the coast and to furnish equal advantages and facilities as to rates time and transportation over such through line an answer was filed by the defendants much testimony was taken and a decree was entered dismissing the petition one of the three circuit judges who heard the case dissenting fed the central pacific railroad company of california was incorporated under the laws of california in for the purposes of constructing a railroad from sacramento to the eastern boundary of california in and congress by proper legislation incorporated the union pacific railroad company to build from the missouri river westward and authorized the central pacific to build eastwardly from the pacific coast at or near san francisco to a common meeting point with the union pacific these acts of congress authorized the issue of first mortgage bonds and also second mortgage bonds and made a land grant of public lands for each linear mile of railroad construction these acts provided that these two railroads should be operated as one continuous line and that neither should discriminate in favor of or against the other leland stanford charles crocker c p huntington and mark hopkins acquired a large part of the capital stock of the central pacific company the central pacific assigned to the western pacific a portion of the construction namely that from sacramento to san jose this with the approval of congress chapter § stat the pacific railroads were constructed from to from the missouri river to the pacific coast—from omaha to ogden by the union pacific from ogden to sacramento by the central pacific from sacramento to san jose by the western pacific afterwards consolidated with the central pacific these are denominated in the defendants brief as the bond aided lines what they call the nonbond aided lines of the central pacific system are those from niles to oakland from lathrop to goshen and from roseville to redding which were constructed in the state of california from the years to in the central pacific absorbed in consolidation the western pacific which built from sacramento to san jose the alameda company which built from niles to oakland the san joaquin which built from lathrop to goshen and the california oregon company which built from roseville north en route to the oregon line the southern pacific railroad company was incorporated in under the laws of california for the purpose of constructing a railroad from san francisco bay by the way of san diego to the eastern boundary of california in congress passed an act to incorporate the atlantic pacific railroad company to construct a railroad near the thirty fifth parallel of latitude from springfield mo to the pacific ocean this act authorized the southern pacific to connect with the atlantic pacific near the eastern boundary of california and both companies were granted public lands in the southern pacific changed its route to the eastward so as not to go as far south as originally contemplated this act was ratified by congress and the legislature of california in in congress incorporated the texas pacific railroad company to build a line of railroad near the thirty second parallel of latitude from marshall tex by the way of el paso to the pacific ocean at san diego and to connect on the east with other railroads and on the west with the southern pacific railroad the southern pacific was authorized to construct a railroad from tehachapi pass to a junction of the texas pacific railroad at the eastern boundary of california land grants were made to both companies in the central pacific had extended its lines to goshen about the promoters of the central pacific obtained control of the southern pacific and subsequently the latter was constructed from goshen through tehachapi pass with one fork to a junction with the atlantic pacific at the needles near thirty fifth parallel on the colorado river at the eastern boundary of the state and the other fork to the southeastern corner of the state near thirty second parallel thence across arizona and new mexico to a junction in texas with the texas pacific thence to a connection at el paso with the galveston harrisburg san antonio railroad the sections of the southern line were leased for a series of years to the central in the southern pacific made a junction with the atchison topeka santa fe at demming n m in the southern made a junction with the texas pacific at sierra blanca tex in the direct line of the southern connection with the galveston harrisburg san antonio and its eastern connections was completed through to new orleans the same year it made its junction with the atlantic pacific at the needles the section from mohave to needles was leased to the santa fe in and sold to it in from to the central pacific was the lessee owner of a system of leases and the system was known as the central pacific railroad and leased lines in february after the formation of the southern pacific company of kentucky that road became the lessee we shall have occasion to deal more particularly with that lease later without familiarity with the geography of the region described and the location of the points named this description means little the outstanding facts and those essential to be considered in the view which we take of this branch of the case are the central pacific railroad extends from the bay of san francisco to ogden utah with a branch extending north from roseville in central california to the northern boundary of california and to kirk in oregon and a branch extending south from lathrop in central california to goshen cal and a branch extending south from hazen in nevada to mojave in california and a branch from fernley in nevada to susanville cal and a short line in oregon from oakridge to natron at ogden the central pacific connects with the union pacific extending to omaha neb and to kansas city mo connecting at ogden with the denver rio grande railroad and with other connecting roads eastwardly to the missouri river and from the missouri river to the central and eastern parts of the united states the southern pacific system extends from san francisco bay by way of el paso to galveston tex and to new orleans there connecting with steamship lines to new york city controlled by the southern pacific at el paso it connects with the rock island which runs to omaha and chicago at new orleans it connects with roads extending to points in the central and eastern parts of the united states it owns branches in texas arizona new mexico oregon and many in california the central pacific with its eastern connection at ogden forms one great system of transportation between the east and the west and the southern pacific with its roads and connections and steamboat lines forms another great transcontinental system for transportation from coast to coast the central pacific constitutes some miles of the transcontinental line of which it is a part the southern pacific system has practically its own line of railroads and steamboat connections to new york via galveston and new orleans under principles settled in the union pacific case u s sup ct l ed the acquisition by the southern pacific company of the stock of the central pacific railway company in unless justified by the special circumstances relied upon to be hereinafter considered constituted a combination in restraint of trade because it fetters the free and normal flow of competition in interstate traffic and tends to monopolization in the union pacific case this court held that the acquisition by the union pacific which constituted about miles of the transcontinental system to which we have referred of enough stock in the southern pacific to dominate and control it was violative of the sherman act this case differs from that not at all in principle these two great systems are normally competitive for the carrying trade in some parts from the east and middle west to the coast and for the traffic moving to and from central and northern california including a great volume of ocean borne traffic which lands on the coast destined across the continent to the atlantic seaboard and intermediate western and eastern points or is destined from the latter points to foreign ports via san francisco or other pacific coast points counsel for the defendants evidently realizing this situation make elaborate argument to distinguish the union pacific case the claim is made that the decision there rested only on the fact that a then existing competition was restrained through the purchase by the union pacific of the control of the southern pacific in but the principle of that decision and of the previous cases upon which it rested was broader than the mere effect upon existing competition between the two systems such combinations not the result of normal and natural growth and development but springing from the formation of holding companies or stock purchases resulting in the unified control of different roads or systems naturally competitive constitute a menace and a restraint upon that freedom of commerce which congress intended to recognize and protect and which the public is entitled to have protected northern securities co v united states u s sup ct l ed this principle was restated and applied in united states v union pacific co supra it was reiterated and approved by the court as recently as the october term united states v reading co u s sup ct l ed these cases collectively establish that one system of railroad transportation cannot acquire another nor a substantial and vital part thereof when the effect of such acquisition is to suppress or materially reduce the free and normal flow of competition in the channels of interstate trade in the instant case we are not dealing with the principle in the abstract the proof is ample that the policy of the southern pacific system has been to favor transportation on its line by securing for itself whenever practicable the carriage of freight which would normally move eastward or westward over the shorter line of the central pacific railroad and its connections for its own much longer and wholly owned southern route this course was limited by an arbitrary rule during the time the union pacific dominated the southern pacific from the stock purchase in until the so called unmerger in as a result of the decision of this court in the union pacific case the compelling motive of this course of conduct is obvious the southern pacific owns and controls the southerly route and receives per cent of the compensation for freight transported by its road and water lines over the central pacific route it receives but a fraction of the freight because the union pacific with its eastern connections take up the carrying from ogden to the east self interest dictates the solicitation and procurement of freight for the longer haul by the southern pacific lines while many practices formerly in vogue are eliminated by the legislation of congress regulating interstate commerce and through rates and transportation may be had under public supervision there are elements of competition in the granting of special facilities the prompt carrying and delivery of freight the ready and agreeable adjustment and settlement of claims and other elements which that legislation does not control it is conceded in the brief of counsel for the defendants that it is true of all such systems that other things being equal freight is preferentially solicited for the per cent haul we reach the conclusion that the stock ownership in the central pacific acquired by the southern pacific is violative of the sherman act within the principles settled by this court certainly since the decision in the northern securities case in and that such stock ownership must be divested from the southern pacific company unless the special circumstanses and defenses set up and relied upon by the defendants are to prevail in the opinion of the majority of the judges sitting in the district court it was set forth that these companies the southern pacific and the central pacific constituted practically a single system of railroads this was held to be particularly true of so much of the systems as are in california and oregon it was said that the roads of the central pacific system appear on the map as natural links and parts of the southern pacific system and that the spurs branches and tributary feeders of the central pacific belong to the southern pacific it was maintained that the construction and control of these systems had substantially united them before the acts complained of which are alleged to be violative of the sherman act true the central pacific was incorporated by and for a portion of the time under consideration its stock was owned by messrs stanford hopkins huntington and crocker perhaps as early as the same group gained control of anc continued to dominate the policy of the southern pacific the roads were always separate and distinct corporations they were so recognized in the acts of congress making land grants to them authorizing their construction and operation from one state or territory to another and otherwise conferring rights on them which only congress could confer for a good part of the time the roads had boards of directors not consisting of the same persons at times the majority of the stock was separately held in the central pacific when the lease of to the southern pacific was made only one fourth of the stock was held by the group to which we have referred it had been sold and was widely owned in the united states and europe the dominating control was maintained from the fact that the stock had not been transferred by its true owners on the company s books and much of it was held in the name of employees who were used in voting it by the original promoters their successors and survivors we cannot accept the theory of prior practical consolidation as a justification for a violation of the sherman act resulting from the stock control acquired in much stress is placed on the lease in february of the central pacific to the southern pacific for a term of years in the southern pacific a holding company was organized as a corporation of the state of kentucky the organization of this company which acquired the stocks of the southern pacific system and became the lessee of the central pacific was the result of a meeting in new york of messrs stanford huntington crocker and timothy hopkins the successor of mark hopkins the plan was then discussed and the necessary measures directed to carry it into execution the lease of is set up in the answer and relied upon as showing an existing legal acquisition before the transfer of the central pacific stock in this lease made february was modified in january and on december a lease was entered into which recited that the agreements of lease between the same parties the southern pacific and the central pacific dated february and january respectively should be canceled except in so far as they relate to the operation of the demised premises prior to january and to the adjustment of accounts in respect to such operation it is contended by the government that this lease in itself constituted a combination in restraint of interstate commerce however this may be this court has repeatedly recognized the fact that the central pacific was a corporation receiving much of its authroity and power from acts of congress california v pacific railroad co u s sup ct l ed in central pacific railroad co v california u s sup ct l ed it was held by this court that on the return for taxation by the central pacific railroad company of the value of its franchise and roadway roadbed and rails within the state of california the same might be taxed under the laws of that state this conclusion was reached against the elaborately stated and strongly expressed dissents of mr justice field and mr justice harlan in the prevailing opinion delivered by mr chief justice fuller it was recognized u s page sup ct l ed that important franchises conferred upon the central pacific were of federal creation including that of constructing a railroad from the pacific ocean to ogden in the then territory of utah it is true as is argued at length by counsel for the defendants that congress had opportunity by the reports of its committees and otherwise to learn of this lease but we are referred to no legislation passed by congress authorizing or approving of it in our view the lease for years by the central pacific to a rival and competitive company could not legally be made without authorization by federal legislation in the absence of such action the central pacific had not the corporate capacity to make the lease pennsylvania railroad co et al v st louis alton terre haute railroad co u s sup ct l ed central transportation co v pullman s palace car co u s sup ct l ed moreover it is authoritatively settled by decisions of this court that no previous contracts or combinations can prevent the application of the sherman act to compel the discontinuation of illegal combinations after congress exercises its authority to regulate interstate commerce conduct becomes illegal which has the effect of contracts conspiracies or combinations to restrain the freedom of interstate trade or to monopolize the same in whole or in part addyston pipe steel co v united states u s sup ct l ed the principle has often been declared and applied in this court it is stated and the previous cases reviewed in philadelphia baltimore washington railroad co v schubert u s sup ct l ed we find nothing in these leases to the southern pacific company which justifies the continued control of the central pacific by the southern pacific after the sherman act became effective we come now to the settlement of the central pacific debt in which the court below held to be a practical construction of the sherman act and to warrant the conclusion that the southern pacific control of the central pacific was not within its condemnation after hearings and reports and attempted legislation congress passed the act to create a commission to settle the indebtedness of the central pacific and western pacific railroads to the united states this act was passed july stat c p and constituted the secretary of the treasury the secretary of the interior and the attorney gneral a commission with full power to settle the indebtedness to the government growing out of the issue of bonds in aid of construction of the central pacific and western pacific bond aided roads upon such terms and in such a manner as might be agreed upon between them and the owners of said road the act also provided that the settlement should not be binding until approved by the president of the united states that the commission should not agree to accept a less sum than the full amount of principal and interest and all amounts necessary to reimburse the united states for moneys paid for interest or otherwise it provided that the rate of interest upon installments should be not less than per cent per annum payable semiannually with such security as the commission might deem expedient that the final discharge of the indebtedness should not be postponed beyond years that the whole amount principal and interest should be paid in equal semiannual installments within that period that any settlement made should provide that if a default were made in the payment of either principal or interest the whole sum and all installments should immediately become due and payable that unless the settlement authorized should be perfected within one year the president of the united states should at once proceed to foreclose all liens held by the united states against the railroad companies to collect the indebtedness sought to be settled under the act and that nothing therein contained should be held to waive or release any right lien or cause of action held by the united states under this act a settlement was effected as of date february the central pacific s debt to the united states for government aid in the construction of lines between sacramento and ogden and sacramento and san jose amounted to one half of this amount was accrued interest it was secured by a statutory lien on the bondaided lines subject to prior first mortgages the central pacific s bonded debt amounted to largely secured by first mortgages on its various lines of railroad the outstanding stock was messrs speyer co new york bankers undertook to formulate the plan and the agreement of settlement was signed by the commissioners in behalf of the united states the central pacific railroad company and messrs speyer co it was approved by the president by the agreement of settlement the central pacific was to execute to the united states promissory notes dated february payable respectively on or before each months for years each note for the sum of being one twentieth of the debt bearing interest at per cent payable semiannually all to mature on default in payment of any one of them under the agreement gold bonds not exceeding were to be issued secured by first mortgage on all the central pacific lines bond aided or not this mortgage to be prior in lien to any lease of the railroads of the central pacific railroad company the bonds were secured by the guaranty of the southern pacific no such agreement of guaranty was embodied in the written settlement but it was known to the commissioners that the plan contemplated such guaranty of these bonds were to be deposited with the treasurer of the united states as security for the installment notes speyer co within one month after the execution and delivery of the notes were to purchase from the united states the notes first maturing by paying the face thereof for the same and interest a proportionate amount of the collateral mortgage bonds was to go with the notes on february the commission reported the agreement to the house of representatives no reference to the guaranty of the southern pacific upon the bonds appeared in the report in the annual report of the attorney general to the senate and house of november the completion of the settlement which had been made was set out and the guaranty of the southern pacific was stated no doubt by inadvertence to be upon the notes instead of upon the bonds the notes held in the treasury of the united states were paid primarily by the checks of the southern pacific and charged by that road against the central pacific on march stat congress authorized the secretary of the treasury to dispose of any notes in his possession touching the indebtedness of the central pacific railroad company to the united states on march stat comp st § the secretary of the treasury was authorized and directed to settle claims for interest growing out of transportation services for the government over nonbond aided lines of the southern pacific and the central pacific by crediting the amounts on the central pacific notes neither the agreement between the commissioners the railroad company and speyer co nor the report of the commission to congress contained any reference to the proposed acquisition of the stock of the central pacific by the southern pacific the speyer plan for the adjustment of the affairs of the central pacific was dated february was put out february th of the same year was extensively published in american and european financial circles and was given publicity in the commercial and financial chronicle in the february issues of that journal under the terms of the plan the central pacific railway company successors to the central pacific railroad company was organized as a corporation of the state of utah on july the central pacific railroad company the old corporation of conveyed all of its property to the new company on august the central pacific railway company executed a refunding mortgage of to the central trust company of new york trustee and a mortgage of to the union trust company of new york the southern pacific company executed instruments subordinating its lease to the lien of these mortgages thereupon carrying out the speyer plan for the southern pacific to acquire the stock of the new central pacific railway company of the preferred shares of the latter company were issued which were taken by the southern pacific at par the outstanding stock of the old central pacific was taken by the southern pacific share for share plus per cent in the bonds of the southern pacific to consummate this transaction southern pacific mortgage bonds amounting to were issued thereof were used to acquire the new central pacific preferred stock the balance to provide the per cent in bonds required to aid in the share for share exchange of the outstanding central pacific stock in the hands of private owners thus the southern pacific under the speyer plan was to become the owner of the central pacific railway company stock in the opinion of the district court it is said we do not say that the commission was authorized to violate or to sanction the violation of the act of congress but the adjustment they effected necessarily involved the question of its pertinence to the business in hand the acceptance of the guaranty of the southern pacific was a recognition that it had sufficient corporate interest in the central pacific to justify it without such interest its accommodation guaranty of of bonds of another company would manifestly have been ultra vires—a gross indefensible excess of its corporate powers again the acceptance of the guaranty implied a recognition of its possible natural result that is to say the enforcements of the rights of a guarantor against the property of a debtor the addition of the stock ownership by the southern pacific to its long leasehold interest did not so change the situation as to make unlawful what was not so before we are unable to accept this view the commission with the approval of the president was authorized to settle the central pacific debt in accordance with the terms of the act of it did not undertake to exercise authority not conferred upon it by giving immunity from the penalties of the sherman act the attorney general testified that the act was not mentioned in the course of the discussion the southern pacific company s guaranty of the new bonds was made so far as that company was concerned from motives of self interest sufficient in the opinion of those who controlled it to warrant such action the commissioners acting for the government accepted such guaranty they did not thereby condone or intend to condone any act which had the effect to violate the sherman act nor could this settlement estop the government from prosecuting an action under the provisions of the act it is insisted that the decree in the union pacific case is decisive of this controversy and amounts to an adjudication against the government of the issues involved the conclusive answer to this contention is that the central pacific was not a party to that suit up to the final decree in this court that suit and the present one do not relate to the same subject matter the issues and questions therein decided are not the ones presented for decision here cromwell v county of sac u s l ed united shoe machinery co v united states decided april u s sup ct l ed —— the defendants contend that the suit is barred by laches on part of the government in failing to institute it earlier without deciding that this defense is available when an action is brought under an act of congress embodying as does the sherman act an expression of public policy enforceable by criminal prosecution and by civil suit instituted by the attorney general we are unable to discover that laches exists in the failure to more promptly prosecute the suit the stock acquisition complained of was in in the union pacific acquired control of the southern pacific by purchase of sufficient stock to accomplish that purpose the union pacific case was begun in and a final decree reached in and in this suit was begun other points are insisted upon in the oral argument and the elaborate briefs of the defendants we have considered them but they do not overcome the conclusions here inbefore stated which in our view dispose of this cause and require a reversal of the decree of the district court we do not find it necessary to pass upon the government s contention that the leases to the southern pacific and the acquisition by it of central pacific stock were in and of themselves violative of the pacific railroad acts of congress of and subsequent supplemental legislation we direct that a decree be entered severing the control by the southern pacific of the central pacific by stock ownership or by lease but in accomplishing this purpose so far as compatible therewith the mortgage lien asserted in the brief filed for the central union trust company shall be protected in addition the several terminal lines and cut offs leading to san francisco bay which have been constructed or acquired during the unified control of the two systems for the purpose of affording direct or convenient access to the bay and to the principal terminal facilities about the bay should be dealt with either by way of apportionment or by provisions for joint or common use in such manner as will secure to both companies such full convenient and ready access to the bay and to terminal facilities thereon that each company will be able freely to compete with the other to serve the public efficiently and to accomplish the purpose of the legislation under which it was constructed and a like course should be pursued in dealing with the lines extending from san francisco bay to sacramento and to portland or to the end that an appropriate decree may be framed the district court may and should bring in additional parties whenever that may become advisable in executing our directions reversed and remanded accordingly mr justice mcreynolds and mr justice brandeis took no part in the consideration or decision of this case mr justice mckenna dissenting i am unable to concur in the opinion and judgment of the court to this i feel constrained because i think it is unjust for the government to enforce a dissolution of the relation existing between the central pacific railway company and the southern pacific company i put my action on that ground alone though much can be said on the other grounds urged by the government and contested by the appellee companies prior to this relation another existed between the two companies or systems they may be said to have had that pretention and extent constituted by a lease for years executed in by the central pacific railroad company to the southern pacific giving to the latter the dominion of a proprietor waskey v chambers u s sup ct l ed ann cas d the central pacific railroad company was a bond aided road and on account of it was under obligation to repay the government the aid it had received and congress by an act passed july stat created a commission with power to settle the indebtedness an agreement of settlement was made in which the southern pacific was a participant and by it assured the payment of the securities provided for in the agreement of settlement between the central pacific railroad company and the government this participation was contemplated in the scheme submitted by speyer company to the commission and the present relation of the company is the outcome of the settlement and it may be said is the substitute of the rights and control the southern pacific as lessee had of the central pacific railroad company was it a justifiable substitute the answer should be in the affirmative when the act of was passed the situation was serious the problem complex and because the problem was complex three cabinet officers were selected to solve it these were the secretary of the treasury the secretary of the interior and the attorney general their prominence in the government their official concern with the subject matter assured fidelity in the execution of the trust and repels charge or intimation that they were or could be actuated by anything other than a strict consideration of duty and the exercise of their trust and their ability assured judgment in the selection of means the problem it is to be remembered was something more than to ascertain the amount of the debt it involved it might be foreclosure of the government s liens and it might also be government ownership and all that that meant the debt was known to be it was secured by a mortgage on the lines of the central pacific railroad company it is true but the mortgage was subordinate to other mortgages for about the same amount it was to be rescued from this subordination and given independent and certain solvency the power given to the commissioners was necessary to and commensurate with the purpose the power was to settle the indebtedness upon such terms and in such manner as might be agreed upon and to take such security as might seem expedient the only limitation was that the payment was not to be extended more than years necessarily therefore there was power to view the situation and judge of it its legal and practical aspects and what was possible in law and fact in the interest of all concerned to be done and it may be presumed that the commission found that there was nothing exigent in the situation or that demanded the separation of the southern pacific from the central pacific and that the guaranty of the former could be accepted and all that would follow from it and it is to be remembered that the action of the commission received the sanction of the president and was reported to congress it either had objected the settlement as planned could not have been accomplished and both would have objected if they had discerned anything sinister or inimical to law in it or that would result from it it is said however that there was no affirmative approval by congress and that its approval cannot be assumed from nonaction the government makes much of this ignoring all else and ventures in a kind of desperation against the circumstances the incredible assertion that congress was ignorant of the guaranty of the southern pacific and its contributing efficiency and this against an irresistible presumption to the contrary and in defiance of the fact that the attorney general reported to congress the terms of settlement and that the notes taken in settlement were guaranteed by the southern pacific and in defiance of the further fact that the bonds that it was provided were to be deposited as security for the notes with the secretary of the treasury had indorsed upon them the guaranty of the southern pacific and that the financial and commercial journals of the country addressing the business world—the world that was to accept the notes which congress authorized the secretary to sell—explained the settlement and the relation of the southern pacific to it and the assurance of safety and value the guarantee of the southern pacific gave i need not dwell on the contention of the government the court has not been impressed by it the court s view is rejecting that of the district court that there was no acceptance by the commission of the southern pacific s guaranty which carried obligation and that the guaranty was the prompting of interest on the part of the southern pacific i concede the latter the enterprise that is necessary and is exhibited in the conduct of great railroad systems whose traffic is concerned with a continent is not induced by the altruistic it is and naturally must be prompted by interest but it as other transactions of the business world is entitled to legal sanction and remedy the court asserts an interest in the southern pacific that urged its guaranty but does not explain the interest it is of pertinent concern to consider what it was it manifestly was no other than the relation of the company to the central pacific railway company through stock ownership the company would necessarily have no concern or interest in the central pacific the new company or the payment of the old company s debts to the government if it was to be separated from the central pacific and declared a competitor and a business antagonist and this must have been apparent to every one connected with the transactions if they gave any reflection to them—anything but a haphazard and reckless attention inconsiderate of practical and legal consequences this cannot be assumed and the contrary must be that is that the guaranty of the southern pacific was accepted as necessary to the settlement of the debt i repeat and summarize that the situation was of great concern to the government its solution was the consummation desired and through the aid of the southern pacific the company s guaranty was assurance to the business world that behind the notes and bonds of the central pacific were the great properties of the southern pacific and the competency of its management and the company made sacrifices in addition to the guaranty and they and it were accepted by the government and therefore the benefit that the company expected cannot be denied it there was no thought in any one s mind that the acquisition of stock by the southern pacific in the central pacific would be a restraint upon competition or a detriment to the public interest the attitude of those concerned in the transaction can be accurately realized by the reflection that the interest—control if it may be so called—that the southern pacific acquired in or over the new company the railway company was not greater nor more offensive to law than it had in or over the old company the railroad company the latter control existed from the enactment of the law until it was superseded by the agreement a period of years and there was no revulsion against or condemnation of the control—not by the government whose duty it was to proceed against it if it violated the anti trust law not by any business interest though for such interest the law was enacted as a protection this suit was not brought until years after the agreement not however by the government of the agreement but by the government of a much later time i think however that the decree of the district court should be affirmed james speyer of the firm of speyer co being on the witness stand the following is part of his testimony q please state whether the central pacific could have complied with the conditions imposed by that act of congress act of without a financial readjustment of their affairs of the kind contained in the readjustment which you arranged for the question was objected to but the witness answered the witness without some kind of readjustment they could not have complied i am not prepared to say that the adjustment we made was the only kind but some kind of adjustment seemed absolutely necessary by mr blair q and you made a plan of readjustment a we did q i put before you for convenience of reference the plan of readjustment which was used when mr ruhlender was testifying you recognize that as the plan of readjustment which was arranged for a i do q mr speyer when you started to work upon that plan of readjustment did you expect and count upon the intervention and aid of the southern pacific company a i knew i could not carry it through without the help of the southern pacific or some other railroad company in case the southern pacific had not come to assist q did you ever contemplate or work upon any plan which did not involve the intervention and aid of the southern pacific company a i did not q and that plan could not have been carried through without the intervention and aid of the southern pacific company the question was objected to q mr speyer considering the terms required by the act of congress namely the requirement that the entire debt of in round numbers would have to be paid in years in semianual installments would any one at all familiar with the central pacific affairs know that the central pacific with its own resources and credit could not comply with those conditions a he would q it would be obvious to any one at all familiar with the affairs of the central pacific that it could not with its own resources and credit comply with the terms of that act a yes sir q in making the agreement which you participated in with the united states what did you count upon to enable you to carry out the agreement with the united states a the co operation of the security holders of the central pacific and of the southern pacific company \n",
      " u.s s.ct l.ed miles collector of internal revenue v.safe deposit trust co of baltimore no argued dec decided may syllabus from intentionally omitted mr wm c herron of washington d c for plaintiff in error mr arthur w machen jr of baltimore md for defendant in error mr justice pitney delivered the opinion of the court defendant in error a corporation organized under the laws of maryland and authorized to act as guardian was on january appointed by the orphans court guardian of frank r brown an infant whose father had died intestate about a year before the son as next of kin became entitled to shares of the stock of the hartford fire insurance company and they were transferred to defendant in error as such guardian and still are held by it in that capacity at that time the capital stock of the insurance company issued and outstanding consisted of shares of the par value of each later in the year that company under statutory authority increased its capital stock to shares of the same par value the resolution of the stockholders sanctioning the increase provided that the right to subscribe to the new issue should be offered to the stockholders at the price of per share in the proportion of one share of new stock to each share of stock held by them subscriptions to be payable in installments and the directors to have power to dispose of shares not so subscribed and paid for in such manner as they might determine to be for the best interests of the company in july defendant in error pursuant to an order of the orphans court sold the subscription right to shares owned by its ward for equivalent to per share the commissioner of internal revenue holding that this entire amount was income for the year under the provisions of the act approved february c stat comp st ann supp § a et seq assessed and plaintiff in error collected a tax amounting to by reason of it defendant in error having paid this under protest and unavailingly appealed to the commissioner claiming that none of the amount so received was income within the meaning either of the act or of the sixteenth amendment brought this action against the collector to recover the entire amount of tax so assessed and paid the case was tried before the district court without a jury on stipulated facts and evidence plaintiff s extreme contention that the subscription right to new stock and also the proceeds of the sale of the right were wholly capital and not in any part subject to be taxed as income was overruled upon the authority of merchants loan trust co v smietanka u s sup ct l ed a l r then recently decided the trial court in the second place held that of the proceeds of the sale of the subscription rights so much only as represented a realized profit over and above the cost to plaintiff of what was sold was taxable as income in order to compute the amount of the profit the court commenced with the value of the old shares prior to authorization of the stock increase which upon the basis of evidence contained in the stipulation was taken to be what they were assessed at by the united states for purpose of the estate tax at the death of the ward s father viz per share and added the necessary to be paid by a stockholder or his assignee in order to obtain a share of the new stock making the cost of two shares one old and new and half of this the cost of one share the sale of the subscription rights at the purchaser to pay the issuing company per share was treated as equivalent to a sale of the fully paid shares at each or in excess of the which represented their cost to plaintiff and this difference multiplied by the number of shares or rights sold yielded as the gain realized out of the entire transaction upon this the court held plaintiff to have been properly taxable and upon nothing more no income tax being assessable with respect to the shares still retained because although they were considered worth more ex rights than the per share found to be their cost the difference could not be regarded as a taxable profit unless or until realized by actual sale fed to review the final judgment entered pursuant to the findings and opinion which sustained only in part plaintiff s demand for a refund of the tax paid the collector of internal revenue prosecuted a direct writ of error from this court under section judicial code comp st § because of the constitutional questions involved there is but one assignment of error based upon a single exception which denied that plaintiff was entitled to recover anything whatever hence the correctness of the particular recovery awarded is not in form raised but the trial judge having the complete facts before him almost of necessity passed upon them in their entirety in order to determine according to truth and substance how much of what plaintiff received was and how much was not income in the proper sense as is proper in a case involving the application of the sixteenth amendment eisner v macomber u s sup ct l ed a l r united states v phellis november u s sup ct l ed and in order to review the judgment it will be proper for us to analyze the reasoning upon which it was based it is not in dispute that the hartford fire insurance company is a corporation of the state of connecticut and that the stock increase in question was made under authority of certain acts of the legislature and certain resolutions of the stockholders by which the right to subscribe to the new issue was offered to existing stockholders upon the terms mentioned it is evident we think that such a distribution in and of itself constituted no division of any part of the accumulated profits or surplus of the company or even of its capital it was in effect an opportunity given to stockholders to share in contributing additional capital not to participate in distribution it was a recognition by the company that the condition of its affairs warranted an increase of its capital stock to double the par value of that already outstanding and that the new stock would have a value to the recipients in excess of per share a determination that it should be issued pro rata to the existing stockholders or so many of them as would pay that price this privilege of itself was not a fruit of stock ownership in the nature of a profit nor was it a division of any part of the assets of the company the right to subscribe to the new stock was but a right to participate in preference to strangers and on equal terms with other existing stockholders in the privilege of contributing new capital called for by the corporation—an equity that inheres in stock ownership under such circumstances as a quality inseparable from the capital interest represented by the old stock recognized so universally as to have become axiomatic in american corporation law gray v portland bank mass am dec atkins v albree allen mass jones v morrison minn n w eidman v bowman ill am rep humboldt driving park ass n v stevens neb n w am st rep electric co v electric co pa atl wall v utah copper co n j eq et seq atl stokes v continental trust co n y n e l r a n s ann cas evidently this inherent equity was recognized in the statute and the resolution under which the new stock here in question was offered and issued the stockholder s right to take his part of the new shares therefore—assuming their intrinsic value to have exceeded the issuing price—was essentially analogous to a stock dividend so far as the issuing price was concerned payment of this was a condition precedent to participation coupled with an opportunity to increase his capital investment in either aspect or both the subscription right of itself constituted no gain profit or income taxable without apportionment under the sixteenth amendment eisner v macomber u s sup ct l ed a l r is conclusive to this effect but in that case it was recognized u s sup ct l ed a l r that a gain through sale of dividend stock at a profit was taxable as income the same as a gain derived through sale of some of the original shares would be in that as in other recent cases this court has interpreted income as including gains and profits derived through sale or conversion of capital assets whether done by a dealer or trader or casually by a non trader as by a trustee in the course of changing investments merchants loan trust co v smietanka u s sup ct l ed a l r hence the district court rightly held defendant in error liable to income tax as to so much of the proceeds of sale of the subscription rights as represented a realized profit over and above the cost to it of what was sold how the gain should be computed is a matter of some contention by the government in this court but it admits of little doubt to treat the stockholder s right to the new shares as something new and independent of the old and as if it actually cost nothing leaving the entire proceeds of sale as gain would ignore the essence of the matter and the suggestion cannot be accepted the district court proceeded correctly in treating the subscription rights as an increase inseparable from the old shares not in the way of income but as capital in treating the new shares if and when issued as indistinguishable legally and in the market sense from the old and in regarding the sale of the rights as a sale of a portion of a capital interest that included the old shares what would have happened had defendant in error decided to accept the new shares and pay the issuing price instead of selling the rights is of no consequence in that event there would have been no realized profit hence no taxable income what resulted or might have resulted to defendant in error s retained interest in the company depending upon whether the purchaser exercised his right to subscribe or allowed it to lapse or whether in the latter event the stock was sold by the directors is of speculative interest only defendant in error resorted to the market for the sale of a part of its capital interest concededly sold at an advance over cost and what the profit actually was is the sole concern here not whether it might have been more or less nor whether the purchaser disposed of the stock to advantage that a comparison of the cost at acquisition and the selling price is proper under section a of the act stat comp st ann supp § bb where as here the property was acquired and sold within the same taxing year we understand to be conceded under the stipulation the court below was warranted in finding per share to have been the fair market value of the old stock when turned over to the guardian and treating this as its cost to the trust it was proper to add to this the required to be paid to the company and treat the total as the cost to plaintiff of each two shares one of which was to pass to the purchaser this in essence is the method adopted by the treasury department in the case of a sale of dividend stock in regulations ed art which reads art sale of stock received as dividend.—stock in a corporation received as a dividend does not constitute taxable income to a stockholder in such corporation but any profit derived by the stockholder from the sale of such stock is taxable income to him following eisner v macomber supra for the purpose of ascertaining the gain or loss derived from the sale of such stock or from the sale of the stock with respect to which it is issued the cost used to include also where required the fair market value as of march of both the old and new shares is to be determined in accordance with the following rules where the stock issued as a dividend is all of substantially the same character or preference as the stock upon which the stock dividend is paid the cost of each share of both the old and new stock will be the quotient of the cost or fair market value as of march if acquired prior to that date of the old shares of stock divided by the total number of the old and new shares that the averaging of cost might present more administrative difficulty in a case more complicated than the present as where the old shares were acquired at different times is not a sufficient ground for denying the soundness of the method itself various suggestions more or less ingenious as to how the profit ought to be computed made by counsel for defendant in error and by an amicus curiae have been examined and found faulty for reasons unnecessary to be mentioned upon the whole we are satisfied that the method adopted by the district court led to a correct result judgment affirmed \n",
      " u.s s.ct l.ed carlisle packing co.v.sandanger no argued march decided may messrs j harry covington of washingto d c and james a kerr and evan s mccord both of seattle wash for petitioner argument of counsel from pages intentionally omitted mr maurice mcmicken of seattle wash for respondent mr justice mcreynolds delivered the opinion of the court the supreme court of washington affirmed a judgment against petitioner packing company rendered by the trial court upon a verdict for damages on account of injuries which respondent suffered while employed upon petitioner s motorboat afloat in navigable alaskan waters respondent claimed that prior to the departure of the boat upon a trip intended to occupy perhaps six or eight hours petitioner or its agents negligently filled with gasoline and placed thereon a can which ordinarily contained coal oil and was so labeled for use according to the prevailing custom in those waters to start fires in the small stove where meals were cooked and water heated without knowledge of the substitution respondent poured the gasoline upon the firewood applied a match an explosion resulted and he was badly burned he further claimed that no life preservers had been placed on board and that his injuries were aggravated by delay attending search for one before he jumped into the water to extinguish his flaming clothes the trial court held the basis of the action is negligence and instructed by jury according to the common law rules in respect thereto it said that if petitioner or its authorized agents negligently filled the can with gasoline and placed it upon the boat and if by reason of such negligence respondent suffered injury he was entitled to recover compensatory damages therefor provided he himself had not been guilty of contributory negligence further that if the injuries resulting directly from the explosion were aggravated because no life preservers had been placed on board then additional compensation could be awarded for such aggravation also that if the explosion occurred without petitioner s negligence but the absence of life preservers caused aggravation of respondent s injuries he would be entitled to recover for such injuries as resulted directly from the negligence in respect of the life preservers but not for those caused solely by the explosion we have heretofore announced the general doctrine concerning rights and liabilities of the parties when one of a crew sustains injuries while on a vessel in navigable waters the vessel and her owners are liable in case a seaman falls sick or is wounded in the service of the ship to the extent of his maintenance and cure and to his wages at least so long as the voyage is continued the vessel and her owner are both a by english and american law liable to an indemnity for injuries received by seamen in consequence of the unseaworthiness of the ship or a failure to supply and keep in order the proper appliances appurtenant to the ship all the members of the crew except perhaps the master are as between themselves fellow servants and hence seamen cannot recover for injuries sustained through the negligence of another member of the crew beyond the expense of their maintenance and cure the seaman is not allowed to recover an indemnity for the negligence of the master or any member of the crew but is entitled to maintenance and cure whether the injuries were received by negligence or accident the osceola u s sup ct l ed chelentis v luckenbach steamship co u s sup ct l ed the general rules of the maritime law apply whether the proceeding be instituted in an admiralty or common law court chelentis v luckenbach steamship co supra knickerbocker ice co v stewart u s sup ct l ed a l r here the trial court did not instruct the jury in consonance with these rules and by failing so to do fell into error but mere error without more is not enough to upset the judgment if the record discloses that no injury could have resulted therefrom west v camden u s sup ct l ed considering the custom prevailing in those waters and other clearly established facts in the present cause we think the trial court might have told the jury that without regard to negligence the vessel was unseaworthy when she left the dock if the can marked coal oil contained gasoline also that she was unseaworthy if no life preservers were then on board and that if thus unseaworthy and one of the crew received damage as the direct result thereof he was entitled to recover compensatory damages the silvia u s sup ct l ed the southwark u s sup ct l ed the verdict shows that the jury found gasoline had been negligently placed in the can or that through negligence no life preservers were put on board or that both of these defaults existed and that as a result of one or both respondent suffered injury without contributory negligence on his part in effect the charge was more favorable to the petitioner than it could have demanded and we think no damage could have resulted from the erroneous theory adopted by the trial court the caledonia u s sup ct l ed thompson towing wrecking ass n v mcgregor fed c c a petitioner asked an instruction that section of the revised statutes comp st § applied and that under it the verdict could not exceed the value of the vessel in a state court when there is only one possible claimant and one owner the advantage of this section may be obtained by proper pleading the lotta d c fed delaware river ferry co v amos d c fed here the privelege was not set up or claimed in the answer and it could not be first presented upon request for a charge to the jury the judgment below must be affirmed mr justice clarke concurs in the result sec the liability of the owner of any vessel for any embezzlement loss or destruction by any person of any property goods or merchandise shipped or put on board of such vessel or for any loss damage or injury by collision or for any act matter or thing lost damage or forfeiture done occasioned or incurred without the privity or knowledge of such owner or owners shall in no case exceed the amount or value of the interest of such owner in such vessel and her freight then pending \n",
      " u.s s.ct l.ed olinv.kitzmiller et al no argued april decided may messrs arthur i moulton and james e fenton both of portland or for appellant messrs willis s moore and w w banks both of portland or for appellees mr justice mcreynolds delivered the opinion of the court the bill was dismissed upon motion by the trial court for want of equity and the circuit court of appeals affirmed this action fed appellant—a native of russia who has declared his intention to become a citizen of the united states—claims the right to fish in specified locations in the columbia river and seeks a mandatory injunction requiring the master fish warden and other officers of oregon to issue a license therefor his prayer is based upon the theory that so much of chapter general laws of oregon as directs that no fishing license shall be issued to any person who is not a citizen of the united states impairs the obligation const art § of the compact and agreement between the states of washington and oregon ratified by an act of congress approved april — stat c —which follows the congress of the united states of america hereby consents to and ratifies the compact and agreement entered into between the states of oregon and washington relative to regulating protecting and preserving fish in the boundary waters of the columbia river and other waters which compact and agreement is contained in section twenty of chapter one hundred and eighty eight of the general laws of oregon for nineteen hundred and fifteen and section one hundred and sixteen chapter thirty one of the session laws of washington for nineteen hundred and fifteen and is as follows all laws and regulations now existing or which may be necessary for regulating protecting or preserving fish in the waters of the columbia river over which the states of oregon and washington have concurrent jurisdiction or any other waters within either of said states which would affect said concurrent jurisdiction shall be made changed altered and amended in whole or in part only with the mutual consent and approbation of both states nothing herein contained shall be construed to affect the right of the united states to regulate commerce or the jurisdiction of the united states over navigable waters the statutes in which the states accepted the compact are not identical but each one provides no license for taking or catching salmon or other food or shell fish required by laws of this state shall be issued to any person who is not a citizen of the united states unless such person has declared his intention to become a citizen and is and has been an actual resident of the state for one year immediately preceding the application for such license nor shall any license be issued to a corporation unless it is authorized to do business in this state oregon act laws p § —washington act laws p § appellant s postulate is that the quoted provision read in connection with the compact inhibits each state from restricting its fishing licenses to citizens of the united states without consent of the other if this is unsound no foundation exists for his claim and all other questions may be disregarded considering the object and nature of the compact and the two acts of we cannot conclude that the parties intended by the identical provision to obligate themselves to issue any fishing license the purpose was to limit the classes of persons who might have them—beyond which the state might not go there is no inhibition against narrowing these classes nor indeed against a refusal to issue any license the oregon legislature acted in harmony with the compact when it excluded aliens there was no impairment and the judgment of the court below must be affirmed \n",
      " u.s s.ct l.ed state industrial commission of state of new yorkv.nordenholt corporation et al no argued march decided may mr e clarence aiken deputy atty gen of new york for petitioner argument of counsel from pages intentionally omitted mr e c sherwood of new york city for respondents argument of counsel from pages intentionally omitted mr justice mcreynolds delivered the opinion of the court sebastiana insana mother of guiseppe insana asked of the new york state industrial commission an allowance under the workmen s compensation law consol laws c on account of her son s death which she claimed resulted from accidental injuries received may in the course of his employment as a longshoreman by the nordenholt corporation then unloading a vessel lying in navigable waters at brooklyn the cargo consisted of bags of cement these were hoisted to the dock and there tiered up by insana and other longshoremen while thus engaged he slipped and fell on the dock the commission found the accidental injuries which the said deceased sustained while working for his employer when he fell from the pile of bags to the floor were the activating cause of his death and his death was a direct result of the injuries sustained by him while engaged in the regular course of his employment and awarded compensation as specified by the statute upon authority of matter of keator v rock plaster manufacturing co n y n e and matter of anderson v johnson lighterage co n y n e the appellate division reversed the award insana v nordenholt corporation app div n y supp and the court of appeals affirmed its action without opinion october n y n e in both the matter of keator and of anderson the employee suffered injuries on land while helping to unload a vessel lying in navigable waters the court of appeals held when so injured he was performing a maritime contract and that for reasons stated in matter of doey v howland co inc n y n e the industrial commission had no jurisdiction to make an award while making repairs on an oceangoing vessel lying at the dock in navigable waters doey fell down the hatchway and sustained fatal injuries the appellate division reversed an award of compensation and the court of appeals affirmed its action holding that as doey was performing a maritime contract the commission had no jurisdiction under the doctrine of southern pacific co v jensen u s sup ct l ed l r a c ann cas e and clyde steamship co v walker u s sup ct l ed it said two questions are presented a was doey at the time of his death engaged in the performance of a maritime contract if the first question be answered in the affirmative then it necessarily follows from the decisions of the supreme court of the united states above referred to southern pacific co v jensen and clyde steamship co v walker that the commission had no authority to make the award in question in determining whether a contract be of maritime nature locality is not controlling since the true test is the subject matter of the contract—the nature and character of the work to be done erie r r co v welsh u s in torts the rule is different there jurisdiction depends solely upon the place where the tort was committed which must have been upon the high seas or other navigable waters atlantic transport co of w va v imbrovek u s an award under the workmen s compensation law is not made on the theory that a tort has been committed on the contrary it is upon the theory that the statute giving the commission power to make an award is read into and becomes a part of the contract matter of post v burger gohlke n y the contract of employment by virtue of the statute contains an implied provision that the employer if the employee be injured will pay to him a certain sum to compensate for the injuries sustained or if death results a certain sum to dependents these payments are made irrespective of whether or not the employer was guilty of wrongdoing it is a part of the compensation agreed to be paid for services rendered in the course of the employment in the present case upon the conceded facts i am of the opinion that doey was at the time he met his death engaged in the performance of a maritime contract his employer had taken a contract to repair an ocean going vessel preparatory to its taking on a cargo of grain doey was one of several carpenters employed to make the necessary changes he was at the time he was killed engaged in such work on a steamship then in navigable waters the contract to make the changes was certainly maritime in its nature preparing a steamship to receive a cargo is as much maritime in nature as putting the cargo on or taking it from the ship nor was the nature of the contract changed in any way because the contractor did not actually do the work himself but employed others to do it for him doey s contract of employment was just as much of a maritime nature as was that of his employer an award to newham injured on the dock while checking freight and doing work similar to that of a foreman of stevedores was set aside in newham v chile exploration co n y n e october the court said we have held in matter of doey v howland co n y and in matter of anderson v johnson lighterage co n y and in matter of keator v rock plaster manufacturing co n u that if the employee was engaged at the time of his injury in the performance of a maritime contract the state did not have jurisdiction of the matter and the workmen s compensation law did not apply this is the deduction which we have made from the cases of southern pacific co v jensen u s and knickerbocker ice co v stewart u s the court below has made deductions from southern pacific co v jensen clyde steamship co v walker and knickerbocker ice co v stewart u s sup ct l ed a l r which we think are unwarranted and has proceeded upon an erroneous view of the federal law when an employee working on board a vessel in navigable waters sustains personal injuries there and seeks damages from the employer the applicable legal principles are very different from those which would control if he had been injured on land while unloading the vessel in the former situation the liability of employer must be determined under the maritime law in the latter no general maritime rule prescribes the liability and the local law has always been applied the liability of the employer for damages on account of injuries received on shipboard by an employee under a maritime contract is matter within the admiralty jurisdiction but not so when the accident occurs on land the injuries out of which southern pacific co v jensen arose occurred on navigable waters and the consequent rights and liabilities of the parties were prescribed by the maritime law the question there was whether these rules could be superseded by the workmen s compensation statute of the state and this court held they could not in the opinion citing atlantic transport co v imbrovek u s sup ct l ed l r a n s we said the work of a stevedore in which the deceased jensen was engaging is maritime in its nature his employment was a maritime contract the injuries which he received were likewise maritime and the rights and liabilities of the parties in connection therewith were matters clearly within the admiralty jurisdiction the doctrine that locality is the exclusive test of admiralty jurisdiction in matters of tort had been questioned in the imbrovek case and to show beyond any doubt that the maritime rules applied as to jensen s injuries we used the quoted language later in grant smith porter ship co v rohde january u s sup ct l ed we said the general doctrine that in contract matters admiralty jurisdiction depends upon the nature of the transaction and in tort matters upon the locality has been so frequently asserted by this court that it must now be treated as settled in chelentis v luckenbach steamship co u s sup ct l ed —an action at law seeking full indemnity for injuries received by a sailor on shipboard—this was said the work about which petitioner was engaged is maritime in its nature his employment was a maritime contract the injuries received were likewise maritime and the parties rights and liabilities were matters clearly within the admiralty jurisdiction atlantic transportation co v imbrovek u s and unless in some way there was imposed upon the owners a liability different from that prescribed by maritime law petitioner could properly demand only wages maintenance and cure under the doctrine approved in southern pacific co v jensen no state has power to abolish the well recognized maritime rule concerning measure of recovery and substitute stitute therefor the full indemnity rule of the common law such a substitution would distinctly and definitely change or add to the settled maritime law and it would be destructive of the uniformity and consistency at which the constitution aimed on all subjects of a commercial character affecting the intercourse of the states with each other or with foreign states see also peters v veasey u s sup ct l ed knickerbocker ice co v stewart u s sup ct l ed a l r in union fish co v erickson u s sup ct l ed it was held that when entering into maritime contracts the parties contemplate the system of maritime law and its well known rules control their rights and liabilities to the exclusion of state statutes in western fuel co v garcia december u s sup ct l ed it was held that where a stevedore s death on a ship within the state resulted from injuries there received an admiralty court in the absence of federal statute or positive maritime rule would recognize and apply the state statute giving an action for damages on account of death the subject is maritime and local in character and the specified modification of or supplement to the rule applied in admiralty courts when following the common law will not work material prejudice to the characteristic features of the general maritime law nor interfere with the proper harmony and uniformity of that law in its international and interstate relations in grant smith porter ship co v rhode a carpenter proceeding in admiralty sought damages for injuries received while at work on a partially completed vessel lying in the willamette river the oregon workmen s compensation law prescribed an exclusive remedy and the question presented was whether to give it effect would work material prejudice to the general maritime law the accident occurred on navigable waters and the cause was of a kind ordinarily within the admiralty jurisdiction neither the general employment contracted for nor the workman s activities at the time had any direct relation to navigation or commerce—it was essentially a local matter—and we said under such circumstances regulation of the rights obligations and consequent liabilities of the parties as between themselves by a local rule would not necessarily work material prejudice to any characteristic feature of the general maritime law or interfere with the proper harmony or uniformity of that law in its international or interstate relations in western fuel company v garcia we recently pointed out that as to certain local matters regulation of which would work no material prejudice to the general maritime law the rules of the latter might be modified or supplemented by state statutes the present case is controlled by that principle the statute of the state applies and defines the rights and liabilities of the parties the employee may assert his claim against the industrial accident fund to which both he and the employer have contributed as provided by the statute but he cannot recover damages in an admiralty court insana was injured upon the dock an extension of the land cleveland etc r r co v cleveland s s co u s sup ct l ed ann cas and certainly prior to the workmen s compensation act the employer s liability for damages would have depended upon the common law and the state statutes consequently when the compensation act superseded other state laws touching the liability in question it did not come into conflict with any superior maritime law and this is true whether awards under the act are made as upon implied agreements or otherwise the stevedore s contract of employment did not contemplate any dominant federal rule concerning the master s liability for personal injuries received on land in jensen s case rights and liabilities were definitely fixed by maritime rules whose uniformity was essential with these the local law came into conflict here no such antagonism exists there is no pertinent federal statute and application of the local law will not work material prejudice to any characteristic feature of the general maritime law compare new york central r r co v winfield u s sup ct l ed l r a c ann cas d the judgment of the court below must be reversed and the cause remanded for further proceedings not inconsistent with this opinion \n"
     ]
    }
   ],
   "source": [
    "for x in range(10):\n",
    "    with open(jsons[x]) as j:\n",
    "        print(get_cleaned_text(json.load(j)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['ca1', 'ca2', 'ca3', 'ca4', 'ca5', 'ca6', 'ca7', 'ca8', 'ca9', 'ca10', 'ca11', 'cadc', 'cafc']\n"
     ]
    }
   ],
   "source": [
    "# ok now let's try a parallel streaming download, because yolo. \n",
    "numbered_appellate_courts = [\"ca\" + str(x) for x in range(1, 12)]\n",
    "appellate_courts = numbered_appellate_courts + [\"cadc\", \"cafc\"]\n",
    "print(appellate_courts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['https://www.courtlistener.com/api/bulk-data/opinions/ca1.tar.gz', 'https://www.courtlistener.com/api/bulk-data/opinions/ca2.tar.gz', 'https://www.courtlistener.com/api/bulk-data/opinions/ca3.tar.gz', 'https://www.courtlistener.com/api/bulk-data/opinions/ca4.tar.gz', 'https://www.courtlistener.com/api/bulk-data/opinions/ca5.tar.gz', 'https://www.courtlistener.com/api/bulk-data/opinions/ca6.tar.gz', 'https://www.courtlistener.com/api/bulk-data/opinions/ca7.tar.gz', 'https://www.courtlistener.com/api/bulk-data/opinions/ca8.tar.gz', 'https://www.courtlistener.com/api/bulk-data/opinions/ca9.tar.gz', 'https://www.courtlistener.com/api/bulk-data/opinions/ca10.tar.gz', 'https://www.courtlistener.com/api/bulk-data/opinions/ca11.tar.gz', 'https://www.courtlistener.com/api/bulk-data/opinions/cadc.tar.gz', 'https://www.courtlistener.com/api/bulk-data/opinions/cafc.tar.gz']\n"
     ]
    }
   ],
   "source": [
    "urls = [f\"https://www.courtlistener.com/api/bulk-data/opinions/{x}.tar.gz\" for x in appellate_courts]\n",
    "print(urls)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['ca3.tar.gz', 'ca1.tar.gz', 'ca4.tar.gz', 'ca2.tar.gz', 'ca7.tar.gz', 'ca8.tar.gz', 'ca10.tar.gz', 'ca11.tar.gz', 'cadc.tar.gz', 'cafc.tar.gz', 'ca9.tar.gz', 'ca6.tar.gz', 'ca5.tar.gz']\n",
      "[]\n"
     ]
    }
   ],
   "source": [
    "from big_download import parallel_download\n",
    "successes, failures = parallel_download(urls, delay=0.1)\n",
    "print(successes)\n",
    "print(failures)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# bam.  Ok, let's get all the rest.\n",
    "already_downloaded = {x.partition(\".\")[0] for x in successes}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'ca10', 'cafc', 'ca1', 'ca8', 'scotus', 'ca7', 'ca2', 'ca9', 'cadc', 'ca6', 'ca5', 'ca3', 'ca11', 'ca4'}\n"
     ]
    }
   ],
   "source": [
    "already_downloaded.add(\"scotus\")\n",
    "print(already_downloaded)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "jxnspage = BeautifulSoup(requests.get(\"https://www.courtlistener.com/api/jurisdictions/\").text, \"lxml\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "jxntable = jxnspage.findAll(\"table\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "jxnrows = jxntable[0].tbody.findAll('tr')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "419"
      ]
     },
     "execution_count": 116,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(jxnrows)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'ca2'"
      ]
     },
     "execution_count": 120,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "jxnrows[2].findAll(\"td\")[4].text.strip()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'scotus'"
      ]
     },
     "execution_count": 121,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "jxnrows[0].findAll(\"td\")[4].text.strip()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "newjxns = {row.findAll(\"td\")[4].text.strip() for row in jxnrows} - already_downloaded"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'acca',\n",
       " 'afcca',\n",
       " 'ag',\n",
       " 'akb',\n",
       " 'akd',\n",
       " 'ala',\n",
       " 'alacivapp',\n",
       " 'alacrimapp',\n",
       " 'alactapp',\n",
       " 'alaska',\n",
       " 'alaskactapp',\n",
       " 'ald',\n",
       " 'almb',\n",
       " 'almd',\n",
       " 'alnb',\n",
       " 'alnd',\n",
       " 'alsb',\n",
       " 'alsd',\n",
       " 'arb',\n",
       " 'areb',\n",
       " 'ared',\n",
       " 'ariz',\n",
       " 'arizctapp',\n",
       " 'arizsuper',\n",
       " 'ariztaxct',\n",
       " 'ark',\n",
       " 'arkag',\n",
       " 'arkctapp',\n",
       " 'arkworkcompcom',\n",
       " 'armfor',\n",
       " 'arwb',\n",
       " 'arwd',\n",
       " 'asbca',\n",
       " 'azd',\n",
       " 'bap1',\n",
       " 'bap10',\n",
       " 'bap2',\n",
       " 'bap6',\n",
       " 'bap8',\n",
       " 'bap9',\n",
       " 'bapma',\n",
       " 'bapme',\n",
       " 'bva',\n",
       " 'caca',\n",
       " 'cacb',\n",
       " 'cacd',\n",
       " 'caeb',\n",
       " 'caed',\n",
       " 'cal',\n",
       " 'calag',\n",
       " 'calappdeptsuper',\n",
       " 'calctapp',\n",
       " 'californiad',\n",
       " 'canalzoned',\n",
       " 'canb',\n",
       " 'cand',\n",
       " 'casb',\n",
       " 'casd',\n",
       " 'cavc',\n",
       " 'cc',\n",
       " 'ccpa',\n",
       " 'circtdel',\n",
       " 'circtnc',\n",
       " 'circttenn',\n",
       " 'cit',\n",
       " 'cjdpa',\n",
       " 'cob',\n",
       " 'cod',\n",
       " 'colo',\n",
       " 'coloag',\n",
       " 'coloctapp',\n",
       " 'coloworkcompcom',\n",
       " 'com',\n",
       " 'conn',\n",
       " 'connappct',\n",
       " 'connsuperct',\n",
       " 'connworkcompcom',\n",
       " 'ctb',\n",
       " 'ctd',\n",
       " 'cusc',\n",
       " 'dc',\n",
       " 'dcb',\n",
       " 'dcd',\n",
       " 'deb',\n",
       " 'ded',\n",
       " 'del',\n",
       " 'delch',\n",
       " 'delctcompl',\n",
       " 'delfamct',\n",
       " 'deljudct',\n",
       " 'delsuperct',\n",
       " 'eca',\n",
       " 'fisc',\n",
       " 'fiscr',\n",
       " 'fla',\n",
       " 'flaag',\n",
       " 'fladistctapp',\n",
       " 'fld',\n",
       " 'flmb',\n",
       " 'flmd',\n",
       " 'flnb',\n",
       " 'flnd',\n",
       " 'flsb',\n",
       " 'flsd',\n",
       " 'ga',\n",
       " 'gactapp',\n",
       " 'gad',\n",
       " 'gamb',\n",
       " 'gamd',\n",
       " 'ganb',\n",
       " 'gand',\n",
       " 'gasb',\n",
       " 'gasd',\n",
       " 'gub',\n",
       " 'gud',\n",
       " 'haw',\n",
       " 'hawapp',\n",
       " 'hib',\n",
       " 'hid',\n",
       " 'iad',\n",
       " 'ianb',\n",
       " 'iand',\n",
       " 'iasb',\n",
       " 'iasd',\n",
       " 'idaho',\n",
       " 'idahoctapp',\n",
       " 'idb',\n",
       " 'idd',\n",
       " 'ilcb',\n",
       " 'ilcd',\n",
       " 'ill',\n",
       " 'illappct',\n",
       " 'illinoisd',\n",
       " 'illinoised',\n",
       " 'ilnb',\n",
       " 'ilnd',\n",
       " 'ilsb',\n",
       " 'ilsd',\n",
       " 'ind',\n",
       " 'indctapp',\n",
       " 'indianad',\n",
       " 'indtc',\n",
       " 'innb',\n",
       " 'innd',\n",
       " 'insb',\n",
       " 'insd',\n",
       " 'iowa',\n",
       " 'iowactapp',\n",
       " 'jpml',\n",
       " 'kan',\n",
       " 'kanag',\n",
       " 'kanctapp',\n",
       " 'kingsbench',\n",
       " 'ksb',\n",
       " 'ksd',\n",
       " 'ky',\n",
       " 'kyctapp',\n",
       " 'kyctapphigh',\n",
       " 'kyd',\n",
       " 'kyeb',\n",
       " 'kyed',\n",
       " 'kywb',\n",
       " 'kywd',\n",
       " 'la',\n",
       " 'laag',\n",
       " 'lactapp',\n",
       " 'lad',\n",
       " 'laeb',\n",
       " 'laed',\n",
       " 'lamb',\n",
       " 'lamd',\n",
       " 'lawb',\n",
       " 'lawd',\n",
       " 'mab',\n",
       " 'mad',\n",
       " 'mass',\n",
       " 'massappct',\n",
       " 'massdistct',\n",
       " 'masssuperct',\n",
       " 'maworkcompcom',\n",
       " 'mc',\n",
       " 'md',\n",
       " 'mdag',\n",
       " 'mdb',\n",
       " 'mdctspecapp',\n",
       " 'mdd',\n",
       " 'me',\n",
       " 'meb',\n",
       " 'med',\n",
       " 'mich',\n",
       " 'michctapp',\n",
       " 'michd',\n",
       " 'mieb',\n",
       " 'mied',\n",
       " 'minn',\n",
       " 'minnag',\n",
       " 'minnctapp',\n",
       " 'miss',\n",
       " 'missctapp',\n",
       " 'missd',\n",
       " 'miwb',\n",
       " 'miwd',\n",
       " 'mnb',\n",
       " 'mnd',\n",
       " 'mo',\n",
       " 'moag',\n",
       " 'moctapp',\n",
       " 'mod',\n",
       " 'moeb',\n",
       " 'moed',\n",
       " 'mont',\n",
       " 'montag',\n",
       " 'monttc',\n",
       " 'mowb',\n",
       " 'mowd',\n",
       " 'msnb',\n",
       " 'msnd',\n",
       " 'mspb',\n",
       " 'mssb',\n",
       " 'mssd',\n",
       " 'mtb',\n",
       " 'mtd',\n",
       " 'nc',\n",
       " 'ncctapp',\n",
       " 'ncd',\n",
       " 'nceb',\n",
       " 'nced',\n",
       " 'ncmb',\n",
       " 'ncmd',\n",
       " 'ncsuperct',\n",
       " 'ncwb',\n",
       " 'ncwd',\n",
       " 'ncworkcompcom',\n",
       " 'nd',\n",
       " 'ndb',\n",
       " 'ndctapp',\n",
       " 'ndd',\n",
       " 'neb',\n",
       " 'nebag',\n",
       " 'nebctapp',\n",
       " 'nebraskab',\n",
       " 'ned',\n",
       " 'nev',\n",
       " 'nh',\n",
       " 'nhb',\n",
       " 'nhd',\n",
       " 'nj',\n",
       " 'njb',\n",
       " 'njch',\n",
       " 'njd',\n",
       " 'njsuperctappdiv',\n",
       " 'njtaxct',\n",
       " 'nm',\n",
       " 'nmb',\n",
       " 'nmcca',\n",
       " 'nmctapp',\n",
       " 'nmd',\n",
       " 'nmib',\n",
       " 'nmid',\n",
       " 'nvb',\n",
       " 'nvd',\n",
       " 'ny',\n",
       " 'nyag',\n",
       " 'nyappdiv',\n",
       " 'nyappterm',\n",
       " 'nycivct',\n",
       " 'nycrimct',\n",
       " 'nyd',\n",
       " 'nyeb',\n",
       " 'nyed',\n",
       " 'nyfamct',\n",
       " 'nynb',\n",
       " 'nynd',\n",
       " 'nysb',\n",
       " 'nysd',\n",
       " 'nysupct',\n",
       " 'nysurct',\n",
       " 'nywb',\n",
       " 'nywd',\n",
       " 'ohio',\n",
       " 'ohioctapp',\n",
       " 'ohioctcl',\n",
       " 'ohiod',\n",
       " 'ohnb',\n",
       " 'ohnd',\n",
       " 'ohsb',\n",
       " 'ohsd',\n",
       " 'okeb',\n",
       " 'oked',\n",
       " 'okla',\n",
       " 'oklaag',\n",
       " 'oklacivapp',\n",
       " 'oklacoj',\n",
       " 'oklacrimapp',\n",
       " 'oklajeap',\n",
       " 'oknb',\n",
       " 'oknd',\n",
       " 'okwb',\n",
       " 'okwd',\n",
       " 'or',\n",
       " 'orb',\n",
       " 'orctapp',\n",
       " 'ord',\n",
       " 'orld',\n",
       " 'ortc',\n",
       " 'pa',\n",
       " 'pacommwct',\n",
       " 'paeb',\n",
       " 'paed',\n",
       " 'pamb',\n",
       " 'pamd',\n",
       " 'pasuperct',\n",
       " 'pawb',\n",
       " 'pawd',\n",
       " 'pennsylvaniad',\n",
       " 'prb',\n",
       " 'prd',\n",
       " 'reglrailreorgct',\n",
       " 'ri',\n",
       " 'rib',\n",
       " 'rid',\n",
       " 'risuperct',\n",
       " 'sc',\n",
       " 'scb',\n",
       " 'scctapp',\n",
       " 'scd',\n",
       " 'sd',\n",
       " 'sdb',\n",
       " 'sdd',\n",
       " 'southcarolinaed',\n",
       " 'southcarolinawd',\n",
       " 'stp',\n",
       " 'sttex',\n",
       " 'tax',\n",
       " 'tecoa',\n",
       " 'tenn',\n",
       " 'tenncrimapp',\n",
       " 'tennctapp',\n",
       " 'tennessed',\n",
       " 'tennesseeb',\n",
       " 'tennsuperct',\n",
       " 'tex',\n",
       " 'texag',\n",
       " 'texapp',\n",
       " 'texcrimapp',\n",
       " 'texd',\n",
       " 'texjpml',\n",
       " 'texreview',\n",
       " 'tneb',\n",
       " 'tned',\n",
       " 'tnmb',\n",
       " 'tnmd',\n",
       " 'tnwb',\n",
       " 'tnwd',\n",
       " 'txeb',\n",
       " 'txed',\n",
       " 'txnb',\n",
       " 'txnd',\n",
       " 'txsb',\n",
       " 'txsd',\n",
       " 'txwb',\n",
       " 'txwd',\n",
       " 'uscfc',\n",
       " 'usjc',\n",
       " 'utah',\n",
       " 'utahctapp',\n",
       " 'utb',\n",
       " 'utd',\n",
       " 'va',\n",
       " 'vactapp',\n",
       " 'vad',\n",
       " 'vaeb',\n",
       " 'vaed',\n",
       " 'vawb',\n",
       " 'vawd',\n",
       " 'vib',\n",
       " 'vid',\n",
       " 'vt',\n",
       " 'vtb',\n",
       " 'vtd',\n",
       " 'waeb',\n",
       " 'waed',\n",
       " 'wash',\n",
       " 'washag',\n",
       " 'washctapp',\n",
       " 'washd',\n",
       " 'wawb',\n",
       " 'wawd',\n",
       " 'wieb',\n",
       " 'wied',\n",
       " 'wis',\n",
       " 'wisag',\n",
       " 'wisctapp',\n",
       " 'wisd',\n",
       " 'wiwb',\n",
       " 'wiwd',\n",
       " 'wva',\n",
       " 'wvad',\n",
       " 'wvnb',\n",
       " 'wvnd',\n",
       " 'wvsb',\n",
       " 'wvsd',\n",
       " 'wyb',\n",
       " 'wyd',\n",
       " 'wyo'}"
      ]
     },
     "execution_count": 123,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "newjxns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import os\n",
    "os.chdir(\"zips/\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m~/code/lawvec/big_download.py\u001b[0m in \u001b[0;36mparallel_download\u001b[0;34m(urls, maxthreads, headers, delay, timeout)\u001b[0m\n\u001b[1;32m     39\u001b[0m         \u001b[0mresults\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mexecutor\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msubmit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0m__download\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0murl\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mheaders\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdelay\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtimeout\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0murl\u001b[0m \u001b[0;32min\u001b[0m \u001b[0murls\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 40\u001b[0;31m         \u001b[0;32mfor\u001b[0m \u001b[0mfuture\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mconcurrent\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfutures\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mas_completed\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresults\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     41\u001b[0m             \u001b[0mres\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfuture\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mresult\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.pyenv/versions/3.6.3/lib/python3.6/concurrent/futures/_base.py\u001b[0m in \u001b[0;36mas_completed\u001b[0;34m(fs, timeout)\u001b[0m\n\u001b[1;32m    239\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 240\u001b[0;31m             \u001b[0mwaiter\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mevent\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwait\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mwait_timeout\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    241\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.pyenv/versions/3.6.3/lib/python3.6/threading.py\u001b[0m in \u001b[0;36mwait\u001b[0;34m(self, timeout)\u001b[0m\n\u001b[1;32m    550\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0msignaled\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 551\u001b[0;31m                 \u001b[0msignaled\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_cond\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwait\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtimeout\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    552\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0msignaled\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.pyenv/versions/3.6.3/lib/python3.6/threading.py\u001b[0m in \u001b[0;36mwait\u001b[0;34m(self, timeout)\u001b[0m\n\u001b[1;32m    294\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mtimeout\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 295\u001b[0;31m                 \u001b[0mwaiter\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0macquire\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    296\u001b[0m                 \u001b[0mgotit\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: ",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-133-5e38b6e3ff14>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0msuccesses\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfailures\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mparallel_download\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0murls\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdelay\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m0.1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      2\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msuccesses\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfailures\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0mos\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mchdir\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"..\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/code/lawvec/big_download.py\u001b[0m in \u001b[0;36mparallel_download\u001b[0;34m(urls, maxthreads, headers, delay, timeout)\u001b[0m\n\u001b[1;32m     47\u001b[0m                 \u001b[0murl\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mres\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     48\u001b[0m                 \u001b[0msuccesses\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0murl\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 49\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0msuccesses\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfailures\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m~/.pyenv/versions/3.6.3/lib/python3.6/concurrent/futures/_base.py\u001b[0m in \u001b[0;36m__exit__\u001b[0;34m(self, exc_type, exc_val, exc_tb)\u001b[0m\n\u001b[1;32m    609\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    610\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m__exit__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mexc_type\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mexc_val\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mexc_tb\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 611\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshutdown\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mwait\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    612\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.pyenv/versions/3.6.3/lib/python3.6/concurrent/futures/thread.py\u001b[0m in \u001b[0;36mshutdown\u001b[0;34m(self, wait)\u001b[0m\n\u001b[1;32m    150\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mwait\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    151\u001b[0m             \u001b[0;32mfor\u001b[0m \u001b[0mt\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_threads\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 152\u001b[0;31m                 \u001b[0mt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjoin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    153\u001b[0m     \u001b[0mshutdown\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__doc__\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_base\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mExecutor\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshutdown\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__doc__\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.pyenv/versions/3.6.3/lib/python3.6/threading.py\u001b[0m in \u001b[0;36mjoin\u001b[0;34m(self, timeout)\u001b[0m\n\u001b[1;32m   1054\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1055\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mtimeout\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1056\u001b[0;31m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_wait_for_tstate_lock\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1057\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1058\u001b[0m             \u001b[0;31m# the behavior of a negative timeout isn't documented, but\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.pyenv/versions/3.6.3/lib/python3.6/threading.py\u001b[0m in \u001b[0;36m_wait_for_tstate_lock\u001b[0;34m(self, block, timeout)\u001b[0m\n\u001b[1;32m   1070\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mlock\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m  \u001b[0;31m# already determined that the C code is done\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1071\u001b[0m             \u001b[0;32massert\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_is_stopped\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1072\u001b[0;31m         \u001b[0;32melif\u001b[0m \u001b[0mlock\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0macquire\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mblock\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtimeout\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1073\u001b[0m             \u001b[0mlock\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrelease\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1074\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_stop\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "successes, failures = parallel_download(urls, delay=0.1)\n",
    "print(successes)\n",
    "print(failures)\n",
    "os.chdir(\"..\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# well, I broke 4 of my existing zips.  Time to download them again.  That sucks. \n",
    "newjxns.update({\"ca1\", \"ca2\", \"ca3\", \"ca4\"})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['https://www.courtlistener.com/api/bulk-data/opinions/connworkcompcom.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/bap8.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/ilcb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/wisd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/coloctapp.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/flaag.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/mass.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/ccpa.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/ohnd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/coloworkcompcom.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/ohioctcl.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/vaed.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/moed.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/nycivct.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/ksb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/arb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/dc.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/ctb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/alactapp.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/ded.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/mad.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/moag.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/californiad.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/oklacivapp.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/or.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/alaska.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/com.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/txsb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/rib.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/utb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/vib.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/txnd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/idd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/ilnd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/nmctapp.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/miss.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/prb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/bapma.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/illinoisd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/mtd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/kyctapp.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/texd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/bap10.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/dcb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/bap6.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/almd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/tennessed.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/monttc.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/tned.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/vid.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/washag.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/wisag.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/bap1.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/utah.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/nh.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/gasd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/wieb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/tennesseeb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/flsb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/montag.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/conn.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/tax.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/indianad.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/pamb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/ca3.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/sdb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/okwb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/tennctapp.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/mssd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/msnb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/nysb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/mod.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/pacommwct.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/innb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/nynd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/deljudct.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/iad.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/uscfc.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/indctapp.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/ca4.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/idahoctapp.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/armfor.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/masssuperct.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/wvad.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/vawb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/wvsb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/oklacoj.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/tenn.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/connappct.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/va.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/nebctapp.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/ark.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/iowa.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/kan.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/lactapp.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/md.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/ncmd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/connsuperct.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/lamd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/nywd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/wawd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/kywb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/kyed.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/del.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/ky.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/texreview.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/wvnb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/paeb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/akd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/calag.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/mspb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/arwb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/texapp.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/mowd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/njb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/mc.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/ohnb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/wied.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/vawd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/alsb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/waed.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/nmd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/southcarolinaed.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/nvb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/delch.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/missctapp.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/vtd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/wvnd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/me.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/vad.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/ganb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/bva.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/flmb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/texag.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/gamd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/nyappterm.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/tecoa.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/lamb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/illinoised.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/cusc.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/mnd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/tex.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/gad.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/kyd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/alnd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/insd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/tnmb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/ianb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/pa.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/arizsuper.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/asbca.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/minn.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/circtnc.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/nmid.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/ca1.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/cod.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/wvsd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/scctapp.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/sttex.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/cacd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/ohiod.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/ned.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/kyctapphigh.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/tenncrimapp.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/texcrimapp.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/usjc.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/nvd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/fisc.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/laeb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/kywd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/areb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/wyd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/delsuperct.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/ncd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/oknb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/utd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/calctapp.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/ny.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/maworkcompcom.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/michd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/njtaxct.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/mont.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/casb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/nyappdiv.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/innd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/ncwd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/ri.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/iasd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/cand.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/sc.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/ilcd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/akb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/ncwb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/txed.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/nebraskab.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/delctcompl.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/ilnb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/nm.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/tnwb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/utahctapp.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/miwd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/hid.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/alacivapp.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/njsuperctappdiv.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/ind.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/bap2.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/alacrimapp.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/gactapp.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/nd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/med.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/tnwd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/ncctapp.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/eca.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/gamb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/mowb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/rid.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/reglrailreorgct.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/ilsd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/nysd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/indtc.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/insb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/nmcca.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/haw.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/dcd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/mtb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/okwd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/arwd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/nyed.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/orctapp.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/nycrimct.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/wyb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/flsd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/lawd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/minnctapp.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/coloag.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/cit.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/txeb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/iand.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/oknd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/ca2.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/ariztaxct.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/washd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/ndctapp.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/massdistct.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/nywb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/missd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/idb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/wawb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/pasuperct.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/jpml.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/mied.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/vt.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/ortc.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/mieb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/txwd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/nc.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/ctd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/cc.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/ohsb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/southcarolinawd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/illappct.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/wva.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/ndd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/alnb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/ala.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/wis.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/bap9.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/nyd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/mnb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/oklacrimapp.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/mdag.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/canb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/scb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/txwb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/cavc.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/pamd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/texjpml.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/fiscr.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/okla.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/gub.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/ariz.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/txnb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/prd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/iowactapp.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/meb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/tennsuperct.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/laed.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/wiwb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/nmb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/hib.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/njch.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/fla.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/ald.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/moeb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/txsd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/colo.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/mdctspecapp.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/afcca.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/nev.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/nmib.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/minnag.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/oked.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/ncworkcompcom.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/tnmd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/arkag.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/mdb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/wash.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/ohio.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/nyeb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/waeb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/lad.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/almb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/oklaag.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/nceb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/stp.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/cjdpa.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/ill.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/flnd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/hawapp.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/mdd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/circtdel.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/alaskactapp.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/caca.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/ord.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/ndb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/azd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/nebag.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/orld.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/risuperct.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/ared.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/sdd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/la.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/laag.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/ncmb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/ilsb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/massappct.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/mich.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/kanag.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/ohsd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/caeb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/mab.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/mo.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/pawb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/nynb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/calappdeptsuper.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/oklajeap.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/paed.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/washctapp.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/kanctapp.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/nyfamct.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/nyag.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/wiwd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/bapme.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/vaeb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/wisctapp.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/ksd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/njd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/ag.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/vactapp.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/orb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/circttenn.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/michctapp.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/caed.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/msnd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/flnb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/gud.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/fld.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/nj.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/cob.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/deb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/alsd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/fladistctapp.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/kingsbench.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/ga.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/ncsuperct.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/nysupct.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/gasb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/pennsylvaniad.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/arkworkcompcom.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/flmd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/acca.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/neb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/cacb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/kyeb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/pawd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/okeb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/mssb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/scd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/ohioctapp.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/arizctapp.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/wyo.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/nhb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/casd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/nced.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/cal.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/idaho.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/sd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/gand.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/vtb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/miwb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/lawb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/tneb.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/delfamct.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/canalzoned.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/arkctapp.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/nhd.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/nysurct.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/moctapp.tar.gz',\n",
       " 'https://www.courtlistener.com/api/bulk-data/opinions/iasb.tar.gz']"
      ]
     },
     "execution_count": 136,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "newurls = [f\"https://www.courtlistener.com/api/bulk-data/opinions/{x}.tar.gz\" for x in newjxns]\n",
    "newurls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['wisd.tar.gz', 'connworkcompcom.tar.gz', 'ilcb.tar.gz', 'bap8.tar.gz', 'coloctapp.tar.gz', 'mass.tar.gz', 'coloworkcompcom.tar.gz', 'flaag.tar.gz', 'ccpa.tar.gz', 'ohioctcl.tar.gz', 'nycivct.tar.gz', 'ksb.tar.gz', 'moed.tar.gz', 'vaed.tar.gz', 'arb.tar.gz', 'ctb.tar.gz', 'alactapp.tar.gz', 'ded.tar.gz', 'moag.tar.gz', 'californiad.tar.gz', 'mad.tar.gz', 'oklacivapp.tar.gz', 'ohnd.tar.gz', 'com.tar.gz', 'txsb.tar.gz', 'rib.tar.gz', 'utb.tar.gz', 'vib.tar.gz', 'dc.tar.gz', 'idd.tar.gz', 'or.tar.gz', 'txnd.tar.gz', 'nmctapp.tar.gz', 'prb.tar.gz', 'bapma.tar.gz', 'illinoisd.tar.gz', 'mtd.tar.gz', 'alaska.tar.gz', 'texd.tar.gz', 'bap10.tar.gz', 'dcb.tar.gz', 'bap6.tar.gz', 'ilnd.tar.gz', 'tennessed.tar.gz', 'monttc.tar.gz', 'tned.tar.gz', 'vid.tar.gz', 'washag.tar.gz', 'wisag.tar.gz', 'bap1.tar.gz', 'almd.tar.gz', 'nh.tar.gz', 'miss.tar.gz', 'gasd.tar.gz', 'tennesseeb.tar.gz', 'flsb.tar.gz', 'montag.tar.gz', 'wieb.tar.gz', 'kyctapp.tar.gz', 'indianad.tar.gz', 'pamb.tar.gz', 'tax.tar.gz', 'sdb.tar.gz', 'okwb.tar.gz', 'utah.tar.gz', 'mssd.tar.gz', 'msnb.tar.gz', 'nysb.tar.gz', 'mod.tar.gz', 'pacommwct.tar.gz', 'innb.tar.gz', 'nynd.tar.gz', 'deljudct.tar.gz', 'iad.tar.gz', 'conn.tar.gz', 'tennctapp.tar.gz', 'indctapp.tar.gz', 'idahoctapp.tar.gz', 'armfor.tar.gz', 'masssuperct.tar.gz', 'wvad.tar.gz', 'vawb.tar.gz', 'wvsb.tar.gz', 'oklacoj.tar.gz', 'uscfc.tar.gz', 'connappct.tar.gz', 'ca4.tar.gz', 'tenn.tar.gz', 'nebctapp.tar.gz', 'va.tar.gz', 'ark.tar.gz', 'ca3.tar.gz', 'iowa.tar.gz', 'kan.tar.gz', 'ncmd.tar.gz', 'lamd.tar.gz', 'nywd.tar.gz', 'wawd.tar.gz', 'kywb.tar.gz', 'kyed.tar.gz', 'md.tar.gz', 'del.tar.gz', 'texreview.tar.gz', 'wvnb.tar.gz', 'ky.tar.gz', 'akd.tar.gz', 'calag.tar.gz', 'mspb.tar.gz', 'arwb.tar.gz', 'paeb.tar.gz', 'mowd.tar.gz', 'connsuperct.tar.gz', 'mc.tar.gz', 'njb.tar.gz', 'ohnb.tar.gz', 'vawd.tar.gz', 'alsb.tar.gz', 'waed.tar.gz', 'nmd.tar.gz', 'southcarolinaed.tar.gz', 'nvb.tar.gz', 'delch.tar.gz', 'wied.tar.gz', 'vtd.tar.gz', 'wvnd.tar.gz', 'missctapp.tar.gz', 'vad.tar.gz', 'ganb.tar.gz', 'bva.tar.gz', 'flmb.tar.gz', 'me.tar.gz', 'texag.tar.gz', 'nyappterm.tar.gz', 'tecoa.tar.gz', 'lamb.tar.gz', 'illinoised.tar.gz', 'gamd.tar.gz', 'cusc.tar.gz', 'mnd.tar.gz', 'gad.tar.gz', 'kyd.tar.gz', 'alnd.tar.gz', 'insd.tar.gz', 'tnmb.tar.gz', 'ianb.tar.gz', 'lactapp.tar.gz', 'arizsuper.tar.gz', 'asbca.tar.gz', 'tex.tar.gz', 'circtnc.tar.gz', 'nmid.tar.gz', 'minn.tar.gz', 'cod.tar.gz', 'wvsd.tar.gz', 'scctapp.tar.gz', 'sttex.tar.gz', 'pa.tar.gz', 'ohiod.tar.gz', 'ca1.tar.gz', 'ned.tar.gz', 'kyctapphigh.tar.gz', 'cacd.tar.gz', 'usjc.tar.gz', 'nvd.tar.gz', 'fisc.tar.gz', 'laeb.tar.gz', 'kywd.tar.gz', 'areb.tar.gz', 'tenncrimapp.tar.gz', 'wyd.tar.gz', 'ncd.tar.gz', 'oknb.tar.gz', 'delsuperct.tar.gz', 'utd.tar.gz', 'ny.tar.gz', 'maworkcompcom.tar.gz', 'michd.tar.gz', 'njtaxct.tar.gz', 'mont.tar.gz', 'casb.tar.gz', 'texcrimapp.tar.gz', 'innd.tar.gz', 'ncwd.tar.gz', 'nyappdiv.tar.gz', 'iasd.tar.gz', 'ri.tar.gz', 'sc.tar.gz', 'ilcd.tar.gz', 'akb.tar.gz', 'ncwb.tar.gz', 'txed.tar.gz', 'nebraskab.tar.gz', 'delctcompl.tar.gz', 'ilnb.tar.gz', 'nm.tar.gz', 'tnwb.tar.gz', 'cand.tar.gz', 'utahctapp.tar.gz', 'hid.tar.gz', 'miwd.tar.gz', 'alacivapp.tar.gz', 'ind.tar.gz', 'bap2.tar.gz', 'alacrimapp.tar.gz', 'njsuperctappdiv.tar.gz', 'nd.tar.gz', 'med.tar.gz', 'tnwd.tar.gz', 'calctapp.tar.gz', 'eca.tar.gz', 'gamb.tar.gz', 'mowb.tar.gz', 'ncctapp.tar.gz', 'reglrailreorgct.tar.gz', 'ilsd.tar.gz', 'rid.tar.gz', 'indtc.tar.gz', 'texapp.tar.gz', 'nmcca.tar.gz', 'insb.tar.gz', 'haw.tar.gz', 'nysd.tar.gz', 'mtb.tar.gz', 'okwd.tar.gz', 'arwd.tar.gz', 'dcd.tar.gz', 'nycrimct.tar.gz', 'wyb.tar.gz', 'gactapp.tar.gz', 'orctapp.tar.gz', 'lawd.tar.gz', 'coloag.tar.gz', 'cit.tar.gz', 'txeb.tar.gz', 'iand.tar.gz', 'oknd.tar.gz', 'flsd.tar.gz', 'ariztaxct.tar.gz', 'washd.tar.gz', 'ndctapp.tar.gz', 'massdistct.tar.gz', 'nywb.tar.gz', 'missd.tar.gz', 'idb.tar.gz', 'wawb.tar.gz', 'minnctapp.tar.gz', 'jpml.tar.gz', 'pasuperct.tar.gz', 'vt.tar.gz', 'ortc.tar.gz', 'mieb.tar.gz', 'txwd.tar.gz', 'nyed.tar.gz', 'ctd.tar.gz', 'mied.tar.gz', 'ohsb.tar.gz', 'southcarolinawd.tar.gz', 'cc.tar.gz', 'nc.tar.gz', 'ndd.tar.gz', 'alnb.tar.gz', 'ca2.tar.gz', 'wva.tar.gz', 'ala.tar.gz', 'nyd.tar.gz', 'bap9.tar.gz', 'mnb.tar.gz', 'mdag.tar.gz', 'canb.tar.gz', 'scb.tar.gz', 'txwb.tar.gz', 'cavc.tar.gz', 'pamd.tar.gz', 'texjpml.tar.gz', 'fiscr.tar.gz', 'oklacrimapp.tar.gz', 'gub.tar.gz', 'okla.tar.gz', 'txnb.tar.gz', 'prd.tar.gz', 'wis.tar.gz', 'meb.tar.gz', 'tennsuperct.tar.gz', 'ariz.tar.gz', 'wiwb.tar.gz', 'nmb.tar.gz', 'hib.tar.gz', 'njch.tar.gz', 'iowactapp.tar.gz', 'ald.tar.gz', 'moeb.tar.gz', 'laed.tar.gz', 'colo.tar.gz', 'txsd.tar.gz', 'afcca.tar.gz', 'mdctspecapp.tar.gz', 'nmib.tar.gz', 'minnag.tar.gz', 'oked.tar.gz', 'ncworkcompcom.tar.gz', 'tnmd.tar.gz', 'arkag.tar.gz', 'mdb.tar.gz', 'nev.tar.gz', 'ohio.tar.gz', 'illappct.tar.gz', 'waeb.tar.gz', 'lad.tar.gz', 'almb.tar.gz', 'nyeb.tar.gz', 'nceb.tar.gz', 'stp.tar.gz', 'cjdpa.tar.gz', 'oklaag.tar.gz', 'flnd.tar.gz', 'hawapp.tar.gz', 'mdd.tar.gz', 'circtdel.tar.gz', 'alaskactapp.tar.gz', 'caca.tar.gz', 'fla.tar.gz', 'ndb.tar.gz', 'azd.tar.gz', 'nebag.tar.gz', 'orld.tar.gz', 'ord.tar.gz', 'risuperct.tar.gz', 'sdd.tar.gz', 'ared.tar.gz', 'laag.tar.gz', 'ncmb.tar.gz', 'ilsb.tar.gz', 'massappct.tar.gz', 'la.tar.gz', 'kanag.tar.gz', 'wash.tar.gz', 'caeb.tar.gz', 'mab.tar.gz', 'ohsd.tar.gz', 'pawb.tar.gz', 'nynb.tar.gz', 'calappdeptsuper.tar.gz', 'oklajeap.tar.gz', 'ill.tar.gz', 'paed.tar.gz', 'kanctapp.tar.gz', 'nyfamct.tar.gz', 'nyag.tar.gz', 'mich.tar.gz', 'bapme.tar.gz', 'vaeb.tar.gz', 'wiwd.tar.gz', 'ksd.tar.gz', 'mo.tar.gz', 'ag.tar.gz', 'wisctapp.tar.gz', 'washctapp.tar.gz', 'circttenn.tar.gz', 'orb.tar.gz', 'caed.tar.gz', 'msnd.tar.gz', 'flnb.tar.gz', 'gud.tar.gz', 'fld.tar.gz', 'nj.tar.gz', 'cob.tar.gz', 'deb.tar.gz', 'alsd.tar.gz', 'vactapp.tar.gz', 'kingsbench.tar.gz', 'ga.tar.gz', 'ncsuperct.tar.gz', 'nysupct.tar.gz', 'gasb.tar.gz', 'pennsylvaniad.tar.gz', 'arkworkcompcom.tar.gz', 'flmd.tar.gz', 'acca.tar.gz', 'njd.tar.gz', 'cacb.tar.gz', 'kyeb.tar.gz', 'pawd.tar.gz', 'okeb.tar.gz', 'mssb.tar.gz', 'scd.tar.gz', 'fladistctapp.tar.gz', 'michctapp.tar.gz', 'wyo.tar.gz', 'nhb.tar.gz', 'casd.tar.gz', 'nced.tar.gz', 'neb.tar.gz', 'arizctapp.tar.gz', 'sd.tar.gz', 'idaho.tar.gz', 'vtb.tar.gz', 'miwb.tar.gz', 'lawb.tar.gz', 'gand.tar.gz', 'tneb.tar.gz', 'delfamct.tar.gz', 'canalzoned.tar.gz', 'nhd.tar.gz', 'nysurct.tar.gz', 'arkctapp.tar.gz', 'iasb.tar.gz', 'moctapp.tar.gz', 'cal.tar.gz', 'ohioctapp.tar.gz']\n",
      "[]\n"
     ]
    }
   ],
   "source": [
    "successes, failures = parallel_download(newurls, delay=0.1)\n",
    "print(successes)\n",
    "print(failures)\n",
    "os.chdir(\"..\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 139,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(glob.glob(\"zips/\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "419"
      ]
     },
     "execution_count": 142,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# oops\n",
    "len(glob.glob(\"zips/*\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 145,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "36232491616"
      ]
     },
     "execution_count": 145,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sum(os.path.getsize(f) for f in glob.glob('zips/*') if os.path.isfile(f))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 146,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "36232.491616"
      ]
     },
     "execution_count": 146,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "36232491616 / 1000000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "oh boy.  36 gigs of data.  So this is my change to see if I can figure out how to deal with files too big to fit in memory I guess...\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 147,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "the kernel still hasn't crashed!\n"
     ]
    }
   ],
   "source": [
    "print(\"the kernel still hasn't crashed!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I need a new sentence tokenization approach---I'd like to use gensim's [PathLineSentences](https://radimrehurek.com/gensim/models/word2vec.html#gensim.models.word2vec.LineSentence) corpus to build the model from those 36 gigs by streaming from file, but if I were to treat whole documents as sentences I'd have to make ridiculous guesses about how long documents are to pass the max length, or, I suppose, iterate over the whole corpus and count or something annoying like that.  I'd rather actually do sentence-level tokenization. \n",
    "\n",
    "So I'm going to try two different approaches.  First, I'm going to use the abbreviations in [the Indigo Book](https://law.resource.org/pub/us/code/blue/IndigoBook.html)---scrape it, build an equivalence table, and use that to expand abbreviations to the extent possible.  I may open-source this to allow people to add modern abbreviations. \n",
    "\n",
    "Second, Dan and Mike have put together a law-aware sentence tokenizer in [LexNLP](https://github.com/LexPredict/lexpredict-lexnlp), I'm going to try that. \n",
    "\n",
    "I'll start with the indigo book thing---and also start a new notebook, because this is murdering memory. It's going to be a bit of a mess, because the data in the indigo book isn't super structured---both abbreviation and non-abbreviation data show up in tables, and tables are of varying width with abbreviations in various positions.  Also some abbreviations have multiple comma-separated forms (e.g. \"F.2d, F.3d\")  But after getting them into a basic mapping of some sort, I come up with a cleanup strategy.  Onward"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "lawvec-fixed",
   "language": "python",
   "name": "lawvec-fixed"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}