![\"Open](\"https://colab.research.google.com/assets/colab-badge.svg\"){kind=icon}
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "BrpyeNL6QY7u",
+ "colab_type": "text"
+ },
+ "source": [
+ "# Analyzing Genomic Data with Biomedical Data Commons\n",
+ "Datacommons is intended for various data science tasks. This tutorial introduces the datacommons knowledge graph and discusses two tools to help integrate its data into your data science projects: (1) the [datacommons browser](https://browser.datacommons.org/) and (2) the [Python API](https://github.com/datacommonsorg/api-python). Before getting started, we will need to install the Python API package.\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "qGeY05pnQSeW",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "source": [
+ "# Install datacommons\n",
+ "!pip install --upgrade --quiet datacommons"
+ ],
+ "execution_count": 1,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "vvXt7uRwQgJE",
+ "colab_type": "text"
+ },
+ "source": [
+ "#What is Biomedical Data Commons?\n",
+ "Data Commons is an open knowledge graph of structured data. It contains statements about real world objects such as\n",
+ "* The genome assembly [hg38](https://browser.datacommons.org/kg?dcid=bio%2Fhg38) is a reference genome for the species *[Homo sapiens](https://browser.datacommons.org/kg?dcid=bio%2Fhs)*.\n",
+ "* For the [hg38](https://browser.datacommons.org/kg?dcid=bio%2Fhg38) genome assembly [chr17](https://browser.datacommons.org/kg?dcid=bio%2Fhg38_chr17) has [83,257,441 base pairs](https://browser.datacommons.org/kg?dcid=BasePairs83257441).\n",
+ "* [BRCA1](https://browser.datacommons.org/kg?dcid=bio/hg38_BRCA1) genomic coordinates are [chr17](https://browser.datacommons.org/kg?dcid=bio%2Fhg38_chr17):[43,044,294-43,125,483](https://browser.datacommons.org/kg?dcid=Position43044294To43125483) for genome assembly [hg38](https://browser.datacommons.org/kg?dcid=bio%2Fhg38).\n",
+ "\n",
+ "In the graph, [entities](https://en.wikipedia.org/wiki/Entity) like the genome assembly [hg38](https://browser.datacommons.org/kg?dcid=bio%2Fhg38) are represented by nodes. Every node has a type corresponding to what the node represents. For example, *[Homo sapiens](https://browser.datacommons.org/kg?dcid=bio%2Fhs)* is a [Species](https://browser.datacommons.org/kg?dcid=Species). Relations between entities are represented by edges between these nodes. For example, the statement \"The genome assembly hg38 is a reference genome for the species *Homo sapiens*.\" is represented in the graph as two nodes: \"hg38\" and \"HomoSapiens\" with an edge labeled \"[is_species](https://browser.datacommons.org/kg?dcid=is_chromosome)\" pointing from \"hg38\" to \"HomoSapiens\". Data Commons follows the [Schema.org data model](https://schema.org/docs/datamodel.html) and leverages schema.org schema to provide a common set of types and properties. To accomodate biological data this schema has been expanded to reflect the schema that has been used across biological databases created by the scientific community."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "S3GA2E_pQmet",
+ "colab_type": "text"
+ },
+ "source": [
+ "#Data Commons Browser\n",
+ "The Data Commons browser provides a way to explore the data in a human-readable format. It is the best way to explore what is in Data Commons. Searching in the browser for an entity like [BRACA1](https://browser.datacommons.org/kg?dcid=bio/hg38_BRCA1), takes you to a page about the entity, including properties like [refSeqID](https://browser.datacommons.org/kg?dcid=refSeqID) and [typeOfGene](https://browser.datacommons.org/kg?dcid=typeOfGene).\n",
+ "\n",
+ "An important property for all entities is the dcid. The dcid (DataCommons identifier) is a unique identifier assigned to each entity in the knowledge graph. With this identifier, you will be able to search for and query information on the given entity in ways that we will discuss later. The dcid is listed at the top of the page next to \"About: \" and also in the list of properties."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "wKXZNB9bQowR",
+ "colab_type": "text"
+ },
+ "source": [
+ "# Python API\n",
+ "\n",
+ "The [Python API](https://github.com/datacommonsorg/api-python) provides functions for users to extract structured information from Data Commons programmatically and view them in different formats such as Python `dict`'s and [Pandas](https://pandas.pydata.org/) DataFrames. DataFrames allow access to all the data processing, analytical and visualization tools provided by packages such as Pandas, NumPy, SciPy, and Matplotlib. For more information check out the [documentation](https://datacommons.readthedocs.io/en/latest/modules.html) on the Python API's modules.\n",
+ "\n",
+ "Every notebook begins by loading the datacommons library as follows:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "49VGt66oQjeq",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "source": [
+ "# Import Data Commons\n",
+ "import datacommons as dc\n",
+ "\n",
+ "# Import other required libraries\n",
+ "import matplotlib.pyplot as plt\n",
+ "import matplotlib.patches as mpatches\n",
+ "import pandas as pd\n",
+ "import requests\n",
+ "import json"
+ ],
+ "execution_count": 2,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "LRntKMDWQ1Ml",
+ "colab_type": "text"
+ },
+ "source": [
+ "##Example: Identify the Genome Assemblies Supported by Biomedical Data Commons\n",
+ "For this exercise we will identify the genome assemblies and their related species that are currently supported by Biomedical Data Commons. We will start by looking up the dcid for '[GenomeAssembly](https://browser.datacommons.org/kg?dcid=GenomeAssembly)'.\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "WGEx_kC5Q2Oz",
+ "colab_type": "text"
+ },
+ "source": [
+ "## Using get_property_value to Access Node Properties\n",
+ "Our first task for this tutorial will be to extract the genome assemblies from Data Commons using the Python API and view it in a Pandas DataFrame. For all properties, one can use [**`get_property_values`**](https://datacommons.readthedocs.io/en/latest/_autosummary/datacommons_core/datacommons.core.get_property_values.html) to get the associated values. Let's look up the dcid of [GenomeAssembly](https://browser.datacommons.org/kg?dcid=GenomeAssembly). We would then like to initialize our Pandas dataframe for the dcid bio/GenomeAssembly.\n",
+ "\n",
+ "For all properties, one can use `get_property_values` to get the associated values. We would like to know the instances of \"GenomeAssembly\" by getting the the typeOf instances that are oriented towards the \"GenomeAssembly\" identified by \"bio/GenomeAssembly\". `get_property_values` accepts the following parameters:\n",
+ "\n",
+ "\n",
+ "* **`dcids`** - A list of dcids to get property values for.\n",
+ "* **`prop`** - The property to get property values for.\n",
+ "* **`out`**`[=True]` - An optional flag that indicates the property is oriented away from the given nodes if true.\n",
+ "* **`value_type`**`[=None]` - An optional parameter which filters property values by the given type.\n",
+ "* **`limit`**`[=100]` - An optional parameter which limits the total number of property values returned aggregated over all given nodes.\n",
+ "\n",
+ "When the dcids are given as a Pandas Series, the returned list of property values is a Pandas Series where the i-th entry corresponds to property values associated with the i-th given dcid. Some properties, like containedInPlace, may have many property values. Consequently, the cells of the returned series will always contain a list of property values. Let's take a look:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "wKxlsd5CQ6dJ",
+ "colab_type": "code",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 36
+ },
+ "outputId": "349f1b0e-0dda-48b6-876b-25f9009a2395"
+ },
+ "source": [
+ "# Call get_property_values. The return value is a dict keyed by 'GenomeAssembly'.\n",
+ "genomeAssembly_dcids = dc.get_property_values(['GenomeAssembly'], 'typeOf', out=False)['GenomeAssembly']\n",
+ "# Display the frame\n",
+ "print(genomeAssembly_dcids)"
+ ],
+ "execution_count": 3,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ "['bio/ce10', 'bio/ce9', 'bio/danRer10', 'bio/danRer11', 'bio/dm3', 'bio/dm6', 'bio/galGal5', 'bio/galGal6', 'bio/hg19', 'bio/hg38', 'bio/mm10', 'bio/mm9', 'bio/sacCer3', 'bio/xenLae2']\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "s9etKR8DQ84d",
+ "colab_type": "text"
+ },
+ "source": [
+ "## Example: List All Genome Assemblies in Human Readable Format\n",
+ "Let's continue learning about the genome assemblies supported by Biomedical Data Commons. We are next going to find the names and species of the genome assemblies that are associated with the list of dcids of genome assemblies that we found. We are going to display the information in a human readable table."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "Tvl9YAzJQ-z7",
+ "colab_type": "code",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 302
+ },
+ "outputId": "8e06043a-f323-4d7d-da71-1cf10d021fe0"
+ },
+ "source": [
+ "# Intialize the Data Frame\n",
+ "df_genomeAssemblies = pd.DataFrame()\n",
+ "\n",
+ "# Add genome assemblies name and dcid\n",
+ "df_genomeAssemblies['name'] = pd.Series(dc.get_property_values(genomeAssembly_dcids, 'name'))\n",
+ "df_genomeAssemblies.reset_index(level=0, inplace=True)\n",
+ "df_genomeAssemblies = df_genomeAssemblies.rename(columns={\"index\": \"dcid\"}).explode('name')\n",
+ "\n",
+ "# Add Species dcid\n",
+ "df_genomeAssemblies['species_dcid'] = df_genomeAssemblies['dcid'].map(\n",
+ " dc.get_property_values(df_genomeAssemblies['dcid'], 'ofSpecies'))\n",
+ "df_genomeAssemblies = df_genomeAssemblies.explode('species_dcid')\n",
+ "\n",
+ "# Add Species name\n",
+ "df_genomeAssemblies['species_name'] = df_genomeAssemblies['species_dcid'].map(\n",
+ " dc.get_property_values(df_genomeAssemblies['species_dcid'], 'name'))\n",
+ "df_genomeAssemblies = df_genomeAssemblies.explode('species_name')\n",
+ "\n",
+ "print(df_genomeAssemblies)"
+ ],
+ "execution_count": 4,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ " dcid name species_dcid species_name\n",
+ "0 bio/ce10 ce10 bio/ce CaenorhabditisElegans\n",
+ "1 bio/ce9 ce9 bio/ce CaenorhabditisElegans\n",
+ "2 bio/danRer10 danRer10 bio/danRer DanioRerio\n",
+ "3 bio/danRer11 danRer11 bio/danRer DanioRerio\n",
+ "4 bio/dm3 dm3 bio/dm DrosophilaMelanogaster\n",
+ "5 bio/dm6 dm6 bio/dm DrosophilaMelanogaster\n",
+ "6 bio/galGal5 galGal5 bio/galGal GallusGallus\n",
+ "7 bio/galGal6 galGal6 bio/galGal GallusGallus\n",
+ "8 bio/hg19 hg19 bio/hs HomoSapiens\n",
+ "9 bio/hg38 hg38 bio/hs HomoSapiens\n",
+ "10 bio/mm10 mm10 bio/mm MusMusculus\n",
+ "11 bio/mm9 mm9 bio/mm MusMusculus\n",
+ "12 bio/sacCer3 sacCer3 bio/sacCer SaccharomycesCerevisiae\n",
+ "13 bio/xenLae2 xenLae2 bio/xenLae XenopusLaevis\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "vXeKzZxlRBc8",
+ "colab_type": "text"
+ },
+ "source": [
+ "Congratulations! You've found the basic information on all the genome assemblies and species currently supported by Biomedical Data Commons. As you can see we currently support data from 8 model organisms across 14 different genome assemblies."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "W4_hSQvJREsA",
+ "colab_type": "text"
+ },
+ "source": [
+ "# Example: Analyze Genetic Variants within RUNX1\n",
+ "For this exercise, we will be analyzing genetic variants within the gene RUNX1. We will start by identifying the genetic variants within the gene region and then limit our list to those within the coding region of BRCA1 and then to those with known clinical significance. First, let's start by looking up the dcid for '[RUNX1](https://browser.datacommons.org/kg?dcid=bio/hg38_RUNX1)'. \n",
+ "\n",
+ "Note that 'Gene' defines the Data Commons type. Let's start by using [**`get_property_labels`**](https://datacommons.readthedocs.io/en/latest/_autosummary/datacommons_core/datacommons.core.get_property_labels.html) to identify all the properties associated with RUNX1. `get_property_labels` accepts the following parameters:\n",
+ "- **`dcids`** `[list of str]` – A list of nodes identified by their dcids.\n",
+ "- **`out`** `[bool, optional]` – Whether or not the property points away from the given list of nodes.\n",
+ "\n",
+ "The output of `get_property_labels` is a dict mapping dcids to lists of property labels. If out is True, then property labels correspond to edges directed away from given nodes. Otherwise, they correspond to edges directed towards the given nodes."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "IHiMmiBERG3B",
+ "colab_type": "code",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 397
+ },
+ "outputId": "37911039-8999-4656-bbec-d6d100ba5796"
+ },
+ "source": [
+ "# Call get_property_labels\n",
+ "dc.get_property_labels(['bio/hg38_RUNX1'])"
+ ],
+ "execution_count": 5,
+ "outputs": [
+ {
+ "output_type": "execute_result",
+ "data": {
+ "text/plain": [
+ "{'bio/hg38_RUNX1': ['description',\n",
+ " 'fullName',\n",
+ " 'geneSymbol',\n",
+ " 'genomicCoordinates',\n",
+ " 'hasRNATranscript',\n",
+ " 'inChromosome',\n",
+ " 'inGenomeAssembly',\n",
+ " 'mRNA',\n",
+ " 'mapLocation',\n",
+ " 'modificationDate',\n",
+ " 'name',\n",
+ " 'ncbiGeneID',\n",
+ " 'ncbiTaxonID',\n",
+ " 'nomenclatureStatus',\n",
+ " 'ofSpecies',\n",
+ " 'provenance',\n",
+ " 'refSeqID',\n",
+ " 'strandOrientation',\n",
+ " 'typeOf',\n",
+ " 'typeOfGene']}"
+ ]
+ },
+ "metadata": {
+ "tags": []
+ },
+ "execution_count": 5
+ }
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "_CXvFjSJRI4_",
+ "colab_type": "text"
+ },
+ "source": [
+ "### Identify RUNX1 Genomic Coordinates\n",
+ "Great, now we see the type of information known about RUNX1. To identify the genetic variants within the gene region we need to know what the chromosome and ther genomic coordinates of RUNX1. Let's grab that information using `get_property_values`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "d75k6KVhRMzp",
+ "colab_type": "code",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 55
+ },
+ "outputId": "23135fce-93d6-4b27-97d7-52b411c89fc3"
+ },
+ "source": [
+ "# Initialize the Data Frame\n",
+ "df_RUNX1 = pd.DataFrame({'gene': ['bio/hg38_RUNX1']})\n",
+ "\n",
+ "# Grab the chromosome and genomic coordinates using get_property_values\n",
+ "df_RUNX1['chromosome'] = df_RUNX1['gene'].map(dc.get_property_values(df_RUNX1['gene'], 'inChromosome'))\n",
+ "df_RUNX1['genomicCoordinates'] = df_RUNX1['gene'].map(dc.get_property_values(df_RUNX1['gene'], 'genomicCoordinates'))\n",
+ "\n",
+ "# display the genomic coordinates\n",
+ "print(df_RUNX1)\n",
+ "\n",
+ "# define start and stop of RUNX1\n",
+ "start, stop = df_RUNX1['genomicCoordinates'][0][0].strip('Position').split('To')\n",
+ "start = int(start)\n",
+ "stop = int(stop)"
+ ],
+ "execution_count": 6,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ " gene chromosome genomicCoordinates\n",
+ "0 bio/hg38_RUNX1 [bio/hg38_chr21] [Position34787800To35049344]\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "kJoGt1wrRPHu",
+ "colab_type": "text"
+ },
+ "source": [
+ "## Find All The Genetic Variants Within RUNX1\n",
+ "We found the coordinates of RUNX1 in the hg38 genome: chr21:34787800-35049344. Now that we know this let's find all the genetic variants of class [GeneticVariant](https://browser.datacommons.org/kg?dcid=GeneticVariant) that occur in the region. But first let's establish the information that we know on genetic variants using `get_property_values`.\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "ozoN1sE6RROW",
+ "colab_type": "code",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 56
+ },
+ "outputId": "87898f5a-29f7-4358-a5c3-027850c7ea74"
+ },
+ "source": [
+ "# Identify all the properties of genetic variants\n",
+ "print(dc.get_property_values(['GeneticVariant'], 'domainIncludes', value_type='Property', out=False))"
+ ],
+ "execution_count": 7,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ "{'GeneticVariant': ['alleleOrigin', 'alleleType', 'averageHeterozygosity', 'averageHeterozygositySE', 'clinVarAlleleID', 'clinVarFilterStatus', 'clinVarID', 'clinVarQualityScore', 'clinVarReviewStatus', 'clinicalSignificance', 'clinicalSignificanceConflicting', 'clinicalSignificanceType', 'clinicalSource', 'dbSNPBuildID', 'dbVarID', 'diseaseDescription', 'diseaseName', 'experimentalFactorOntologyID', 'geneID', 'geneReviewsID', 'geneSymbol', 'geneticTestingRegistryID', 'geneticVariantAlignmentQuality', 'geneticVariantAttribute', 'geneticVariantClass', 'geneticVariantExceptions', 'geneticVariantFunctionalCategory', 'geneticVariantImpercise', 'geneticVariantLength', 'geneticVariantLocType', 'geneticVariantSubmitterCount', 'geneticVariantValidationStatus', 'geneticsHomeReferenceID', 'genomicPosition', 'hg19GenomicLocation', 'hg19GenomicPosition', 'hg38GenomicLocation', 'hg38GenomicPosition', 'humanGenomeVariationSocietyNomenclature', 'humanPhenotypeOntologyID', 'medGenID', 'medicalGeneticSummariesID', 'medicalSubjectHeadingID', 'mm10GenomicLocation', 'mm9GenomicLocation', 'molecularType', 'mondoID', 'ncbiGeneID', 'numberOfAllelesWithFreq', 'observedAllele', 'omimID', 'orphaNumber', 'pharmGKBID', 'referenceAlleleNCBI', 'referenceAlleleUCSC', 'referenceSNPClusterID', 'sequenceOntologyID', 'snomedCT', 'submitter', 'suspectReasonCode', 'translocationToChromosome', 'variationEndCI', 'variationPositionCI', 'variationType']}\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "AiMokiMfRTe4",
+ "colab_type": "text"
+ },
+ "source": [
+ "### Using SPARQL and get_property_values to find genetic variants in RUNX1\n",
+ "Like genes, genetic variants also point to the chromosome on which they reside. Their positions on the chromosome is specified by hg38GenomicPosition. Using [**`query`**](https://datacommons.readthedocs.io/en/latest/_autosummary/datacommons.query.html) we will identify the dcids of all genetic variants on chr21. `query` accepts the following parameter\n",
+ "- **`query`** `[query_string[, select]]` – Returns the results of executing a SPARQL query on the Data Commons graph.\n",
+ "\n",
+ "`query` parameter is a SPARQL query that quickly searches and returns the data that matches the query on multiple parameters. There is no limit to the number of values that can be returned by a SPARQL query. In our query here we will be specifying that we want all genetic variants on chr21. Then we will format all the returned genetic variant dcids into a list and use `get_property_values` again to filter for genetic variants whose hg38GenomicPosition is within RUNX1. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "4rFcLP0rRVz5",
+ "colab_type": "code",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 207
+ },
+ "outputId": "c2f6f4e5-6bd9-4ef1-aea3-53a9525d11ae"
+ },
+ "source": [
+ "# Query for genetic variants associated with RUNX1\n",
+ "query = '''\n",
+ "SELECT ?gv ?p\n",
+ "WHERE { \n",
+ " ?chr dcid \"bio/hg38_chr21\" . \n",
+ " ?gv inChromosome ?chr .\n",
+ " ?gv typeOf GeneticVariant .\n",
+ " ?gv hg38GenomicPosition ?p\n",
+ "}\n",
+ "'''\n",
+ "print(query)\n",
+ "rows = dc.query(query)\n",
+ "dcids = set()\n",
+ "for row in rows:\n",
+ " dcids.add(row['?gv'])\n",
+ "dcids = list(dcids)\n",
+ "print(len(dcids))"
+ ],
+ "execution_count": 8,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ "\n",
+ "SELECT ?gv ?p\n",
+ "WHERE { \n",
+ " ?chr dcid \"bio/hg38_chr21\" . \n",
+ " ?gv inChromosome ?chr .\n",
+ " ?gv typeOf GeneticVariant .\n",
+ " ?gv hg38GenomicPosition ?p\n",
+ "}\n",
+ "\n",
+ "8028\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "Zp3uhwfJRYcA",
+ "colab_type": "code",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 55
+ },
+ "outputId": "f5effb12-eacc-4182-c570-a0d930c44cbe"
+ },
+ "source": [
+ "# filter all genetic variants for the ones within RUNX1\n",
+ "RUNX1_geneticVariants = []\n",
+ "gen_positions = dc.get_property_values(dcids, 'hg38GenomicPosition')\n",
+ "data = pd.DataFrame(gen_positions).transpose()\n",
+ "data.reset_index(level=0, inplace=True)\n",
+ "data = data.rename(columns={\"index\": \"dcid\", 0: \"position\"})\n",
+ "data['position'] = pd.to_numeric(data['position'])\n",
+ "data = data[data['position'] >= start]\n",
+ "data = data[data['position'] < stop]\n",
+ "RUNX1_geneticVariants = list(set(data['dcid']))\n",
+ "\n",
+ "# print the first few genetic variants\n",
+ "print(RUNX1_geneticVariants[:5])\n",
+ "\n",
+ "# check how many genetic variants are in RUNX1\n",
+ "print(len(RUNX1_geneticVariants))"
+ ],
+ "execution_count": 9,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ "['bio/rs1569084170', 'bio/643883', 'bio/rs1569002337', 'bio/839054', 'bio/rs150481777']\n",
+ "504\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "0Zb9dsK_RfMD",
+ "colab_type": "text"
+ },
+ "source": [
+ "## Identify Which Genetic Variants Are In Coding Regions\n",
+ "We've identified 360 genetic varaints within RUNX1. However, these can be in introns or exons. Let's further restrict genetic variant list to ones in the coding region of RUNX1. To do this we need to identify the positions of the exons of RUNX1. We know that RUNX1 has a property called rnaTranscript. Let's use `get_property_values` to find out more information on the RUNX1 transcript."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "rbJMK_pbRf2Z",
+ "colab_type": "code",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 93
+ },
+ "outputId": "9b274fb7-0b75-4508-e980-4e4ce3b000c5"
+ },
+ "source": [
+ "# Get property values for rnaTranscript of RUNX1\n",
+ "df_RUNX1['rnaTranscript'] = df_RUNX1['gene'].map(dc.get_property_values(df_RUNX1['gene'], 'hasRNATranscript'))\n",
+ "print(df_RUNX1)"
+ ],
+ "execution_count": 10,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ " gene ... rnaTranscript\n",
+ "0 bio/hg38_RUNX1 ... [bio/hg38_ENST00000300305.7, bio/hg38_ENST0000...\n",
+ "\n",
+ "[1 rows x 4 columns]\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "QmXRJpriRift",
+ "colab_type": "text"
+ },
+ "source": [
+ "### Identify the properties of RNATranscripts\n",
+ "There are several dcids associated with this property which are pointing to nodes of class [RNATranscript](https://browser.datacommons.org/kg?dcid=RNATranscript). Let's verify that this is indeed the case and then check the properties of RNATrancript using `get_property_values`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "Cbwu_Et1Rk3C",
+ "colab_type": "code",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 150
+ },
+ "outputId": "64b4424a-ae33-4e40-d8a7-83bb012150a1"
+ },
+ "source": [
+ "# Specify an rnaTranscript associated with RUNX1\n",
+ "RUNX1_transcript = df_RUNX1.iloc[0]['rnaTranscript'][0]\n",
+ "\n",
+ "# Check what type the rnaTranscript is\n",
+ "print(dc.get_property_values([RUNX1_transcript], 'typeOf'))\n",
+ "\n",
+ "# Identify properties of RNATranscript\n",
+ "dict_temp = dc.get_property_values(['RNATranscript'], 'domainIncludes', value_type='Property', out=False)\n",
+ "for prop in dict_temp['RNATranscript']:\n",
+ " print(prop)"
+ ],
+ "execution_count": 11,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ "{'bio/hg38_ENST00000300305.7': ['RNATranscript']}\n",
+ "codingCoordinates\n",
+ "exonCoordinates\n",
+ "exonFrame\n",
+ "makesProtein\n",
+ "refSeqID\n",
+ "transcriptionCoordinates\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "rZXoXHFzRm7Q",
+ "colab_type": "text"
+ },
+ "source": [
+ "### Explore the difference between codingCoordinates and exonCoordinates\n",
+ "There are two properties that may be useful for us in identifying which genetic variants are in the coding region of RUNX1. Let's figure out which one that we'd like to use moving forward by grabbing the values associated with both these properties using `get_property_values`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "OSH4nzL7RpEy",
+ "colab_type": "code",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 264
+ },
+ "outputId": "2e49a555-4fc6-4138-c19e-55b883d50468"
+ },
+ "source": [
+ "# Get the values for codingCoordinates and exonCoordiantes\n",
+ "temp_dict = dc.get_property_values([RUNX1_transcript], 'codingCoordinates')\n",
+ "print('Coding Coordinate Values:')\n",
+ "for value in temp_dict[RUNX1_transcript]:\n",
+ " print(value)\n",
+ "print('\\n')\n",
+ "temp_dict = dc.get_property_values([RUNX1_transcript], 'exonCoordinates')\n",
+ "print('Exon Coordinate Values:')\n",
+ "for value in temp_dict[RUNX1_transcript]:\n",
+ " print(value)"
+ ],
+ "execution_count": 12,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ "Coding Coordinate Values:\n",
+ "Position34792134To35048899\n",
+ "\n",
+ "\n",
+ "Exon Coordinate Values:\n",
+ "Position34787800To34792610\n",
+ "Position34799300To34799462\n",
+ "Position34834409To34834601\n",
+ "Position34859473To34859578\n",
+ "Position34880556To34880713\n",
+ "Position34886842To34887096\n",
+ "Position34892924To34892963\n",
+ "Position35048841To35049344\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "BqtiSfKFRrIP",
+ "colab_type": "text"
+ },
+ "source": [
+ "### Find the exon coordinates reported for all RNA transcripts of RUNX1\n",
+ "From the values of codingCoordinates and exonCoordinates we observe that coding coordinates contains the range of base pairs spanning the entire coding region of RUNX1 including introns. Whereas exonCoordinates reports the genomic coordinates of all exons of RUNX1. We want to find all genetic variants in exons of RUNX1, so going forward we want to grab the exonCoordinates of transcripts. There are multiple RNA transcripts reported for RUNX1. Let's make a unique list of all exonCoordinates recorded for RUNX1 using `get_property_values`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "OHd3PtgLRtkv",
+ "colab_type": "code",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 55
+ },
+ "outputId": "9a927890-275d-4ee3-b53b-4f2c971ce453"
+ },
+ "source": [
+ "# Initiate an empty set\n",
+ "RUNX1_exonCoordinates = set()\n",
+ "\n",
+ "# Using get_property_values get all the exon coordinates for all rnaTranscripts\n",
+ "for rnaTranscript_dcid in df_RUNX1['rnaTranscript'][0]:\n",
+ " temp_dict = dc.get_property_values([rnaTranscript_dcid], 'exonCoordinates')\n",
+ " for item in temp_dict[rnaTranscript_dcid]:\n",
+ " RUNX1_exonCoordinates.add(item)\n",
+ "\n",
+ "# check the firs few exons\n",
+ "RUNX1_exonCoordinates = list(RUNX1_exonCoordinates)\n",
+ "print(RUNX1_exonCoordinates[:5])\n",
+ "\n",
+ "# check how many unique exon coordinates have been reported for RUNX1\n",
+ "print(len(RUNX1_exonCoordinates))"
+ ],
+ "execution_count": 13,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ "['Position35049167To35049298', 'Position34791754To34792610', 'Position35988129To35988171', 'Position35984571To35984830', 'Position34859473To34859578']\n",
+ "41\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "4E26lXFDRvYe",
+ "colab_type": "text"
+ },
+ "source": [
+ "### Identify the genetic variants in the exon coding regions\n",
+ "Now that we know all the possible reported exon coordinates for RUNX1 we can identify which genetic variants are in exons. Note that RUNX1 has 4 - 9 variants depending on the isoform. Many of these exon coordinates from transcripts of different isoforms are overlapping with each other, but not exact resulting in 41 unique coordinate ranges to be observed. We are interested in genetic variants within any reported exon coordinates in this example and will therefore use them all for our next filtering step. For filtering by overlap in position please remember that the range of the coordinates is [) in which the first number, but not the last number is considered as inside of the range."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "b5i3FNxxRyge",
+ "colab_type": "code",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 36
+ },
+ "outputId": "125d623a-3a4e-4e56-fe91-0d0ae4a9c10a"
+ },
+ "source": [
+ "# initialize empty list for storing genetic variants in RUNX1 exons\n",
+ "RUNX1_exon_geneticVariants = []\n",
+ "\n",
+ "# for each genetic variant identify their hg38GenomicPosition and check if it's in an exon\n",
+ "for geneticVariant_dcid in RUNX1_geneticVariants:\n",
+ " position = int(dc.get_property_values([geneticVariant_dcid], 'hg38GenomicPosition')[geneticVariant_dcid][0])\n",
+ " for exonCoordinates in RUNX1_exonCoordinates:\n",
+ " start, stop = exonCoordinates.strip('Position').split('To')\n",
+ " start, stop = int(start), int(stop)\n",
+ " # filter for variants within RUNX1 exons\n",
+ " if position >= start and position < stop:\n",
+ " RUNX1_exon_geneticVariants.append(geneticVariant_dcid)\n",
+ " break\n",
+ "RUNX1_exon_geneticVariants = list(set(RUNX1_exon_geneticVariants))\n",
+ "\n",
+ "# check how many of the genetic variants are in exons\n",
+ "print(len(RUNX1_exon_geneticVariants))"
+ ],
+ "execution_count": 14,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ "474\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "6LPPL03eR0ft",
+ "colab_type": "text"
+ },
+ "source": [
+ "### Filter genetic variants for ones that have been clinically studied\n",
+ "Great! We've identified 345 genetic variants in exons that are worth further consideration. Let's narrow our candidate list further by identifying genetic variants with clinical data associated with them (reported in ClinVar). We can do this by checking if the genetic variant has the property `clinVarID` using `get_property_values`. For ones that have been clinically reported we are going to grab the following additional clinical information for the genetic variant: `diseaseName`, `clinicalSignificance`, and `clinVarReviewStatus`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "uGmUfs8TR2Fb",
+ "colab_type": "code",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 188
+ },
+ "outputId": "ef66e1f5-33a9-431d-80b9-9d0e10d759a3"
+ },
+ "source": [
+ "# initialize Empty Data Frame\n",
+ "column_names = ['name', 'clinVarAlleleID', 'diseaseName', 'clinicalSignificance', 'clinVarReviewStatus', 'dcid']\n",
+ "df_RUNX1_genVar_clinical = pd.DataFrame(columns=column_names)\n",
+ "\n",
+ "for geneticVariant_dcid in RUNX1_exon_geneticVariants:\n",
+ " clinVarID = dc.get_property_values([geneticVariant_dcid], 'clinVarAlleleID')[geneticVariant_dcid][0]\n",
+ " if clinVarID.isdigit():\n",
+ " name = geneticVariant_dcid.strip('bio/')\n",
+ " diseaseName = dc.get_property_values( [geneticVariant_dcid], 'diseaseName')[geneticVariant_dcid][0]\n",
+ " clinicalSignificance = dc.get_property_values( [geneticVariant_dcid], 'clinicalSignificance')[geneticVariant_dcid][0]\n",
+ " clinVarReviewStatus = dc.get_property_values( [geneticVariant_dcid], 'clinVarReviewStatus')[geneticVariant_dcid][0]\n",
+ " df_RUNX1_genVar_clinical = df_RUNX1_genVar_clinical.append({'name': name, 'clinVarAlleleID': clinVarID, 'diseaseName': diseaseName, \\\n",
+ " 'clinicalSignificance': clinicalSignificance, 'clinVarReviewStatus': clinVarReviewStatus, \\\n",
+ " 'dcid': geneticVariant_dcid}, ignore_index=True)\n",
+ "\n",
+ "# visualize the head of the dataframe containing the clinical info\n",
+ "print(df_RUNX1_genVar_clinical.head())\n",
+ "\n",
+ "# see how many clinical genetic variants in RUNX1\n",
+ "print(df_RUNX1_genVar_clinical.shape[0])"
+ ],
+ "execution_count": 15,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ " name ... dcid\n",
+ "0 rs1569084170 ... bio/rs1569084170\n",
+ "1 643883 ... bio/643883\n",
+ "2 rs1569002337 ... bio/rs1569002337\n",
+ "3 839054 ... bio/839054\n",
+ "4 rs150481777 ... bio/rs150481777\n",
+ "\n",
+ "[5 rows x 6 columns]\n",
+ "474\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "4m57MS1gR4Wp",
+ "colab_type": "text"
+ },
+ "source": [
+ "## Filter For Pathogenic Genetic Variants\n",
+ "There are 345 genetic variants in the exons of RUNX1 that have recorded clinical information on them. Filter out the ones that are benign to establish our final candidate list of genetic variants that effect the function of RUNX1. We'll do this by filtering our pandas dataframe with the clinical information on the genetic variants."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "PNoAw2XjR6M5",
+ "colab_type": "code",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 55
+ },
+ "outputId": "2ddae2f0-5b5c-4655-ec85-fbe0d8b775be"
+ },
+ "source": [
+ "# identify the clinical significance types for the genetic variants\n",
+ "print(df_RUNX1_genVar_clinical.clinicalSignificance.unique())"
+ ],
+ "execution_count": 16,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ "['ClinSigPathogenic' 'ClinSigUncertain' 'ClinSigBenign'\n",
+ " 'ClinSigConflictingPathogenicity']\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "xpmj5i3dR8sb",
+ "colab_type": "code",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 549
+ },
+ "outputId": "953deea3-fea1-4f84-b4a5-0653010abe74"
+ },
+ "source": [
+ "# filter genetic variants for those with pathogenicity\n",
+ "clinSig = ['ClinSigPathogenic']\n",
+ "df_final_geneticVariants = df_RUNX1_genVar_clinical[df_RUNX1_genVar_clinical.clinicalSignificance.isin(clinSig)]\n",
+ "\n",
+ "# check how many genetic variants made the final cut\n",
+ "print(df_final_geneticVariants.shape[0])\n",
+ "\n",
+ "# identify the diseases associated with these pathogenic variants\n",
+ "print(df_final_geneticVariants.diseaseName.unique())\n",
+ "\n",
+ "# print the final genetic variant dataframe\n",
+ "print(df_final_geneticVariants)"
+ ],
+ "execution_count": 17,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ "21\n",
+ "['not provided'\n",
+ " 'Familial platelet disorder with associated myeloid malignancy'\n",
+ " 'Acute myeloid leukemia']\n",
+ " name ... dcid\n",
+ "0 rs1569084170 ... bio/rs1569084170\n",
+ "26 871175 ... bio/871175\n",
+ "64 rs1555889984 ... bio/rs1555889984\n",
+ "65 rs1569084082 ... bio/rs1569084082\n",
+ "90 rs587776811 ... bio/rs587776811\n",
+ "95 rs74315451 ... bio/rs74315451\n",
+ "129 rs1569008655 ... bio/rs1569008655\n",
+ "143 rs1569084530 ... bio/rs1569084530\n",
+ "183 rs1555899813 ... bio/rs1555899813\n",
+ "192 rs74315450 ... bio/rs74315450\n",
+ "205 rs1060499616 ... bio/rs1060499616\n",
+ "207 rs121912498 ... bio/rs121912498\n",
+ "240 840868 ... bio/840868\n",
+ "266 rs121912499 ... bio/rs121912499\n",
+ "321 647118 ... bio/647118\n",
+ "326 rs1057519748 ... bio/rs1057519748\n",
+ "377 rs1569061762 ... bio/rs1569061762\n",
+ "394 rs1569061831 ... bio/rs1569061831\n",
+ "408 rs1569061768 ... bio/rs1569061768\n",
+ "444 rs1555884790 ... bio/rs1555884790\n",
+ "463 869209 ... bio/869209\n",
+ "\n",
+ "[21 rows x 6 columns]\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "KF3dr_kAR8Xk",
+ "colab_type": "text"
+ },
+ "source": [
+ "##Conclusion of RUNX1 Analysis\n",
+ "In this exercise we were able to filter millions of genetic variants to identify a handful that meet our specific parameters. To do this we used multiple datasets from UCSC Genome Browser, NCBI/gene, and ClinVar all in a single python notebook. We found that there are 16 genetic variants in exons of RUNX1, which have been reported to be pathogenic by clinical data. We also learned that these pathogenic variants are associated with Familial platelet disorder with associated myeloid malignancy. Such analyses can preformed using any gene and is a way to easily identify candidate genetic variants of interest. The synthesis of data of different types acorss multiple databases enables easy complex analyses all in a single python notebook."
+ ]
+ }
+ ]
+}
\ No newline at end of file
From 746b0d2a72528eaa71fe8cf9102106f56617e977 Mon Sep 17 00:00:00 2001
From: TJann <18621425+tjann@users.noreply.github.com>
Date: Tue, 28 Jul 2020 12:27:57 -0700
Subject: [PATCH 2/5] Created using Colaboratory
---
..._Study_Analyzing_Income_Distribution.ipynb | 642 ++++++++++++++++++
1 file changed, 642 insertions(+)
create mode 100644 Case_Study_Analyzing_Income_Distribution.ipynb
diff --git a/Case_Study_Analyzing_Income_Distribution.ipynb b/Case_Study_Analyzing_Income_Distribution.ipynb
new file mode 100644
index 00000000..81855e02
--- /dev/null
+++ b/Case_Study_Analyzing_Income_Distribution.ipynb
@@ -0,0 +1,642 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Case Study: Analyzing Income Distribution",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "nnY4S5m_boXI",
+ "colab_type": "text"
+ },
+ "source": [
+ "Copyright 2020 Google LLC.\n",
+ "SPDX-License-Identifier: Apache-2.0\n",
+ "\n",
+ "**Notebook Version** - 1.0.1"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "9_SFWWxoSoMn",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "source": [
+ "# Install datacommons\n",
+ "!pip install datacommons --upgrade --quiet"
+ ],
+ "execution_count": 1,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "vWo_v5cI-JXC",
+ "colab_type": "text"
+ },
+ "source": [
+ "# Analyzing Income Distribution\n",
+ "\n",
+ "The American Community Survey (published by the US Census) annually reports the number of individuals in a given income bracket at the State level. We can use this information, stored in Data Commons, to visualize disparity in income for each State in the US. Our goal for this tutorial will be to generate a plot that visualizes the total number of individuals across a given set of income brackets for a given state. \n",
+ "\n",
+ "Before we begin, we'll setup our notebook."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "OwquZnRa-JXD",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "source": [
+ "# Import the Data Commons library\n",
+ "import datacommons as dc\n",
+ "\n",
+ "# Import other libraries\n",
+ "import pandas as pd\n",
+ "import matplotlib.pyplot as plt\n",
+ "import numpy as np\n"
+ ],
+ "execution_count": 2,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Hnk34Wgr-JXG",
+ "colab_type": "text"
+ },
+ "source": [
+ "## Preparing the Data\n",
+ "\n",
+ "We'll begin by creating a dataframe with states and their total population. We can use **`get_places_in`** to get all States within the United States. We can then call **`get_stats`** to get the population of each State."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "4tXBoR3h-JXI",
+ "colab_type": "code",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 210
+ },
+ "outputId": "a486ae30-f530-4388-f9d3-4f1b12d940e1"
+ },
+ "source": [
+ "# Initialize a DataFrame holding the states\n",
+ "data = pd.DataFrame({'state': dc.get_places_in(['country/USA'], 'State')['country/USA']})\n",
+ "\n",
+ "# Get all state names and store it in a column \"name\"\n",
+ "# Get the first name, if there are multiple for a state\n",
+ "data['name'] = data['state'].map(dc.get_property_values(data['state'], 'name')).str[0]\n",
+ "\n",
+ "# Get the total count of all persons for each state\n",
+ "data['Count_Person'] = data['state'].map(dc.get_stats(data['state'], \"Count_Person\", ['2017']))\n",
+ "\n",
+ "# Extract just the number for population\n",
+ "get_value = lambda row: row.get('data').get('2017')\n",
+ "data['Count_Person'] = data['Count_Person'].apply(get_value).dropna()\n",
+ "\n",
+ "# Display the first five rows of the table\n",
+ "data.head(5)"
+ ],
+ "execution_count": 3,
+ "outputs": [
+ {
+ "output_type": "execute_result",
+ "data": {
+ "text/html": [
+ "| \n", + " | state | \n", + "name | \n", + "Count_Person | \n", + "
|---|---|---|---|
| 0 | \n", + "geoId/01 | \n", + "Alabama | \n", + "4874486 | \n", + "
| 1 | \n", + "geoId/02 | \n", + "Alaska | \n", + "739700 | \n", + "
| 2 | \n", + "geoId/04 | \n", + "Arizona | \n", + "7044008 | \n", + "
| 3 | \n", + "geoId/05 | \n", + "Arkansas | \n", + "3001345 | \n", + "
| 4 | \n", + "geoId/06 | \n", + "California | \n", + "39358497 | \n", + "
| \n", + " | state | \n", + "name | \n", + "Count_Person | \n", + "Count_Household_IncomeOfUpto10000USDollar | \n", + "Count_Household_IncomeOf10000To14999USDollar | \n", + "Count_Household_IncomeOf15000To19999USDollar | \n", + "Count_Household_IncomeOf20000To24999USDollar | \n", + "Count_Household_IncomeOf25000To29999USDollar | \n", + "Count_Household_IncomeOf30000To34999USDollar | \n", + "Count_Household_IncomeOf35000To39999USDollar | \n", + "Count_Household_IncomeOf40000To44999USDollar | \n", + "Count_Household_IncomeOf45000To49999USDollar | \n", + "Count_Household_IncomeOf50000To59999USDollar | \n", + "Count_Household_IncomeOf60000To74999USDollar | \n", + "Count_Household_IncomeOf75000To99999USDollar | \n", + "Count_Household_IncomeOf100000To124999USDollar | \n", + "Count_Household_IncomeOf125000To149999USDollar | \n", + "Count_Household_IncomeOf150000To199999USDollar | \n", + "Count_Household_IncomeOf200000OrMoreUSDollar | \n", + "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", + "geoId/01 | \n", + "Alabama | \n", + "4874486 | \n", + "170650 | \n", + "121688 | \n", + "116273 | \n", + "112406 | \n", + "102590 | \n", + "100223 | \n", + "92139 | \n", + "88987 | \n", + "76070 | \n", + "149492 | \n", + "176589 | \n", + "206436 | \n", + "133618 | \n", + "76939 | \n", + "71748 | \n", + "60847 | \n", + "
| 1 | \n", + "geoId/02 | \n", + "Alaska | \n", + "739700 | \n", + "9650 | \n", + "7838 | \n", + "8013 | \n", + "8854 | \n", + "8383 | \n", + "8914 | \n", + "8998 | \n", + "10163 | \n", + "8536 | \n", + "18810 | \n", + "26224 | \n", + "36745 | \n", + "29925 | \n", + "19752 | \n", + "23596 | \n", + "18135 | \n", + "
| 2 | \n", + "geoId/04 | \n", + "Arizona | \n", + "7044008 | \n", + "178384 | \n", + "118676 | \n", + "123075 | \n", + "131903 | \n", + "127741 | \n", + "127570 | \n", + "119431 | \n", + "125103 | \n", + "107245 | \n", + "208318 | \n", + "254294 | \n", + "303811 | \n", + "200818 | \n", + "119347 | \n", + "119778 | \n", + "116817 | \n", + "
| 3 | \n", + "geoId/05 | \n", + "Arkansas | \n", + "3001345 | \n", + "97226 | \n", + "78535 | \n", + "74451 | \n", + "75194 | \n", + "70211 | \n", + "67958 | \n", + "61588 | \n", + "60814 | \n", + "50978 | \n", + "95836 | \n", + "111618 | \n", + "120034 | \n", + "72752 | \n", + "40780 | \n", + "35900 | \n", + "33416 | \n", + "
| 4 | \n", + "geoId/06 | \n", + "California | \n", + "39358497 | \n", + "694945 | \n", + "604666 | \n", + "536203 | \n", + "568994 | \n", + "523563 | \n", + "539988 | \n", + "500571 | \n", + "509113 | \n", + "456152 | \n", + "895758 | \n", + "1199773 | \n", + "1568843 | \n", + "1206637 | \n", + "818690 | \n", + "1008388 | \n", + "1255844 | \n", + "
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "srAnaUPPbrH6",
+ "colab_type": "text"
+ },
+ "source": [
+ "Copyright 2020 Google LLC.\n",
+ "SPDX-License-Identifier: Apache-2.0\n",
+ "\n",
+ "**Notebook Version** - 1.0.1"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "7Et41jikt2yA",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "source": [
+ "from sklearn.linear_model import LinearRegression\n",
+ "from sklearn.model_selection import train_test_split\n",
+ "from google.colab import drive\n",
+ "\n",
+ "import matplotlib.pyplot as plt\n",
+ "import numpy as np\n",
+ "import pandas as pd\n",
+ "\n",
+ "import json"
+ ],
+ "execution_count": 1,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "yjBCrzJfqHtM",
+ "colab_type": "text"
+ },
+ "source": [
+ "# Case Study: Prevalence of Obesity in 500 US Cities\n",
+ "\n",
+ "Obesity is well known to correlate with health factors such as high blood pressure, but is also known to correlate with economic factors such as low-income, unemployment, etc [[1]](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3198075/)[[2]](https://www.ncbi.nlm.nih.gov/pubmed/26562758). The Center for Disease Control (CDC) provides prevalence percentages on health conditions such as [obesity](https://browser.datacommons.org/kg?dcid=dc/p/8f94jymf10dh7), [high blood pressure](https://browser.datacommons.org/kg?dcid=dc/p/lddprn26my0wh), and [high cholesterol](https://browser.datacommons.org/kg?dcid=dc/p/ybn085073e7z6) for approximately 500 major cities in the US (e.g. [San Francisco](https://browser.datacommons.org/kg?dcid=geoId/0667000), [New York](https://browser.datacommons.org/kg?dcid=geoId/3651000), and [Austin](https://browser.datacommons.org/kg?dcid=geoId/4805000)). Meanwhile, the US Bureau of Labor Statistics provides [unemployment rates](https://browser.datacommons.org/kg?dcid=dc/p/820mxx8ejlrj7) while the US Census provides [poverty rates](https://browser.datacommons.org/kg?dcid=dc/p/mrbgbmbe5lvyh) for most cities across the United States. \n",
+ "\n",
+ "Even though these statistics come from different datasets across different government agencies with different storage formats, Data Commons surfaces each of these in a single, uniform knowledge graph. In fact, you can see this in the [browser](https://browser.datacommons.org/kg?dcid=dc/9xkkc71) by looking at the *provenance* column. Let's use the data in Data Commons to create a linear regression model that incorporates variables:\n",
+ "\n",
+ "- Prevalence of high blood pressure\n",
+ "- Unemployment rate\n",
+ "- Percent of population living with income below the poverty line\n",
+ "\n",
+ "to predict the prevalence of obesity in the 500 cities that the CDC provides data for. One thing you may note is that the US Census also provides employment statistics (you can see this by navigating to the \"employment\" and \"employmentStatus\" sections for [San Francisco](https://browser.datacommons.org/kg?dcid=geoId/0667000) and observing the different provenances). Our choice of using statistics from the Bureau of Labor Statistics is purely demonstrative, but it would be interesting to see if similar results can be reproduced using US Census employment statistics. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "GQrede0Bdm-k",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "source": [
+ "!pip install datacommons --upgrade --quiet\n",
+ "\n",
+ "import datacommons as dc"
+ ],
+ "execution_count": 2,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "pNHUbzq_dnGc",
+ "colab_type": "text"
+ },
+ "source": [
+ "To get started, we'll need the cities that CDC provides data for. We can query Data Commons for Cities that are members of [CDC500_City](https://browser.datacommons.org/kg?dcid=CDC500_City). The default return limit is 100, so we'll explicitly set it to 500."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "4CoM0LD4kZSN",
+ "colab_type": "code",
+ "cellView": "both",
+ "colab": {}
+ },
+ "source": [
+ "city_dcids = dc.get_property_values([\"CDC500_City\"], \"member\", limit=500)"
+ ],
+ "execution_count": 3,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "2JmxK2KrnCcQ",
+ "colab_type": "text"
+ },
+ "source": [
+ "We then initialize a `pandas.DataFrame` with the dcids in this list."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "IM4ajcwoNtXr",
+ "colab_type": "code",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 210
+ },
+ "outputId": "d02424fe-8a0a-407f-c279-b7dc82d66050"
+ },
+ "source": [
+ "# Create the Pandas DataFrame \n",
+ "data = pd.DataFrame(city_dcids)\n",
+ "# Simplify the column name\n",
+ "data.columns = ['City']\n",
+ "# Get the name of each city\n",
+ "# If a city has multiple common names (e.g. \"New York\" and \"New York City\"), use the first one\n",
+ "data['CityName'] = data['City'].map(dc.get_property_values(data['City'], 'name')).str[0]\n",
+ "data.dropna(inplace=True)\n",
+ "\n",
+ "# Show the top five rows\n",
+ "data.head(5)"
+ ],
+ "execution_count": 4,
+ "outputs": [
+ {
+ "output_type": "execute_result",
+ "data": {
+ "text/html": [
+ "| \n", + " | City | \n", + "CityName | \n", + "
|---|---|---|
| 0 | \n", + "geoId/0107000 | \n", + "Birmingham | \n", + "
| 1 | \n", + "geoId/0135896 | \n", + "Hoover | \n", + "
| 2 | \n", + "geoId/0137000 | \n", + "Huntsville | \n", + "
| 3 | \n", + "geoId/0150000 | \n", + "Mobile | \n", + "
| 4 | \n", + "geoId/0151000 | \n", + "Montgomery | \n", + "
| \n", + " | City | \n", + "CityName | \n", + "Obesity | \n", + "HighBp | \n", + "PersonCount | \n", + "Unemployment | \n", + "PovertyCount | \n", + "
|---|---|---|---|---|---|---|---|
| 0 | \n", + "geoId/0107000 | \n", + "Birmingham | \n", + "42.7 | \n", + "44.1 | \n", + "212211.0 | \n", + "6.6 | \n", + "63503.0 | \n", + "
| 1 | \n", + "geoId/0135896 | \n", + "Hoover | \n", + "28.3 | \n", + "29.7 | \n", + "83547.0 | \n", + "3.9 | \n", + "5003.0 | \n", + "
| 2 | \n", + "geoId/0137000 | \n", + "Huntsville | \n", + "34.4 | \n", + "36.3 | \n", + "185594.0 | \n", + "5.2 | \n", + "32412.0 | \n", + "
| 3 | \n", + "geoId/0150000 | \n", + "Mobile | \n", + "35.3 | \n", + "41.0 | \n", + "194669.0 | \n", + "6.8 | \n", + "43468.0 | \n", + "
| 4 | \n", + "geoId/0151000 | \n", + "Montgomery | \n", + "39.4 | \n", + "40.8 | \n", + "202967.0 | \n", + "5.8 | \n", + "47371.0 | \n", + "
| \n", + " | CityName | \n", + "Obesity | \n", + "HighBp | \n", + "Unemployment | \n", + "PovertyRate | \n", + "
|---|---|---|---|---|---|
| 0 | \n", + "Birmingham | \n", + "42.7 | \n", + "44.1 | \n", + "6.6 | \n", + "29.924462 | \n", + "
| 1 | \n", + "Hoover | \n", + "28.3 | \n", + "29.7 | \n", + "3.9 | \n", + "5.988246 | \n", + "
| 2 | \n", + "Huntsville | \n", + "34.4 | \n", + "36.3 | \n", + "5.2 | \n", + "17.463927 | \n", + "
| 3 | \n", + "Mobile | \n", + "35.3 | \n", + "41.0 | \n", + "6.8 | \n", + "22.329184 | \n", + "
| 4 | \n", + "Montgomery | \n", + "39.4 | \n", + "40.8 | \n", + "5.8 | \n", + "23.339262 | \n", + "
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "ZXL-qge7bmCa",
+ "colab_type": "text"
+ },
+ "source": [
+ "Copyright 2019 Google LLC.\n",
+ "SPDX-License-Identifier: Apache-2.0\n",
+ "\n",
+ "**Notebook Version** - 1.0.0"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "knFLVFogCWq0",
+ "colab_type": "text"
+ },
+ "source": [
+ "# Analyzing Census Data with Data Commons\n",
+ "\n",
+ "Every year, the American Community Survey (published by the US Census) reports thousands of variables about demographics, economics, housing, and more. This information, stored in Data Commons, is available to everyone for data science projects, education, and exploration. This tutorial introduces the Data Commons graph and two of its tools to help integrate its data into your data science projects: (1) the [browser](https://browser.datacommons.org/) and (2) the [Python API](https://github.com/datacommonsorg/api-python).\n",
+ "\n",
+ "## What is Data Commons?\n",
+ "\n",
+ "Data Commons is an open knowledge repository that combines data from public datasets using mapped common entities. It contains statements about real world objects such as\n",
+ "\n",
+ "* [Santa Clara County](https://browser.datacommons.org/kg?dcid=geoId/06085) is contained in the [State of California](https://browser.datacommons.org/kg?dcid=geoId/06)\n",
+ "* The latitude of [Berkeley, CA](https://browser.datacommons.org/kg?dcid=geoId/0606000) is 37.8703\n",
+ "* [The population of Maryland](https://browser.datacommons.org/kg?dcid=dc/p/psjx4xy30nws1) was [6,003,435 in 2018](https://browser.datacommons.org/kg?dcid=dc%2Fo%2Fx1tlfg4ll9yr9).\n",
+ "\n",
+ "In the graph, [*entities*](https://en.wikipedia.org/wiki/Entity) like [Santa Clara County](https://browser.datacommons.org/kg?dcid=geoId/06085) are represented by nodes. Every node has a type corresponding to what the node represents. For example, [California](https://browser.datacommons.org/kg?dcid=geoId/06) is a [State](https://schema.org/State). *Relations* between entities are represented by edges between these nodes. For example, the statement \"Santa Clara County is contained in the State of California\" is represented in the graph as two nodes: \"Santa Clara County\" and \"California\" with an edge labeled \"[containedInPlace](https://schema.org/containedInPlace)\" pointing from Santa Clara to California. Data Commons closely follows the [Schema.org data model](https://schema.org/docs/datamodel.html) and leverages Schema.org schema to provide a common set of types and properties.\n",
+ "\n",
+ "## Data Commons Browser\n",
+ "\n",
+ "The [Data Commons browser](https://browser.datacommons.org/) provides a way to explore the data in a human-readable format. It is the best way to explore what is in Data Commons. Searching in the browser for an entity like [Mountain View](https://browser.datacommons.org/kg?dcid=geoId/0649670), takes you to a page about the entity, including properties like [containedInPlace](https://browser.datacommons.org/kg?dcid=containedInPlace) and [timezone](https://browser.datacommons.org/kg?dcid=timezone).\n",
+ "\n",
+ "An important property for all entities is the **`dcid`**. The `dcid` (Data Commons identifier) is a unique identifier assigned to each entity in the knowledge graph. With this identifier, you will be able to search for and query information on the given entity in ways that we will discuss later. The `dcid` is listed at the top of the page next to \"About: \" and also in the list of properties.\n",
+ "\n",
+ "## Python API\n",
+ "\n",
+ "The [Python API](https://github.com/datacommonsorg/api-python) provides functions for users to extract structured information from Data Commons programmatically and view them in different formats such as Python `dict`s and [Pandas](https://pandas.pydata.org/) DataFrames. DataFrames allow access to all the data processing, analytical and visualization tools provided by packages such as Pandas, NumPy, SciPy, and Matplotlib.\n",
+ "\n",
+ "Let's begin by loading the Data Commons library and the standard data science libraries:\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "zxg_UfUgIq-Y",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "source": [
+ "# Install datacommons\n",
+ "!pip install datacommons --upgrade --quiet\n",
+ "# Import Data Commons\n",
+ "import datacommons as dc\n",
+ "\n",
+ "# Import other required libraries\n",
+ "import matplotlib.pyplot as plt\n",
+ "import matplotlib.patches as mpatches\n",
+ "import pandas as pd\n",
+ "\n",
+ "import json"
+ ],
+ "execution_count": 1,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "rrkDVNp0JI6g",
+ "colab_type": "text"
+ },
+ "source": [
+ "## Example: Median Age vs Population by State\n",
+ "For this exercise, we will be comparing the median ages and population count for US states, counties, and cities. First, let's lookup the dcid for the '[United States](https://browser.datacommons.org/kg?dcid=country/USA)'."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "0peszosKQU_g",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "source": [
+ "# In the browser, we saw that the dcid for United States is country/USA\n",
+ "usa = 'country/USA'"
+ ],
+ "execution_count": 2,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "kiPXhAJkDyjk",
+ "colab_type": "text"
+ },
+ "source": [
+ "### Using `get_places_in` to Query Administrative Areas\n",
+ "\n",
+ "The Python API defines a number of convenience functions for building Pandas dataframes with information in the datacommons graph. We will be using **`get_places_in`** which requires three arguments:\n",
+ "\n",
+ "- `dcids` - A list or `pandas.Series` of dcids identifying administrative areas that we wish to get containing places for.\n",
+ "- `place_type` - The type of the administrative area that we wish to query for.\n",
+ "\n",
+ "In Data Commons, the 'containedInPlace' property relates administrative areas to its containing administrative areas. Concretely, every US 'State' node has a directed edge to the USA 'Country' node where the name of this edge is 'containedInPlace'. The same also goes for 'County' to 'State' nodes and 'City' to 'County'.\n",
+ "\n",
+ "When we provide a list of places to `get_places_in`, we get back a `dict` mapping each place to all the places contained inside it."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "MXy_j6VS04g5",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "source": [
+ "# Create three DataFrames storing states, counties, and cities within the United States respectively.\n",
+ "state = pd.DataFrame({'state': dc.get_places_in([usa], 'State')[usa]})\n",
+ "county = pd.DataFrame({'county': dc.get_places_in([usa], 'County')[usa]})\n",
+ "city = pd.DataFrame({'city': dc.get_places_in([usa], 'City')[usa]})"
+ ],
+ "execution_count": 3,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "fKbexXS60mLw",
+ "colab_type": "text"
+ },
+ "source": [
+ "Let's see what each table has."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "5ZnJp5ns0nXt",
+ "colab_type": "code",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 210
+ },
+ "outputId": "b28601c1-7ec7-4784-d920-66b9d62c64e4"
+ },
+ "source": [
+ "# Display the state data\n",
+ "state.head(5)"
+ ],
+ "execution_count": 4,
+ "outputs": [
+ {
+ "output_type": "execute_result",
+ "data": {
+ "text/html": [
+ "| \n", + " | state | \n", + "
|---|---|
| 0 | \n", + "geoId/01 | \n", + "
| 1 | \n", + "geoId/02 | \n", + "
| 2 | \n", + "geoId/04 | \n", + "
| 3 | \n", + "geoId/05 | \n", + "
| 4 | \n", + "geoId/06 | \n", + "
| \n", + " | county | \n", + "
|---|---|
| 0 | \n", + "geoId/01001 | \n", + "
| 1 | \n", + "geoId/01003 | \n", + "
| 2 | \n", + "geoId/01005 | \n", + "
| 3 | \n", + "geoId/01007 | \n", + "
| 4 | \n", + "geoId/01009 | \n", + "
| \n", + " | city | \n", + "
|---|---|
| 0 | \n", + "geoId/0100100 | \n", + "
| 1 | \n", + "geoId/0100124 | \n", + "
| 2 | \n", + "geoId/0100460 | \n", + "
| 3 | \n", + "geoId/0100484 | \n", + "
| 4 | \n", + "geoId/0100676 | \n", + "
| \n", + " | state | \n", + "state_name | \n", + "
|---|---|---|
| 0 | \n", + "geoId/01 | \n", + "Alabama | \n", + "
| 1 | \n", + "geoId/02 | \n", + "Alaska | \n", + "
| 2 | \n", + "geoId/04 | \n", + "Arizona | \n", + "
| 3 | \n", + "geoId/05 | \n", + "Arkansas | \n", + "
| 4 | \n", + "geoId/06 | \n", + "California | \n", + "
| \n", + " | county | \n", + "county_name | \n", + "
|---|---|---|
| 0 | \n", + "geoId/01001 | \n", + "Autauga County | \n", + "
| 1 | \n", + "geoId/01003 | \n", + "Baldwin County | \n", + "
| 2 | \n", + "geoId/01005 | \n", + "Barbour County | \n", + "
| 3 | \n", + "geoId/01007 | \n", + "Bibb County | \n", + "
| 4 | \n", + "geoId/01009 | \n", + "Blount County | \n", + "
| \n", + " | city | \n", + "city_name | \n", + "
|---|---|---|
| 0 | \n", + "geoId/0100100 | \n", + "Abanda | \n", + "
| 1 | \n", + "geoId/0100124 | \n", + "Abbeville | \n", + "
| 2 | \n", + "geoId/0100460 | \n", + "Adamsville | \n", + "
| 3 | \n", + "geoId/0100484 | \n", + "Addison | \n", + "
| 4 | \n", + "geoId/0100676 | \n", + "Akron | \n", + "
| \n", + " | state | \n", + "state_name | \n", + "count | \n", + "med_age | \n", + "
|---|---|---|---|---|
| 0 | \n", + "geoId/01 | \n", + "Alabama | \n", + "4874486 | \n", + "38.7 | \n", + "
| 1 | \n", + "geoId/02 | \n", + "Alaska | \n", + "739700 | \n", + "33.9 | \n", + "
| 2 | \n", + "geoId/04 | \n", + "Arizona | \n", + "7044008 | \n", + "37.2 | \n", + "
| 3 | \n", + "geoId/05 | \n", + "Arkansas | \n", + "3001345 | \n", + "37.9 | \n", + "
| 4 | \n", + "geoId/06 | \n", + "California | \n", + "39358497 | \n", + "36.1 | \n", + "
| \n", + " | county | \n", + "county_name | \n", + "count | \n", + "med_age | \n", + "
|---|---|---|---|---|
| 0 | \n", + "geoId/01001 | \n", + "Autauga County | \n", + "55390.0 | \n", + "37.8 | \n", + "
| 1 | \n", + "geoId/01003 | \n", + "Baldwin County | \n", + "212521.0 | \n", + "42.6 | \n", + "
| 2 | \n", + "geoId/01005 | \n", + "Barbour County | \n", + "25157.0 | \n", + "39.7 | \n", + "
| 3 | \n", + "geoId/01007 | \n", + "Bibb County | \n", + "22550.0 | \n", + "39.8 | \n", + "
| 4 | \n", + "geoId/01009 | \n", + "Blount County | \n", + "57787.0 | \n", + "40.9 | \n", + "
| \n", + " | city | \n", + "city_name | \n", + "count | \n", + "med_age | \n", + "
|---|---|---|---|---|
| 0 | \n", + "geoId/0100100 | \n", + "Abanda | \n", + "174.0 | \n", + "32.7 | \n", + "
| 1 | \n", + "geoId/0100124 | \n", + "Abbeville | \n", + "2566.0 | \n", + "44.9 | \n", + "
| 2 | \n", + "geoId/0100460 | \n", + "Adamsville | \n", + "4342.0 | \n", + "42.7 | \n", + "
| 3 | \n", + "geoId/0100484 | \n", + "Addison | \n", + "728.0 | \n", + "47.6 | \n", + "
| 4 | \n", + "geoId/0100676 | \n", + "Akron | \n", + "332.0 | \n", + "40.3 | \n", + "
"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "ZXL-qge7bmCa",
- "colab_type": "text"
- },
- "source": [
- "Copyright 2019 Google LLC.\n",
- "SPDX-License-Identifier: Apache-2.0\n",
- "\n",
- "**Notebook Version** - 1.0.0"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "knFLVFogCWq0",
- "colab_type": "text"
- },
- "source": [
- "# Analyzing Census Data with Data Commons\n",
- "\n",
- "Every year, the American Community Survey (published by the US Census) reports thousands of variables about demographics, economics, housing, and more. This information, stored in Data Commons, is available to everyone for data science projects, education, and exploration. This tutorial introduces the Data Commons graph and two of its tools to help integrate its data into your data science projects: (1) the [browser](https://browser.datacommons.org/) and (2) the [Python API](https://github.com/datacommonsorg/api-python).\n",
- "\n",
- "## What is Data Commons?\n",
- "\n",
- "Data Commons is an open knowledge repository that combines data from public datasets using mapped common entities. It contains statements about real world objects such as\n",
- "\n",
- "* [Santa Clara County](https://browser.datacommons.org/kg?dcid=geoId/06085) is contained in the [State of California](https://browser.datacommons.org/kg?dcid=geoId/06)\n",
- "* The latitude of [Berkeley, CA](https://browser.datacommons.org/kg?dcid=geoId/0606000) is 37.8703\n",
- "* [The population of Maryland](https://browser.datacommons.org/kg?dcid=dc/p/psjx4xy30nws1) was [6,003,435 in 2018](https://browser.datacommons.org/kg?dcid=dc%2Fo%2Fx1tlfg4ll9yr9).\n",
- "\n",
- "In the graph, [*entities*](https://en.wikipedia.org/wiki/Entity) like [Santa Clara County](https://browser.datacommons.org/kg?dcid=geoId/06085) are represented by nodes. Every node has a type corresponding to what the node represents. For example, [California](https://browser.datacommons.org/kg?dcid=geoId/06) is a [State](https://schema.org/State). *Relations* between entities are represented by edges between these nodes. For example, the statement \"Santa Clara County is contained in the State of California\" is represented in the graph as two nodes: \"Santa Clara County\" and \"California\" with an edge labeled \"[containedInPlace](https://schema.org/containedInPlace)\" pointing from Santa Clara to California. Data Commons closely follows the [Schema.org data model](https://schema.org/docs/datamodel.html) and leverages Schema.org schema to provide a common set of types and properties.\n",
- "\n",
- "## Data Commons Browser\n",
- "\n",
- "The [Data Commons browser](https://browser.datacommons.org/) provides a way to explore the data in a human-readable format. It is the best way to explore what is in Data Commons. Searching in the browser for an entity like [Mountain View](https://browser.datacommons.org/kg?dcid=geoId/0649670), takes you to a page about the entity, including properties like [containedInPlace](https://browser.datacommons.org/kg?dcid=containedInPlace) and [timezone](https://browser.datacommons.org/kg?dcid=timezone).\n",
- "\n",
- "An important property for all entities is the **`dcid`**. The `dcid` (Data Commons identifier) is a unique identifier assigned to each entity in the knowledge graph. With this identifier, you will be able to search for and query information on the given entity in ways that we will discuss later. The `dcid` is listed at the top of the page next to \"About: \" and also in the list of properties.\n",
- "\n",
- "## Python API\n",
- "\n",
- "The [Python API](https://github.com/datacommonsorg/api-python) provides functions for users to extract structured information from Data Commons programmatically and view them in different formats such as Python `dict`s and [Pandas](https://pandas.pydata.org/) DataFrames. DataFrames allow access to all the data processing, analytical and visualization tools provided by packages such as Pandas, NumPy, SciPy, and Matplotlib.\n",
- "\n",
- "Let's begin by loading the Data Commons library and the standard data science libraries:\n"
- ]
- },
- {
- "cell_type": "code",
- "metadata": {
- "id": "zxg_UfUgIq-Y",
- "colab_type": "code",
- "colab": {}
- },
- "source": [
- "# Install datacommons\n",
- "!pip install datacommons --upgrade --quiet\n",
- "# Import Data Commons\n",
- "import datacommons as dc\n",
- "\n",
- "# Import other required libraries\n",
- "import matplotlib.pyplot as plt\n",
- "import matplotlib.patches as mpatches\n",
- "import pandas as pd\n",
- "\n",
- "import json"
- ],
- "execution_count": 1,
- "outputs": []
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "rrkDVNp0JI6g",
- "colab_type": "text"
- },
- "source": [
- "## Example: Median Age vs Population by State\n",
- "For this exercise, we will be comparing the median ages and population count for US states, counties, and cities. First, let's lookup the dcid for the '[United States](https://browser.datacommons.org/kg?dcid=country/USA)'."
- ]
- },
- {
- "cell_type": "code",
- "metadata": {
- "id": "0peszosKQU_g",
- "colab_type": "code",
- "colab": {}
- },
- "source": [
- "# In the browser, we saw that the dcid for United States is country/USA\n",
- "usa = 'country/USA'"
- ],
- "execution_count": 2,
- "outputs": []
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "kiPXhAJkDyjk",
- "colab_type": "text"
- },
- "source": [
- "### Using `get_places_in` to Query Administrative Areas\n",
- "\n",
- "The Python API defines a number of convenience functions for building Pandas dataframes with information in the datacommons graph. We will be using **`get_places_in`** which requires three arguments:\n",
- "\n",
- "- `dcids` - A list or `pandas.Series` of dcids identifying administrative areas that we wish to get containing places for.\n",
- "- `place_type` - The type of the administrative area that we wish to query for.\n",
- "\n",
- "In Data Commons, the 'containedInPlace' property relates administrative areas to its containing administrative areas. Concretely, every US 'State' node has a directed edge to the USA 'Country' node where the name of this edge is 'containedInPlace'. The same also goes for 'County' to 'State' nodes and 'City' to 'County'.\n",
- "\n",
- "When we provide a list of places to `get_places_in`, we get back a `dict` mapping each place to all the places contained inside it."
- ]
- },
- {
- "cell_type": "code",
- "metadata": {
- "id": "MXy_j6VS04g5",
- "colab_type": "code",
- "colab": {}
- },
- "source": [
- "# Create three DataFrames storing states, counties, and cities within the United States respectively.\n",
- "state = pd.DataFrame({'state': dc.get_places_in([usa], 'State')[usa]})\n",
- "county = pd.DataFrame({'county': dc.get_places_in([usa], 'County')[usa]})\n",
- "city = pd.DataFrame({'city': dc.get_places_in([usa], 'City')[usa]})"
- ],
- "execution_count": 3,
- "outputs": []
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "fKbexXS60mLw",
- "colab_type": "text"
- },
- "source": [
- "Let's see what each table has."
- ]
- },
- {
- "cell_type": "code",
- "metadata": {
- "id": "5ZnJp5ns0nXt",
- "colab_type": "code",
- "colab": {
- "base_uri": "https://localhost:8080/",
- "height": 210
- },
- "outputId": "b28601c1-7ec7-4784-d920-66b9d62c64e4"
- },
- "source": [
- "# Display the state data\n",
- "state.head(5)"
- ],
- "execution_count": 4,
- "outputs": [
- {
- "output_type": "execute_result",
- "data": {
- "text/html": [
- "| \n", - " | state | \n", - "
|---|---|
| 0 | \n", - "geoId/01 | \n", - "
| 1 | \n", - "geoId/02 | \n", - "
| 2 | \n", - "geoId/04 | \n", - "
| 3 | \n", - "geoId/05 | \n", - "
| 4 | \n", - "geoId/06 | \n", - "
| \n", - " | county | \n", - "
|---|---|
| 0 | \n", - "geoId/01001 | \n", - "
| 1 | \n", - "geoId/01003 | \n", - "
| 2 | \n", - "geoId/01005 | \n", - "
| 3 | \n", - "geoId/01007 | \n", - "
| 4 | \n", - "geoId/01009 | \n", - "
| \n", - " | city | \n", - "
|---|---|
| 0 | \n", - "geoId/0100100 | \n", - "
| 1 | \n", - "geoId/0100124 | \n", - "
| 2 | \n", - "geoId/0100460 | \n", - "
| 3 | \n", - "geoId/0100484 | \n", - "
| 4 | \n", - "geoId/0100676 | \n", - "
| \n", - " | state | \n", - "state_name | \n", - "
|---|---|---|
| 0 | \n", - "geoId/01 | \n", - "Alabama | \n", - "
| 1 | \n", - "geoId/02 | \n", - "Alaska | \n", - "
| 2 | \n", - "geoId/04 | \n", - "Arizona | \n", - "
| 3 | \n", - "geoId/05 | \n", - "Arkansas | \n", - "
| 4 | \n", - "geoId/06 | \n", - "California | \n", - "
| \n", - " | county | \n", - "county_name | \n", - "
|---|---|---|
| 0 | \n", - "geoId/01001 | \n", - "Autauga County | \n", - "
| 1 | \n", - "geoId/01003 | \n", - "Baldwin County | \n", - "
| 2 | \n", - "geoId/01005 | \n", - "Barbour County | \n", - "
| 3 | \n", - "geoId/01007 | \n", - "Bibb County | \n", - "
| 4 | \n", - "geoId/01009 | \n", - "Blount County | \n", - "
| \n", - " | city | \n", - "city_name | \n", - "
|---|---|---|
| 0 | \n", - "geoId/0100100 | \n", - "Abanda | \n", - "
| 1 | \n", - "geoId/0100124 | \n", - "Abbeville | \n", - "
| 2 | \n", - "geoId/0100460 | \n", - "Adamsville | \n", - "
| 3 | \n", - "geoId/0100484 | \n", - "Addison | \n", - "
| 4 | \n", - "geoId/0100676 | \n", - "Akron | \n", - "
| \n", - " | state | \n", - "state_name | \n", - "count | \n", - "med_age | \n", - "
|---|---|---|---|---|
| 0 | \n", - "geoId/01 | \n", - "Alabama | \n", - "4874486 | \n", - "38.7 | \n", - "
| 1 | \n", - "geoId/02 | \n", - "Alaska | \n", - "739700 | \n", - "33.9 | \n", - "
| 2 | \n", - "geoId/04 | \n", - "Arizona | \n", - "7044008 | \n", - "37.2 | \n", - "
| 3 | \n", - "geoId/05 | \n", - "Arkansas | \n", - "3001345 | \n", - "37.9 | \n", - "
| 4 | \n", - "geoId/06 | \n", - "California | \n", - "39358497 | \n", - "36.1 | \n", - "
| \n", - " | county | \n", - "county_name | \n", - "count | \n", - "med_age | \n", - "
|---|---|---|---|---|
| 0 | \n", - "geoId/01001 | \n", - "Autauga County | \n", - "55390.0 | \n", - "37.8 | \n", - "
| 1 | \n", - "geoId/01003 | \n", - "Baldwin County | \n", - "212521.0 | \n", - "42.6 | \n", - "
| 2 | \n", - "geoId/01005 | \n", - "Barbour County | \n", - "25157.0 | \n", - "39.7 | \n", - "
| 3 | \n", - "geoId/01007 | \n", - "Bibb County | \n", - "22550.0 | \n", - "39.8 | \n", - "
| 4 | \n", - "geoId/01009 | \n", - "Blount County | \n", - "57787.0 | \n", - "40.9 | \n", - "
| \n", - " | city | \n", - "city_name | \n", - "count | \n", - "med_age | \n", - "
|---|---|---|---|---|
| 0 | \n", - "geoId/0100100 | \n", - "Abanda | \n", - "174.0 | \n", - "32.7 | \n", - "
| 1 | \n", - "geoId/0100124 | \n", - "Abbeville | \n", - "2566.0 | \n", - "44.9 | \n", - "
| 2 | \n", - "geoId/0100460 | \n", - "Adamsville | \n", - "4342.0 | \n", - "42.7 | \n", - "
| 3 | \n", - "geoId/0100484 | \n", - "Addison | \n", - "728.0 | \n", - "47.6 | \n", - "
| 4 | \n", - "geoId/0100676 | \n", - "Akron | \n", - "332.0 | \n", - "40.3 | \n", - "
"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "nnY4S5m_boXI",
- "colab_type": "text"
- },
- "source": [
- "Copyright 2020 Google LLC.\n",
- "SPDX-License-Identifier: Apache-2.0\n",
- "\n",
- "**Notebook Version** - 1.0.1"
- ]
- },
- {
- "cell_type": "code",
- "metadata": {
- "id": "9_SFWWxoSoMn",
- "colab_type": "code",
- "colab": {}
- },
- "source": [
- "# Install datacommons\n",
- "!pip install datacommons --upgrade --quiet"
- ],
- "execution_count": 1,
- "outputs": []
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "vWo_v5cI-JXC",
- "colab_type": "text"
- },
- "source": [
- "# Analyzing Income Distribution\n",
- "\n",
- "The American Community Survey (published by the US Census) annually reports the number of individuals in a given income bracket at the State level. We can use this information, stored in Data Commons, to visualize disparity in income for each State in the US. Our goal for this tutorial will be to generate a plot that visualizes the total number of individuals across a given set of income brackets for a given state. \n",
- "\n",
- "Before we begin, we'll setup our notebook."
- ]
- },
- {
- "cell_type": "code",
- "metadata": {
- "id": "OwquZnRa-JXD",
- "colab_type": "code",
- "colab": {}
- },
- "source": [
- "# Import the Data Commons library\n",
- "import datacommons as dc\n",
- "\n",
- "# Import other libraries\n",
- "import pandas as pd\n",
- "import matplotlib.pyplot as plt\n",
- "import numpy as np\n"
- ],
- "execution_count": 2,
- "outputs": []
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "Hnk34Wgr-JXG",
- "colab_type": "text"
- },
- "source": [
- "## Preparing the Data\n",
- "\n",
- "We'll begin by creating a dataframe with states and their total population. We can use **`get_places_in`** to get all States within the United States. We can then call **`get_stats`** to get the population of each State."
- ]
- },
- {
- "cell_type": "code",
- "metadata": {
- "id": "4tXBoR3h-JXI",
- "colab_type": "code",
- "colab": {
- "base_uri": "https://localhost:8080/",
- "height": 210
- },
- "outputId": "a486ae30-f530-4388-f9d3-4f1b12d940e1"
- },
- "source": [
- "# Initialize a DataFrame holding the states\n",
- "data = pd.DataFrame({'state': dc.get_places_in(['country/USA'], 'State')['country/USA']})\n",
- "\n",
- "# Get all state names and store it in a column \"name\"\n",
- "# Get the first name, if there are multiple for a state\n",
- "data['name'] = data['state'].map(dc.get_property_values(data['state'], 'name')).str[0]\n",
- "\n",
- "# Get the total count of all persons for each state\n",
- "data['Count_Person'] = data['state'].map(dc.get_stats(data['state'], \"Count_Person\", ['2017']))\n",
- "\n",
- "# Extract just the number for population\n",
- "get_value = lambda row: row.get('data').get('2017')\n",
- "data['Count_Person'] = data['Count_Person'].apply(get_value).dropna()\n",
- "\n",
- "# Display the first five rows of the table\n",
- "data.head(5)"
- ],
- "execution_count": 3,
- "outputs": [
- {
- "output_type": "execute_result",
- "data": {
- "text/html": [
- "| \n", - " | state | \n", - "name | \n", - "Count_Person | \n", - "
|---|---|---|---|
| 0 | \n", - "geoId/01 | \n", - "Alabama | \n", - "4874486 | \n", - "
| 1 | \n", - "geoId/02 | \n", - "Alaska | \n", - "739700 | \n", - "
| 2 | \n", - "geoId/04 | \n", - "Arizona | \n", - "7044008 | \n", - "
| 3 | \n", - "geoId/05 | \n", - "Arkansas | \n", - "3001345 | \n", - "
| 4 | \n", - "geoId/06 | \n", - "California | \n", - "39358497 | \n", - "
| \n", - " | state | \n", - "name | \n", - "Count_Person | \n", - "Count_Household_IncomeOfUpto10000USDollar | \n", - "Count_Household_IncomeOf10000To14999USDollar | \n", - "Count_Household_IncomeOf15000To19999USDollar | \n", - "Count_Household_IncomeOf20000To24999USDollar | \n", - "Count_Household_IncomeOf25000To29999USDollar | \n", - "Count_Household_IncomeOf30000To34999USDollar | \n", - "Count_Household_IncomeOf35000To39999USDollar | \n", - "Count_Household_IncomeOf40000To44999USDollar | \n", - "Count_Household_IncomeOf45000To49999USDollar | \n", - "Count_Household_IncomeOf50000To59999USDollar | \n", - "Count_Household_IncomeOf60000To74999USDollar | \n", - "Count_Household_IncomeOf75000To99999USDollar | \n", - "Count_Household_IncomeOf100000To124999USDollar | \n", - "Count_Household_IncomeOf125000To149999USDollar | \n", - "Count_Household_IncomeOf150000To199999USDollar | \n", - "Count_Household_IncomeOf200000OrMoreUSDollar | \n", - "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", - "geoId/01 | \n", - "Alabama | \n", - "4874486 | \n", - "170650 | \n", - "121688 | \n", - "116273 | \n", - "112406 | \n", - "102590 | \n", - "100223 | \n", - "92139 | \n", - "88987 | \n", - "76070 | \n", - "149492 | \n", - "176589 | \n", - "206436 | \n", - "133618 | \n", - "76939 | \n", - "71748 | \n", - "60847 | \n", - "
| 1 | \n", - "geoId/02 | \n", - "Alaska | \n", - "739700 | \n", - "9650 | \n", - "7838 | \n", - "8013 | \n", - "8854 | \n", - "8383 | \n", - "8914 | \n", - "8998 | \n", - "10163 | \n", - "8536 | \n", - "18810 | \n", - "26224 | \n", - "36745 | \n", - "29925 | \n", - "19752 | \n", - "23596 | \n", - "18135 | \n", - "
| 2 | \n", - "geoId/04 | \n", - "Arizona | \n", - "7044008 | \n", - "178384 | \n", - "118676 | \n", - "123075 | \n", - "131903 | \n", - "127741 | \n", - "127570 | \n", - "119431 | \n", - "125103 | \n", - "107245 | \n", - "208318 | \n", - "254294 | \n", - "303811 | \n", - "200818 | \n", - "119347 | \n", - "119778 | \n", - "116817 | \n", - "
| 3 | \n", - "geoId/05 | \n", - "Arkansas | \n", - "3001345 | \n", - "97226 | \n", - "78535 | \n", - "74451 | \n", - "75194 | \n", - "70211 | \n", - "67958 | \n", - "61588 | \n", - "60814 | \n", - "50978 | \n", - "95836 | \n", - "111618 | \n", - "120034 | \n", - "72752 | \n", - "40780 | \n", - "35900 | \n", - "33416 | \n", - "
| 4 | \n", - "geoId/06 | \n", - "California | \n", - "39358497 | \n", - "694945 | \n", - "604666 | \n", - "536203 | \n", - "568994 | \n", - "523563 | \n", - "539988 | \n", - "500571 | \n", - "509113 | \n", - "456152 | \n", - "895758 | \n", - "1199773 | \n", - "1568843 | \n", - "1206637 | \n", - "818690 | \n", - "1008388 | \n", - "1255844 | \n", - "
"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "srAnaUPPbrH6",
- "colab_type": "text"
- },
- "source": [
- "Copyright 2020 Google LLC.\n",
- "SPDX-License-Identifier: Apache-2.0\n",
- "\n",
- "**Notebook Version** - 1.0.1"
- ]
- },
- {
- "cell_type": "code",
- "metadata": {
- "id": "7Et41jikt2yA",
- "colab_type": "code",
- "colab": {}
- },
- "source": [
- "from sklearn.linear_model import LinearRegression\n",
- "from sklearn.model_selection import train_test_split\n",
- "from google.colab import drive\n",
- "\n",
- "import matplotlib.pyplot as plt\n",
- "import numpy as np\n",
- "import pandas as pd\n",
- "\n",
- "import json"
- ],
- "execution_count": 1,
- "outputs": []
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "yjBCrzJfqHtM",
- "colab_type": "text"
- },
- "source": [
- "# Case Study: Prevalence of Obesity in 500 US Cities\n",
- "\n",
- "Obesity is well known to correlate with health factors such as high blood pressure, but is also known to correlate with economic factors such as low-income, unemployment, etc [[1]](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3198075/)[[2]](https://www.ncbi.nlm.nih.gov/pubmed/26562758). The Center for Disease Control (CDC) provides prevalence percentages on health conditions such as [obesity](https://browser.datacommons.org/kg?dcid=dc/p/8f94jymf10dh7), [high blood pressure](https://browser.datacommons.org/kg?dcid=dc/p/lddprn26my0wh), and [high cholesterol](https://browser.datacommons.org/kg?dcid=dc/p/ybn085073e7z6) for approximately 500 major cities in the US (e.g. [San Francisco](https://browser.datacommons.org/kg?dcid=geoId/0667000), [New York](https://browser.datacommons.org/kg?dcid=geoId/3651000), and [Austin](https://browser.datacommons.org/kg?dcid=geoId/4805000)). Meanwhile, the US Bureau of Labor Statistics provides [unemployment rates](https://browser.datacommons.org/kg?dcid=dc/p/820mxx8ejlrj7) while the US Census provides [poverty rates](https://browser.datacommons.org/kg?dcid=dc/p/mrbgbmbe5lvyh) for most cities across the United States. \n",
- "\n",
- "Even though these statistics come from different datasets across different government agencies with different storage formats, Data Commons surfaces each of these in a single, uniform knowledge graph. In fact, you can see this in the [browser](https://browser.datacommons.org/kg?dcid=dc/9xkkc71) by looking at the *provenance* column. Let's use the data in Data Commons to create a linear regression model that incorporates variables:\n",
- "\n",
- "- Prevalence of high blood pressure\n",
- "- Unemployment rate\n",
- "- Percent of population living with income below the poverty line\n",
- "\n",
- "to predict the prevalence of obesity in the 500 cities that the CDC provides data for. One thing you may note is that the US Census also provides employment statistics (you can see this by navigating to the \"employment\" and \"employmentStatus\" sections for [San Francisco](https://browser.datacommons.org/kg?dcid=geoId/0667000) and observing the different provenances). Our choice of using statistics from the Bureau of Labor Statistics is purely demonstrative, but it would be interesting to see if similar results can be reproduced using US Census employment statistics. "
- ]
- },
- {
- "cell_type": "code",
- "metadata": {
- "id": "GQrede0Bdm-k",
- "colab_type": "code",
- "colab": {}
- },
- "source": [
- "!pip install datacommons --upgrade --quiet\n",
- "\n",
- "import datacommons as dc"
- ],
- "execution_count": 2,
- "outputs": []
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "pNHUbzq_dnGc",
- "colab_type": "text"
- },
- "source": [
- "To get started, we'll need the cities that CDC provides data for. We can query Data Commons for Cities that are members of [CDC500_City](https://browser.datacommons.org/kg?dcid=CDC500_City). The default return limit is 100, so we'll explicitly set it to 500."
- ]
- },
- {
- "cell_type": "code",
- "metadata": {
- "id": "4CoM0LD4kZSN",
- "colab_type": "code",
- "cellView": "both",
- "colab": {}
- },
- "source": [
- "city_dcids = dc.get_property_values([\"CDC500_City\"], \"member\", limit=500)"
- ],
- "execution_count": 3,
- "outputs": []
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "2JmxK2KrnCcQ",
- "colab_type": "text"
- },
- "source": [
- "We then initialize a `pandas.DataFrame` with the dcids in this list."
- ]
- },
- {
- "cell_type": "code",
- "metadata": {
- "id": "IM4ajcwoNtXr",
- "colab_type": "code",
- "colab": {
- "base_uri": "https://localhost:8080/",
- "height": 210
- },
- "outputId": "d02424fe-8a0a-407f-c279-b7dc82d66050"
- },
- "source": [
- "# Create the Pandas DataFrame \n",
- "data = pd.DataFrame(city_dcids)\n",
- "# Simplify the column name\n",
- "data.columns = ['City']\n",
- "# Get the name of each city\n",
- "# If a city has multiple common names (e.g. \"New York\" and \"New York City\"), use the first one\n",
- "data['CityName'] = data['City'].map(dc.get_property_values(data['City'], 'name')).str[0]\n",
- "data.dropna(inplace=True)\n",
- "\n",
- "# Show the top five rows\n",
- "data.head(5)"
- ],
- "execution_count": 4,
- "outputs": [
- {
- "output_type": "execute_result",
- "data": {
- "text/html": [
- "| \n", - " | City | \n", - "CityName | \n", - "
|---|---|---|
| 0 | \n", - "geoId/0107000 | \n", - "Birmingham | \n", - "
| 1 | \n", - "geoId/0135896 | \n", - "Hoover | \n", - "
| 2 | \n", - "geoId/0137000 | \n", - "Huntsville | \n", - "
| 3 | \n", - "geoId/0150000 | \n", - "Mobile | \n", - "
| 4 | \n", - "geoId/0151000 | \n", - "Montgomery | \n", - "
| \n", - " | City | \n", - "CityName | \n", - "Obesity | \n", - "HighBp | \n", - "PersonCount | \n", - "Unemployment | \n", - "PovertyCount | \n", - "
|---|---|---|---|---|---|---|---|
| 0 | \n", - "geoId/0107000 | \n", - "Birmingham | \n", - "42.7 | \n", - "44.1 | \n", - "212211.0 | \n", - "6.6 | \n", - "63503.0 | \n", - "
| 1 | \n", - "geoId/0135896 | \n", - "Hoover | \n", - "28.3 | \n", - "29.7 | \n", - "83547.0 | \n", - "3.9 | \n", - "5003.0 | \n", - "
| 2 | \n", - "geoId/0137000 | \n", - "Huntsville | \n", - "34.4 | \n", - "36.3 | \n", - "185594.0 | \n", - "5.2 | \n", - "32412.0 | \n", - "
| 3 | \n", - "geoId/0150000 | \n", - "Mobile | \n", - "35.3 | \n", - "41.0 | \n", - "194669.0 | \n", - "6.8 | \n", - "43468.0 | \n", - "
| 4 | \n", - "geoId/0151000 | \n", - "Montgomery | \n", - "39.4 | \n", - "40.8 | \n", - "202967.0 | \n", - "5.8 | \n", - "47371.0 | \n", - "
| \n", - " | CityName | \n", - "Obesity | \n", - "HighBp | \n", - "Unemployment | \n", - "PovertyRate | \n", - "
|---|---|---|---|---|---|
| 0 | \n", - "Birmingham | \n", - "42.7 | \n", - "44.1 | \n", - "6.6 | \n", - "29.924462 | \n", - "
| 1 | \n", - "Hoover | \n", - "28.3 | \n", - "29.7 | \n", - "3.9 | \n", - "5.988246 | \n", - "
| 2 | \n", - "Huntsville | \n", - "34.4 | \n", - "36.3 | \n", - "5.2 | \n", - "17.463927 | \n", - "
| 3 | \n", - "Mobile | \n", - "35.3 | \n", - "41.0 | \n", - "6.8 | \n", - "22.329184 | \n", - "
| 4 | \n", - "Montgomery | \n", - "39.4 | \n", - "40.8 | \n", - "5.8 | \n", - "23.339262 | \n", - "
"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "ZXL-qge7bmCa",
- "colab_type": "text"
- },
- "source": [
- "Copyright 2019 Google LLC.\n",
- "SPDX-License-Identifier: Apache-2.0\n",
- "\n",
- "**Notebook Version** - 1.0.0"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "knFLVFogCWq0",
- "colab_type": "text"
- },
- "source": [
- "# Analyzing Census Data with Data Commons\n",
- "\n",
- "Every year, the American Community Survey (published by the US Census) reports thousands of variables about demographics, economics, housing, and more. This information, stored in Data Commons, is available to everyone for data science projects, education, and exploration. This tutorial introduces the Data Commons graph and two of its tools to help integrate its data into your data science projects: (1) the [browser](https://browser.datacommons.org/) and (2) the [Python API](https://github.com/datacommonsorg/api-python).\n",
- "\n",
- "## What is Data Commons?\n",
- "\n",
- "Data Commons is an open knowledge repository that combines data from public datasets using mapped common entities. It contains statements about real world objects such as\n",
- "\n",
- "* [Santa Clara County](https://browser.datacommons.org/kg?dcid=geoId/06085) is contained in the [State of California](https://browser.datacommons.org/kg?dcid=geoId/06)\n",
- "* The latitude of [Berkeley, CA](https://browser.datacommons.org/kg?dcid=geoId/0606000) is 37.8703\n",
- "* [The population of Maryland](https://browser.datacommons.org/kg?dcid=dc/p/psjx4xy30nws1) was [6,003,435 in 2018](https://browser.datacommons.org/kg?dcid=dc%2Fo%2Fx1tlfg4ll9yr9).\n",
- "\n",
- "In the graph, [*entities*](https://en.wikipedia.org/wiki/Entity) like [Santa Clara County](https://browser.datacommons.org/kg?dcid=geoId/06085) are represented by nodes. Every node has a type corresponding to what the node represents. For example, [California](https://browser.datacommons.org/kg?dcid=geoId/06) is a [State](https://schema.org/State). *Relations* between entities are represented by edges between these nodes. For example, the statement \"Santa Clara County is contained in the State of California\" is represented in the graph as two nodes: \"Santa Clara County\" and \"California\" with an edge labeled \"[containedInPlace](https://schema.org/containedInPlace)\" pointing from Santa Clara to California. Data Commons closely follows the [Schema.org data model](https://schema.org/docs/datamodel.html) and leverages Schema.org schema to provide a common set of types and properties.\n",
- "\n",
- "## Data Commons Browser\n",
- "\n",
- "The [Data Commons browser](https://browser.datacommons.org/) provides a way to explore the data in a human-readable format. It is the best way to explore what is in Data Commons. Searching in the browser for an entity like [Mountain View](https://browser.datacommons.org/kg?dcid=geoId/0649670), takes you to a page about the entity, including properties like [containedInPlace](https://browser.datacommons.org/kg?dcid=containedInPlace) and [timezone](https://browser.datacommons.org/kg?dcid=timezone).\n",
- "\n",
- "An important property for all entities is the **`dcid`**. The `dcid` (Data Commons identifier) is a unique identifier assigned to each entity in the knowledge graph. With this identifier, you will be able to search for and query information on the given entity in ways that we will discuss later. The `dcid` is listed at the top of the page next to \"About: \" and also in the list of properties.\n",
- "\n",
- "## Python API\n",
- "\n",
- "The [Python API](https://github.com/datacommonsorg/api-python) provides functions for users to extract structured information from Data Commons programmatically and view them in different formats such as Python `dict`s and [Pandas](https://pandas.pydata.org/) DataFrames. DataFrames allow access to all the data processing, analytical and visualization tools provided by packages such as Pandas, NumPy, SciPy, and Matplotlib.\n",
- "\n",
- "Let's begin by loading the Data Commons client and the standard data science libraries:\n"
- ]
- },
- {
- "cell_type": "code",
- "metadata": {
- "id": "zxg_UfUgIq-Y",
- "colab_type": "code",
- "colab": {
- "base_uri": "https://localhost:8080/",
- "height": 36
- },
- "outputId": "e7c4fda3-90f5-4a1b-b0ab-a1e9f9269f3e"
- },
- "source": [
- "# Install datacommons\n",
- "!pip install git+https://github.com/datacommonsorg/api-python.git --upgrade --quiet\n",
- "# Import Data Commons\n",
- "import datacommons as dc\n",
- "\n",
- "# Import other required libraries\n",
- "import matplotlib.pyplot as plt\n",
- "import matplotlib.patches as mpatches\n",
- "import pandas as pd\n",
- "\n",
- "import json"
- ],
- "execution_count": null,
- "outputs": [
- {
- "output_type": "stream",
- "text": [
- " Building wheel for datacommons (setup.py) ... \u001b[?25l\u001b[?25hdone\n"
- ],
- "name": "stdout"
- }
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "rrkDVNp0JI6g",
- "colab_type": "text"
- },
- "source": [
- "## Example: Median Age vs Population by State\n",
- "For this exercise, we will be comparing the median ages and population count for US states, counties, and cities. First, let's lookup the dcid for the '[United States](https://browser.datacommons.org/kg?dcid=country/USA)'."
- ]
- },
- {
- "cell_type": "code",
- "metadata": {
- "id": "0peszosKQU_g",
- "colab_type": "code",
- "colab": {}
- },
- "source": [
- "# In the browser, we saw that the dcid for United States is country/USA\n",
- "usa = 'country/USA'"
- ],
- "execution_count": null,
- "outputs": []
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "kiPXhAJkDyjk",
- "colab_type": "text"
- },
- "source": [
- "### Using `get_places_in` to Query Administrative Areas\n",
- "\n",
- "The Python API defines a number of convenience functions for building Pandas dataframes with information in the datacommons graph. We will be using **`get_places_in`** which requires three arguments:\n",
- "\n",
- "- `dcids` - A list or `pandas.Series` of dcids identifying administrative areas that we wish to get containing places for.\n",
- "- `place_type` - The type of the administrative area that we wish to query for.\n",
- "\n",
- "In Data Commons, the 'containedInPlace' property relates administrative areas to its containing administrative areas. Concretely, every US 'State' node has a directed edge to the USA 'Country' node where the name of this edge is 'containedInPlace'. The same also goes for 'County' to 'State' nodes and 'City' to 'County'.\n",
- "\n",
- "When we provide a list of places to `get_places_in`, we get back a `dict` mapping each place to all the places contained inside it."
- ]
- },
- {
- "cell_type": "code",
- "metadata": {
- "id": "MXy_j6VS04g5",
- "colab_type": "code",
- "colab": {}
- },
- "source": [
- "# Create three DataFrames storing states, counties, and cities within the United States respectively.\n",
- "state = pd.DataFrame({'state': dc.get_places_in([usa], 'State')[usa]})\n",
- "county = pd.DataFrame({'county': dc.get_places_in([usa], 'County')[usa]})\n",
- "city = pd.DataFrame({'city': dc.get_places_in([usa], 'City')[usa]})"
- ],
- "execution_count": null,
- "outputs": []
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "fKbexXS60mLw",
- "colab_type": "text"
- },
- "source": [
- "Let's see what each table has."
- ]
- },
- {
- "cell_type": "code",
- "metadata": {
- "id": "5ZnJp5ns0nXt",
- "colab_type": "code",
- "colab": {
- "base_uri": "https://localhost:8080/",
- "height": 210
- },
- "outputId": "87f858cf-3c05-4f95-d78b-c2272117aba0"
- },
- "source": [
- "# Display the state data\n",
- "state.head(5)"
- ],
- "execution_count": null,
- "outputs": [
- {
- "output_type": "execute_result",
- "data": {
- "text/html": [
- "| \n", - " | state | \n", - "
|---|---|
| 0 | \n", - "geoId/01 | \n", - "
| 1 | \n", - "geoId/02 | \n", - "
| 2 | \n", - "geoId/04 | \n", - "
| 3 | \n", - "geoId/05 | \n", - "
| 4 | \n", - "geoId/06 | \n", - "
| \n", - " | county | \n", - "
|---|---|
| 0 | \n", - "geoId/01001 | \n", - "
| 1 | \n", - "geoId/01003 | \n", - "
| 2 | \n", - "geoId/01005 | \n", - "
| 3 | \n", - "geoId/01007 | \n", - "
| 4 | \n", - "geoId/01009 | \n", - "
| \n", - " | city | \n", - "
|---|---|
| 0 | \n", - "geoId/0100100 | \n", - "
| 1 | \n", - "geoId/0100124 | \n", - "
| 2 | \n", - "geoId/0100460 | \n", - "
| 3 | \n", - "geoId/0100484 | \n", - "
| 4 | \n", - "geoId/0100676 | \n", - "
| \n", - " | state | \n", - "state_name | \n", - "
|---|---|---|
| 0 | \n", - "geoId/01 | \n", - "Alabama | \n", - "
| 1 | \n", - "geoId/02 | \n", - "Alaska | \n", - "
| 2 | \n", - "geoId/04 | \n", - "Arizona | \n", - "
| 3 | \n", - "geoId/05 | \n", - "Arkansas | \n", - "
| 4 | \n", - "geoId/06 | \n", - "California | \n", - "
| \n", - " | county | \n", - "county_name | \n", - "
|---|---|---|
| 0 | \n", - "geoId/01001 | \n", - "Autauga County | \n", - "
| 1 | \n", - "geoId/01003 | \n", - "Baldwin County | \n", - "
| 2 | \n", - "geoId/01005 | \n", - "Barbour County | \n", - "
| 3 | \n", - "geoId/01007 | \n", - "Bibb County | \n", - "
| 4 | \n", - "geoId/01009 | \n", - "Blount County | \n", - "
| \n", - " | city | \n", - "city_name | \n", - "
|---|---|---|
| 0 | \n", - "geoId/0100100 | \n", - "Abanda | \n", - "
| 1 | \n", - "geoId/0100124 | \n", - "Abbeville | \n", - "
| 2 | \n", - "geoId/0100460 | \n", - "Adamsville | \n", - "
| 3 | \n", - "geoId/0100484 | \n", - "Addison | \n", - "
| 4 | \n", - "geoId/0100676 | \n", - "Akron | \n", - "
| \n", - " | state | \n", - "state_name | \n", - "count | \n", - "med_age | \n", - "
|---|---|---|---|---|
| 0 | \n", - "geoId/01 | \n", - "Alabama | \n", - "4874486 | \n", - "38.7 | \n", - "
| 1 | \n", - "geoId/02 | \n", - "Alaska | \n", - "739700 | \n", - "33.9 | \n", - "
| 2 | \n", - "geoId/04 | \n", - "Arizona | \n", - "7044008 | \n", - "37.2 | \n", - "
| 3 | \n", - "geoId/05 | \n", - "Arkansas | \n", - "3001345 | \n", - "37.9 | \n", - "
| 4 | \n", - "geoId/06 | \n", - "California | \n", - "39358497 | \n", - "36.1 | \n", - "
| \n", - " | county | \n", - "county_name | \n", - "count | \n", - "med_age | \n", - "
|---|---|---|---|---|
| 0 | \n", - "geoId/01001 | \n", - "Autauga County | \n", - "55390.0 | \n", - "37.8 | \n", - "
| 1 | \n", - "geoId/01003 | \n", - "Baldwin County | \n", - "212521.0 | \n", - "42.6 | \n", - "
| 2 | \n", - "geoId/01005 | \n", - "Barbour County | \n", - "25157.0 | \n", - "39.7 | \n", - "
| 3 | \n", - "geoId/01007 | \n", - "Bibb County | \n", - "22550.0 | \n", - "39.8 | \n", - "
| 4 | \n", - "geoId/01009 | \n", - "Blount County | \n", - "57787.0 | \n", - "40.9 | \n", - "
| \n", - " | city | \n", - "city_name | \n", - "count | \n", - "med_age | \n", - "
|---|---|---|---|---|
| 0 | \n", - "geoId/0100100 | \n", - "Abanda | \n", - "174.0 | \n", - "32.7 | \n", - "
| 1 | \n", - "geoId/0100124 | \n", - "Abbeville | \n", - "2566.0 | \n", - "44.9 | \n", - "
| 2 | \n", - "geoId/0100460 | \n", - "Adamsville | \n", - "4342.0 | \n", - "42.7 | \n", - "
| 3 | \n", - "geoId/0100484 | \n", - "Addison | \n", - "728.0 | \n", - "47.6 | \n", - "
| 4 | \n", - "geoId/0100676 | \n", - "Akron | \n", - "332.0 | \n", - "40.3 | \n", - "
"
]
},
{
"cell_type": "markdown",
"metadata": {
- "id": "knFLVFogCWq0",
+ "id": "ZXL-qge7bmCa",
"colab_type": "text"
},
"source": [
- "# Analyzing Census Data with Data Commons\n",
+ "Copyright 2019 Google LLC.\n",
+ "SPDX-License-Identifier: Apache-2.0\n",
"\n",
- "Datacommons is intended for various data science tasks. This tutorial introduces the datacommons knowledge graph and discusses two tools to help integrate its data into your data science projects: (1) the [datacommons browser](https://browser.datacommons.org/) and (2) the [Python API](https://github.com/google/datacommons). Before getting started, we will need to install the Python API package. \n"
- ]
- },
- {
- "cell_type": "code",
- "metadata": {
- "id": "S-GaEOefIpq5",
- "colab_type": "code",
- "outputId": "19ad4a69-a034-4e0a-d722-74014bee7a57",
- "colab": {
- "base_uri": "https://localhost:8080/",
- "height": 34
- }
- },
- "source": [
- "# Install datacommons\n",
- "!pip install --upgrade --quiet git+https://github.com/datacommonsorg/api-python.git@stable-1.x"
- ],
- "execution_count": 0,
- "outputs": [
- {
- "output_type": "stream",
- "text": [
- " Building wheel for datacommons (setup.py) ... \u001b[?25l\u001b[?25hdone\n"
- ],
- "name": "stdout"
- }
+ "**Notebook Version** - 1.0.0"
]
},
{
"cell_type": "markdown",
"metadata": {
- "id": "3U5niWKlDg3A",
+ "id": "knFLVFogCWq0",
"colab_type": "text"
},
"source": [
+ "# Analyzing Census Data with Data Commons\n",
+ "\n",
+ "Every year, the American Community Survey (published by the US Census) reports thousands of variables about demographics, economics, housing, and more. This information, stored in Data Commons, is available to everyone for data science projects, education, and exploration. This tutorial introduces the Data Commons graph and two of its tools to help integrate its data into your data science projects: (1) the [browser](https://browser.datacommons.org/) and (2) the [Python API](https://github.com/datacommonsorg/api-python).\n",
+ "\n",
"## What is Data Commons?\n",
"\n",
- "Data Commons is an open knowledge graph of structured data. It contains statements about real world objects such as\n",
+ "Data Commons is an open knowledge repository that combines data from public datasets using mapped common entities. It contains statements about real world objects such as\n",
"\n",
"* [Santa Clara County](https://browser.datacommons.org/kg?dcid=geoId/06085) is contained in the [State of California](https://browser.datacommons.org/kg?dcid=geoId/06)\n",
"* The latitude of [Berkeley, CA](https://browser.datacommons.org/kg?dcid=geoId/0606000) is 37.8703\n",
- "* The [population of all persons in Maryland ](https://browser.datacommons.org/kg?dcid=dc/o/6w1c9qk7hxjch)has a total count of 5,996,080.\n",
+ "* [The population of Maryland](https://browser.datacommons.org/kg?dcid=dc/p/psjx4xy30nws1) was [6,003,435 in 2018](https://browser.datacommons.org/kg?dcid=dc%2Fo%2Fx1tlfg4ll9yr9).\n",
+ "\n",
+ "In the graph, [*entities*](https://en.wikipedia.org/wiki/Entity) like [Santa Clara County](https://browser.datacommons.org/kg?dcid=geoId/06085) are represented by nodes. Every node has a type corresponding to what the node represents. For example, [California](https://browser.datacommons.org/kg?dcid=geoId/06) is a [State](https://schema.org/State). *Relations* between entities are represented by edges between these nodes. For example, the statement \"Santa Clara County is contained in the State of California\" is represented in the graph as two nodes: \"Santa Clara County\" and \"California\" with an edge labeled \"[containedInPlace](https://schema.org/containedInPlace)\" pointing from Santa Clara to California. Data Commons closely follows the [Schema.org data model](https://schema.org/docs/datamodel.html) and leverages Schema.org schema to provide a common set of types and properties.\n",
"\n",
- "In the graph, [*entities*](https://en.wikipedia.org/wiki/Entity) like [Santa Clara County](https://browser.datacommons.org/kg?dcid=geoId/06085) are represented by nodes. Every node has a type corresponding to what the node represents. For example, [California](https://browser.datacommons.org/kg?dcid=geoId/06) is a [State](https://schema.org/State). *Relations* between entities are represented by edges between these nodes. For example, the statement \"Santa Clara County is contained in the State of California\" is represented in the graph as two nodes: \"Santa Clara County\" and \"California\" with an edge labeled \"[containedInPlace](https://schema.org/containedInPlace)\" pointing from Santa Clara to California. Data Commons closely follows the [Schema.org data model](https://schema.org/docs/datamodel.html) and leverages schema.org schema to provide a common set of types and properties."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "dwT0Iz493DcL",
- "colab_type": "text"
- },
- "source": [
"## Data Commons Browser\n",
"\n",
"The [Data Commons browser](https://browser.datacommons.org/) provides a way to explore the data in a human-readable format. It is the best way to explore what is in Data Commons. Searching in the browser for an entity like [Mountain View](https://browser.datacommons.org/kg?dcid=geoId/0649670), takes you to a page about the entity, including properties like [containedInPlace](https://browser.datacommons.org/kg?dcid=containedInPlace) and [timezone](https://browser.datacommons.org/kg?dcid=timezone).\n",
"\n",
- "An important property for all entities is the **`dcid`**. The `dcid` (DataCommons identifier) is a unique identifier assigned to each entity in the knowledge graph. With this identifier, you will be able to search for and query information on the given entity in ways that we will discuss later. The `dcid` is listed at the top of the page next to \"About: \" and also in the list of properties. "
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "sN2cWYLF01Zk",
- "colab_type": "text"
- },
- "source": [
+ "An important property for all entities is the **`dcid`**. The `dcid` (Data Commons identifier) is a unique identifier assigned to each entity in the knowledge graph. With this identifier, you will be able to search for and query information on the given entity in ways that we will discuss later. The `dcid` is listed at the top of the page next to \"About: \" and also in the list of properties.\n",
+ "\n",
"## Python API\n",
"\n",
- "The [Python API](https://github.com/datacommonsorg/api-python) provides functions for users to extract structured information from Data Commons programmatically and view them in different formats such as Python `dict`'s and [Pandas](https://pandas.pydata.org/) DataFrames. DataFrames allow access to all the data processing, analytical and visualization tools provided by packages such as Pandas, NumPy, SciPy, and Matplotlib.\n",
+ "The [Python API](https://github.com/datacommonsorg/api-python) provides functions for users to extract structured information from Data Commons programmatically and view them in different formats such as Python `dict`s and [Pandas](https://pandas.pydata.org/) DataFrames. DataFrames allow access to all the data processing, analytical and visualization tools provided by packages such as Pandas, NumPy, SciPy, and Matplotlib.\n",
"\n",
- "Every notebook begins by loading the dataCommons client as follows:\n"
+ "Let's begin by loading the Data Commons library and the standard data science libraries:\n"
]
},
{
@@ -119,6 +80,8 @@
"colab": {}
},
"source": [
+ "# Install datacommons\n",
+ "!pip install datacommons --upgrade --quiet\n",
"# Import Data Commons\n",
"import datacommons as dc\n",
"\n",
@@ -129,67 +92,34 @@
"\n",
"import json"
],
- "execution_count": 0,
+ "execution_count": 1,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
- "id": "5-BG0JClVne-",
+ "id": "rrkDVNp0JI6g",
"colab_type": "text"
},
"source": [
- "We will also need to provide an API key to access the Data Commons Python API. This notebook is setup to read from a `json` file stored at `key_path` in your Google Drive. This text file should have one line:\n",
- "\n",
- " ```\n",
- "{\n",
- " \"dc_api_key\": \"YOUR-API-KEY\"\n",
- "}\n",
- " ```\n",
- " \n",
- " If you want to make a copy of this notebook, make sure to replace `key_path` with the path to a file containing your API key. For more detail, visit the [Creating an API Key](https://datacommons.readthedocs.io/en/latest/started.html#creating-an-api-key) page of the Python API documentation.\n",
- "\n"
+ "## Example: Median Age vs Population by State\n",
+ "For this exercise, we will be comparing the median ages and population count for US states, counties, and cities. First, let's lookup the dcid for the '[United States](https://browser.datacommons.org/kg?dcid=country/USA)'."
]
},
{
"cell_type": "code",
"metadata": {
- "id": "iM0YkAZVV0qF",
+ "id": "0peszosKQU_g",
"colab_type": "code",
"colab": {}
},
"source": [
- "# The key is stored in a secret file. To use this notebook, you will need to \n",
- "# make a copy and point this line to the file containing the key in your drive!\n",
- "from google.colab import drive\n",
- "\n",
- "# Mount the Drive\n",
- "drive.mount('/content/drive', force_remount=True)\n",
- "\n",
- "# REPLACE THIS with the path to your key.\n",
- "key_path = '/content/drive/My Drive/DataCommons/secret.json'\n",
- "\n",
- "# Read the key in and provide it to the Data Commons API\n",
- "with open(key_path, 'r') as f:\n",
- " secrets = json.load(f)\n",
- " dc.set_api_key(secrets['dc_api_key'])"
+ "# In the browser, we saw that the dcid for United States is country/USA\n",
+ "usa = 'country/USA'"
],
- "execution_count": 0,
+ "execution_count": 2,
"outputs": []
},
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "rrkDVNp0JI6g",
- "colab_type": "text"
- },
- "source": [
- "## Example: Median Age vs Population by State\n",
- "For this exercise, we will be comparing the median ages and population count for US states, counties, and cities. First, let's lookup the dcid for the [United States](https://browser.datacommons.org/kg?dcid=country/USA)'. \n",
- "\n",
- "Note that \"Country\" defines the Data Commons type, and \"country\" is the name we assign to this column. \n"
- ]
- },
{
"cell_type": "markdown",
"metadata": {
@@ -199,14 +129,14 @@
"source": [
"### Using `get_places_in` to Query Administrative Areas\n",
"\n",
- "The Client API defines a number of convenience functions for building Pandas dataframes with information in the datacommons graph. We will be using **`get_places_in`** which requires three arguments:\n",
+ "The Python API defines a number of convenience functions for building Pandas dataframes with information in the datacommons graph. We will be using **`get_places_in`** which requires three arguments:\n",
"\n",
"- `dcids` - A list or `pandas.Series` of dcids identifying administrative areas that we wish to get containing places for.\n",
"- `place_type` - The type of the administrative area that we wish to query for.\n",
"\n",
- "In the datacommons knowledge graph, the 'containedInPlace' property relates a administrative area types to its containing administrative area type. Concretely, every 'state' node has a directed edge to a 'country' node where the name of this edge is 'containedInPlace'. To confirm this, you can check the browser page for '[United States](https://browser.datacommons.org/kg?dcid=country/USA)'. The same also goes for 'county' to 'state' nodes and 'city' to 'county'.\n",
+ "In Data Commons, the 'containedInPlace' property relates administrative areas to its containing administrative areas. Concretely, every US 'State' node has a directed edge to the USA 'Country' node where the name of this edge is 'containedInPlace'. The same also goes for 'County' to 'State' nodes and 'City' to 'County'.\n",
"\n",
- "When we provide a `pandas.Series` to `get_places_in`, we get back a `pandas.Series` with the all places contained in the Administrative Areas identified by the given series of dcids."
+ "When we provide a list of places to `get_places_in`, we get back a `dict` mapping each place to all the places contained inside it."
]
},
{
@@ -217,18 +147,12 @@
"colab": {}
},
"source": [
- "# Create a three DataFrames with the dcid of the USA storing state, county and \n",
- "# city data respectively.\n",
- "state = pd.DataFrame({'country': ['country/USA']})\n",
- "county = pd.DataFrame({'country': ['country/USA']})\n",
- "city = pd.DataFrame({'country': ['country/USA']})\n",
- "\n",
- "# Get all states, counties, and cities within the United States\n",
- "state['state'] = dc.get_places_in(state['country'], 'State')\n",
- "county['county'] = dc.get_places_in(county['country'], 'County')\n",
- "city['city'] = dc.get_places_in(county['country'], 'City')"
+ "# Create three DataFrames storing states, counties, and cities within the United States respectively.\n",
+ "state = pd.DataFrame({'state': dc.get_places_in([usa], 'State')[usa]})\n",
+ "county = pd.DataFrame({'county': dc.get_places_in([usa], 'County')[usa]})\n",
+ "city = pd.DataFrame({'city': dc.get_places_in([usa], 'City')[usa]})"
],
- "execution_count": 0,
+ "execution_count": 3,
"outputs": []
},
{
@@ -246,17 +170,17 @@
"metadata": {
"id": "5ZnJp5ns0nXt",
"colab_type": "code",
- "outputId": "04316337-a12c-4332-cf96-bbe7a03d79b4",
"colab": {
"base_uri": "https://localhost:8080/",
- "height": 80
- }
+ "height": 210
+ },
+ "outputId": "b28601c1-7ec7-4784-d920-66b9d62c64e4"
},
"source": [
"# Display the state data\n",
"state.head(5)"
],
- "execution_count": 0,
+ "execution_count": 4,
"outputs": [
{
"output_type": "execute_result",
@@ -280,29 +204,47 @@
" \n",
" | \n", - " | country | \n", - "state | \n", - "|
|---|---|---|---|
| 0 | \n", - "country/USA | \n", - "geoId/01 | \n", + "geoId/0100100 | \n", "
| 1 | \n", - "country/USA | \n", - "geoId/02 | \n", + "geoId/0100124 | \n", "
| 2 | \n", - "country/USA | \n", - "geoId/04 | \n", + "geoId/0100460 | \n", "
| 3 | \n", - "country/USA | \n", - "geoId/05 | \n", + "geoId/0100484 | \n", "
| 4 | \n", - "country/USA | \n", - "geoId/06 | \n", + "geoId/0100676 | \n", "
| \n", - " | country | \n", - "state | \n", - "state_name | \n", - "all_persons_pop | \n", - "
|---|---|---|---|---|
| 0 | \n", - "country/USA | \n", - "geoId/01 | \n", - "Alabama | \n", - "dc/p/dft95nqhpkkgd | \n", - "
| 1 | \n", - "country/USA | \n", - "geoId/02 | \n", - "Alaska | \n", - "dc/p/th9yyf8ej6m64 | \n", - "
| 2 | \n", - "country/USA | \n", - "geoId/04 | \n", - "Arizona | \n", - "dc/p/8hyhwze6s5tn6 | \n", - "
| 3 | \n", - "country/USA | \n", - "geoId/05 | \n", - "Arkansas | \n", - "dc/p/rp61n0st0vtr | \n", - "
| 4 | \n", - "country/USA | \n", - "geoId/06 | \n", - "California | \n", - "dc/p/f1jbe2m7w6f58 | \n", - "
"
+ ]
+ },
{
"cell_type": "markdown",
"metadata": {
@@ -22,10 +33,10 @@
"colab_type": "text"
},
"source": [
- "Copyright 2019 Google LLC.\n",
+ "Copyright 2020 Google LLC.\n",
"SPDX-License-Identifier: Apache-2.0\n",
"\n",
- "**Notebook Version** - 1.0.0"
+ "**Notebook Version** - 1.0.1"
]
},
{
@@ -33,26 +44,14 @@
"metadata": {
"id": "9_SFWWxoSoMn",
"colab_type": "code",
- "outputId": "eef38f1d-8c4e-4288-b6a3-4d0557cb619f",
- "colab": {
- "base_uri": "https://localhost:8080/",
- "height": 34
- }
+ "colab": {}
},
"source": [
"# Install datacommons\n",
- "!pip install --upgrade --quiet git+https://github.com/datacommonsorg/api-python.git@stable-1.x"
+ "!pip install datacommons --upgrade --quiet"
],
- "execution_count": 4,
- "outputs": [
- {
- "output_type": "stream",
- "text": [
- " Building wheel for datacommons (setup.py) ... \u001b[?25l\u001b[?25hdone\n"
- ],
- "name": "stdout"
- }
- ]
+ "execution_count": 1,
+ "outputs": []
},
{
"cell_type": "markdown",
@@ -65,7 +64,7 @@
"\n",
"The American Community Survey (published by the US Census) annually reports the number of individuals in a given income bracket at the State level. We can use this information, stored in Data Commons, to visualize disparity in income for each State in the US. Our goal for this tutorial will be to generate a plot that visualizes the total number of individuals across a given set of income brackets for a given state. \n",
"\n",
- "Before we begin, we'll setup our notebook"
+ "Before we begin, we'll setup our notebook."
]
},
{
@@ -82,58 +81,11 @@
"# Import other libraries\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
- "import numpy as np\n",
- "import json\n",
- "\n",
- "from google.colab import drive"
+ "import numpy as np\n"
],
- "execution_count": 0,
+ "execution_count": 2,
"outputs": []
},
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "vpZ81m1ckA8J",
- "colab_type": "text"
- },
- "source": [
- "We will also need to provide the API with an API key. See the [Analyzing Statistics in Data Commons Using the Python API](https://colab.research.google.com/drive/1ZNXTHu3J0W3vo9Mg3kNUpk0hnD6Ce1u6#scrollTo=ijxoBhFHjo3Z) to see how to set this up for a Colab Notebook."
- ]
- },
- {
- "cell_type": "code",
- "metadata": {
- "id": "PmcH0ApJkNbD",
- "colab_type": "code",
- "outputId": "daa26713-71a3-4ffa-942a-39a77915491a",
- "colab": {
- "base_uri": "https://localhost:8080/",
- "height": 34
- }
- },
- "source": [
- "# Mount the Drive\n",
- "drive.mount('/content/drive', force_remount=True)\n",
- "\n",
- "# REPLACE THIS with the path to your key.\n",
- "key_path = '/content/drive/My Drive/DataCommons/secret.json'\n",
- "\n",
- "# Read the key in and provide it to the Data Commons API\n",
- "with open(key_path, 'r') as f:\n",
- " secrets = json.load(f)\n",
- " dc.set_api_key(secrets['dc_api_key'])"
- ],
- "execution_count": 6,
- "outputs": [
- {
- "output_type": "stream",
- "text": [
- "Mounted at /content/drive\n"
- ],
- "name": "stdout"
- }
- ]
- },
{
"cell_type": "markdown",
"metadata": {
@@ -143,7 +95,7 @@
"source": [
"## Preparing the Data\n",
"\n",
- "We'll begin by creating a dataframe with states and their total population. We can use **`get_places_in`** to get all States within the United States. We can then call **`get_populations`** and **`get_observations`** to get the population of all persons in each State."
+ "We'll begin by creating a dataframe with states and their total population. We can use **`get_places_in`** to get all States within the United States. We can then call **`get_stats`** to get the population of each State."
]
},
{
@@ -151,38 +103,31 @@
"metadata": {
"id": "4tXBoR3h-JXI",
"colab_type": "code",
- "outputId": "cdc24988-8e51-4db3-e468-c98513a339e7",
"colab": {
"base_uri": "https://localhost:8080/",
- "height": 204
- }
+ "height": 210
+ },
+ "outputId": "a486ae30-f530-4388-f9d3-4f1b12d940e1"
},
"source": [
- "# Initialize a DataFrame holding the USA.\n",
- "data = pd.DataFrame({'country': ['country/USA']})\n",
- "\n",
- "# Add a column for states and get their names\n",
- "data['state'] = dc.get_places_in(data['country'], 'State')\n",
- "data = dc.flatten_frame(data)\n",
+ "# Initialize a DataFrame holding the states\n",
+ "data = pd.DataFrame({'state': dc.get_places_in(['country/USA'], 'State')['country/USA']})\n",
"\n",
"# Get all state names and store it in a column \"name\"\n",
- "data['name'] = dc.get_property_values(data['state'], 'name')\n",
- "data = dc.flatten_frame(data)\n",
+ "# Get the first name, if there are multiple for a state\n",
+ "data['name'] = data['state'].map(dc.get_property_values(data['state'], 'name')).str[0]\n",
"\n",
- "# Get StatisticalPopulations representing all persons in each state. \n",
- "data['all_pop'] = dc.get_populations(data['state'], 'Person')\n",
+ "# Get the total count of all persons for each state\n",
+ "data['Count_Person'] = data['state'].map(dc.get_stats(data['state'], \"Count_Person\", ['2017']))\n",
"\n",
- "# Get the total count of all persons in each population\n",
- "data['all'] = dc.get_observations(data['all_pop'], \n",
- " 'count',\n",
- " 'measuredValue',\n",
- " '2017', \n",
- " measurement_method='CenusACS5yrSurvey')\n",
+ "# Extract just the number for population\n",
+ "get_value = lambda row: row.get('data').get('2017')\n",
+ "data['Count_Person'] = data['Count_Person'].apply(get_value).dropna()\n",
"\n",
- "# Display the first five rows of the table.\n",
+ "# Display the first five rows of the table\n",
"data.head(5)"
],
- "execution_count": 7,
+ "execution_count": 3,
"outputs": [
{
"output_type": "execute_result",
@@ -206,71 +151,59 @@
" \n",
" | \n", - " | name | \n", - "all | \n", - "USDollarUpto10000 | \n", - "USDollar10000To14999 | \n", - "USDollar15000To19999 | \n", - "USDollar20000To24999 | \n", - "USDollar25000To29999 | \n", - "USDollar30000To34999 | \n", - "USDollar35000To39999 | \n", - "USDollar40000To44999 | \n", - "USDollar45000To49999 | \n", - "USDollar50000To59999 | \n", - "USDollar60000To74999 | \n", - "USDollar75000To99999 | \n", - "USDollar100000To124999 | \n", - "USDollar125000To149999 | \n", - "USDollar150000To199999 | \n", - "USDollar200000Onwards | \n", - "|||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", - "Alabama | \n", - "4850771.0 | \n", - "170650.0 | \n", - "121688.0 | \n", - "116273.0 | \n", - "112406.0 | \n", - "102590.0 | \n", - "100223.0 | \n", - "92139.0 | \n", - "88987.0 | \n", - "76070.0 | \n", - "149492.0 | \n", - "176589.0 | \n", - "206436.0 | \n", - "133618.0 | \n", - "76939.0 | \n", - "71748.0 | \n", - "60847.0 | \n", - "|||||||||||||||||
| 1 | \n", - "Alaska | \n", - "738565.0 | \n", - "9650.0 | \n", - "7838.0 | \n", - "8013.0 | \n", - "8854.0 | \n", - "8383.0 | \n", - "8914.0 | \n", - "8998.0 | \n", - "10163.0 | \n", - "8536.0 | \n", - "18810.0 | \n", - "26224.0 | \n", - "36745.0 | \n", - "29925.0 | \n", - "19752.0 | \n", - "23596.0 | \n", - "18135.0 | \n", - "|||||||||||||||||
| 2 | \n", - "Arizona | \n", - "6809946.0 | \n", - "178384.0 | \n", - "118676.0 | \n", - "123075.0 | \n", - "131903.0 | \n", - "127741.0 | \n", - "127570.0 | \n", - "119431.0 | \n", - "125103.0 | \n", - "107245.0 | \n", - "208318.0 | \n", - "254294.0 | \n", - "303811.0 | \n", - "200818.0 | \n", - "119347.0 | \n", - "119778.0 | \n", - "116817.0 | \n", - "|||||||||||||||||
| 3 | \n", - "Arkansas | \n", - "2977944.0 | \n", - "97226.0 | \n", - "78535.0 | \n", - "74451.0 | \n", - "75194.0 | \n", - "70211.0 | \n", - "67958.0 | \n", - "61588.0 | \n", - "60814.0 | \n", - "50978.0 | \n", - "95836.0 | \n", - "111618.0 | \n", - "120034.0 | \n", - "72752.0 | \n", - "40780.0 | \n", - "35900.0 | \n", - "33416.0 | \n", - "|||||||||||||||||
| 4 | \n", - "California | \n", - "38982847.0 | \n", - "694945.0 | \n", - "604666.0 | \n", - "536203.0 | \n", - "568994.0 | \n", - "523563.0 | \n", - "539988.0 | \n", - "500571.0 | \n", - "509113.0 | \n", - "456152.0 | \n", - "895758.0 | \n", - "1199773.0 | \n", - "1568843.0 | \n", - "1206637.0 | \n", - "818690.0 | \n", - "1008388.0 | \n", - "1255844.0 | \n", + "39358497 | \n", + "694945 | \n", + "604666 | \n", + "536203 | \n", + "568994 | \n", + "523563 | \n", + "539988 | \n", + "500571 | \n", + "509113 | \n", + "456152 | \n", + "895758 | \n", + "1199773 | \n", + "1568843 | \n", + "1206637 | \n", + "818690 | \n", + "1008388 | \n", + "1255844 | \n", "
"
+ ]
+ },
{
"cell_type": "markdown",
"metadata": {
@@ -23,10 +31,10 @@
"colab_type": "text"
},
"source": [
- "Copyright 2019 Google LLC.\n",
+ "Copyright 2020 Google LLC.\n",
"SPDX-License-Identifier: Apache-2.0\n",
"\n",
- "**Notebook Version** - 1.0.0"
+ "**Notebook Version** - 1.0.1"
]
},
{
@@ -47,7 +55,7 @@
"\n",
"import json"
],
- "execution_count": 0,
+ "execution_count": 1,
"outputs": []
},
{
@@ -59,15 +67,15 @@
"source": [
"# Case Study: Prevalence of Obesity in 500 US Cities\n",
"\n",
- "Obesity is well known to correlate with health factors such as high blood pressure, but is also known to correlate with economic factors such as low-income, unemployment, etc [[1]](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3198075/)[[2]](https://www.ncbi.nlm.nih.gov/pubmed/26562758). The Center for Disease Control (CDC) provides prevalence percentages on health conditions such as [obesity](https://browser.datacommons.org/kg?type=Observation&dcid=dc/o/znbdp41vejt84&r=0), [high blood pressure](https://browser.datacommons.org/kg?type=Observation&dcid=dc/o/wsj0e6pjndf74&r=0), and [high cholesterol](https://browser.datacommons.org/kg?type=Observation&dcid=dc/o/5l1fm8514pgxc&r=0) for approximately 500 major cities in the US (e.g. [San Francisco](https://browser.datacommons.org/kg?type=City&dcid=dc/3eknvn2), [New York](https://browser.datacommons.org/kg?type=City&dcid=dc/prehdd2&r=1), and [Austin](https://browser.datacommons.org/kg?type=City&dcid=dc/khxv4f2&r=1)). Meanwhile, the US Bureau of Labor Statistics provides [unemployment rates]() while the US Census provides [poverty rates]() for most cities across the United States. \n",
+ "Obesity is well known to correlate with health factors such as high blood pressure, but is also known to correlate with economic factors such as low-income, unemployment, etc [[1]](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3198075/)[[2]](https://www.ncbi.nlm.nih.gov/pubmed/26562758). The Center for Disease Control (CDC) provides prevalence percentages on health conditions such as [obesity](https://browser.datacommons.org/kg?dcid=dc/p/8f94jymf10dh7), [high blood pressure](https://browser.datacommons.org/kg?dcid=dc/p/lddprn26my0wh), and [high cholesterol](https://browser.datacommons.org/kg?dcid=dc/p/ybn085073e7z6) for approximately 500 major cities in the US (e.g. [San Francisco](https://browser.datacommons.org/kg?dcid=geoId/0667000), [New York](https://browser.datacommons.org/kg?dcid=geoId/3651000), and [Austin](https://browser.datacommons.org/kg?dcid=geoId/4805000)). Meanwhile, the US Bureau of Labor Statistics provides [unemployment rates](https://browser.datacommons.org/kg?dcid=dc/p/820mxx8ejlrj7) while the US Census provides [poverty rates](https://browser.datacommons.org/kg?dcid=dc/p/mrbgbmbe5lvyh) for most cities across the United States. \n",
"\n",
- "Even though these statistics come from different datasets across different government agencies with different storage formats, dataCommons surfaces each of these in a single, uniform knowledge graph. In fact, you can see this in the [browser](https://browser.datacommons.org/kg?dcid=dc/re83dx3&r=0) by looking at the *provenance* column. Let's use the data in dataCommons to create a linear regression model that incorporates variables:\n",
+ "Even though these statistics come from different datasets across different government agencies with different storage formats, Data Commons surfaces each of these in a single, uniform knowledge graph. In fact, you can see this in the [browser](https://browser.datacommons.org/kg?dcid=dc/9xkkc71) by looking at the *provenance* column. Let's use the data in Data Commons to create a linear regression model that incorporates variables:\n",
"\n",
"- Prevalence of high blood pressure\n",
"- Unemployment rate\n",
"- Percent of population living with income below the poverty line\n",
"\n",
- "to predict the prevalence of obesity in the 500 cities that the CDC provides data for. One thing you may note is that the US Census also provides employment statistics (you can see this by navigating to the \"employment\" and \"employmentStatus\" sections for [San Francisco](https://browser.datacommons.org/kg?type=City&dcid=dc/3eknvn2) and observing the different provenances). Our choice of using statistics from the Bureau of Labor Statistics is purely demonstrative, but it would be interesting to see if similar results can be reproduced using US Census employment statistics. "
+ "to predict the prevalence of obesity in the 500 cities that the CDC provides data for. One thing you may note is that the US Census also provides employment statistics (you can see this by navigating to the \"employment\" and \"employmentStatus\" sections for [San Francisco](https://browser.datacommons.org/kg?dcid=geoId/0667000) and observing the different provenances). Our choice of using statistics from the Bureau of Labor Statistics is purely demonstrative, but it would be interesting to see if similar results can be reproduced using US Census employment statistics. "
]
},
{
@@ -75,71 +83,15 @@
"metadata": {
"id": "GQrede0Bdm-k",
"colab_type": "code",
- "outputId": "3509bade-4032-4e36-a991-78a014575f8c",
- "colab": {
- "base_uri": "https://localhost:8080/",
- "height": 34
- }
+ "colab": {}
},
"source": [
- "!pip install --upgrade --quiet git+https://github.com/datacommonsorg/api-python.git@stable-1.x\n",
+ "!pip install datacommons --upgrade --quiet\n",
"\n",
"import datacommons as dc"
],
- "execution_count": 5,
- "outputs": [
- {
- "output_type": "stream",
- "text": [
- " Building wheel for datacommons (setup.py) ... \u001b[?25l\u001b[?25hdone\n"
- ],
- "name": "stdout"
- }
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "HH3P0arRk7Mv",
- "colab_type": "text"
- },
- "source": [
- "We will also need to provide the API with an API key. See the [Analyzing Statistics in Data Commons Using the Python API](https://colab.research.google.com/drive/1ZNXTHu3J0W3vo9Mg3kNUpk0hnD6Ce1u6#scrollTo=ijxoBhFHjo3Z) to see how to set this up for a Colab Notebook."
- ]
- },
- {
- "cell_type": "code",
- "metadata": {
- "id": "_am1Pfb0lBNh",
- "colab_type": "code",
- "outputId": "01be5f1c-cc52-487d-cb62-7e5a726ae39c",
- "colab": {
- "base_uri": "https://localhost:8080/",
- "height": 34
- }
- },
- "source": [
- "# Mount the Drive\n",
- "drive.mount('/content/drive', force_remount=True)\n",
- "\n",
- "# REPLACE THIS with the path to your key.\n",
- "key_path = '/content/drive/My Drive/DataCommons/secret.json'\n",
- "\n",
- "# Read the key in and provide it to the Data Commons API\n",
- "with open(key_path, 'r') as f:\n",
- " secrets = json.load(f)\n",
- " dc.set_api_key(secrets['dc_api_key'])"
- ],
- "execution_count": 6,
- "outputs": [
- {
- "output_type": "stream",
- "text": [
- "Mounted at /content/drive\n"
- ],
- "name": "stdout"
- }
- ]
+ "execution_count": 2,
+ "outputs": []
},
{
"cell_type": "markdown",
@@ -148,7 +100,7 @@
"colab_type": "text"
},
"source": [
- "To get started, we'll need the cities that CDC provides data for. We've listed the DCIDs of all cities with CDC obesity prevalence statistics and stored it as a variable `city_dcids`. For now we will hide this cell!"
+ "To get started, we'll need the cities that CDC provides data for. We can query Data Commons for Cities that are members of [CDC500_City](https://browser.datacommons.org/kg?dcid=CDC500_City). The default return limit is 100, so we'll explicitly set it to 500."
]
},
{
@@ -156,182 +108,13 @@
"metadata": {
"id": "4CoM0LD4kZSN",
"colab_type": "code",
- "cellView": "form",
+ "cellView": "both",
"colab": {}
},
"source": [
- "#@title\n",
- "city_dcids = [\n",
- " \"geoId/5370000\", \"geoId/0524550\", \"geoId/0820000\", \n",
- " \"geoId/3502000\", \"geoId/0665000\", \"geoId/1239550\", \n",
- " \"geoId/2148006\", \"geoId/1232000\", \"geoId/1757225\", \n",
- " \"geoId/0622804\", \"geoId/0807850\", \"geoId/4848072\", \n",
- " \"geoId/3921000\", \"geoId/0680000\", \"geoId/3650617\", \n",
- " \"geoId/4813024\", \"geoId/0644000\", \"geoId/0918430\", \n",
- " \"geoId/5147672\", \"geoId/3345140\", \"geoId/4856000\", \n",
- " \"geoId/1714351\", \"geoId/1982425\", \"geoId/2674900\", \n",
- " \"geoId/1258050\", \"geoId/4943660\", \"geoId/0541000\", \n",
- " \"geoId/0659451\", \"geoId/1269700\", \"geoId/0616350\", \n",
- " \"geoId/0653980\", \"geoId/4459000\", \"geoId/0650258\", \n",
- " \"geoId/0622230\", \"geoId/4845744\", \"geoId/0662938\", \n",
- " \"geoId/0150000\", \"geoId/0673080\", \"geoId/3725580\", \n",
- " \"geoId/0603526\", \"geoId/0846465\", \"geoId/1656260\", \n",
- " \"geoId/2523000\", \"geoId/2241155\", \"geoId/0427820\", \n",
- " \"geoId/2970000\", \"geoId/0622020\", \"geoId/3728000\", \n",
- " \"geoId/5101000\", \"geoId/0681204\", \"geoId/1851876\", \n",
- " \"geoId/1768003\", \"geoId/4224000\", \"geoId/4815976\", \n",
- " \"geoId/2688940\", \"geoId/0606000\", \"geoId/0454050\", \n",
- " \"geoId/3006550\", \"geoId/1751622\", \"geoId/4748000\", \n",
- " \"geoId/5164000\", \"geoId/1077580\", \"geoId/0681666\", \n",
- " \"geoId/5548000\", \"geoId/5584250\", \"geoId/0681344\", \n",
- " \"geoId/0623182\", \"geoId/2915670\", \"geoId/1368516\", \n",
- " \"geoId/4260000\", \"geoId/0615044\", \"geoId/1765000\", \n",
- " \"geoId/4819000\", \"geoId/1921000\", \"geoId/1212875\", \n",
- " \"geoId/3413690\", \"geoId/1253000\", \"geoId/0827425\", \n",
- " \"geoId/0603666\", \"geoId/0616000\", \"geoId/2255000\", \n",
- " \"geoId/5539225\", \"geoId/3710740\", \"geoId/3663000\", \n",
- " \"geoId/4843888\", \"geoId/2567000\", \"geoId/2646000\", \n",
- " \"geoId/2954074\", \"geoId/1150000\", \"geoId/0668084\", \n",
- " \"geoId/3665508\", \"geoId/3436000\", \"geoId/1270600\", \n",
- " \"geoId/0980000\", \"geoId/2706616\", \"geoId/4548535\", \n",
- " \"geoId/1840788\", \"geoId/0677000\", \"geoId/1779293\", \n",
- " \"geoId/2836000\", \"geoId/4983470\", \"geoId/0636770\", \n",
- " \"geoId/2680700\", \"geoId/4803000\", \"geoId/1239425\", \n",
- " \"geoId/2509000\", \"geoId/2036000\", \"geoId/0446000\", \n",
- " \"geoId/2582000\", \"geoId/0816000\", \"geoId/0477000\", \n",
- " \"geoId/0616532\", \"geoId/0658072\", \"geoId/4856348\", \n",
- " \"geoId/2511000\", \"geoId/5303180\", \"geoId/0877290\", \n",
- " \"geoId/4847892\", \"geoId/5305280\", \"geoId/1257425\", \n",
- " \"geoId/0653000\", \"geoId/3912000\", \"geoId/2649000\", \n",
- " \"geoId/1912000\", \"geoId/0639892\", \"geoId/4957300\", \n",
- " \"geoId/4962470\", \"geoId/0680812\", \"geoId/0640130\", \n",
- " \"geoId/1822000\", \"geoId/4147000\", \"geoId/0660466\", \n",
- " \"geoId/1823278\", \"geoId/2758000\", \"geoId/0681554\", \n",
- " \"geoId/1207875\", \"geoId/0135896\", \"geoId/3702140\", \n",
- " \"geoId/4261000\", \"geoId/1207300\", \"geoId/2669035\", \n",
- " \"geoId/4810768\", \"geoId/0680854\", \"geoId/5135000\", \n",
- " \"geoId/0535710\", \"geoId/1712385\", \"geoId/0107000\", \n",
- " \"geoId/0655156\", \"geoId/1772000\", \"geoId/4419180\", \n",
- " \"geoId/4751560\", \"geoId/0630000\", \"geoId/3916000\", \n",
- " \"geoId/5566000\", \"geoId/0614218\", \"geoId/5367000\", \n",
- " \"geoId/2038900\", \"geoId/2545000\", \"geoId/0659920\", \n",
- " \"geoId/1938595\", \"geoId/0648256\", \"geoId/3825700\", \n",
- " \"geoId/0845970\", \"geoId/4752006\", \"geoId/4841464\", \n",
- " \"geoId/0678582\", \"geoId/0613392\", \"geoId/3260600\", \n",
- " \"geoId/4041850\", \"geoId/4819972\", \"geoId/4827000\", \n",
- " \"geoId/3240000\", \"geoId/0608786\", \"geoId/0636000\", \n",
- " \"geoId/4714000\", \"geoId/5357745\", \"geoId/4105800\", \n",
- " \"geoId/0455000\", \"geoId/1703012\", \"geoId/1367284\", \n",
- " \"geoId/4848804\", \"geoId/0665042\", \"geoId/0684200\", \n",
- " \"geoId/1827000\", \"geoId/1714000\", \"geoId/1342425\", \n",
- " \"geoId/5010675\", \"geoId/4513330\", \"geoId/0611194\", \n",
- " \"geoId/0621712\", \"geoId/0650076\", \"geoId/3712000\", \n",
- " \"geoId/0660018\", \"geoId/1831000\", \"geoId/4982950\", \n",
- " \"geoId/4715160\", \"geoId/1243975\", \"geoId/4822660\", \n",
- " \"geoId/0843000\", \"geoId/5531000\", \"geoId/5167000\", \n",
- " \"geoId/4105350\", \"geoId/4123850\", \"geoId/1652120\", \n",
- " \"geoId/0602000\", \"geoId/4810912\", \"geoId/0628000\", \n",
- " \"geoId/0804000\", \"geoId/5176432\", \"geoId/2621000\", \n",
- " \"geoId/2754880\", \"geoId/4561405\", \"geoId/4829000\", \n",
- " \"geoId/1973335\", \"geoId/5380010\", \"geoId/0646898\", \n",
- " \"geoId/4740000\", \"geoId/2717000\", \"geoId/0668000\", \n",
- " \"geoId/1230000\", \"geoId/5157000\", \"geoId/0624680\", \n",
- " \"geoId/1216725\", \"geoId/0651182\", \"geoId/0636546\", \n",
- " \"geoId/0523290\", \"geoId/0669070\", \"geoId/0137000\", \n",
- " \"geoId/0659962\", \"geoId/3128000\", \"geoId/0613210\", \n",
- " \"geoId/4801924\", \"geoId/0812815\", \"geoId/0485540\", \n",
- " \"geoId/0613214\", \"geoId/4009050\", \"geoId/0600884\", \n",
- " \"geoId/0662000\", \"geoId/1245025\", \"geoId/3961000\", \n",
- " \"geoId/2629000\", \"geoId/4804000\", \"geoId/3570500\", \n",
- " \"geoId/5182000\", \"geoId/4841980\", \"geoId/0611530\", \n",
- " \"geoId/3722920\", \"geoId/2146027\", \"geoId/2052575\", \n",
- " \"geoId/1759000\", \"geoId/3474000\", \"geoId/1224000\", \n",
- " \"geoId/0648354\", \"geoId/4861796\", \"geoId/1254200\", \n",
- " \"geoId/15003\", \"geoId/5305210\", \"geoId/4269000\", \n",
- " \"geoId/4807000\", \"geoId/4879000\", \"geoId/0629000\", \n",
- " \"geoId/0656000\", \"geoId/2935000\", \"geoId/2071000\", \n",
- " \"geoId/4835000\", \"geoId/1235000\", \"geoId/4075000\", \n",
- " \"geoId/4853388\", \"geoId/0617918\", \"geoId/4055000\", \n",
- " \"geoId/1871000\", \"geoId/5116000\", \"geoId/0668196\", \n",
- " \"geoId/5374060\", \"geoId/4830464\", \"geoId/2537490\", \n",
- " \"geoId/1238250\", \"geoId/2360545\", \"geoId/0832155\", \n",
- " \"geoId/0664000\", \"geoId/4801000\", \"geoId/0647766\", \n",
- " \"geoId/0672016\", \"geoId/2684000\", \"geoId/4858016\", \n",
- " \"geoId/0465000\", \"geoId/3563460\", \"geoId/0646842\", \n",
- " \"geoId/3775000\", \"geoId/0682996\", \"geoId/1707133\", \n",
- " \"geoId/2627440\", \"geoId/2239475\", \"geoId/2603000\", \n",
- " \"geoId/5335415\", \"geoId/1723074\", \"geoId/0632548\", \n",
- " \"geoId/0670000\", \"geoId/1836003\", \"geoId/0669000\", \n",
- " \"geoId/5168000\", \"geoId/3268400\", \"geoId/3774440\", \n",
- " \"geoId/3457000\", \"geoId/2964550\", \"geoId/4876000\", \n",
- " \"geoId/5414600\", \"geoId/0656700\", \"geoId/3539380\", \n",
- " \"geoId/0657792\", \"geoId/0670098\", \"geoId/0952000\", \n",
- " \"geoId/0666000\", \"geoId/0973000\", \"geoId/0639496\", \n",
- " \"geoId/0566080\", \"geoId/0600562\", \"geoId/3673000\", \n",
- " \"geoId/1245060\", \"geoId/0669196\", \"geoId/3350260\", \n",
- " \"geoId/0862000\", \"geoId/2676460\", \"geoId/0955990\", \n",
- " \"geoId/1810342\", \"geoId/0626000\", \"geoId/1738570\", \n",
- " \"geoId/3137000\", \"geoId/4052500\", \"geoId/4857200\", \n",
- " \"geoId/3601000\", \"geoId/0608954\", \"geoId/0619766\", \n",
- " \"geoId/3456550\", \"geoId/3474630\", \"geoId/4131250\", \n",
- " \"geoId/4164900\", \"geoId/1319000\", \"geoId/2555745\", \n",
- " \"geoId/3714100\", \"geoId/3231900\", \"geoId/0602364\", \n",
- " \"geoId/4550875\", \"geoId/0427400\", \"geoId/1369000\", \n",
- " \"geoId/1225175\", \"geoId/0654652\", \"geoId/1304204\", \n",
- " \"geoId/4202000\", \"geoId/3918000\", \"geoId/2829700\", \n",
- " \"geoId/2270000\", \"geoId/5367167\", \"geoId/0937000\", \n",
- " \"geoId/4263624\", \"geoId/4845384\", \"geoId/5613900\", \n",
- " \"geoId/2941348\", \"geoId/0682590\", \"geoId/1258715\", \n",
- " \"geoId/4874144\", \"geoId/0682954\", \"geoId/1825000\", \n",
- " \"geoId/0908000\", \"geoId/0633434\", \"geoId/0653896\", \n",
- " \"geoId/4955980\", \"geoId/0151000\", \"geoId/0645484\", \n",
- " \"geoId/1216475\", \"geoId/3728080\", \"geoId/2545560\", \n",
- " \"geoId/0883835\", \"geoId/3410000\", \"geoId/2642160\", \n",
- " \"geoId/2537000\", \"geoId/4848768\", \"geoId/1724582\", \n",
- " \"geoId/4965330\", \"geoId/0627000\", \"geoId/0652526\", \n",
- " \"geoId/0649670\", \"geoId/4870808\", \"geoId/4206088\", \n",
- " \"geoId/1919000\", \"geoId/3755000\", \"geoId/0636448\", \n",
- " \"geoId/0641992\", \"geoId/5553000\", \"geoId/0684550\", \n",
- " \"geoId/0675000\", \"geoId/5323515\", \"geoId/3684000\", \n",
- " \"geoId/2404000\", \"geoId/0613588\", \"geoId/4652980\", \n",
- " \"geoId/3651000\", \"geoId/2562535\", \"geoId/1718823\", \n",
- " \"geoId/1245000\", \"geoId/2507000\", \"geoId/0604982\", \n",
- " \"geoId/0664224\", \"geoId/2240735\", \"geoId/0678120\", \n",
- " \"geoId/3451000\", \"geoId/2751730\", \"geoId/0667000\", \n",
- " \"geoId/4659020\", \"geoId/5363000\", \"geoId/2622000\", \n",
- " \"geoId/1255775\", \"geoId/4863500\", \"geoId/0685292\", \n",
- " \"geoId/0950370\", \"geoId/1217200\", \"geoId/0643000\", \n",
- " \"geoId/3734200\", \"geoId/4837000\", \"geoId/0412000\", \n",
- " \"geoId/4842508\", \"geoId/1303440\", \"geoId/4967000\", \n",
- " \"geoId/0669088\", \"geoId/0660620\", \"geoId/0203000\", \n",
- " \"geoId/1702154\", \"geoId/2686000\", \"geoId/4806128\", \n",
- " \"geoId/0669084\", \"geoId/1304000\", \"geoId/1301052\", \n",
- " \"geoId/0649270\", \"geoId/1210275\", \"geoId/0613014\", \n",
- " \"geoId/4474300\", \"geoId/5322640\", \"geoId/2743000\", \n",
- " \"geoId/0803455\", \"geoId/1271000\", \"geoId/1380508\", \n",
- " \"geoId/0602252\", \"geoId/4134100\", \"geoId/4159000\", \n",
- " \"geoId/3901000\", \"geoId/1263000\", \"geoId/3421000\", \n",
- " \"geoId/4817000\", \"geoId/1245975\", \"geoId/0404720\", \n",
- " \"geoId/4454640\", \"geoId/0633000\", \"geoId/3611000\", \n",
- " \"geoId/5156000\", \"geoId/3977000\", \"geoId/4845000\", \n",
- " \"geoId/0633182\", \"geoId/0668252\", \"geoId/3731400\", \n",
- " \"geoId/1706613\", \"geoId/2634000\", \"geoId/4839148\", \n",
- " \"geoId/4516000\", \"geoId/2707966\", \"geoId/1349000\", \n",
- " \"geoId/2053775\", \"geoId/1214400\", \"geoId/4864472\", \n",
- " \"geoId/2938000\", \"geoId/3251800\", \"geoId/2965000\", \n",
- " \"geoId/4865000\", \"geoId/3649121\", \"geoId/4827684\", \n",
- " \"geoId/0471510\", \"geoId/1254000\", \"geoId/4023200\", \n",
- " \"geoId/3050200\", \"geoId/1276600\", \"geoId/0660102\", \n",
- " \"geoId/0644574\", \"geoId/3719000\", \"geoId/1608830\", \n",
- " \"geoId/0653322\", \"geoId/5502375\", \"geoId/2534550\", \n",
- " \"geoId/4967440\", \"geoId/0473000\", \"geoId/5335275\", \n",
- " \"geoId/3915000\", \"geoId/1805860\", \"geoId/4824000\", \n",
- " \"geoId/4805000\", \"geoId/2079000\", \"geoId/2205000\", \n",
- " \"geoId/0624638\", \"geoId/3988000\", \"geoId/0668378\", \n",
- " \"geoId/0177256\", \"geoId/0680238\",\n",
- "]"
+ "city_dcids = dc.get_property_values([\"CDC500_City\"], \"member\", limit=500)"
],
- "execution_count": 0,
+ "execution_count": 3,
"outputs": []
},
{
@@ -349,24 +132,26 @@
"metadata": {
"id": "IM4ajcwoNtXr",
"colab_type": "code",
- "outputId": "f1bb3296-1bff-48a9-ba56-d1a36677548d",
"colab": {
"base_uri": "https://localhost:8080/",
- "height": 204
- }
+ "height": 210
+ },
+ "outputId": "d02424fe-8a0a-407f-c279-b7dc82d66050"
},
"source": [
"# Create the Pandas DataFrame \n",
- "data = pd.DataFrame({'City': city_dcids})\n",
- "\n",
+ "data = pd.DataFrame(city_dcids)\n",
+ "# Simplify the column name\n",
+ "data.columns = ['City']\n",
"# Get the name of each city\n",
- "data['CityName'] = dc.get_property_values(data['City'], 'name')\n",
- "data = dc.flatten_frame(data)\n",
+ "# If a city has multiple common names (e.g. \"New York\" and \"New York City\"), use the first one\n",
+ "data['CityName'] = data['City'].map(dc.get_property_values(data['City'], 'name')).str[0]\n",
+ "data.dropna(inplace=True)\n",
"\n",
"# Show the top five rows\n",
"data.head(5)"
],
- "execution_count": 8,
+ "execution_count": 4,
"outputs": [
{
"output_type": "execute_result",
@@ -397,46 +182,46 @@
" \n",
"