Dataset documentation and questions

CMS Open Payments

CMS Open Payments

Information reported by manufactures and group purchasing organizations (GPOs, see here and here) about payments to physicians and teaching hospitals. Useful links:

• natures of payments
• glossary and acronyms
• see also resources and FAQs
• video about the 2014 program overview

1. OP_DTL_OWNRSHP_PGYR2014_P06302015.csv - value of ownership and investment interest of physician in Manu/GPO
2. OP_DTL_GNRL_PGYR2014_P06302015.csv - general payments
3. OP_DTL_RSRCH_PGYR2014_P06302015.csv -research payments

OP_DTL_OWNRSHP_PGYR2014_P06302015.csv

Columns

• Physician_Profile_ID
• Physician_First_Name
• Physician_Middle_Name
• Physician_Last_Name
• Physician_Name_Suffix
• Recipient_Primary_Business_Street_Address_Line1
• Recipient_Primary_Business_Street_Address_Line2
• Recipient_City
• Recipient_State
• Recipient_Zip_Code
• Recipient_Country
• Recipient_Province
• Recipient_Postal_Code
• Physician_Primary_Type
• Physician_Specialty
• Record_ID
• Program_Year
• Total_Amount_Invested_USDollars
• Value_of_Interest
• Terms_of_Interest
• Submitting_Applicable_Manufacturer_or_Applicable_GPO_Name
• Applicable_Manufacturer_or_Applicable_GPO_Making_Payment_ID
• Applicable_Manufacturer_or_Applicable_GPO_Making_Payment_Name
• Applicable_Manufacturer_or_Applicable_GPO_Making_Payment_State
• Applicable_Manufacturer_or_Applicable_GPO_Making_Payment_Country
• Dispute_Status_for_Publication
• Interest_Held_by_Physician_or_an_Immediate_Family_Member
• Payment_Publication_Date

OP_DTL_GNRL_PGYR2014_P06302015.csv

Columns

(color codes: black = looked at, pink = long text field, not looked at)

• Covered_Recipient_Type

• Teaching_Hospital_ID

• Teaching_Hospital_Name

• Physician_Profile_ID

• Physician_First_Name

• Physician_Middle_Name

• Physician_Last_Name

• Physician_Name_Suffix

• Recipient_Primary_Business_Street_Address_Line1

• Recipient_Primary_Business_Street_Address_Line2

• Recipient_City

• Recipient_State

• Recipient_Zip_Code

• Recipient_Country: 23 countries

• Recipient_Province

• Recipient_Postal_Code

• Physician_Primary_Type:

• Chiropractor

• Doctor of Dentistry

• Doctor of Optometry

• Doctor of Osteopathy

• Doctor of Podiatric Medicine

• Medical Doctor

• Physician_Specialty

• Physician_License_State_code1

• Physician_License_State_code2

• Physician_License_State_code3

• Physician_License_State_code4

• Physician_License_State_code5

• Submitting_Applicable_Manufacturer_or_Applicable_GPO_Name

• Applicable_Manufacturer_or_Applicable_GPO_Making_Payment_ID

• Applicable_Manufacturer_or_Applicable_GPO_Making_Payment_Name

• Applicable_Manufacturer_or_Applicable_GPO_Making_Payment_State

• Applicable_Manufacturer_or_Applicable_GPO_Making_Payment_Country

• Total_Amount_of_Payment_USDollars

• Date_of_Payment

• Number_of_Payments_Included_in_Total_Amount

• Form_of_Payment_or_Transfer_of_Value

• Cash or cash equivalent

• Dividend, profit or other return on investment

• In-kind items and services

• Stock, stock option, or any other ownership interest

• Nature_of_Payment_or_Transfer_of_Value

• City_of_Travel

• State_of_Travel

• Country_of_Travel: 96 countries

• Physician_Ownership_Indicator

• Third_Party_Payment_Recipient_Indicator

• Name_of_Third_Party_Entity_Receiving_Payment_or_Transfer_of_Value

• Charity_Indicator

• Third_Party_Equals_Covered_Recipient_Indicator

• Contextual_Information

• Delay_in_Publication_Indicator

• Record_ID

• Dispute_Status_for_Publication

• Product_Indicator

• Name_of_Associated_Covered_Drug_or_Biological1

• Name_of_Associated_Covered_Drug_or_Biological2

• Name_of_Associated_Covered_Drug_or_Biological3

• Name_of_Associated_Covered_Drug_or_Biological4

• Name_of_Associated_Covered_Drug_or_Biological5

• NDC_of_Associated_Covered_Drug_or_Biological1

• NDC_of_Associated_Covered_Drug_or_Biological2

• NDC_of_Associated_Covered_Drug_or_Biological3

• NDC_of_Associated_Covered_Drug_or_Biological4

• NDC_of_Associated_Covered_Drug_or_Biological5

• Name_of_Associated_Covered_Device_or_Medical_Supply1

• Name_of_Associated_Covered_Device_or_Medical_Supply2

• Name_of_Associated_Covered_Device_or_Medical_Supply3

• Name_of_Associated_Covered_Device_or_Medical_Supply4

• Name_of_Associated_Covered_Device_or_Medical_Supply5

• Program_Year: 2014 only

####Suggested Columns:

• general: Teaching_Hospital_ID, Teaching_Hospital_Name, Physician_Profile_ID, Recipient_Country, Physician_License_State_code1
• Submitting_Applicable_Manufacturer_or_Applicable_GPO_Name
• Applicable_Manufacturer_or_Applicable_GPO_Making_Payment_Name
• Applicable_Manufacturer_or_Applicable_GPO_Making_Payment_Country
• Total_Amount_of_Payment_USDollars, Date_of_Payment
• Number_of_Payments_Included_in_Total_Amount
• Form_of_Payment_or_Transfer_of_Value, Nature_of_Payment_or_Transfer_of_Value

####Categorical Variables:

• Covered_Recipient_Type
• Physician_Primary_Type
• Form_of_Payment_or_Transfer_of_Value
• Nature_of_Payment_or_Transfer_of_Value
• Country_of_Travel
• Physician_Ownership_Indicator
• Third_Party_Payment_Recipient_Indicator
• Charity_Indicator
• Third_Party_Equals_Covered_Recipient_Indicator
• Delay_in_Publication_Indicator
• Dispute_Status_for_Publication
• Product_Indicator

keys: Record_ID, Applicable_Manufacturer_or_Applicable_GPO_Making_Payment_ID

OP_DTL_RSRCH_PGYR2014_P06302015.csv

Columns

• Covered_Recipient_Type
• Noncovered_Recipient_Entity_Name
• Teaching_Hospital_ID
• Teaching_Hospital_Name
• Physician_Profile_ID
• Physician_First_Name
• Physician_Middle_Name
• Physician_Last_Name
• Physician_Name_Suffix
• Recipient_Primary_Business_Street_Address_Line1
• Recipient_Primary_Business_Street_Address_Line2
• Recipient_City
• Recipient_State
• Recipient_Zip_Code
• Recipient_Country
• Recipient_Province
• Recipient_Postal_Code
• Physician_Primary_Type
• Physician_Specialty
• Physician_License_State_code1
• Physician_License_State_code2
• Physician_License_State_code3
• Physician_License_State_code4
• Physician_License_State_code5
• Principal_Investigator_1_Profile_ID
• Principal_Investigator_1_First_Name
• Principal_Investigator_1_Middle_Name
• Principal_Investigator_1_Last_Name
• Principal_Investigator_1_Name_Suffix
• Principal_Investigator_1_Business_Street_Address_Line1
• Principal_Investigator_1_Business_Street_Address_Line2
• Principal_Investigator_1_City
• Principal_Investigator_1_State
• Principal_Investigator_1_Zip_Code
• Principal_Investigator_1_Country
• Principal_Investigator_1_Province
• Principal_Investigator_1_Postal_Code
• Principal_Investigator_1_Primary_Type
• Principal_Investigator_1_Specialty
• Principal_Investigator_1_License_State_code1
• Principal_Investigator_1_License_State_code2
• Principal_Investigator_1_License_State_code3
• Principal_Investigator_1_License_State_code4
• Principal_Investigator_1_License_State_code5
• Principal_Investigator_2_Profile_ID
• Principal_Investigator_2_First_Name
• Principal_Investigator_2_Middle_Name
• Principal_Investigator_2_Last_Name
• Principal_Investigator_2_Name_Suffix
• Principal_Investigator_2_Business_Street_Address_Line1
• Principal_Investigator_2_Business_Street_Address_Line2
• Principal_Investigator_2_City
• Principal_Investigator_2_State
• Principal_Investigator_2_Zip_Code
• Principal_Investigator_2_Country
• Principal_Investigator_2_Province
• Principal_Investigator_2_Postal_Code
• Principal_Investigator_2_Primary_Type
• Principal_Investigator_2_Specialty
• Principal_Investigator_2_License_State_code1
• Principal_Investigator_2_License_State_code2
• Principal_Investigator_2_License_State_code3
• Principal_Investigator_2_License_State_code4
• Principal_Investigator_2_License_State_code5
• Principal_Investigator_3_Profile_ID
• Principal_Investigator_3_First_Name
• Principal_Investigator_3_Middle_Name
• Principal_Investigator_3_Last_Name
• Principal_Investigator_3_Name_Suffix
• Principal_Investigator_3_Business_Street_Address_Line1
• Principal_Investigator_3_Business_Street_Address_Line2
• Principal_Investigator_3_City
• Principal_Investigator_3_State
• Principal_Investigator_3_Zip_Code
• Principal_Investigator_3_Country
• Principal_Investigator_3_Province
• Principal_Investigator_3_Postal_Code
• Principal_Investigator_3_Primary_Type
• Principal_Investigator_3_Specialty
• Principal_Investigator_3_License_State_code1
• Principal_Investigator_3_License_State_code2
• Principal_Investigator_3_License_State_code3
• Principal_Investigator_3_License_State_code4
• Principal_Investigator_3_License_State_code5
• Principal_Investigator_4_Profile_ID
• Principal_Investigator_4_First_Name
• Principal_Investigator_4_Middle_Name
• Principal_Investigator_4_Last_Name
• Principal_Investigator_4_Name_Suffix
• Principal_Investigator_4_Business_Street_Address_Line1
• Principal_Investigator_4_Business_Street_Address_Line2
• Principal_Investigator_4_City
• Principal_Investigator_4_State
• Principal_Investigator_4_Zip_Code
• Principal_Investigator_4_Country
• Principal_Investigator_4_Province
• Principal_Investigator_4_Postal_Code
• Principal_Investigator_4_Primary_Type
• Principal_Investigator_4_Specialty
• Principal_Investigator_4_License_State_code1
• Principal_Investigator_4_License_State_code2
• Principal_Investigator_4_License_State_code3
• Principal_Investigator_4_License_State_code4
• Principal_Investigator_4_License_State_code5
• Principal_Investigator_5_Profile_ID
• Principal_Investigator_5_First_Name
• Principal_Investigator_5_Middle_Name
• Principal_Investigator_5_Last_Name
• Principal_Investigator_5_Name_Suffix
• Principal_Investigator_5_Business_Street_Address_Line1
• Principal_Investigator_5_Business_Street_Address_Line2
• Principal_Investigator_5_City
• Principal_Investigator_5_State
• Principal_Investigator_5_Zip_Code
• Principal_Investigator_5_Country
• Principal_Investigator_5_Province
• Principal_Investigator_5_Postal_Code
• Principal_Investigator_5_Primary_Type
• Principal_Investigator_5_Specialty
• Principal_Investigator_5_License_State_code1
• Principal_Investigator_5_License_State_code2
• Principal_Investigator_5_License_State_code3
• Principal_Investigator_5_License_State_code4
• Principal_Investigator_5_License_State_code5
• Submitting_Applicable_Manufacturer_or_Applicable_GPO_Name
• Applicable_Manufacturer_or_Applicable_GPO_Making_Payment_ID
• Applicable_Manufacturer_or_Applicable_GPO_Making_Payment_Name
• Applicable_Manufacturer_or_Applicable_GPO_Making_Payment_State
• Applicable_Manufacturer_or_Applicable_GPO_Making_Payment_Country
• Product_Indicator
• Name_of_Associated_Covered_Drug_or_Biological1
• Name_of_Associated_Covered_Drug_or_Biological2
• Name_of_Associated_Covered_Drug_or_Biological3
• Name_of_Associated_Covered_Drug_or_Biological4
• Name_of_Associated_Covered_Drug_or_Biological5
• NDC_of_Associated_Covered_Drug_or_Biological1
• NDC_of_Associated_Covered_Drug_or_Biological2
• NDC_of_Associated_Covered_Drug_or_Biological3
• NDC_of_Associated_Covered_Drug_or_Biological4
• NDC_of_Associated_Covered_Drug_or_Biological5
• Name_of_Associated_Covered_Device_or_Medical_Supply1
• Name_of_Associated_Covered_Device_or_Medical_Supply2
• Name_of_Associated_Covered_Device_or_Medical_Supply3
• Name_of_Associated_Covered_Device_or_Medical_Supply4
• Name_of_Associated_Covered_Device_or_Medical_Supply5
• Total_Amount_of_Payment_USDollars
• Date_of_Payment
• Form_of_Payment_or_Transfer_of_Value
• Expenditure_Category1
• Expenditure_Category2
• Expenditure_Category3
• Expenditure_Category4
• Expenditure_Category5
• Expenditure_Category6
• Preclinical_Research_Indicator
• Delay_in_Publication_Indicator
• Name_of_Study
• Dispute_Status_for_Publication
• Record_ID
• Program_Year
• Payment_Publication_Date
• ClinicalTrials_Gov_Identifier
• Research_Information_Link
• Context_of_Research-

---

Elhadad

Data

The goal was to predict when an article is newsworthy. To do that they built two datasets:

1. Articles from a diverse range of biomedical and health journal and kept the articles that were considered newsworthy because they were covered by Reuters. (1,431 rows)
2. Articles from JAMA journals and kept the one that were they were given press release by the journal editors. (1,007 rows)

Columns for *_article_info.csv:

• PubMed ID,
• citation title
• journal,
• authors,
• affiliation (or missing),
• abstract,
• MeSH terms: Medical Subject Headings (MeSH) is a comprehensive controlled vocabulary for the purpose of indexing journal articles and books in the life sciences; it serves as a thesaurus that facilitates searching.

To add columns to the dataframe create by pd.read CSV do the following: reuters.columns=['PubMed_ID', 'citation_title', 'journal', 'authors', 'affiliation', 'abstract', 'MeSH terms']

There are 2 other datasets that they call negative instances. These contain articles having some similarities but not considered newsworthy. The two datasets are :

1. Negative instances linked to the article from Reuters. (28,359 rows)
2. Negative instances linked to articles from JAMA. (10,026 rows)

These datasets are the datasets jama_article_info.csv and all_reuters_artcile_info.csv.

Columns for matched_articles_.csv:

• PubMed ID,[ This is the pmid of the corresponding article for which the row was 'matched']
• PubMed ID,
• citation title
• journal,
• authors,
• affiliation (or missing),
• abstract,
• MeSH terms

There are 2 other datasets that they call negative instances. These contain articles having some similarities but not considered newsworthy. The two datasets are :

3. Negative instances linked to the article from Reuters. (28,359 rows)
4. Negative instances linked to articles from JAMA. (10,026 rows)

These are the datasets all_reuters_matched_articles_filtered.csv and jama_pmids.txt_matched_articles_filtered.csv. Negative instances contain X articles for each positive article (1) or (2) published in the same journal in the same year that did not receive coverage in the Reuters corpus or no press release was issued for JAMA. They also used several filtering heuristics to do this « matched sampling ». Not more explanations on how they matched this negative instances.

The columns are the same as (1) and (2) except that the first column corresponds to the PubMed ID from the newsworthy corresponding article.

What they did in the paper

They built citation features: journal name, institution of first author, extract words (uni and bi-grams) from titles, abstracts and MeSH terms. At the end they had 14,614 features. Then they used a L2 regularized regression to predict the newsworthiness.

---

Chronic Conditions

Each cell is a record defined by all combinations of age category, gender, various chronic conditions, and dual-eligibility status (eligible for both Medicare and Medicaid) of the beneficiaries.

2010_Chronic_Conditions_PUF.csv 3.6 MB, 22003 rows, 55 columns - see data dictionary

COLUMNS

###Profile features:

• BENE_SEX_IDENT_CD: 1 or 2
• BENE_AGE_CAT_CD: 6 groups

###Conditions:

• CC_ALZHDMTA

• CC_CANCER

• CC_CHF

• CC_CHRNKIDN

• CC_COPD

• CC_DEPRESSN

• CC_DIABETES

• CC_ISCHMCHT

• CC_OSTEOPRS

• CC_RA_OA

• CC_STRKETIA

• CC_2_OR_MORE

• DUAL_STUS

###Claims variables (in the form of counts and averages): ####Part A: less than 12 months

• BENE_COUNT_PA_LT_12
• AVE_MO_EN_PA_LT_12
• AVE_PA_PAY_PA_LT_12
• AVE_IP_PAY_PA_LT_12
• AVE_SNF_PAY_PA_LT_12
• AVE_OTH_PAY_PA_LT_12
• AVE_IP_ADM_PA_LT_12
• AVE_SNF_DAYS_PA_LT_12

####Part A: equal to 12 months

• BENE_COUNT_PA_EQ_12
• AVE_PA_PAY_PA_EQ_12
• AVE_IP_PAY_PA_EQ_12
• AVE_SNF_PAY_PA_EQ_12
• AVE_OTH_PAY_PA_EQ_12
• AVE_IP_ADM_PA_EQ_12
• AVE_SNF_DAYS_PA_EQ_12

####Part B: less than 12 months

• BENE_COUNT_PB_LT_12
• AVE_MO_EN_PB_LT_12
• AVE_PB_PAY_PB_LT_12
• AVE_CA_PAY_PB_LT_12
• AVE_OP_PAY_PB_LT_12
• AVE_OTH_PAY_PB_LT_12
• AVE_CA_VST_PB_LT_12
• AVE_OP_VST_PB_LT_12

####Part B: equal to 12 months

• BENE_COUNT_PB_EQ_12
• AVE_PB_PAY_PB_EQ_12
• AVE_CA_PAY_PB_EQ_12
• AVE_OP_PAY_PB_EQ_12
• AVE_OTH_PAY_PB_EQ_12
• AVE_CA_VST_PB_EQ_12
• AVE_OP_VST_PB_EQ_12

####Part C:

• BENE_COUNT_PC_LT_12
• AVE_MO_EN_PC_LT_12
• BENE_COUNT_PC_EQ_12

####Part D:

• BENE_COUNT_PD_LT_12
• AVE_MO_EN_PD_LT_12
• AVE_PDE_CST_PD_LT_12
• AVE_PDE_PD_LT_12
• BENE_COUNT_PD_EQ_12
• AVE_PDE_CST_PD_EQ_12
• AVE_PDE_PD_EQ_12

Here is a useful read about how the data can be used.

####About Medicare Part A, B, C, D Different parts of Medicare cover different services. Original Medicare, made up of Parts A and B, is administered by the federal government:

1. Part A (Hospital Insurance) covers most medically necessary hospital, skilled nursing facility, home health and hospice care. No monthly premium if you’ve worked more than 10 years.
2. Part B (Medical Insurance) covers most medically necessary doctors’ services, preventive care, durable medical equipment, hospital outpatient services, laboratory tests, x-rays, mental health care, and some home health and ambulance services. You pay a monthly premium for this coverage.
3. Part C is the part of Medicare policy that allows private health insurance companies to provide Medicare benefits.
4. Part D provides prescription drug coverage and is provided only through private insurance.

Notes: payments distribution is very skewed (90% of cost comes from 1% of patients). Consider taking logs.

---

NPDB Malpractice Claims

---

Medicare Provider Utilization and Payment Data

---

SPARCS

Hospital inpatient discharges 2012-2013: datasets. When you open the dataset on the website you can access the definitions of each variables directly.

More information about the Clinical Classification Software (CCS) may be found here.

All Patient Refined Diagnosis Related Groups and Major Diagnostics Category (APR-DRG and APR-MDC) information are here.

The Emergency Department Indicator is set based on the submitted revenue codes. If the record contained an emergency department revenue code of 045X, the indicator is set to Y, otherwise it will be N.

SPARCS 2012 (Hospital_Inpatient_Discharges__SPARCS_De-Identified___2012.csv): 2,529,422 rows and 38 columns 963.3 Mo.

SPARCS 2013 (Hospital_Inpatient_Discharges__SPARCS_De-Identified___2013.csv): 2,418,874 rows and 38 columns 913.5 Mo.

Columns of Discrete Variables

• Health_Service_Area
• Hospital_County
• Operating_Certificate_Number
• Facility_Id
• Facility_Name
• Age_Group
• Zip_Code_3_digits
• Gender
• Race
• Ethnicity
• Admit_Day_of_Week
• Type_of_Admission
• Patient_Disposition
• Discharge_Year
• Discharge_Day_of_Week
• CCS_Diagnosis_Code
• CCS_Diagnosis_Description
• CCS_Procedure_Code
• CCS_Procedure_Description
• APR_DRG_Code
• APR_DRG_Description
• APR_MDC_Code
• APR_MDC_Description
• APR_Severity_of_Illness_Code
• APR_Severity_of_Illness_Description
• APR_Risk_of_Mortality
• APR_Medical_Surgical_Description
• Source_of_Payment_1
• Source_of_Payment_2
• Source_of_Payment_3
• Attending_Provider_License_Number
• Operating_Provider_License_Number
• Other_Provider_License_Number
• Abortion_Edit_Indicator
• Emergency_Department_Indicator

CCS= Clinical Classification.

APR_DRG= All Patient Refined Diagnosis Related Groups.

APR_MDC=All Patient Refined Major Diagnostics Category is set based on the submitted revenue codes. If the record contained an Emergency Department revenue code of 045X, the indicator is set to "Y", otherwise it will be "N".

Columns of Continuous Variables

• Length_of_Stay
• Birth_Weight
• Total_Charges