Postprocessing Negative Controls

Erica Voss edited this page Aug 26, 2018 · 2 revisions

Post Processing - Negative Controls

What is a Negative Control

Exposure-outcome pairs for which there is no known causal relationship [1]. They are used to identify, estimate, and resolve residual confounding [2]. There are two types of negative controls: (1) negative control exposures are exposures known to have no association with the outcome of interest and (2) negative control outcomes are outcomes known to have no association with the exposures of interest.

Motivation for Negative Controls

Often epidemiologic studies on observational data declare results statistically significant when the “p < 0.05” saying that there is only a 5% probability that the observed effect is by chance alone. However, observational data is vulnerable to systematic error such as bias and confounding [2]. In 2013, the OHDSI community published work showing that empirical calibration “reduced spurious results to the desired 5% level” [2]. The empirical calibration framework is based on modeling the observed null distribution of negative controls. “Using the empirical distributions of negative controls, we can compute a better estimate of the probability that a value at least as extreme as a certain effect estimate could have been observed under the null hypothesis” [2]. The OHDSI community recommends that observational study always include negative controls to derive the empirical null distribution and use these to calibrate p-values.

Prep for Building Negative Controls

Typically, negative controls are processed for a user given a set of concepts of interest. However, some data processing can occur prior to a given user’s request for negative controls. Find detailed documentation here on the preparatory processing.

Negative Controls Prep

Where to Obtain Negative Controls

OHDSI has built a tool called ATLAS where negative controls can be processed calling evidence from the CommonEvidenceModel. See detailed information here on how to perform this process.

Getting Negative Controls from ATLAS

Best Practices of Obtaining Negative Controls
  1. For your given study, generate one concept set with all drugs or conditions that you are trying to study. We recommend when choosing negative controls for drugs, using the classes of target and comparator drug.
  2. Via ATLAS generate and view evidence based on that one concept set.
  3. Export concept set to CSV.
  4. Reviewing in the “Sort Order” (smallest to largest) review concepts till you get between 50 and 60 concepts. 50 negative controls are recommended for calibration so shooting for 60 gives you a few additional concepts if it is determined later that they should not be included. If you are reviewing conditions for a set of drugs, open the drug labels to help in your review.
  5. Once you have a subset of believed negative controls, submit the list for clinical review.

Some notes to think about when generating a list of negative controls:

  • Negative controls do not preclude an expected association between exposure and outcome. For example, when studying celecoxib, a good negative control outcome would be ingrown nails. We do expect an association, because celecoxib tends to be given to older people, who are also more at risk of ingrown nails, but there is no causal association. One way to think about causation is whether an intervention would have an effect. If we were to intervene and switch all people on celecoxib to another medication, would we expect a difference in the number of ingrown nails? If the answer is no, then it is likely a good negative control.
  • A causal relationship can be positive or negative. If a drug causes an outcome to happen, then it cannot be a negative control. Also, if a drug is known to prevent an outcome, it cannot be used as a negative control. For a negative control we expect the true relative risk to be exactly 1, not bigger but also not smaller.
  • Comparative effect study: we define negative controls as target-comparator-outcome triplets where neither target nor comparator is believed to cause the outcome.
Description of Process

If you would like to describe the process of selecting negative controls from the CommonEvidenceModel in published literature, please feel free to use this verbiage:

Negative controls are concepts known to not be associated with the target or comparator cohorts, such that we can assume the true relative risk between the two cohorts is 1. Negative controls are selected using a similar process to that outlined by Voss et al [3]. Person counts of all potential drug-condition pairs are reviewed in a diverse set observational data; this person count data helps determine which pairs are even probable for use in calibration as well as provide a priority for which pairs should be reviewed first. Given the list of potential drug-condition pairs, the concepts in the pairs must meet the following requirements to be considered as negative controls: (1) have no published literature association between the drug-condition pair of interest, either exactly as the evidence was mapped, a lower lever concept contained evidence, a direct parent concept contained evidence, or an ancestor contained evidence, (2) not existing on the US product label in the “Adverse Drug Reactions” or “Postmarketing” section, either exactly as the evidence was mapped, a lower lever concept contained evidence, a direct parent concept contained evidence, or an ancestor contained evidence, (3) not considered a FAERS signal [4], either exactly as the evidence was mapped, a lower lever concept contained evidence, a direct parent concept contained evidence, or an ancestor contained evidence, (4) have no indication or contraindication listed in the OMOP Vocabulary for the pair, (5) are not considered a broad concepts, (6) are not considered a drug induced concept, or (7) not considered a pregnancy related concept. The remaining concepts are “optimized”, meaning parent concepts remove children as defined by the OMOP Vocabulary (e.g. if both “Non-Hodgkin’s Lymphoma” and “B-Cell Lymphoma” we selected, child concept “B-Cell Lymphoma would be removed for its parent “Non-Hodgkin’s Lymphoma”). Once potential negative control candidates were selected, manual clinical review to exclude any pairs that may still be in a causal relationship or similar to the study outcome was be performed to select the top concepts by patient exposure.


  1. Lipsitch, M., E. Tchetgen Tchetgen, and T. Cohen, Negative controls: a tool for detecting confounding and bias in observational studies. Epidemiology, 2010. 21(3): p. 383-8.
  2. Schuemie, M.J., et al., Interpreting observational studies: why empirical calibration is needed to correct p-values. Stat Med, 2014. 33(2): p. 209-18.
  3. Voss, E.A., et al., Accuracy of an automated knowledge base for identifying drug adverse reactions. J Biomed Inform, 2017. 66: p. 72-81.
  4. Evans, S.J., P.C. Waller, and S. Davis, Use of proportional reporting ratios (PRRs) for signal generation from spontaneous adverse drug reaction reports. Pharmacoepidemiol Drug Saf, 2001. 10(6): p. 483-6.
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.