Repository navigation
Coalescent Thinking
Before considering species trees and gene trees, let us first examine the ancestry of genes within a single population. Suppose we sample two gene copies from a population and trace their ancestry backward through time. Eventually, both copies must originate from the same ancestral gene copy. The point at which these two lineages merge is called a coalescent event. Rather than modeling the entire population, coalescent theory focuses only on the ancestry of sampled lineages. This perspective was formalized in Kingman coalescent that provides a simple stochastic model describing how ancestral lineages merge as we move backward through time.
Figure 1. Illustration of the Kingman coalescent process. Time flows from bottom to top (backward in time). Red lineages represent gene copies sampled in the present-day population and merge through successive coalescent events as they are traced backward to a common ancestor. Modified from "Introduction to Population Genetics (Sinauer Associates, 2013)", after Figure 3.5.
An important feature of the coalescent process is that it is inherently random. Even under identical population conditions, different simulations can produce different genealogical histories. Consequently, a population does not have a single genealogy, but rather a distribution of possible genealogies. Two factors strongly influence the probability that lineages coalesce: population size (width of Figure 1) and time (or number of generations; height of Figure 1). In small populations, lineages are more likely to share recent common ancestors and therefore coalesce quickly, whereas in large populations lineages are less likely to encounter the same ancestor in any given generation. Likewise, the further we trace lineages into the past, the more opportunities there are for coalescent events to occur.
The Kingman coalescent relies on several simplifying assumptions. The classical model assumes a single randomly mating population of constant size with non-overlapping generations. It further assumes neutrality, meaning that all gene copies are equally likely to be inherited, and ignores processes such as migration, recombination, and population subdivision. Although real populations often violate one or more of these assumptions, the coalescent nevertheless provides a remarkably useful approximation for understanding patterns of genetic ancestry. Many modern extensions of coalescent theory relax these assumptions by incorporating changing population sizes, geographic structure, migration, recombination, and natural selection.
A key insight from the coalescent is that even within a single population, genealogical histories are stochastic. If we repeatedly sample the same number of individuals from the same population and trace their ancestry backward through time, we will generally obtain different genealogies. Rather than producing a single ancestral history, the coalescent generates a distribution of possible genealogies. Understanding this variation within a single population is the first step toward understanding why genealogies can become even more complex when multiple species are involved.
Important
To better understand how population size and time influence coalescence, we will perform a simple class activity using an interactive simulator available at https://phyleauxsim.github.io/coalescent/ (created by Jeremy Brown at Louisiana State University and supported by the Society of Systematic Biologists). First, play around with it. Then, select various combinations of population size and number of generations while fixing the sample size at 2. Then click Draw New Sampled History and observe whether the two sampled lineages coalesce before reaching the oldest generation shown in the simulation.
Enter your observations into the class spreadsheet. Record the parameter values and whether the two lineages coalesced. Enter a value of 1 if the lineages coalesced and 0 if they did not. Repeat this process 10 or more times using different parameter combinations. After the class has submitted all results, we will visualize the data using a spreadsheet plot. As you examine the results, consider how increasing population size affects the probability of coalescence, how increasing the number of generations changes this probability, and why large populations combined with short time intervals often cause lineages to remain uncoalesced.
A species tree represents the evolutionary history of species. One useful way to think about a species tree is as a collection of populations connected through time. Each branch corresponds to a population that exists for a particular period, while speciation events divide ancestral populations into descendant populations. Genes evolve within these populations, and their ancestry is shaped by the movement of genetic lineages through the branches of the species tree. However, genes within those species also have their own evolutionary histories, known as gene trees. Importantly, gene trees do not always match the species tree, and the coalescent process helps explain why these differences occur.
Figure 2. Illustration of the relationship between the species tree and populations used in the multispecies coalescent (MSC). Each branch of the species tree corresponds to a population through which gene lineages are traced backward in time. Figure adapted from the BEAST2 website.
In coalescent theory, we often think backward in time. Rather than asking how species split into descendant species, we instead ask where present-day gene copies originated. As genetic lineages are traced backward through ancestral populations, they eventually merge into common ancestors. This merging process is called coalescence.
For example, consider the following species tree:
Species tree: (((Human,Chimpanzee),Gorilla),Orangutan);
One might expect genes to follow the same evolutionary history as the species tree.
Gene tree 1: (((Human,Chimpanzee),Gorilla),Orangutan);
However, individual genes can sometimes produce different gene trees due to processes such as incomplete lineage sorting.
Gene tree 2: (((Human,Gorilla),Chimpanzee),Orangutan);
Gene tree 3: (((Chimpanzee,Gorilla),Human),Orangutan);
or
Gene tree 4: ((Chimpanzee,Gorilla),(Human,Orangutan));
and many other possibilities.
These differences occur because ancestral populations may simultaneously contain multiple genetic lineages, and the genealogical history of a particular gene is influenced by random ancestry patterns like random survival or loss of ancestral alleles or genetic drift.
When gene trees differ from the species tree due to these stochastic ancestral processes, the phenomenon is called Incomplete Lineage Sorting (ILS). ILS is especially common when speciation events occur rapidly in succession, leaving little time for genetic lineages to coalesce within ancestral populations before additional speciation events occur.
This has important implications for phylogenetic inference. Different genes may support different evolutionary histories, meaning that a single gene tree is often insufficient for accurately reconstructing species relationships. As a result, modern phylogenetics increasingly relies on genome-scale datasets and statistical models that jointly consider species trees, gene trees, and coalescent processes.
Importantly, discordance among gene trees is not caused only by ILS. Non-tree-like evolutionary processes can also produce conflicting genealogical histories. Examples include hybridization, introgression, and horizontal gene transfer, all of which involve genetic exchange between distinct evolutionary lineages. In such cases, different parts of the genome may trace back through different evolutionary pathways, making the overall history better represented by a phylogenetic network rather than a strictly branching tree.
Figure 3. Illustration of incomplete lineage sorting (ILS), a process in which gene lineages fail to coalesce within the most recent ancestral population and instead coalesce deeper in the species history. As a result, gene tree topologies may differ from the underlying species tree. Retrieved from [1].
[1] Benjamin Rosenzweig, Andrew Kern, and Matthew Hahn, Accurate Detection of Incomplete Lineage Sorting via Supervised Machine Learning, bioRxiv (2022). DOI: 10.1101/2022.11.09.515828