[ELF] Align the end of PT_GNU_RELRO associated PT_LOAD to a common-pa…

…ge-size boundary

Close #57618: currently we align the end of PT_GNU_RELRO to a common-page-size
boundary, but do not align the end of the associated PT_LOAD. This is benign
when runtime_page_size >= common-page-size.

However, when runtime_page_size < common-page-size, it is possible that
`alignUp(end(PT_LOAD), page_size) < alignDown(end(PT_GNU_RELRO), page_size)`.
In this case, rtld's mprotect call for PT_GNU_RELRO will apply to unmapped
regions and lead to an error, e.g.

```
error while loading shared libraries: cannot apply additional memory protection after relocation: Cannot allocate memory
```

To fix the issue, add a padding section .relro_padding like mold, which
is contained in the PT_GNU_RELRO segment and the associated PT_LOAD
segment. The section also prevents strip from corrupting PT_LOAD program
headers.

.relro_padding has the largest `sortRank` among RELRO sections.
Therefore, it is naturally placed at the end of `PT_GNU_RELRO` segment
in the absence of `PHDRS`/`SECTIONS` commands.

In the presence of `SECTIONS` commands, we place .relro_padding
immediately before a symbol assignment using DATA_SEGMENT_RELRO_END (see
also https://reviews.llvm.org/D124656), if present.
DATA_SEGMENT_RELRO_END is changed to align to max-page-size instead of common-page-size.

Some edge cases worth mentioning:

* ppc64-toc-addis-nop.s: when PHDRS is present, do not append .relro_padding
* avoid-empty-program-headers.s: when the only RELRO section is .tbss,
  it is not part of PT_LOAD segment, therefore we do not append .relro_padding.

---

Close #65002: GNU ld from 2.39 onwards aligns the end of PT_GNU_RELRO to a
max-page-size boundary (https://sourceware.org/PR28824) so that the last page is
protected even if runtime_page_size > common-page-size.

In my opinion, losing protection for the last page when the runtime page size is
larger than common-page-size is not really an issue. Double mapping a page of up
to max-common-page for the protection could cause undesired VM waste. Internally
we had users complaining about 2MiB max-page-size applying to shared objects.

Therefore, the end of .relro_padding is padded to a common-page-size
boundary. Users who are really anxious can set common-page-size to match
their runtime page size.

---

17 tests need updating as there are lots of change detectors.

lld/ELF/Driver.cpp

@@ -1586,8 +1586,8 @@ static void readConfigs(opt::InputArgList &args) {
 
   // Page alignment can be disabled by the -n (--nmagic) and -N (--omagic).
   // As PT_GNU_RELRO relies on Paging, do not create it when we have disabled
-  // it.
-  if (config->nmagic || config->omagic)
+  // it. Also disable RELRO for -r.
+  if (config->nmagic || config->omagic || config->relocatable)
     config->zRelro = false;
 
   std::tie(config->buildId, config->buildIdVector) = getBuildId(args);

lld/ELF/LinkerScript.cpp

@@ -887,6 +887,10 @@ void LinkerScript::diagnoseOrphanHandling() const {
   if (config->orphanHandling == OrphanHandlingPolicy::Place)
     return;
   for (const InputSectionBase *sec : orphanSections) {
+    // .relro_padding is inserted before DATA_SEGMENT_RELRO_END, if present,
+    // automatically. The section is not supposed to be specified by scripts.
+    if (sec == in.relroPadding.get())
+      continue;
     // Input SHT_REL[A] retained by --emit-relocs are ignored by
     // computeInputSections(). Don't warn/error.
     if (isa<InputSection>(sec) &&
@@ -1079,6 +1083,11 @@ void LinkerScript::assignOffsets(OutputSection *sec) {
     }
   }
 
+  // If .relro_padding is present, round up the end to a common-page-size
+  // boundary to protect the last page.
+  if (in.relroPadding && sec == in.relroPadding->getParent())
+    expandOutputSection(alignToPowerOf2(dot, config->commonPageSize) - dot);
+
   // Non-SHF_ALLOC sections do not affect the addresses of other OutputSections
   // as they are not part of the process image.
   if (!(sec->flags & SHF_ALLOC)) {
@@ -1160,6 +1169,7 @@ void LinkerScript::adjustOutputSections() {
   uint64_t flags = SHF_ALLOC;
 
   SmallVector<StringRef, 0> defPhdrs;
+  bool seenRelro = false;
   for (SectionCommand *&cmd : sectionCommands) {
     if (!isa<OutputDesc>(cmd))
       continue;
@@ -1196,9 +1206,17 @@ void LinkerScript::adjustOutputSections() {
     if (sec->sectionIndex != UINT32_MAX)
       maybePropagatePhdrs(*sec, defPhdrs);
 
+    // Discard .relro_padding if we have not seen one RELRO section. Note: when
+    // .tbss is the only RELRO section, there is no associated PT_LOAD segment
+    // (needsPtLoad), so we don't append .relro_padding in the case.
+    if (in.relroPadding && in.relroPadding->getParent() == sec && !seenRelro)
+      discardable = true;
     if (discardable) {
       sec->markDead();
       cmd = nullptr;
+    } else {
+      seenRelro |=
+          sec->relro && !(sec->type == SHT_NOBITS && (sec->flags & SHF_TLS));
     }
   }
 

lld/ELF/LinkerScript.h

@@ -105,6 +105,9 @@ struct SymbolAssignment : SectionCommand {
   bool provide = false;
   bool hidden = false;
 
+  // This assignment references DATA_SEGMENT_RELRO_END.
+  bool dataSegmentRelroEnd = false;
+
   unsigned symOrder;
 
   // Holds file name and line number for error reporting.
@@ -352,6 +355,8 @@ class LinkerScript final {
   SmallVector<PhdrsCommand, 0> phdrsCommands;
 
   bool hasSectionsCommand = false;
+  bool seenDataAlign = false;
+  bool seenRelroEnd = false;
   bool errorOnMissingSection = false;
 
   // List of section patterns specified with KEEP commands. They will

lld/ELF/ScriptParser.cpp

@@ -136,9 +136,6 @@ class ScriptParser final : ScriptLexer {
   // True if a script being read is in the --sysroot directory.
   bool isUnderSysroot = false;
 
-  bool seenDataAlign = false;
-  bool seenRelroEnd = false;
-
   // A set to detect an INCLUDE() cycle.
   StringSet<> seen;
 };
@@ -600,7 +597,7 @@ void ScriptParser::readSections() {
 
   // If DATA_SEGMENT_RELRO_END is absent, for sections after DATA_SEGMENT_ALIGN,
   // the relro fields should be cleared.
-  if (!seenRelroEnd)
+  if (!script->seenRelroEnd)
     for (SectionCommand *cmd : v)
       if (auto *osd = dyn_cast<OutputDesc>(cmd))
         osd->osec.relro = false;
@@ -916,7 +913,7 @@ OutputDesc *ScriptParser::readOutputSectionDescription(StringRef outSec) {
       script->createOutputSection(unquote(outSec), getCurrentLocation());
   OutputSection *osec = &cmd->osec;
   // Maybe relro. Will reset to false if DATA_SEGMENT_RELRO_END is absent.
-  osec->relro = seenDataAlign && !seenRelroEnd;
+  osec->relro = script->seenDataAlign && !script->seenRelroEnd;
 
   size_t symbolsReferenced = script->referencedSymbols.size();
 
@@ -1051,6 +1048,7 @@ SymbolAssignment *ScriptParser::readAssignment(StringRef tok) {
 
   size_t oldPos = pos;
   SymbolAssignment *cmd = nullptr;
+  bool savedSeenRelroEnd = script->seenRelroEnd;
   const StringRef op = peek();
   if (op.starts_with("=")) {
     // Support = followed by an expression without whitespace.
@@ -1071,6 +1069,7 @@ SymbolAssignment *ScriptParser::readAssignment(StringRef tok) {
   }
 
   if (cmd) {
+    cmd->dataSegmentRelroEnd = !savedSeenRelroEnd && script->seenRelroEnd;
     cmd->commandString =
         tok.str() + " " +
         llvm::join(tokens.begin() + oldPos, tokens.begin() + pos, " ");
@@ -1439,7 +1438,7 @@ Expr ScriptParser::readPrimary() {
     expect(",");
     readExpr();
     expect(")");
-    seenDataAlign = true;
+    script->seenDataAlign = true;
     return [=] {
       uint64_t align = std::max(uint64_t(1), e().getValue());
       return (script->getDot() + align - 1) & -align;
@@ -1460,9 +1459,8 @@ Expr ScriptParser::readPrimary() {
     expect(",");
     readExpr();
     expect(")");
-    seenRelroEnd = true;
-    Expr e = getPageSize();
-    return [=] { return alignToPowerOf2(script->getDot(), e().getValue()); };
+    script->seenRelroEnd = true;
+    return [=] { return alignToPowerOf2(script->getDot(), config->maxPageSize); };
   }
   if (tok == "DEFINED") {
     StringRef name = unquote(readParenLiteral());

lld/ELF/SyntheticSections.cpp

@@ -2688,6 +2688,10 @@ size_t IBTPltSection::getSize() const {
 
 bool IBTPltSection::isNeeded() const { return in.plt->getNumEntries() > 0; }
 
+RelroPaddingSection::RelroPaddingSection()
+    : SyntheticSection(SHF_ALLOC | SHF_WRITE, SHT_NOBITS, 1, ".relro_padding") {
+}
+
 // The string hash function for .gdb_index.
 static uint32_t computeGdbHash(StringRef s) {
   uint32_t h = 0;
@@ -3839,6 +3843,7 @@ void InStruct::reset() {
   got.reset();
   gotPlt.reset();
   igotPlt.reset();
+  relroPadding.reset();
   armCmseSGSection.reset();
   ppc64LongBranchTarget.reset();
   mipsAbiFlags.reset();

lld/ELF/SyntheticSections.h

@@ -778,6 +778,16 @@ class IBTPltSection : public SyntheticSection {
   size_t getSize() const override;
 };
 
+// Used to align the end of the PT_GNU_RELRO segment and the associated PT_LOAD
+// segment to a common-page-size boundary. This padding section ensures that all
+// pages in the PT_LOAD segment is covered by at least one section.
+class RelroPaddingSection final : public SyntheticSection {
+public:
+  RelroPaddingSection();
+  size_t getSize() const override { return 0; }
+  void writeTo(uint8_t *buf) override {}
+};
+
 class GdbIndexSection final : public SyntheticSection {
 public:
   struct AddressEntry {
@@ -1333,6 +1343,7 @@ struct InStruct {
   std::unique_ptr<GotSection> got;
   std::unique_ptr<GotPltSection> gotPlt;
   std::unique_ptr<IgotPltSection> igotPlt;
+  std::unique_ptr<RelroPaddingSection> relroPadding;
   std::unique_ptr<SyntheticSection> armCmseSGSection;
   std::unique_ptr<PPC64LongBranchTargetSection> ppc64LongBranchTarget;
   std::unique_ptr<SyntheticSection> mipsAbiFlags;

lld/ELF/Writer.cpp

@@ -458,6 +458,13 @@ template <class ELFT> void elf::createSyntheticSections() {
   add(*in.gotPlt);
   in.igotPlt = std::make_unique<IgotPltSection>();
   add(*in.igotPlt);
+  // Add .relro_padding if DATA_SEGMENT_RELRO_END is used; otherwise, add the
+  // section in the absence of PHDRS/SECTIONS commands.
+  if (config->zRelro && ((script->phdrsCommands.empty() &&
+        !script->hasSectionsCommand) || script->seenRelroEnd)) {
+    in.relroPadding = std::make_unique<RelroPaddingSection>();
+    add(*in.relroPadding);
+  }
 
   if (config->emachine == EM_ARM) {
     in.armCmseSGSection = std::make_unique<ArmCmseSGSection>();
@@ -818,6 +825,9 @@ static bool isRelroSection(const OutputSection *sec) {
   if (sec == in.gotPlt->getParent())
     return config->zNow;
 
+  if (in.relroPadding && sec == in.relroPadding->getParent())
+    return true;
+
   // .dynamic section contains data for the dynamic linker, and
   // there's no need to write to it at runtime, so it's better to put
   // it into RELRO.
@@ -857,7 +867,7 @@ enum RankFlags {
   RF_BSS = 1 << 7,
 };
 
-static unsigned getSectionRank(const OutputSection &osec) {
+static unsigned getSectionRank(OutputSection &osec) {
   unsigned rank = osec.partition * RF_PARTITION;
 
   // We want to put section specified by -T option first, so we
@@ -920,7 +930,9 @@ static unsigned getSectionRank(const OutputSection &osec) {
     // TLS sections directly before the other RELRO sections.
     if (!(osec.flags & SHF_TLS))
       rank |= RF_NOT_TLS;
-    if (!isRelroSection(&osec))
+    if (isRelroSection(&osec))
+      osec.relro = true;
+    else
       rank |= RF_NOT_RELRO;
     // Place .ldata and .lbss after .bss. Making .bss closer to .text alleviates
     // relocation overflow pressure.
@@ -1140,6 +1152,18 @@ findOrphanPos(SmallVectorImpl<SectionCommand *>::iterator b,
               SmallVectorImpl<SectionCommand *>::iterator e) {
   OutputSection *sec = &cast<OutputDesc>(*e)->osec;
 
+  // As a special case, place .relro_padding before the SymbolAssignment using
+  // DATA_SEGMENT_RELRO_END, if present.
+  if (in.relroPadding && sec == in.relroPadding->getParent()) {
+    auto i = std::find_if(b, e, [=](SectionCommand *a) {
+      if (auto *assign = dyn_cast<SymbolAssignment>(a))
+        return assign->dataSegmentRelroEnd;
+      return false;
+    });
+    if (i != e)
+      return i;
+  }
+
   // Find the first element that has as close a rank as possible.
   auto i = std::max_element(b, e, [=](SectionCommand *a, SectionCommand *b) {
     return getRankProximity(sec, a) < getRankProximity(sec, b);
@@ -2334,6 +2358,7 @@ SmallVector<PhdrEntry *, 0> Writer<ELFT>::createPhdrs(Partition &part) {
       relroEnd = sec;
     }
   }
+  relRo->p_align = 1;
 
   for (OutputSection *sec : outputSections) {
     if (!needsPtLoad(sec))
@@ -2677,16 +2702,6 @@ template <class ELFT> void Writer<ELFT>::setPhdrs(Partition &part) {
       if (!p->hasLMA)
         p->p_paddr = first->getLMA();
     }
-
-    if (p->p_type == PT_GNU_RELRO) {
-      p->p_align = 1;
-      // musl/glibc ld.so rounds the size down, so we need to round up
-      // to protect the last page. This is a no-op on FreeBSD which always
-      // rounds up.
-      p->p_memsz =
-          alignToPowerOf2(p->p_offset + p->p_memsz, config->commonPageSize) -
-          p->p_offset;
-    }
   }
 }
 

lld/docs/ELF/linker_script.rst

@@ -172,3 +172,18 @@ description in the ``OVERWRITE_SECTIONS`` command while the insert command
 still applies (possibly after orphan section placement). It is recommended to
 leave the brace empty (i.e. ``section : {}``) for the insert command, because
 its description will be ignored anyway.
+
+Built-in functions
+~~~~~~~~~~~~~~~~~~
+
+``DATA_SEGMENT_RELRO_END(offset, exp)`` defines the end of the ``PT_GNU_RELRO``
+segment when ``-z relro`` (default) is in effect. Sections between
+``DATA_SEGMENT_ALIGN`` and ``DATA_SEGMENT_RELRO_END`` are considered RELRO.
+
+The typical use case is ``. = DATA_SEGMENT_RELRO_END(0, .);`` followed by
+writable but non-RELRO sections. LLD ignores ``offset`` and ``exp`` and aligns
+the current location to a max-page-size boundary, ensuring that the next
+``PT_LOAD`` segment will not overlap with the ``PT_GNU_RELRO`` segment.
+
+LLD will insert ``.relro_padding`` immediately before the symbol assignment
+using ``DATA_SEGMENT_RELRO_END``.

lld/docs/ReleaseNotes.rst

@@ -29,7 +29,8 @@ ELF Improvements
 * ``--fat-lto-objects`` option is added to support LLVM FatLTO.
   Without ``--fat-lto-objects``, LLD will link LLVM FatLTO objects using the
   relocatable object file. (`D146778 <https://reviews.llvm.org/D146778>`_)
-
+* common-page-size can now be larger than the system page-size.
+ (`#57618 <https://github.com/llvm/llvm-project/issues/57618>`_)
 
 Breaking changes
 ----------------

lld/test/ELF/arm-execute-only.s

@@ -13,7 +13,7 @@
 // CHECK:      LOAD           0x000000 0x00000000 0x00000000 0x0016d 0x0016d  R 0x10000
 // CHECK:      LOAD           0x000170 0x00010170 0x00010170 0x{{.*}} 0x{{.*}} R E 0x10000
 // CHECK:      LOAD           0x000174 0x00020174 0x00020174 0x{{.*}} 0x{{.*}}   E 0x10000
-// CHECK:      LOAD           0x000178 0x00030178 0x00030178 0x00038  0x00038  RW  0x10000
+// CHECK:      LOAD           0x000178 0x00030178 0x00030178 0x00038  0x00e88  RW  0x10000
 
 // CHECK: 01     .dynsym .gnu.hash .hash .dynstr
 // CHECK: 02     .text
@@ -22,7 +22,7 @@
 
 // DIFF:      LOAD           0x000000 0x00000000 0x00000000 0x0014d 0x0014d R   0x10000
 // DIFF:      LOAD           0x000150 0x00010150 0x00010150 0x0000c 0x0000c R E 0x10000
-// DIFF:      LOAD           0x00015c 0x0002015c 0x0002015c 0x00038 0x00038 RW  0x10000
+// DIFF:      LOAD           0x00015c 0x0002015c 0x0002015c 0x00038 0x00ea4 RW  0x10000
 
 // DIFF: 01     .dynsym .gnu.hash .hash .dynstr
 // DIFF: 02     .text .foo

lld/test/ELF/end-dso-defined.s

@@ -21,16 +21,16 @@
 # CHECK-NEXT: AddressAlignment:
 # CHECK-NEXT: EntrySize:
 # CHECK-NEXT: SectionData (
-# CHECK-NEXT:   0000: 08232000 00000000 08232000 00000000
+# CHECK-NEXT:   0000: 00302000 00000000 00302000 00000000
 # CHECK-NEXT: )
 
 # CHECK:      Symbol {
 # CHECK:        Name: _end
-# CHECK-NEXT:   Value: 0x202308
+# CHECK-NEXT:   Value: 0x203000
 
 # CHECK:      Symbol {
 # CHECK:        Name: end
-# CHECK-NEXT:   Value: 0x202308
+# CHECK-NEXT:   Value: 0x203000
 
 .global _start
 _start: