Skip to content

QWK.NET ‐ .NDX Files ‐ A Primer

Jump to bottom
Agent 57951 edited this page Jan 19, 2026 · 2 revisions

Index Files (.NDX) - A Primer

QWK index files (.NDX) provide random access to messages within MESSAGES.DAT without requiring sequential scanning. This page explains what they are, how they work, and common issues encountered in real-world packets.

What NDX Files Are Used For

Index files enable efficient message access by mapping message numbers to their locations in MESSAGES.DAT. Each conference has its own index file named by conference number:

  • Conference 0 → 0.NDX
  • Conference 1 → 1.NDX
  • Conference 123 → 123.NDX

Purpose:

  • Random access - Jump directly to a specific message without reading all preceding messages
  • Performance - Faster message lookup than sequential scanning
  • Conference organisation - Each conference maintains its own message index

When Index Files Are Optional:

  • QWK.NET can read packets without index files by scanning MESSAGES.DAT sequentially
  • Missing index files are not an error - the library falls back to sequential enumeration
  • Some BBS implementations did not generate index files, especially for small packets

High-Level Structure

Each .NDX file is a simple binary format:

  • File format: Binary, no header or metadata
  • Entry size: 4 bytes per entry
  • Entry format: Microsoft Binary Format (MSBIN) floating-point number
  • Entry count: One entry per message in the conference (may be zero for empty conferences)

File Size:

  • Must be a multiple of 4 bytes (each entry is exactly 4 bytes)
  • Empty conferences produce 0-byte files
  • File size = (number of messages) × 4 bytes

Example:

  • Conference with 10 messages → 0.NDX is 40 bytes (10 × 4)
  • Conference with 0 messages → 0.NDX is 0 bytes (empty file)

MSBIN Offsets: Conceptual Overview

MSBIN (Microsoft Binary Format) is a floating-point format used in Microsoft BASIC and QuickBASIC. In QWK index files, MSBIN floats encode record offsets into MESSAGES.DAT.

What Is a Record Offset?

QWK messages are stored in 128-byte records:

  • Record 0: First message header (after MESSAGES.DAT header)
  • Record 1: Second message header
  • Record N: (N+1)th message header

The MSBIN float encodes the byte offset divided by 128:

  • MSBIN value = (byte offset) / 128.0
  • Record offset = floor(MSBIN value)

Example:

  • Message at byte offset 1280 → MSBIN encodes 10.0 → Record offset 10
  • Message at byte offset 256 → MSBIN encodes 2.0 → Record offset 2

Why MSBIN Format?

MSBIN was chosen because:

  • Historical compatibility - QWK format originated in the DOS/BASIC era
  • Compact - 4 bytes per entry is efficient for storage
  • Precision - Sufficient for file offsets up to several megabytes

MSBIN Characteristics

  • 4-byte format - Exactly 4 bytes per entry
  • Zero representation - Four zero bytes (0x00 0x00 0x00 0x00) represents zero
  • Big-endian mantissa - Mantissa bytes are stored big-endian
  • Biased exponent - Exponent is biased by 129 (0x81)

QWK.NET preserves the raw MSBIN bytes in each IndexEntry for round-trip fidelity, ensuring that QWK → REP → QWK cycles maintain exact byte representation.

Common Real-World Problems

File Size Not Multiple of 4

Problem: Index file size is not divisible by 4 bytes.

Causes:

  • Truncated file transfers
  • Corrupted archives
  • Software bugs in packet generation
  • Manual file editing errors

Impact:

  • Cannot parse complete entries
  • Last entry may be incomplete or missing

QWK.NET Handling:

  • Strict: Throws exception immediately
  • Lenient/Salvage: Logs warning, parses complete entries only, skips incomplete trailing bytes

Negative Record Offsets

Problem: MSBIN float decodes to a negative number.

Causes:

  • Corrupted MSBIN bytes
  • Bit flips from storage media
  • Software bugs

Impact:

  • Invalid offset cannot point to a real message location

QWK.NET Handling:

  • All modes: Logs warning, skips the invalid entry
  • Strict: May throw if validation is strict enough
  • Sequential message numbering maintained even when entries are skipped

Offsets Beyond File Size

Problem: Record offset points beyond the end of MESSAGES.DAT.

Causes:

  • Index file from a different packet version
  • Messages deleted but index not updated
  • File truncation after index generation
  • Software bugs

Impact:

  • Attempting to read message would fail or read invalid data

QWK.NET Handling:

  • Strict: Throws exception when MESSAGES.DAT size is known
  • Lenient/Salvage: Logs warning, skips invalid entry
  • Validation requires MESSAGES.DAT file size to be provided

Missing Index Files

Problem: Conference has messages but no corresponding .NDX file.

Causes:

  • BBS software that doesn't generate indexes
  • Archive extraction errors
  • Manual packet creation

Impact:

  • No performance impact - library falls back to sequential scanning
  • Random access not available for that conference

QWK.NET Handling:

  • All modes: Not an error - sequential enumeration used instead
  • No warnings logged (missing indexes are acceptable)

Empty Index Files

Problem: Index file exists but is 0 bytes (no entries).

Causes:

  • Conference has no messages
  • File created but never populated

Impact:

  • No messages in conference (expected behaviour)

QWK.NET Handling:

  • All modes: Valid - returns empty index file
  • No warnings logged (empty conferences are normal)

Incorrect Entry Count

Problem: Index file has more or fewer entries than actual messages.

Causes:

  • Messages added/deleted without updating index
  • Index generation bugs
  • Partial index updates

Impact:

  • Some messages may not be accessible via index
  • Extra entries may point to invalid locations

QWK.NET Handling:

  • All modes: Index entries validated against MESSAGES.DAT when size is known
  • Invalid entries skipped with warnings
  • Sequential enumeration still works correctly

Validation Modes and NDX Issues

Strict Mode

Behaviour:

  • Throws QwkFormatException on first structural error
  • File size not multiple of 4 → Exception
  • Offset beyond file size → Exception (when MESSAGES.DAT size known)
  • MSBIN parse failure → Exception

Use When:

  • Packet correctness is critical
  • Invalid indexes indicate serious problems
  • You need guaranteed index integrity

Lenient Mode (Default)

Behaviour:

  • Logs warnings for all issues
  • Continues parsing, skipping invalid entries
  • Returns partial index with valid entries only
  • Falls back to sequential enumeration if index unusable

Use When:

  • Processing packets with minor index issues
  • Partial index data is acceptable
  • Recommended for most scenarios

Salvage Mode

Behaviour:

  • Similar to Lenient but more aggressive recovery
  • Attempts to parse partial entries when possible
  • May recover some entries from corrupted files
  • Maximum data extraction priority

Use When:

  • Processing damaged or corrupted packets
  • Maximum recovery is more important than correctness
  • Forensic analysis of problematic packets

When to Expect Warnings vs Errors

Expected Warnings (Normal)

In Lenient mode, these warnings are normal and don't indicate serious problems:

  • Missing index files - Not logged (acceptable)
  • Empty index files - Not logged (normal for empty conferences)
  • Index file size not multiple of 4 - Warning logged, partial parsing continues

Problematic Warnings

These warnings suggest actual issues that may need attention:

  • Negative record offsets - Indicates corruption or software bugs
  • Offsets beyond file size - Index may be from different packet version
  • MSBIN parse failures - Corrupted binary data

Errors (Strict Mode Only)

In Strict mode, these become exceptions:

  • File size not multiple of 4 - Structural error, cannot parse
  • Offset beyond file size - Invalid index entry, cannot trust index
  • MSBIN parse failure - Corrupted data, cannot decode entry

No Warnings (Normal Behaviour)

These situations are expected and produce no warnings:

  • Missing index files - Library uses sequential enumeration
  • Empty index files - Normal for conferences with no messages
  • Valid index files - No issues, no warnings

Best Practices

Reading Packets

  • Use Lenient mode - Handles index issues gracefully
  • Check validation report - Review warnings for index problems
  • Trust sequential enumeration - Works even with invalid indexes

Writing Packets

  • Generate indexes - Create .NDX files for all conferences with messages
  • Validate offsets - Ensure all offsets point to valid message locations
  • Preserve MSBIN bytes - Use exact MSBIN representation for round-trip fidelity

Handling Index Issues

  • Missing indexes - Not a problem, sequential enumeration works
  • Corrupted indexes - Use Salvage mode for recovery attempts
  • Invalid offsets - Check if index is from correct packet version

Further Reading

Clone this wiki locally