December 2024 Progress Prize Submission: Page Instance Segmentation

An approach for segmenting the scrolls using 3D voxel-based instance segmentation networks.

The region of scroll 1 with the coordinates 09936_03536_04048 (this mask and many more examples available at https://dl.ash2txt.org/community-uploads/tim/09936_region_possible_masks/)

A mostly coherent region of scroll 1 consisting of 16 cube inferences (but manually put together - stitching the inferences together automatically is not done)

This submission contains my pretraining, training, inference, and links to the training data + two best trained weights uploaded to the community uploads section of the sever.

3D Voxel Based Instance Segmentation Networks

This approach segments 256^3 voxel cubes of the scroll scans into distinct papyrus layers for the purposes of stitching together larger areas automatically.

How to use:

• Set up a jupyter kernel with the required dependencies
  • pytorch
  • monai
  • pytorch-lightning
  • pynrrd
  • "pip install -e ." in the root directory of this repo
• Choose either the one-to-many or the one-to-one notebook and run the code
• If you only want inference using my trained weights, just skip to the inference sections of the relevant notebooks and use the linked weights

The pretraining reference volumes, trained weights, and training set is located at https://dl.ash2txt.org/community-uploads/tim/dec_2024_submission_data/

Future work:

These networks work well enough to segment as shown above, but are also still very much a prototype and proof of concept that this approach has a lot of potential:

• Better network architectures
• Better loss functions that disentangle the segmentation signal from the layer-assignment signal
• Better loss functions that inherently understand the shape of the page better (dice loss penalizes equally for an incorrect voxel regardless of distance, but we know pages should be a flat shape)
• Train for longer
• Better pretraining with more variety of page shapes
• Smart inference and oversampling
• Automatic patch stitching to compile larger regions

This approach has been developed in collaboration with John Skinner and Lachlan Parker