Document Layout Analysis on OCR results

[malv007]

Hi,

What do you think would be the best approach of reusing some of the document layout analysis logic available in the PdfPig library for OCR results? In particular, I am interested in being able to reuse reading order detectors and decoration text classifier. I can think of two options:

• Rebuild PDF: Identify the fonts used for each of the blocks using one of the ONNX models available in HF, and rebuild the PDF making sure the text blocks have similar dimensions
• Using abstraction/interfaces: the layout analysis logic I am interested in seems to only use TextBlock, TextLine and Word, if these classes and the doc layout methods used abstraction, the OCR logic could generate similar objects only having to do some bounding box recalculations
• New classes: Create new document layout logic for OCR results

What are your thoughts?

[malv007]

I have actually tried the approach of using abstraction, and it works very well. This is some of the code for converting the list of OCR TextBlock into instances of Word:

    private async Task<List<IReadOnlyList<LayoutAnalysis.Pdf.Word>>> DetectWords(Stream pdf, PdfDocument doc, double lineHeightScale = 0.2d, CancellationToken cancellationToken = default)
    {
      const string modelsFolderName = "data/models/v5";
      string detPath = Path.Combine(modelsFolderName, "ch_PP-OCRv5_mobile_det.onnx");
      string clsPath = Path.Combine(modelsFolderName, "ch_ppocr_mobile_v2.0_cls_infer.onnx");
      string recPath = Path.Combine(modelsFolderName, "latin_PP-OCRv5_rec_mobile_infer.onnx");
      string keysPath = Path.Combine(modelsFolderName, "ppocrv5_latin_dict.txt");

      var resultPages = new List<IReadOnlyList<LayoutAnalysis.Pdf.Word>>();

      var ocrEngin = GetOcrEngine(detPath, clsPath, recPath, keysPath);
      var pageNum = 0;
#pragma warning disable CA1416 // Validate platform compatibility
      await foreach (var image in PDFtoImage.Conversion.ToImagesAsync(pdf, true, null, new PDFtoImage.RenderOptions() { Dpi = 200, Grayscale = true, AntiAliasing = PDFtoImage.PdfAntiAliasing.All }, cancellationToken))
      {
        pageNum++;
        var page = doc.GetPage(pageNum);
        if (cancellationToken.IsCancellationRequested) break;

        RapidOcrNet.OcrResult ocrResult = ocrEngin.Detect(image, new RapidOcrNet.RapidOcrOptions()
        {
          Padding = 100, //The image is surrounded by a white frame to improve the recognition rate. When the text frame does not correctly frame all the text, increase this value.
          ImgResize = 0,
          BoxScoreThresh = 0.5f, // The greater the threshold for whether the text-testing box retains, the lower the recall rate. Range of Values:[0, 1]The default value is 0.5. Text box confidence threshold, when the text box does not correctly frame all the text, reduce this value
          BoxThresh = 0.1f, // The text part and background part of the image are partitioned thresholds. The larger the value, the smaller the text part. Range of Values:[0, 1]The default value is 0.3.
          UnClipRatio = 1.6f, // Control the size of the text detection box, the larger the value, the larger the detection box as a whole. This is related to the size of the image, and the larger the image, the larger the value. Range of Values:[1.6, 2.0]The default value is 1.6.
          DoAngle = false, // Enable (1) / Disable (0) text orientation detection. Text orientation detection is only required when the image is inverted (rotated 90 to 270 degrees).
          MostAngle = false //  Enable (1)/Disable (0) Angle Poll (the entire image is identified by the maximum possible text orientation), which also does not work when text orientation detection is disabled.
        });
        var words = new List<LayoutAnalysis.Pdf.Word>();
        int textSequence = 0;
        foreach (var block in ocrResult.TextBlocks)
        {
          // Figure out the bounding box, text block orientation, and convert SKPoint to PdfPoint
          var points = block.BoxPoints;
          var xDist = SKPoint.Distance(points[0], points[1]);
          var yDist = SKPoint.Distance(points[3], points[0]);
          var isHorizontal = xDist >= yDist;
          var tl = points[0];
          var tr = points[1];
          var br = points[2];
          var bl = points[3];
          if (!isHorizontal)
          {
            tl = points[3];
            tr = points[0];
            br = points[1];
            bl = points[2];
          }
          var topLeft = ToPdfPoint(tl, image, page);
          var topRight = ToPdfPoint(tr, image, page);
          var bottomRight = ToPdfPoint(br, image, page);
          var bottomLeft = ToPdfPoint(bl, image, page);

          // Scale the line height to better fit the text
          var diff = 0.0d;
          if (isHorizontal)
          {
            diff = (topLeft.Y - bottomLeft.Y) * lineHeightScale;
            topLeft = new PdfPoint(topLeft.X, topLeft.Y - diff);
            bottomLeft = new PdfPoint(bottomLeft.X, bottomLeft.Y + diff);
            diff = (topRight.Y - bottomRight.Y) * lineHeightScale;
            topRight = new PdfPoint(topRight.X, topRight.Y - diff);
            bottomRight = new PdfPoint(bottomRight.X, bottomRight.Y + diff);
          }
          else
          {
            diff = (bottomLeft.X - topLeft.X) * lineHeightScale;
            topLeft = new PdfPoint(topLeft.X + diff, topLeft.Y);
            bottomLeft = new PdfPoint(bottomLeft.X - diff, bottomLeft.Y);
            diff = (bottomRight.X - topRight.X) * lineHeightScale;
            topRight = new PdfPoint(topRight.X + diff, topRight.Y);
            bottomRight = new PdfPoint(bottomRight.X - diff, bottomRight.Y);
          }

          // Figure out the angle of the text block, even though we have already determined if it is horizontal or vertical text
          var angle = AngleDegrees(tl, tr);
          var orientation = TextOrientation.Horizontal;
          if (Math.Min(xDist, yDist) / Math.Max(xDist, yDist) < 0.8)
          {
            if (angle >= 340 || angle <= 20)
              orientation = TextOrientation.Horizontal;
            else if (angle >= 160 && angle <= 200)
              orientation = TextOrientation.Rotate180;
            else if (angle >= 70 && angle <= 110)
              orientation = TextOrientation.Rotate90;
            else if (angle >= 250 && angle <= 290)
              orientation = TextOrientation.Rotate270;
            else
              orientation = TextOrientation.Other;
          }

          // Split text blocks into words, assuming a monospace font
          var index = 0;
          var blockLength = block.Chars.Length;
          foreach (var wordText in block.GetText().Split(" "))
          {
            if (!string.IsNullOrWhiteSpace(wordText))
            {
              textSequence++;
              var endIndex = index + wordText.Length;
              var word = new LayoutAnalysis.Pdf.Word(wordText, orientation,
                new PdfRectangle(
                SegmentPoint(topLeft, topRight, index, blockLength),
                SegmentPoint(topLeft, topRight, endIndex, blockLength),
                SegmentPoint(bottomLeft, bottomRight, index, blockLength),
                SegmentPoint(bottomLeft, bottomRight, endIndex, blockLength)
                ), textSequence, block.CharScores.SubList(index, endIndex).Average());
              words.Add(word);
            }
            index = index + wordText.Length + 1;
          }
        }

        resultPages.Add(words);

And below you can see the output of a scanned PDF document that was OCRed using RapidOcrNet. In red you have the identified text blocks by the docstrum logic, and in green/blue you have the fake words.

But for this to work the classes in PdfPig: TextBlock, TextLine and Word (perhaps even Letter) would need to be virtual/abstract or use interfaces. What do you think about making these changes to PdfPig?

[malv007]

Never mind, I actually constructed instances of the "Letter" class with dummy values for font, font size, width, color, etc., in order to initialize the "Word" instances, and it works fine. I am able to reuse all the document layout analysis logic that is included in the library PdfPig.

[BobLd]

@malv007 I did not have time to reply before but everything you said makes a lot of sense. I'm not sure I'll have time to work on that but the topic is gaining traction UglyToad/PdfPig#1285