How to check 2-col pdfs? (Code for checking center of pdf for white space or text)

[DevanshuBrahmbhatt]

def is_two_column_layout(page):
    # Crop the middle 5% of the page and check for text
    center_height = page.height *0.5
    center_width = page.width*0.5
    center_crop = page.crop((center_height, center_width,  page.width-center_width, page.height-center_height))
    extracted_text_1 = center_crop.extract_text()
    print(extracted_text_1)
    # middle_crop = page.crop((0.5 * page.width, 0.5 * page.height, 0.5 * page.width, 0.5 * page.height))
    # extracted_text = middle_crop.extract_text()
    return not extracted_text_1.strip()

[samkit-jain]

Hi @DevanshuBrahmbhatt Appreciate your interest in the library. Are you facing any issue with the code you've provided? Request you to please provide more details on the issue you are facing or what you need help with. Attaching the PDF will be helpful too.

Assuming you want to know how many columns a PDF has, you can try with the following code:

import math

import pdfplumber

pdf = pdfplumber.open("tests/pdfs/federal-register-2020-17221.pdf")  # https://github.com/jsvine/pdfplumber/blob/stable/tests/pdfs/federal-register-2020-17221.pdf
page = pdf.pages[0]

# Crop the top and bottom 5% of the page.
page = page.crop((0.0, 0.05 * float(page.height), float(page.width), 0.95 * float(page.height)))

# Assuming a page is a table having "width" columns,
# this list will tell whether a column has text at a particular column or not.
# If the list becomes like
#   T F F F F F T T F F F T
# Then we have a 2-column PDF as we have 3 vertical lines that don't cut
# through any page.
is_pos_blank = [True] * int(page.width)
for word in page.extract_words():
    for pos in range(int(word["x0"]), math.ceil(word["x1"])):
        is_pos_blank[pos] = False

empty_columns = []

match = False
for pos, is_blank in enumerate(is_pos_blank):
    if is_blank and not match:
        empty_columns.append(pos)
        match = True
    elif not is_blank:
        match = False

print(len(empty_columns) - 1)  # Number of columns in the PDF.

# For debugging.
im = page.to_image(resolution=200)
im.draw_rects(page.extract_words(), stroke_width=2)
im.draw_vlines(empty_columns, stroke_width=5)
im.save("image.png", format="PNG")

Feel free to optimise the code based on your needs.

[DevanshuBrahmbhatt]

Hi @samkit-jain
Thank you so much for the incredible support and guidance from your code, I got the idea to parse pdf line_by_line find columns for each line, and extract the words. I am trying to write that code, I am sharing it with you here the draft version, please let me know your view. That would be a great help to me, as I have been stuck on this problem for a very long time.

`import math
import pdfplumber
pdf = pdfplumber.open("pdfs/test5.pdf")
page = pdf.pages[0]
page = page.crop((0.0, 0.05 * float(page.height), float(page.width), 0.95 * float(page.height)))

text_data=[]
line_columns = []
is_pos_blank = [True] * int(page.width)

current_line_y = 0
line_column_count = 0

for word in page.extract_words():
word_y = int(word["top"])
if current_line_y is None or abs(word_y - current_line_y) < 10:
for pos in range(int(word["x0"]), math.ceil(word["x1"])):
is_pos_blank[pos] = False
else:
if current_line_y !=0:
empty_columns = []
match = False
consecutive_count = 0
for pos, is_blank in enumerate(is_pos_blank):
if is_blank and not match:
consecutive_count += 1
elif not is_blank:
match = False
consecutive_count = 0
if consecutive_count==3 and not match:
empty_columns.append(pos)
match = True
print(len(empty_columns) - 1)
im = page.to_image(resolution=200)
print(empty_columns)
im.draw_rects(page.extract_words(), stroke_width=2)
im.draw_vlines(empty_columns, stroke_width=5)
im.save(f"{word}.png", format="PNG")
is_pos_blank = [True] * int(page.width)
print(len(empty_columns) - 1)
line_columns.append(len(empty_columns) - 1)
current_line_y = word_y
for pos in range(int(word["x0"]), math.ceil(word["x1"])):
is_pos_blank[pos] = False

for i, column_count in enumerate(line_columns, start=1):
print(f"Line {i}: Number of columns = {column_count}")

prev_index = 0
for pos in empty_columns:
if prev_index != pos:
print(page.crop((prev_index, 0, pos, page.height), relative=True).extract_text())
prev_index = pos
print(page.crop((prev_index, 0, page.width, page.height), relative=True).extract_text())`

[samkit-jain]

Okay, glad you have found a way out. Not able to make much sense of the code as it is rendered as text and not a code block. For multiple line code, please use triple backticks ``` and not single backtick `.

[DevanshuBrahmbhatt]

I am sorry, @samkit-jain, For your previous solution .extract_words(use_text_flow=True) It seems working fine for the PDFs, Can you please share in which scenarios this approach could go wrong?

[DevanshuBrahmbhatt]

Line-By-Line Parser: This Code is able to find accurate lines for each page but does not give the correct No.Columns for each line, Since Each line contains many white space among the words. I maintain white_space count if there are more than 3 consecutive white spaces I am considering a gap for multi-columns.

import math
import pdfplumber
pdf = pdfplumber.open("pdfs/test5.pdf") 
page = pdf.pages[0]
page = page.crop((0.0, 0.05 * float(page.height), float(page.width), 0.95 * float(page.height)))

text_data=[]
line_columns = []
is_pos_blank = [True] * int(page.width)

current_line_y = 0
line_column_count = 0

for word in page.extract_words():
        word_y = int(word["top"])
        #marking blank space for current line
        if current_line_y is None or abs(word_y - current_line_y) < 10:
            for pos in range(int(word["x0"]), math.ceil(word["x1"])):
                is_pos_blank[pos] = False
        else:
            if current_line_y !=0:
                # Empty Column Count for each line
                empty_columns = []
                match = False
                consecutive_count = 0
                for pos, is_blank in enumerate(is_pos_blank):
                    if is_blank and not match:
                        consecutive_count += 1
                    elif not is_blank:
                        match = False
                        consecutive_count = 0
                    if consecutive_count==3 and not match:   # Maintaining the consecutive_count for white_space in single line
                        empty_columns.append(pos)
                        match = True
                print(len(empty_columns) - 1)
                
                # Image Generation for image Debugging
                im = page.to_image(resolution=200)
                print(empty_columns)
                im.draw_rects(page.extract_words(), stroke_width=2)
                im.draw_vlines(empty_columns, stroke_width=5)
                im.save(f"{word}.png", format="PNG")
                is_pos_blank = [True] * int(page.width)

                print(len(empty_columns) - 1)
                line_columns.append(len(empty_columns) - 1)

            # marking blank space for first word of new line
            current_line_y = word_y
            for pos in range(int(word["x0"]), math.ceil(word["x1"])):
                is_pos_blank[pos] = False
            
for i, column_count in enumerate(line_columns, start=1):
    print(f"Line {i}: Number of columns = {column_count}")
    
prev_index = 0
for pos in empty_columns:
    if prev_index != pos:
        print(page.crop((prev_index, 0, pos, page.height), relative=True).extract_text())
    prev_index = pos
print(page.crop((prev_index, 0, page.width, page.height), relative=True).extract_text())

[samkit-jain]

For your previous solution .extract_words(use_text_flow=True) It seems working fine for the PDFs, Can you please share in which scenarios this approach could go wrong?

It all depends on how the PDF was produced. Sometimes, even if there are visible columns, the PDF reads the text Left-Right and Top-Bottom, not a mix of both. If the approach is working fine for the PDFs you are dealing with, then I guess it should be considered as the most hassle free solution.

I maintain white_space count if there are more than 3 consecutive white spaces I am considering a gap for multi-columns.

I guess this is the reason. My recommendation would be that instead of considering a new column gap when more than 3 spaces, best is to find all the X-axis positions where no character is present throughout (This assumes that in a n-col PDF, there'd be no intersecting character anywhere).