bug/partition.auto.partition raises ValueError: Invalid file (FileType.UNK) caused by semicolon including Content-Type without libmagic

[ftnext]

Describe the bug

unstructured.partition.auto.partition raises ValueError

>>> from unstructured.partition.auto import partition
>>> html_elements = partition(url="https://www.arxiv-vanity.com/papers/2305.14283/")
libmagic is unavailable but assists in filetype detection on file-like objects. Please consider installing libmagic for better results.
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/.../venv/lib/python3.11/site-packages/unstructured/partition/auto.py", line 490, in partition
    raise ValueError(f"{msg}. The {filetype} file type is not supported in partition.")
ValueError: Invalid file. The FileType.UNK file type is not supported in partition.

partition_html works (but partition does not)

>>> from unstructured.partition.html import partition_html
>>> html_elements = partition_html(url="https://www.arxiv-vanity.com/papers/2305.14283/")

To Reproduce
(See above snippet)

Expected behavior
No error (same as partition_html example)

Environment Info

OS version:  macOS-12.6.6-arm64-arm-64bit
Python version:  3.11.4
unstructured version:  0.11.2
unstructured-inference version:  0.7.15
pytesseract version:  0.3.10
Torch version:  2.1.1
Detectron2 is not installed
PaddleOCR is not installed
Libmagic version: file-5.41
magic file from /usr/share/file/magic
Traceback (most recent call last):
  ...
  File "/.../collect_env.py", line 234, in main
    libreoffice_version = get_libreoffice_version()
                          ^^^^^^^^^^^^^^^^^^^^^^^^^
  ...
FileNotFoundError: [Errno 2] No such file or directory: 'libreoffice'

Additional context
Workaround: Specify content_type

>>> html_elements = partition(url="https://www.arxiv-vanity.com/papers/2305.14283/", content_type="text/html")

Cause:
Content-Type of the response header is 'text/html; charset=utf-8'

unstructured/unstructured/partition/auto.py

Line 516 in 039ae17

content_type = content_type or response.headers.get("Content-Type")

detect_filetype(content_type='text/html; charset=utf-8') returns FileType.UNK

unstructured/unstructured/file_utils/filetype.py

Lines 238 to 242 in 039ae17

	# first check (content_type)
	if content_type:
	filetype = STR_TO_FILETYPE.get(content_type)
	if filetype:
	return filetype

Idea: Add semicolon parse logic

    content_type = content_type or response.headers.get("Content-Type")
+    if content_type:
+        content_type = content_type.split(";")[0]

[christinestraub]

Hi @ftnext

What version of the unstructured library did you use? I tested both partition_html() and partition() with unstructured==0.11.5. It didn't get any errors and worked as expected.

from unstructured.partition.html import partition_html
html_elements = partition_html(url="https://www.arxiv-vanity.com/papers/2305.14283/")

from unstructured.partition.auto import partition
html_elements = partition(url="https://www.arxiv-vanity.com/papers/2305.14283/")

[ftnext]

@christinestraub Thanks for your reply.

What version of the unstructured library did you use?

I used unstructured version: 0.11.2 (I mentioned above)

I also try unstructured==0.11.5

>>> from unstructured.partition.auto import partition
>>> html_elements2 = partition(url="https://www.arxiv-vanity.com/papers/2305.14283/")
libmagic is unavailable but assists in filetype detection on file-like objects. Please consider installing libmagic for better results.
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/.../venv/lib/python3.11/site-packages/unstructured/partition/auto.py", line 490, in partition
    raise ValueError(f"{msg}. The {filetype} file type is not supported in partition.")
ValueError: Invalid file. The FileType.UNK file type is not supported in partition.

I noticed the message "libmagic is unavailable", so I ran brew install libmagic

>>> from unstructured.partition.auto import partition
>>> html_elements2 = partition(url="https://www.arxiv-vanity.com/papers/2305.14283/")

So it seems that partition.auto.partition raises ValueError: Invalid file (FileType.UNK) caused by semicolon including Content-Type without libmagic

[hackgoofer]

I installed libmagic, and then the error disappeared.