diff --git a/.github/workflows/release.yaml b/.github/workflows/release.yaml index 2594ee25..3aaf7639 100644 --- a/.github/workflows/release.yaml +++ b/.github/workflows/release.yaml @@ -41,7 +41,7 @@ jobs: - name: Push to dockerhub if: ${{ success() }} run: | - docker build -f docker/Dockerfile -t dedocproject/dedoc:$GITHUB_REF_NAME . + docker build -f Dockerfile -t dedocproject/dedoc:$GITHUB_REF_NAME . docker login -u ${{ secrets.DOCKERHUB_USERNAME }} -p ${{ secrets.DOCKERHUB_PASSWORD }} docker tag dedocproject/dedoc:$GITHUB_REF_NAME dedocproject/dedoc:latest docker push dedocproject/dedoc:$GITHUB_REF_NAME diff --git a/README.md b/README.md index f821b75d..02b9cb26 100644 --- a/README.md +++ b/README.md @@ -4,10 +4,15 @@ ![Dedoc](https://github.com/ispras/dedoc/raw/master/dedoc_logo.png) -Dedoc is an open universal system for converting documents to a unified output format. It extracts a document’s logical structure and content, its tables, text formatting and metadata. The document’s content is represented as a tree storing headings and lists of any level. Dedoc can be integrated in a document contents and structure analysis system as a separate module. +Dedoc is an open universal system for converting documents to a unified output format. +It extracts a document’s logical structure and content, its tables, text formatting and metadata. +The document’s content is represented as a tree storing headings and lists of any level. +Dedoc can be integrated in a document contents and structure analysis system as a separate module. ## Features and advantages -Dedoc is implemented in Python and works with semi-structured data formats (DOC/DOCX, ODT, XLS/XLSX, CSV, TXT, JSON) and none-structured data formats like images (PNG, JPG etc.), archives (ZIP, RAR etc.), PDF and HTML formats. Document structure extraction is fully automatic regardless of input data type. Metadata and text formatting is also extracted automatically. +Dedoc is implemented in Python and works with semi-structured data formats (DOC/DOCX, ODT, XLS/XLSX, CSV, TXT, JSON) and none-structured data formats like images (PNG, JPG etc.), archives (ZIP, RAR etc.), PDF and HTML formats. +Document structure extraction is fully automatic regardless of input data type. +Metadata and text formatting are also extracted automatically. In 2022, the system won a grant to support the development of promising AI projects from the [Innovation Assistance Foundation (Фонд содействия инновациям)](https://fasie.ru/). @@ -16,45 +21,189 @@ In 2022, the system won a grant to support the development of promising AI proje * Support for extracting document structure out of nested documents having different formats. * Extracting various text formatting features (indentation, font type, size, style etc.). * Working with documents of various origin (statements of work, legal documents, technical reports, scientific papers) allowing flexible tuning for new domains. -* Working with PDF documents containinng a text layer: - * Support to automatically determine the correctness of the text layer in PDF documents; - * Extract containing and formatting from PDF-documents with a text layer using the developed interpreter of the virtual stack machine for printing graphics according to the format specification. -Extracting table data from DOC/DOCX, PDF, HTML, CSV and image formats: +* Working with PDF documents containing a textual layer: + * Support to automatically determine the correctness of the textual layer in PDF documents; + * Extract containing and formatting from PDF-documents with a textual layer using the developed interpreter of the virtual stack machine for printing graphics according to the format specification. +* Extracting table data from DOC/DOCX, PDF, HTML, CSV and image formats: * Recognizing a physical structure and a cell text for complex multipage tables having explicit borders with the help of contour analysis. * Working with scanned documents (image formats and PDF without text layer): * Using Tesseract, an actively developed OCR engine from Google, together with image preprocessing methods. * Utilizing modern machine learning approaches for detecting a document orientation, detecting single/multicolumn document page, detecting bold text and extracting hierarchical structure based on the classification of features extracted from document images. +## Impact +This project may be useful as a first step of automatic document analysis pipeline (e.g. before the NLP part). +Dedoc is in demand for information analytic systems, information leak monitoring systems, as well as for natural language processing systems. +The library is intended for application use by developers of systems for automatic analysis and structuring of electronic documents, including for further search in electronic documents. -This project may be useful as a first step of automatic document analysis pipeline (e.g. before the NLP part) +# Online-Documentation +Relevant documentation of the dedoc is available [here](https://dedoc.readthedocs.io/en/latest/) -This project has REST Api and you can run it in Docker container -To read full Dedoc documentation run the project and go to localhost:1231. - +# Installation instructions +**************************************** +This project has REST Api and you can run it in Docker container. +Also, dedoc can be installed as a library via `pip`. +There are two ways to install and run dedoc as a web application or a library that are described below. -## Run the project -How to build and run the project -Ensure you have Git and Docker installed - -Clone the project +## Install and run dedoc using docker + +You should have [`git`] (https://git-scm.com) and [`docker`](https://www.docker.com) installed for running dedoc by this method. +This method is more flexible because it doesn't depend on the operating system and other user's limitations, +still, the docker application should be installed and configured properly. + +If you don't need to change the application configuration, you may use the built docker image as well. + +### 1. Pull the image ```bash -git clone https://github.com/ispras/dedoc.git +docker pull dedocproject/dedoc +``` -cd dedoc/ +### 2. Run the container +```bash +docker run -p 1231:1231 --rm dedocproject/dedoc python3 /dedoc_root/dedoc/main.py ``` - -Ensure you have Docker installed. -Start 'Dedoc' on the port 1231: - ```bash + +Go to [dockerhub](https://hub.docker.com/r/dedocproject/dedoc) to get more information about available dedoc images. + +If you need to change some application settings, you may update `config.py` according to your needs and re-build the image. +You can build and run image: + +### 1. Clone the repository +```bash +git clone https://github.com/ispras/dedoc +``` + +### 2. Go to the `dedoc` directory +```bash +cd dedoc +``` + +### 3. Build the image and run the application +```bash docker-compose up --build ``` -Start Dedoc with tests: - ```bash +### 4. Run container with tests +```bash test="true" docker-compose up --build - ``` +``` + +If you need to change some application settings, you may update `config.py` according to your needs and re-build the image. + + +## Install dedoc using pip + +If you don't want to use docker for running the application, it's possible to run dedoc locally. +However, it isn't suitable for any operating system (`Ubuntu 20+` is recommended) and +there may be not enough machine's resources for its work. +You should have `python` (`python3.8`, `python3.9` are recommended) and `pip` installed. + +### 1. Install necessary packages: +```bash +sudo apt-get install -y libreoffice djvulibre-bin unzip unrar +``` + +`libreoffice` and `djvulibre-bin` packages are used by converters (doc, odt to docx; xls, ods to xlsx; ppt, odp to pptx; djvu to pdf). +If you don't need converters, you can skip this step. +`unzip` and `unrar` packages are used in the process of extracting archives. + +### 2. Install `Tesseract OCR 5` framework: +You can try any tutorial for this purpose or look [`here`](https://github.com/ispras/dedockerfiles/blob/master/dedoc_p3.9_base.Dockerfile) +to get the example of Tesseract installing for dedoc container or use next commands for building Tesseract OCR 5 from sources: + +#### 2.1. Install compilers and libraries required by the Tesseract OCR: +```bash +sudo apt-get update +sudo apt-get install -y automake binutils-dev build-essential ca-certificates clang g++ g++-multilib gcc-multilib libcairo2 libffi-dev \ +libgdk-pixbuf2.0-0 libglib2.0-dev libjpeg-dev libleptonica-dev libpango-1.0-0 libpango1.0-dev libpangocairo-1.0-0 libpng-dev libsm6 \ +libtesseract-dev libtool libxext6 make pkg-config poppler-utils pstotext shared-mime-info software-properties-common swig zlib1g-dev +``` +#### 2.2. Build Tesseract from sources: +```bash +sudo add-apt-repository -y ppa:alex-p/tesseract-ocr-devel +sudo apt-get update --allow-releaseinfo-change +sudo apt-get install -y tesseract-ocr tesseract-ocr-rus +git clone --depth 1 --branch 5.0.0-beta-20210916 https://github.com/tesseract-ocr/tesseract/ +cd tesseract && ./autogen.sh && sudo ./configure && sudo make && sudo make install && sudo ldconfig && cd .. +export TESSDATA_PREFIX=/usr/share/tesseract-ocr/5/tessdata/ +``` + +## Install the dedoc library via pip. + +You need torch~=1.11.0 and torchvision~=0.12.0 installed. If you already have torch and torchvision in your environment: + +```bash +pip install dedoc +``` + +Or you can install dedoc with torch and torchvision included: + +```bash +pip install "dedoc[torch]" +``` + +## Install and run dedoc from sources + +If you want to run dedoc as a service from sources. it's possible to run dedoc locally. +However, it isn't suitable for any operating system (Ubuntu 20+ is recommended) and +there may be not enough machine's resources for its work. +You should have `python` (python3.8, python3.9 are recommended) and `pip` installed. + +### 1. Install necessary packages: according to instructions [install necessary packages](#1-Install-necessary-packages) + +### 2. Build Tesseract from sources according to instructions [Install Tesseract OCR-5 framework](#2-Install-Tesseract-OCR-5-framework) + +### 3. We recommend to install python's virtual environment (for example, via `virtualenvwrapper`) + +Below are the instructions for installing the package `virtualenvwrapper`: + +```bash +sudo pip3 install virtualenv virtualenvwrapper +mkdir ~/.virtualenvs +export WORKON_HOME=~/.virtualenvs +echo "export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3.8" >> ~/.bashrc +echo ". /usr/local/bin/virtualenvwrapper.sh" >> ~/.bashrc +source ~/.bashrc +mkvirtualenv dedoc_env +``` + +### 4. Install python's requirements and launch dedoc service on default port `1231`: + +```bash +# clone dedoc project +git clone https://github.com/ispras/dedoc.git +cd dedoc +# check on your's python environment +workon dedoc_env +export PYTHONPATH=$PYTHONPATH:$(pwd) +pip install -r requirements.txt +pip install torch=1.11.0 torchvision==0.12.0 -f https://download.pytorch.org/whl/torch_stable.html +python dedoc/main.py -c ./dedoc/config.py +``` +Now you can go to the `localhost:1231` and look at the docs and examples. + +## Option: You can change the port of service: +you need to change environment DOCREADER_PORT + +1. For local service launching on your_port (1166 example). [Install instruction from sources](#Install-and-run-dedoc-from-sources) and launch with environment: +```bash +DOCREADER_PORT=1166 python dedoc/main.py -c ./dedoc/config.py +``` -Now you can go to the localhost:1231 and look at the docs and examples. +2. For service launching in docker-container you need to change port value in DOCREADER_PORT env and field 'ports' in docker-compose.yml file: +```yaml + ... + dedoc: + ... + ports: + - your_port_number:your_port_number + environment: + DOCREADER_PORT: your_port_number + ... + test: + ... + environment: + DOCREADER_PORT: your_port_number +``` -You can change the port and host in the config file 'dedoc/config.py' +Go [here](https://dedoc.readthedocs.io/en/latest/getting_started/installation.html) to get more details about dedoc installation. diff --git a/VERSION b/VERSION index 027934ea..a8839f70 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.11.1 \ No newline at end of file +0.11.2 \ No newline at end of file diff --git a/dedoc/readers/pdf_reader/pdf_txtlayer_reader/tabbypdf/jars/plexus-utils-1.1.jar b/dedoc/readers/pdf_reader/pdf_txtlayer_reader/tabbypdf/jars/plexus-utils-1.1.jar deleted file mode 100644 index 5c50e177..00000000 Binary files a/dedoc/readers/pdf_reader/pdf_txtlayer_reader/tabbypdf/jars/plexus-utils-1.1.jar and /dev/null differ diff --git a/docs/source/changelog.rst b/docs/source/changelog.rst index a300853f..d991dabe 100644 --- a/docs/source/changelog.rst +++ b/docs/source/changelog.rst @@ -1,6 +1,17 @@ Changelog ========= +v0.11.2 (2023-09-06) +-------------------- +Release note: `v0.11.2 `_ + +* Remove plexus-utils-1.1.jar. +* Update installation documentation. +* Add documentation for Tesseract OCR installation. +* Add documentation for annotations. +* Add documentation for secure torch. +* Fix examples. + v0.11.1 (2023-08-30) -------------------- Release note: `v0.11.1 `_ diff --git a/docs/source/getting_started/installation.rst b/docs/source/getting_started/installation.rst index 42ae2ea8..70373dfc 100644 --- a/docs/source/getting_started/installation.rst +++ b/docs/source/getting_started/installation.rst @@ -59,27 +59,54 @@ Install dedoc using pip If you don't want to use docker for running the application, it's possible to run dedoc locally. However, it isn't suitable for any operating system (Ubuntu 20+ is recommended) and there may be not enough machine's resources for its work. -You should have `python` (python3.8+ is recommended) and `pip` installed. +You should have `python` (python3.8, python3.9 are recommended) and `pip` installed. +.. _install_packages: -1. Install `libreoffice` and `djvulibre-bin` packages: +1. Install necessary packages: +****************************** .. code-block:: bash - sudo apt-get install -y libreoffice djvulibre-bin + sudo apt-get install -y libreoffice djvulibre-bin unzip unrar -These packages are used by converters (doc, odt to docx; xls, ods to xlsx; ppt, odp to pptx; djvu to pdf). +`libreoffice` and `djvulibre-bin` packages are used by converters (doc, odt to docx; xls, ods to xlsx; ppt, odp to pptx; djvu to pdf). If you don't need converters, you can skip this step. +`unzip` and `unrar` packages are used in the process of extracting archives. +.. _install_tesseract: 2. Install `Tesseract OCR 5` framework. +*************************************** You can try any tutorial for this purpose or look `here `_ -to get the example of Tesseract installing for dedoc container. +to get the example of Tesseract installing for dedoc container or use next commands for building Tesseract OCR 5 from sources: +2.1. Install compilers and libraries required by the Tesseract OCR: + + .. code-block:: bash + + sudo apt-get update + sudo apt-get install -y automake binutils-dev build-essential ca-certificates clang g++ g++-multilib gcc-multilib libcairo2 libffi-dev \ + libgdk-pixbuf2.0-0 libglib2.0-dev libjpeg-dev libleptonica-dev libpango-1.0-0 libpango1.0-dev libpangocairo-1.0-0 libpng-dev libsm6 \ + libtesseract-dev libtool libxext6 make pkg-config poppler-utils pstotext shared-mime-info software-properties-common swig zlib1g-dev + +2.2. Build Tesseract from sources: + + .. code-block:: bash + + sudo add-apt-repository -y ppa:alex-p/tesseract-ocr-devel + sudo apt-get update --allow-releaseinfo-change + sudo apt-get install -y tesseract-ocr tesseract-ocr-rus + git clone --depth 1 --branch 5.0.0-beta-20210916 https://github.com/tesseract-ocr/tesseract/ + cd tesseract && ./autogen.sh && sudo ./configure && sudo make && sudo make install && sudo ldconfig && cd .. + export TESSDATA_PREFIX=/usr/share/tesseract-ocr/5/tessdata/ + +.. _install_library_via_pip: 3. Install the dedoc library via pip. -To fulfil all the library requirements, you should have `torch~=1.11.0` and `torchvision~=0.12.0` installed. -You can install suitable for you versions of these libraries and install dedoc using pip command: +************************************* +You need `torch~=1.11.0` and `torchvision~=0.12.0` installed. +If you already have torch and torchvision in your environment: .. code-block:: bash @@ -90,3 +117,69 @@ Or you can install dedoc with torch and torchvision included: .. code-block:: bash pip install "dedoc[torch]" + +Install and run dedoc from sources +---------------------------------- + +If you want to run dedoc as a service from sources. it's possible to run dedoc locally. +However, it isn't suitable for any operating system (Ubuntu 20+ is recommended) and +there may be not enough machine's resources for its work. +You should have `python` (python3.8, python3.9 are recommended) and `pip` installed. + +1. Install necessary packages: according to instructions :ref:`install_packages` + +2. Build Tesseract from sources according to instructions :ref:`install_tesseract` + +3. We recommend to install python's virtual environment (for example, via `virtualenvwrapper`) + +Below are the instructions for installing the package `virtualenvwrapper`: + + .. code-block:: bash + + sudo pip3 install virtualenv virtualenvwrapper + mkdir ~/.virtualenvs + export WORKON_HOME=~/.virtualenvs + echo "export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3.8" >> ~/.bashrc + echo ". /usr/local/bin/virtualenvwrapper.sh" >> ~/.bashrc + source ~/.bashrc + mkvirtualenv dedoc_env + +4. Install python's requirements and launch dedoc service on default port `1231`: + + .. code-block:: bash + + # clone dedoc project + git clone https://github.com/ispras/dedoc.git + cd dedoc + # check on your's python environment + workon dedoc_env + export PYTHONPATH=$PYTHONPATH:$(pwd) + pip install -r requirements.txt + pip install torch==1.11.0 torchvision==0.12.0 -f https://download.pytorch.org/whl/torch_stable.html + python dedoc/main.py -c ./dedoc/config.py + + +Install trusted torch (verified version) +---------------------------------------------- + +You can install a trusted library `torch` (as a verified version of the library, verified by tools developed by the Ivannikov Institute for System Programming of the Russian Academy of Sciences). + +First you need to install two required packages.: + + .. code-block:: bash + + sudo apt-get install -y mpich intel-mkl + +Second you need to install torch and torchvision from built wheels: + +For `python3.8`: + .. code-block:: bash + + pip install https://github.com/ispras/dedockerfiles/raw/master/wheels/torch-1.11.0a0+git137096a-cp38-cp38-linux_x86_64.whl + pip install https://github.com/ispras/dedockerfiles/raw/master/wheels/torchvision-0.12.0a0%2B9b5a3fe-cp38-cp38-linux_x86_64.whl + +For `python3.9`: + .. code-block:: bash + + pip install https://github.com/ispras/dedockerfiles/raw/master/wheels/torch-1.11.0a0+git137096a-cp39-cp39-linux_x86_64.whl + pip install https://github.com/ispras/dedockerfiles/raw/master/wheels/torchvision-0.12.0a0%2B9b5a3fe-cp39-cp39-linux_x86_64.whl \ No newline at end of file diff --git a/docs/source/index.rst b/docs/source/index.rst index 8978f173..1cf30272 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -50,6 +50,8 @@ Reading documents using dedoc Dedoc allows to get the common intermediate representation for the documents of various formats. The resulting output of any reader is a class :class:`~dedoc.data_structures.UnstructuredDocument`. +See :ref:`readers' annotations ` and :ref:`readers' line types ` +to get more details about information that can be extracted by each available reader. .. _table_formats: @@ -220,6 +222,13 @@ For a document of unknown or unsupported domain there is an option to use defaul dedoc_api_usage/return_format +.. toctree:: + :maxdepth: 1 + :caption: Readers output + + readers_output/annotations + readers_output/line_types + .. toctree:: :maxdepth: 1 :caption: Structure types diff --git a/docs/source/readers_output/annotations.rst b/docs/source/readers_output/annotations.rst new file mode 100644 index 00000000..e177ed53 --- /dev/null +++ b/docs/source/readers_output/annotations.rst @@ -0,0 +1,159 @@ +.. _readers_annotations: + +Text annotations +================ + +Below the readers are enlisted that can return non-empty list of annotations for document lines: + +* `+` means that the reader can return the annotation. +* `-` means that the reader doesn't return the annotation due to complexity of the task or lack of information provided by the format. + +.. _table_annotations: + +.. list-table:: Annotations returned by each reader + :widths: 20 10 10 10 10 10 10 + :class: tight-table + + * - **Annotation** + - :class:`~dedoc.readers.DocxReader` + - :class:`~dedoc.readers.HtmlReader`, :class:`~dedoc.readers.MhtmlReader`, :class:`~dedoc.readers.EmailReader` + - :class:`~dedoc.readers.RawTextReader` + - :class:`~dedoc.readers.PdfImageReader` + - :class:`~dedoc.readers.PdfTabbyReader` + - :class:`~dedoc.readers.PdfTxtlayerReader` + + * - :class:`~dedoc.data_structures.AttachAnnotation` + - `+` + - `-` + - `-` + - `-` + - `-` + - `+` + + * - :class:`~dedoc.data_structures.TableAnnotation` + - `+` + - `-` + - `-` + - `+` + - `+` + - `+` + + * - :class:`~dedoc.data_structures.LinkedTextAnnotation` + - `+` + - `+` + - `-` + - `-` + - `+` + - `+` + + * - :class:`~dedoc.data_structures.BBoxAnnotation` + - `-` + - `-` + - `-` + - `+` + - `+` + - `+` + + * - :class:`~dedoc.data_structures.AlignmentAnnotation` + - `+` + - `+` + - `-` + - `-` + - `-` + - `-` + + * - :class:`~dedoc.data_structures.IndentationAnnotation` + - `+` + - `-` + - `+` + - `+` + - `+` + - `+` + + * - :class:`~dedoc.data_structures.SpacingAnnotation` + - `+` + - `-` + - `+` + - `+` + - `+` + - `+` + + * - :class:`~dedoc.data_structures.BoldAnnotation` + - `+` + - `+` + - `-` + - `+` + - `+` + - `+` + + * - :class:`~dedoc.data_structures.ItalicAnnotation` + - `+` + - `+` + - `-` + - `-` + - `+` + - `+` + + * - :class:`~dedoc.data_structures.UnderlinedAnnotation` + - `+` + - `+` + - `-` + - `-` + - `-` + - `-` + + * - :class:`~dedoc.data_structures.StrikeAnnotation` + - `+` + - `+` + - `-` + - `-` + - `-` + - `-` + + * - :class:`~dedoc.data_structures.SubscriptAnnotation` + - `+` + - `+` + - `-` + - `-` + - `-` + - `-` + + * - :class:`~dedoc.data_structures.SuperscriptAnnotation` + - `+` + - `+` + - `-` + - `-` + - `-` + - `-` + + * - :class:`~dedoc.data_structures.ColorAnnotation` + - `-` + - `-` + - `-` + - `+` + - `-` + - `+` + + * - :class:`~dedoc.data_structures.SizeAnnotation` + - `+` + - `+` + - `-` + - `+` + - `+` + - `+` + + * - :class:`~dedoc.data_structures.StyleAnnotation` + - `+` + - `+` + - `-` + - `-` + - `+` + - `+` + + * - :class:`~dedoc.data_structures.ConfidenceAnnotation` + - `-` + - `-` + - `-` + - `+` + - `-` + - `-` diff --git a/docs/source/readers_output/line_types.rst b/docs/source/readers_output/line_types.rst new file mode 100644 index 00000000..826a4807 --- /dev/null +++ b/docs/source/readers_output/line_types.rst @@ -0,0 +1,65 @@ +.. _readers_line_types: + +Types of textual lines +====================== + +Each reader returns :class:`~dedoc.data_structures.UnstructuredDocument` with textual lines. +Readers don't fill `hierarchy_level` metadata field (structure extractors do this), but they can fill `hierarchy_level_tag` with information about line types. +Below the readers are enlisted that can return non-empty `hierarchy_level_tag` in document lines metadata: + +* `+` means that the reader can return lines of this type. +* `-` means that the reader doesn't return lines of this type due to complexity of the task or lack of information provided by the format. + +.. _table_line_types: + +.. list-table:: Line types returned by each reader + :widths: 20 20 20 20 20 + :class: tight-table + + * - **Reader** + - **header** + - **list_item** + - **raw_text, unknown** + - **key** + + * - :class:`~dedoc.readers.DocxReader` + - `+` + - `+` + - `+` + - `-` + + * - :class:`~dedoc.readers.HtmlReader`, :class:`~dedoc.readers.MhtmlReader`, :class:`~dedoc.readers.EmailReader` + - `+` + - `+` + - `+` + - `-` + + * - :class:`~dedoc.readers.RawTextReader` + - `-` + - `+` + - `+` + - `-` + + * - :class:`~dedoc.readers.JsonReader` + - `-` + - `+` + - `+` + - `+` + + * - :class:`~dedoc.readers.PdfImageReader` + - `-` + - `+` + - `+` + - `-` + + * - :class:`~dedoc.readers.PdfTabbyReader` + - `+` + - `+` + - `+` + - `-` + + * - :class:`~dedoc.readers.PdfTxtlayerReader` + - `-` + - `+` + - `+` + - `-` diff --git a/examples/README.md b/examples/README.md new file mode 100644 index 00000000..39938547 --- /dev/null +++ b/examples/README.md @@ -0,0 +1,31 @@ +# Examples of document parsing with Dedoc + +This directory contains files with examples of using Dedoc to process different types of documents. + +It's worth starting with the `example_manager_input.py` file. It describes examples of document processing using the `DedocManager` class. +This is the easiest way, since this class automatically determines the format of the file being processed and calls the necessary readers. + +As shown in corresponding examples, you can create this manager with following lines: +``` +from dedoc.config import get_config +from dedoc import DedocManager + +manager = DedocManager(config=get_config()) +``` +And after that you can get parsed document with one simple line, just replace `"your_file_name"` with the path to your chosen file: +``` +parsed_document = manager.parse(file_path="your_file_name") +``` +To get more information, look at Dedoc usage tutorial. + +If you want to call a specific parser, you can look at some examples in this directory. File `example_doc_parser.py` shows how you can use `DocxReader`, +`example_pdf_parser.py` shows examples with PDF file parsing. In order to parse img-like file you can call `PdfImageReader` like it's shown in +`example_img_parser.py`. + +Also you can look at the example of using a post-request to parse documents while Dedoc container is working. This example is written in `example_post.py`. + +You can check an example like this: +```bash +cd examples +python create_structured_document.py +``` \ No newline at end of file diff --git a/examples/create_structured_document.py b/examples/create_structured_document.py new file mode 100644 index 00000000..0a4d468d --- /dev/null +++ b/examples/create_structured_document.py @@ -0,0 +1,11 @@ +# noqa +from create_unstructured_document import unstructured_document +from dedoc.structure_constructors import TreeConstructor + +# to create structured document you can use TreeConstructor and apply it to unstructured document +# in this example we'll use unstructured_document from create_unstructured_document.py +structure_constructor = TreeConstructor() +parsed_document = structure_constructor.structure_document(document=unstructured_document, + structure_type="tree") + +print(parsed_document.to_dict()) diff --git a/examples/create_unstructured_document.py b/examples/create_unstructured_document.py index a8c327d1..de27cd87 100644 --- a/examples/create_unstructured_document.py +++ b/examples/create_unstructured_document.py @@ -1,12 +1,13 @@ +# noqa # in this example we create UnstructuredDocument, lets construct document corresponding to example.docx -from dedoc.data_structures import LineWithMeta -from dedoc.data_structures import ParagraphMetadata -from dedoc.data_structures import Table +from dedoc.data_structures import LineMetadata, Table, UnstructuredDocument from dedoc.data_structures import TableMetadata - +from dedoc.data_structures import LineWithMeta # First of all lets create some table, table consist of cells (list of rows, and row is a list of strings -from dedoc import HierarchyLevel +from dedoc.data_structures import HierarchyLevel +from dedoc.metadata_extractors import BaseMetadataExtractor + table_cells = [ ["N", "Second name", "Name", "Organization", "Phone", "Notes"], @@ -15,7 +16,7 @@ # table also has some metadata, lets assume that our table is on first page table_metadata = TableMetadata(page_id=0) -# finally lets build table +# let's build table table = Table(cells=table_cells, metadata=table_metadata) # Documents also contain some text. @@ -23,13 +24,7 @@ # but unstructured document consist of flat list of lines with text and metadata # hierarchy structure hidden in HierarchyLevel attribute of LineWithMeta # lets build firs line, it is document tree root: -text = "DOCUMENT TITLE" -metadata = ParagraphMetadata( - paragraph_type="title", - predicted_classes=None, - page_id=0, - line_id=0 -) + # hierarchy level define position of this line in document tree. hierarchy_level = HierarchyLevel( @@ -42,8 +37,14 @@ # if can_be_multiline is true than several lines in a row with same level_1, level_2 and paragraph_type # will be merged in one tree node can_be_multiline=True, - paragraph_type="title" + line_type="header" ) +text = "DOCUMENT TITLE" +metadata = LineMetadata(page_id=0, + line_id=1, + tag_hierarchy_level=None, + hierarchy_level=hierarchy_level, + other_fields=None) # Annotations: one may specify some information about some part of the text, for example that some word # written in italic font. @@ -51,11 +52,14 @@ line1 = LineWithMeta( line=text, - hierarchy_level=hierarchy_level, metadata=metadata, annotations=annotations ) +unstructured_document = UnstructuredDocument(tables=[table], + lines=[line1], + attachments=[]) + # I hope you understand some concepts of the LineWithMeta, but you may ask why it need level_1 and level_2 # parameters. Why is only level_1 not enough. Imagine that we have lists like these: # 1. 1.1. 1.2.1.1. and so on, It may be difficult to restrict the length of the list with @@ -63,3 +67,8 @@ # HierarchyLevel(1, 1) for 1. # HierarchyLevel(1, 2) for 1.1. # HierarchyLevel(1, 4) for 1.2.1.1. and so on +unstructured_document = BaseMetadataExtractor().add_metadata(document=unstructured_document, + directory="./", + filename="example.docx", + converted_filename="example.doc", + original_filename="example.docx") diff --git a/examples/example.jpg b/examples/example.jpg new file mode 100644 index 00000000..afd13dc2 Binary files /dev/null and b/examples/example.jpg differ diff --git a/examples/example_converter.py b/examples/example_converter.py deleted file mode 100644 index 3b7be501..00000000 --- a/examples/example_converter.py +++ /dev/null @@ -1,19 +0,0 @@ -# noqa -import os -from werkzeug.datastructures import FileStorage -from dedoc import get_config -from dedoc.manager import DedocThreadedManager - -version_file_path = os.path.abspath(os.path.join(os.path.dirname(__file__), "../src", "..", "VERSION")) -manager = DedocThreadedManager.from_config(config=get_config(), version=open(version_file_path).read().strip()) - - -filename = "example.docx" - -with open(filename, 'rb') as fp: - file = FileStorage(fp, filename=filename) - - parsed_document = manager.parse_file(file, parameters={"document_type": "example"}) - - print(parsed_document) - print(parsed_document.to_dict()) diff --git a/examples/example_doc_parser.py b/examples/example_doc_parser.py index e13c3650..f4be2b43 100644 --- a/examples/example_doc_parser.py +++ b/examples/example_doc_parser.py @@ -1,6 +1,6 @@ # noqa -from dedoc import get_config -from dedoc import DocxReader +from dedoc.config import get_config +from dedoc.readers.docx_reader.docx_reader import DocxReader # create docx reader docx_reader = DocxReader(config=get_config()) @@ -8,9 +8,9 @@ file_name = "example.docx" # we get unstructured file with lines and tables -unstructured_document, can_contain_attachments = docx_reader.read(path=file_name, document_type="example") +unstructured_document = docx_reader.read(path=file_name, document_type="example") -# lets look at content of unstructured_file, it consist of tables and lines +# let's look at content of unstructured_file, it consists of tables and lines print(unstructured_document.tables, unstructured_document.lines) # first of all lets look at the table @@ -23,8 +23,8 @@ # and now lets look at lines. lines it is list of object of class LineWithMeta lines = unstructured_document.lines -# lets look at first line +# let's look at first line line = lines[0] print(line) -# line consist of line (text), metadata and hierarchy_level -print(line.line, line.metadata, line.hierarchy_level) +# line consist of line (text), metadata, hierarchy level +print(line.line, line.metadata, line.metadata.hierarchy_level) diff --git a/examples/example_img_parser.py b/examples/example_img_parser.py new file mode 100644 index 00000000..589d651c --- /dev/null +++ b/examples/example_img_parser.py @@ -0,0 +1,31 @@ +# noqa +from dedoc.config import get_config +from dedoc.readers.pdf_reader.pdf_image_reader.pdf_image_reader import PdfImageReader + +# create img reader +# pdf reader can parse image-like formats +img_reader = PdfImageReader(config=get_config()) +# and read file example.docx +file_name = "example.jpg" + +# we get unstructured file with lines and tables +unstructured_document = img_reader.read(path=file_name, document_type="example") + +# let's look at content of unstructured_file, it consists of tables and lines +print(unstructured_document.tables, unstructured_document.lines) + +# first of all lets look at the table +table = unstructured_document.tables[0] +# table consists of cells (we assume that table is rectangle) +# so cells is list of rows and row is list of strings +print(table.cells) +# there is also some metadata in table +print(table.metadata) + +# and now lets look at lines. lines it is list of object of class LineWithMeta +lines = unstructured_document.lines +# let's look at first line +line = lines[0] +print(line) +# line consist of line (text), metadata, hierarchy level +print(line.line, line.metadata, line.metadata.hierarchy_level) diff --git a/examples/example_manager_input.py b/examples/example_manager_input.py index 925f4b36..bd5a1b41 100644 --- a/examples/example_manager_input.py +++ b/examples/example_manager_input.py @@ -1,18 +1,35 @@ # noqa -import os -from werkzeug.datastructures import FileStorage +import json -from dedoc import get_config -from dedoc.manager import DedocThreadedManager +from dedoc import DedocManager +from dedoc.config import get_config -version_file_path = os.path.abspath(os.path.join(os.path.dirname(__file__), "../src", "..", "VERSION")) -manager = DedocThreadedManager.from_config(config=get_config(), version=open(version_file_path).read().strip()) +manager = DedocManager(config=get_config()) -filename = "example.docx" -with open(filename, 'rb') as fp: - file = FileStorage(fp, filename=filename) +filename_docx = "example.docx" +parsed_docx_document = manager.parse(file_path=filename_docx, parameters={}) - parsed_document = manager.parse_file(file, parameters={"document_type": "example"}) +# save the result +with open("result_docx.json", "w") as outfile: + outfile.write(json.dumps(parsed_docx_document.to_dict())) - print(parsed_document) - print(parsed_document.to_dict()) +filename_jpg = "example.jpg" +parsed_jpg_document = manager.parse(file_path=filename_jpg, parameters={}) + +# save the result +with open("result_jpg.json", "w") as outfile: + outfile.write(json.dumps(parsed_jpg_document.to_dict())) + +filename_pdf_no_text_layer = "example_without_text_layer.pdf" +parsed_pdf_no_text_layer_document = manager.parse(file_path=filename_pdf_no_text_layer, parameters={"pdf_with_text_layer": "false"}) + +# save the result +with open("result_pdf_no_text_layer.json", "w") as outfile: + outfile.write(json.dumps(parsed_pdf_no_text_layer_document.to_dict())) + +filename_pdf_with_text_layer = "example_with_text_layer.pdf" +parsed_pdf_with_text_layer_document = manager.parse(file_path=filename_pdf_with_text_layer, parameters={"pdf_with_text_layer": "true"}) + +# save the result +with open("result_pdf_with_text_layer.json", "w") as outfile: + outfile.write(json.dumps(parsed_pdf_with_text_layer_document.to_dict())) diff --git a/examples/example_pdf_parser.py b/examples/example_pdf_parser.py new file mode 100644 index 00000000..5d7507b8 --- /dev/null +++ b/examples/example_pdf_parser.py @@ -0,0 +1,60 @@ +# noqa +from dedoc.config import get_config +from dedoc.readers.pdf_reader.pdf_txtlayer_reader.pdf_txtlayer_reader import PdfTxtlayerReader +from dedoc.readers.pdf_reader.pdf_image_reader.pdf_image_reader import PdfImageReader + +# create pdf text layer reader +pdf_txt_layer_reader = PdfTxtlayerReader(config=get_config()) +# and read file example.docx +file_name = "example_with_text_layer.pdf" + +# we get unstructured file with lines and tables +unstructured_document = pdf_txt_layer_reader.read(path=file_name, document_type="example") + +# let's look at content of unstructured_file, it consists of tables and lines +print(unstructured_document.tables, unstructured_document.lines) + +# first of all lets look at the table +table = unstructured_document.tables[0] +# table consists of cells (we assume that table is rectangle) +# so cells is list of rows and row is list of strings +print(table.cells) +# there is also some metadata in table +print(table.metadata) + +# and now lets look at lines. lines it is list of object of class LineWithMeta +lines = unstructured_document.lines +# let's look at first line +line = lines[0] +print(line) +# line consist of line (text), metadata, hierarchy level +print(line.line, line.metadata, line.metadata.hierarchy_level) + + +# let's now look at the example of parsing pdf document without text layer +# create pdf reader +pdf_image_reader = PdfImageReader(config=get_config()) +# and read file example.docx +file_name = "example_without_text_layer.pdf" + +# we get unstructured file with lines and tables +unstructured_document = pdf_image_reader.read(path=file_name, document_type="example") + +# let's look at content of unstructured_file, it consists of tables and lines +print(unstructured_document.tables, unstructured_document.lines) + +# first of all lets look at the table +table = unstructured_document.tables[0] +# table consists of cells (we assume that table is rectangle) +# so cells is list of rows and row is list of strings +print(table.cells) +# there is also some metadata in table +print(table.metadata) + +# and now lets look at lines. lines it is list of object of class LineWithMeta +lines = unstructured_document.lines +# let's look at first line +line = lines[0] +print(line) +# line consist of line (text), metadata, hierarchy level +print(line.line, line.metadata, line.metadata.hierarchy_level) diff --git a/examples/example_post.py b/examples/example_post.py index 79bb21d1..97317c81 100644 --- a/examples/example_post.py +++ b/examples/example_post.py @@ -6,6 +6,7 @@ # We want to parse file example.docx # We will send it with requests lib +# To parse other document types just paste correct file name below file_name = "example.docx" file_path = os.path.abspath(file_name) @@ -13,6 +14,7 @@ # file we want to parse files = {'file': (file_name, file)} # dict with additional parameters + # to parse pdf with text layer add parameter "pdf_with_text_layer":"true" data = {"document_type": ""} # and now we send post request with attached file and paremeters. r = requests.post("http://localhost:1231/upload", files=files, data=data) diff --git a/examples/example_with_text_layer.pdf b/examples/example_with_text_layer.pdf new file mode 100644 index 00000000..45b240c7 Binary files /dev/null and b/examples/example_with_text_layer.pdf differ diff --git a/examples/example_without_text_layer.pdf b/examples/example_without_text_layer.pdf new file mode 100644 index 00000000..2db9ff61 Binary files /dev/null and b/examples/example_without_text_layer.pdf differ