fixed examples directory

README.md

@@ -36,7 +36,7 @@ Dedoc is in demand for information analytic systems, information leak monitoring
 The library is intended for application use by developers of systems for automatic analysis and structuring of electronic documents, including for further search in electronic documents. 
 
 # Online-Documentation
-Relevant documentation of the dedoc is available [here]((https://dedoc.readthedocs.io/en/latest/))
+Relevant documentation of the dedoc is available [here](https://dedoc.readthedocs.io/en/latest/)
 
 # Installation instructions
 ****************************************

docs/source/getting_started/installation.rst

@@ -105,8 +105,8 @@ to get the example of Tesseract installing for dedoc container or use next comma
 
 3. Install the dedoc library via pip.
 *************************************
-To fulfil all the library requirements, you should have `torch~=1.11.0` and `torchvision~=0.12.0` installed.
-You can install suitable for you versions of these libraries and install dedoc using pip command:
+You need `torch~=1.11.0` and `torchvision~=0.12.0` installed.
+If you already have torch and torchvision in your environment:
 
   .. code-block:: bash
 
@@ -155,7 +155,7 @@ Below are the instructions for installing the package `virtualenvwrapper`:
     workon dedoc_env
     export PYTHONPATH=$PYTHONPATH:$(pwd)
     pip install -r requirements.txt
-    pip install torch=1.11.0 torchvision==0.12.0 -f https://download.pytorch.org/whl/torch_stable.html
+    pip install torch==1.11.0 torchvision==0.12.0 -f https://download.pytorch.org/whl/torch_stable.html
     python dedoc/main.py -c ./dedoc/config.py
 
 
@@ -164,14 +164,22 @@ Install trusted torch (verified version)
 
 You can install a trusted library `torch` (as a verified version of the library, verified by tools developed by the Ivannikov Institute for System Programming of the Russian Academy of Sciences).
 
-For `python3.9`:
-  .. code-block:: bash
+First you need to install two required packages.:
 
-    pip install https://github.com/ispras/dedockerfiles/raw/master/wheels/torch-1.11.0a0+git137096a-cp39-cp39-linux_x86_64.whl
-    pip install https://github.com/ispras/dedockerfiles/raw/master/wheels/torchvision-0.12.0a0%2B9b5a3fe-cp39-cp39-linux_x86_64.whl
+    .. code-block:: bash
+
+       sudo apt-get install -y mpich intel-mkl
+
+Second you need to install torch and torchvision from built wheels:
 
 For `python3.8`:
   .. code-block:: bash
 
      pip install https://github.com/ispras/dedockerfiles/raw/master/wheels/torch-1.11.0a0+git137096a-cp38-cp38-linux_x86_64.whl
-     pip install https://github.com/ispras/dedockerfiles/raw/master/wheels/torchvision-0.12.0a0%2B9b5a3fe-cp38-cp38-linux_x86_64.whl
+     pip install https://github.com/ispras/dedockerfiles/raw/master/wheels/torchvision-0.12.0a0%2B9b5a3fe-cp38-cp38-linux_x86_64.whl
+
+For `python3.9`:
+  .. code-block:: bash
+
+    pip install https://github.com/ispras/dedockerfiles/raw/master/wheels/torch-1.11.0a0+git137096a-cp39-cp39-linux_x86_64.whl
+    pip install https://github.com/ispras/dedockerfiles/raw/master/wheels/torchvision-0.12.0a0%2B9b5a3fe-cp39-cp39-linux_x86_64.whl

examples/README.md

@@ -0,0 +1,31 @@
+# Examples of document parsing with Dedoc 
+
+This directory contains files with examples of using Dedoc to process different types of documents.
+
+It's worth starting with the `example_manager_input.py` file. It describes examples of document processing using the `DedocManager` class. 
+This is the easiest way, since this class automatically determines the format of the file being processed and calls the necessary readers.
+
+As shown in corresponding examples, you can create this manager with following lines:
+```
+from dedoc.config import get_config
+from dedoc import DedocManager
+
+manager = DedocManager(config=get_config())
+```
+And after that you can get parsed document with one simple line, just replace `"your_file_name"` with the path to your chosen file:
+```
+parsed_document = manager.parse(file_path="your_file_name")
+```
+To get more information, look at Dedoc usage tutorial.
+
+If you want to call a specific parser, you can look at some examples in this directory. File `example_doc_parser.py` shows how you can use `DocxReader`,
+`example_pdf_parser.py` shows examples with PDF file parsing. In order to parse img-like file you can call `PdfImageReader` like it's shown in
+`example_img_parser.py`. 
+
+Also you can look at the example of using a post-request to parse documents while Dedoc container is working. This example is written in `example_post.py`.
+
+You can check an example like this:
+```bash
+cd examples
+python create_structured_document.py
+```

examples/create_structured_document.py

@@ -0,0 +1,11 @@
+# noqa
+from create_unstructured_document import unstructured_document
+from dedoc.structure_constructors import TreeConstructor
+
+# to create structured document you can use TreeConstructor and apply it to unstructured document
+# in this example we'll use unstructured_document from create_unstructured_document.py
+structure_constructor = TreeConstructor()
+parsed_document = structure_constructor.structure_document(document=unstructured_document,
+                                                           structure_type="tree")
+
+print(parsed_document.to_dict())

examples/create_unstructured_document.py

@@ -1,12 +1,13 @@
+# noqa
 #  in this example we create UnstructuredDocument, lets construct document corresponding to example.docx
-from dedoc.data_structures import LineWithMeta
-from dedoc.data_structures import ParagraphMetadata
-from dedoc.data_structures import Table
+from dedoc.data_structures import LineMetadata, Table, UnstructuredDocument
 from dedoc.data_structures import TableMetadata
-
+from dedoc.data_structures import LineWithMeta
 
 #  First of all lets create some table, table consist of cells (list of rows, and row is a list of strings
-from dedoc import HierarchyLevel
+from dedoc.data_structures import HierarchyLevel
+from dedoc.metadata_extractors import BaseMetadataExtractor
+
 
 table_cells = [
     ["N", "Second name", "Name", "Organization", "Phone", "Notes"],
@@ -15,21 +16,15 @@
 # table also has some metadata, lets assume that our table is on first page
 table_metadata = TableMetadata(page_id=0)
 
-# finally lets build table
+# let's build table
 table = Table(cells=table_cells, metadata=table_metadata)
 
 # Documents also contain some text.
 # Logical structure of document may be represented by tree (see  example_tree.png)
 # but unstructured document consist of flat list of lines with text and metadata
 # hierarchy structure hidden in HierarchyLevel attribute of LineWithMeta
 # lets build firs line, it is document tree root:
-text = "DOCUMENT TITLE"
-metadata = ParagraphMetadata(
-    paragraph_type="title",
-    predicted_classes=None,
-    page_id=0,
-    line_id=0
-)
+
 # hierarchy level define position of this line in document tree.
 
 hierarchy_level = HierarchyLevel(
@@ -42,24 +37,38 @@
     # if can_be_multiline is true than several lines in a row with same level_1, level_2 and paragraph_type
     # will be merged in one tree node
     can_be_multiline=True,
-    paragraph_type="title"
+    line_type="header"
 )
+text = "DOCUMENT TITLE"
+metadata = LineMetadata(page_id=0,
+                        line_id=1,
+                        tag_hierarchy_level=None,
+                        hierarchy_level=hierarchy_level,
+                        other_fields=None)
 
 # Annotations: one may specify some information about some part of the text, for example that some word
 # written in italic font.
 annotations = []
 
 line1 = LineWithMeta(
     line=text,
-    hierarchy_level=hierarchy_level,
     metadata=metadata,
     annotations=annotations
 )
 
+unstructured_document = UnstructuredDocument(tables=[table],
+                                             lines=[line1],
+                                             attachments=[])
+
 # I hope you understand some concepts of the LineWithMeta, but you may ask why it need level_1 and level_2
 # parameters. Why is only level_1 not enough. Imagine that we have lists like these:
 # 1.  1.1.  1.2.1.1. and so on, It may be difficult to restrict the length of the list with
 # some pre-set number, so you may define
 # HierarchyLevel(1, 1) for 1.
 # HierarchyLevel(1, 2) for 1.1.
 # HierarchyLevel(1, 4) for 1.2.1.1. and so on
+unstructured_document = BaseMetadataExtractor().add_metadata(document=unstructured_document,
+                                                             directory="./",
+                                                             filename="example.docx",
+                                                             converted_filename="example.doc",
+                                                             original_filename="example.docx")

examples/example_doc_parser.py

@@ -1,16 +1,16 @@
 # noqa
-from dedoc import get_config
-from dedoc import DocxReader
+from dedoc.config import get_config
+from dedoc.readers.docx_reader.docx_reader import DocxReader
 
 # create docx reader
 docx_reader = DocxReader(config=get_config())
 # and read file example.docx
 file_name = "example.docx"
 
 # we get unstructured file with lines and tables
-unstructured_document, can_contain_attachments = docx_reader.read(path=file_name, document_type="example")
+unstructured_document = docx_reader.read(path=file_name, document_type="example")
 
-# lets look at content of unstructured_file, it consist of tables and lines
+# let's look at content of unstructured_file, it consists of tables and lines
 print(unstructured_document.tables, unstructured_document.lines)
 
 # first of all lets look at the table
@@ -23,8 +23,8 @@
 
 # and now lets look at lines. lines it is list of object of class LineWithMeta
 lines = unstructured_document.lines
-# lets look at first line
+# let's look at first line
 line = lines[0]
 print(line)
-# line consist of line (text), metadata and hierarchy_level
-print(line.line, line.metadata, line.hierarchy_level)
+# line consist of line (text), metadata, hierarchy level
+print(line.line, line.metadata, line.metadata.hierarchy_level)

examples/example_img_parser.py

@@ -0,0 +1,31 @@
+# noqa
+from dedoc.config import get_config
+from dedoc.readers.pdf_reader.pdf_image_reader.pdf_image_reader import PdfImageReader
+
+# create img reader
+# pdf reader can parse image-like formats
+img_reader = PdfImageReader(config=get_config())
+# and read file example.docx
+file_name = "example.jpg"
+
+# we get unstructured file with lines and tables
+unstructured_document = img_reader.read(path=file_name, document_type="example")
+
+# let's look at content of unstructured_file, it consists of tables and lines
+print(unstructured_document.tables, unstructured_document.lines)
+
+# first of all lets look at the table
+table = unstructured_document.tables[0]
+# table consists of cells (we assume that table is rectangle)
+# so cells is list of rows and row is list of strings
+print(table.cells)
+# there is also some metadata in table
+print(table.metadata)
+
+# and now lets look at lines. lines it is list of object of class LineWithMeta
+lines = unstructured_document.lines
+# let's look at first line
+line = lines[0]
+print(line)
+# line consist of line (text), metadata, hierarchy level
+print(line.line, line.metadata, line.metadata.hierarchy_level)

examples/example_manager_input.py

@@ -1,18 +1,35 @@
 # noqa
-import os
-from werkzeug.datastructures import FileStorage
+import json
 
-from dedoc import get_config
-from dedoc.manager import DedocThreadedManager
+from dedoc import DedocManager
+from dedoc.config import get_config
 
-version_file_path = os.path.abspath(os.path.join(os.path.dirname(__file__), "../src", "..", "VERSION"))
-manager = DedocThreadedManager.from_config(config=get_config(), version=open(version_file_path).read().strip())
+manager = DedocManager(config=get_config())
 
-filename = "example.docx"
-with open(filename, 'rb') as fp:
-    file = FileStorage(fp, filename=filename)
+filename_docx = "example.docx"
+parsed_docx_document = manager.parse(file_path=filename_docx, parameters={})
 
-    parsed_document = manager.parse_file(file, parameters={"document_type": "example"})
+# save the result
+with open("result_docx.json", "w") as outfile:
+    outfile.write(json.dumps(parsed_docx_document.to_dict()))
 
-    print(parsed_document)
-    print(parsed_document.to_dict())
+filename_jpg = "example.jpg"
+parsed_jpg_document = manager.parse(file_path=filename_jpg, parameters={})
+
+# save the result
+with open("result_jpg.json", "w") as outfile:
+    outfile.write(json.dumps(parsed_jpg_document.to_dict()))
+
+filename_pdf_no_text_layer = "example_without_text_layer.pdf"
+parsed_pdf_no_text_layer_document = manager.parse(file_path=filename_pdf_no_text_layer, parameters={"pdf_with_text_layer": "false"})
+
+# save the result
+with open("result_pdf_no_text_layer.json", "w") as outfile:
+    outfile.write(json.dumps(parsed_pdf_no_text_layer_document.to_dict()))
+
+filename_pdf_with_text_layer = "example_with_text_layer.pdf"
+parsed_pdf_with_text_layer_document = manager.parse(file_path=filename_pdf_with_text_layer, parameters={"pdf_with_text_layer": "true"})
+
+# save the result
+with open("result_pdf_with_text_layer.json", "w") as outfile:
+    outfile.write(json.dumps(parsed_pdf_with_text_layer_document.to_dict()))

examples/example_pdf_parser.py

@@ -0,0 +1,60 @@
+# noqa
+from dedoc.config import get_config
+from dedoc.readers.pdf_reader.pdf_txtlayer_reader.pdf_txtlayer_reader import PdfTxtlayerReader
+from dedoc.readers.pdf_reader.pdf_image_reader.pdf_image_reader import PdfImageReader
+
+# create pdf text layer reader
+pdf_txt_layer_reader = PdfTxtlayerReader(config=get_config())
+# and read file example.docx
+file_name = "example_with_text_layer.pdf"
+
+# we get unstructured file with lines and tables
+unstructured_document = pdf_txt_layer_reader.read(path=file_name, document_type="example")
+
+# let's look at content of unstructured_file, it consists of tables and lines
+print(unstructured_document.tables, unstructured_document.lines)
+
+# first of all lets look at the table
+table = unstructured_document.tables[0]
+# table consists of cells (we assume that table is rectangle)
+# so cells is list of rows and row is list of strings
+print(table.cells)
+# there is also some metadata in table
+print(table.metadata)
+
+# and now lets look at lines. lines it is list of object of class LineWithMeta
+lines = unstructured_document.lines
+# let's look at first line
+line = lines[0]
+print(line)
+# line consist of line (text), metadata, hierarchy level
+print(line.line, line.metadata, line.metadata.hierarchy_level)
+
+
+# let's now look at the example of parsing pdf document without text layer
+# create pdf reader
+pdf_image_reader = PdfImageReader(config=get_config())
+# and read file example.docx
+file_name = "example_without_text_layer.pdf"
+
+# we get unstructured file with lines and tables
+unstructured_document = pdf_image_reader.read(path=file_name, document_type="example")
+
+# let's look at content of unstructured_file, it consists of tables and lines
+print(unstructured_document.tables, unstructured_document.lines)
+
+# first of all lets look at the table
+table = unstructured_document.tables[0]
+# table consists of cells (we assume that table is rectangle)
+# so cells is list of rows and row is list of strings
+print(table.cells)
+# there is also some metadata in table
+print(table.metadata)
+
+# and now lets look at lines. lines it is list of object of class LineWithMeta
+lines = unstructured_document.lines
+# let's look at first line
+line = lines[0]
+print(line)
+# line consist of line (text), metadata, hierarchy level
+print(line.line, line.metadata, line.metadata.hierarchy_level)

examples/example_post.py

@@ -6,13 +6,15 @@
 
 # We want to parse file example.docx
 # We will send it with requests lib
+# To parse other document types just paste correct file name below
 file_name = "example.docx"
 file_path = os.path.abspath(file_name)
 
 with open(file_path, 'rb') as file:
     # file we want to parse
     files = {'file': (file_name, file)}
     # dict with additional parameters
+    # to parse pdf with text layer add parameter "pdf_with_text_layer":"true"
     data = {"document_type": ""}
     # and now we send post request with attached file and paremeters.
     r = requests.post("http://localhost:1231/upload", files=files, data=data)