feat(proto!): make document a recursive structure

[hanxiao]

Estimated Done 25th July
This will be part of v0.5.0

Current test progress

• hello-world can be run without any problem
• unit test is missing for all new QL drivers, will do in another PR

Intro

This will be a super big PR with breaking changes. Simply put, it removes the protobuf spec of "Chunk". Now, a "Chunk" is defined as a "Document" that belongs to another "Document". Hence, "Document" is now a recursive structure. One can visit chunks via doc.chunks and get children List[Document]

This is an upgrade over the previous "Document/Chunk" concept. It makes the whole thing more general, enables advance segmenting and matching (#648)

Affects

• (Thanks to Jina's design btw) this should mostly affect drivers, not executors.
• drivers that previously work with doc/chunk will be refactored by recursively working on the doc structure.
• all previous chunk-related drivers/executors are removed because the interface is now unified.
• Executors do not need to know protobuf exists. They are "document/chunk-agnostic". All structure abstraction is taken care by Drivers.

Breakdown

• Change protobuf, delete the definition of Chunk.
• Add a BaseRecursiveDriver that works on single doc recursively in preorder and postoder.
• Refactoring drivers using the newly added BaseRecursiveDriver
  • craft.py
  • convert.py
  • prune.py
  • control.py: no need to modify
  • encode.py
  • index.py
  • score.py
  • search.py
  • reduce.py
• all YAML resources
• all unit tests (will cover in other PRs)
• all examples, freeze the jina version this time!

Migration guide (draft)

• level is not required in all drivers anymore.

• PruneDriver is now ExcludeQL, no level is required, excludes -> fields, e.g.

    - !ExcludeQL
      with:
        fields:
          - embedding
          - buffer
          - blob
          - text

• DocPruneDriver is removed, one can now use

    - !ExcludeQL
      with:
        fields:
          - buffer
          - chunks

• ChunkPruneDriver is removed, one can now use the following, as one can see it is basically the same as DocPruneDriver , just with different fields.

    - !ExcludeQL
      with:
        fields:
          - embedding
          - buffer
          - blob
          - text

• BaseCraftDriver is now CraftDriver. DocCraftDriver and ChunkCraftDriver are removed
• doc_id and chunk_id are unified as id now
• UnarySegmentDriver is removed. Now EncodeDriver works directly on doc-level embedding, extract_chunks is now refactored as extract_docs works on root-level extraction (previously it works on doc.chunks level extraction)
• BaseChunkCrafter and BaseDocCrafter are removed.
• TopkFilterDriver is removed and now is implemented with Query Language:

!SliceQL
with:
   start: 0
   end: k

• ChunkPbIndex and DocPbIndexer are now removed, please use BinaryPbIndexer
• ReqPruneDriver is removed now use

    - !ExcludeReqQL
      with:
        fields:
          - request

• _merge_topk_docs -> _merge_all

[github-actions]

Jina CLA check ✅ All Contributors have signed the CLA.

[github-actions]

This PR closes: #648

[JoanFM]

@hanxiao better do not activate integration-test as required until the PR #651 is merged, and integration test become a reality

[JoanFM]

Will solve #669