refactor: bytes to bytes_ in predefined documents #1273
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>
anna-charlotte
force-pushed
the
refactor-bytes-fields
branch
from
6e4768f
to
190c6f7
Compare
Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>
|
📝 Docs are deployed on https://ft-refactor-bytes-fields--jina-docs.netlify.app 🎉 |
samsja
approved these changes
Mar 22, 2023
RStar2022
pushed a commit
to RStar2022/docarray
that referenced
this pull request
Apr 2, 2023
* refactor: bytes to bytes_ Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * refactor: missed fields Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> --------- Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> Signed-off-by: RStar2022 <risingstar2880@gmail.com>
jupyterjazz
added a commit
that referenced
this pull request
Apr 3, 2023
* refactor: rename predefined documents (#1208) * refactor: rename Image to ImageDoc Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * refactor: rename Text to TextDoc Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * refactor: rename Audio to AudioDoc Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * refactor: rename Video to VideoDOc Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> --------- Signed-off-by: samsja <sami.jaghouar@hotmail.fr> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * feat: support other text formats (#1207) * feat: add validation function and filetypes feat: add validation function and filetypes fix: formatting Signed-off-by: rik61072@gmail.com <rik61072@gmail.com> * feat: added extension validation tests for TextURL Signed-off-by: rik61072@gmail.com <rik61072@gmail.com> * fix: fix for tests and PR codereview Signed-off-by: rik61072@gmail.com <rik61072@gmail.com> * fix: added internet mark to json dump test for texturl Signed-off-by: rik61072@gmail.com <rik61072@gmail.com> --------- Signed-off-by: rik61072@gmail.com <rik61072@gmail.com> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * refactor: refactor query builder (#1213) * refactor: query builder Signed-off-by: Johannes Messner <messnerjo@gmail.com> * docs: add guidance for query builder Signed-off-by: Johannes Messner <messnerjo@gmail.com> --------- Signed-off-by: Johannes Messner <messnerjo@gmail.com> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * feat(index): automatically convert a dict to Document (#1215) * feat: add _convert_dict_to_doc Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: str cut Signed-off-by: AnneY <evangeline-lun@foxmail.com> * feat: _get_items add return type Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: mypy Signed-off-by: AnneY <evangeline-lun@foxmail.com> * test: add test for _convert_dict_to_doc Signed-off-by: AnneY <evangeline-lun@foxmail.com> * refactor: simplify _convert_to_doc_list Signed-off-by: AnneY <evangeline-lun@foxmail.com> * docs: add docstring for _convert_dict_to_doc Signed-off-by: AnneY <evangeline-lun@foxmail.com> * feat: filter find text add return type Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: type judge Signed-off-by: AnneY <evangeline-lun@foxmail.com> * refactor: change search_field default value Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: always check length Signed-off-by: AnneY <evangeline-lun@foxmail.com> * refactor: create private result class Signed-off-by: AnneY <evangeline-lun@foxmail.com> * refactor: seperate helper method Signed-off-by: AnneY <evangeline-lun@foxmail.com> * refactor: create private result class Signed-off-by: AnneY <evangeline-lun@foxmail.com> --------- Signed-off-by: AnneY <evangeline-lun@foxmail.com> Signed-off-by: Anne Yang <evangeline-lun@foxmail.com> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * refactor: da stack full column wise (#1183) * refactor: wip add storage class Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: remove impl Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * refactor: move some tests Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add storage view Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add storage view Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add storage view Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add document view Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add tests Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * refactor: rename storage to column storage Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * refactor: order of function in da stacked Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * refactor: add inner doc in test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * refactor: rename storage to column Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add from storage Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * refactor: rename file Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * refactor: storage take columns as init Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add back getitiem Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix getatr view problem Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add get array attribute Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add setitem back Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix some tests Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix some tests Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix some tests Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add to Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add ttests Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add unstack Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add validation da set atr Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add da and any Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: remove deleitem overload Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: remove context manager Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: move tests Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: add tidi Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix copy in ndarray mypy Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add indexing sequence mixin Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add list index class Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: add cast from tuple to list Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix indexing test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add back traversle flat Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add to protobuf Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add to protobuf Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add to protobuf Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add to protobuf Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: remove unstack mode Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add proto Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add init mixin Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: add back staticmethod Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: add len to mixin Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: add del to index seq Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy del Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy del Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * docs: change DocumentArrayStacked docstring Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * docs: add docstring dor ColumnStorage Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * refactor: move docs to column storage to da stack Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * refactor: remove useless arg Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy 2 Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy 2 Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy 2 Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix nested array in da stack Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: remove type hint from setitiem Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix getitem Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * merge: merge featrewrite v2 Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: black does nto format pb file Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: remove comment Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix proto3 Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix multimodal dataset tests The initialization of all the subclasses is no longer necessary Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: fix del Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix scalar value Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: remove useless test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: raise error when docs are empty Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix tests Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix tests Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix tests Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix tests Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix type hint Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix settatr Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: apply suggestion Co-authored-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com> Co-authored-by: Joan Fontanals <joan.martinez@jina.ai> Signed-off-by: samsja <55492238+samsja@users.noreply.github.com> * fix: docstring Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: last fix hopefully Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: apply johannes suggestion Co-authored-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com> Signed-off-by: samsja <55492238+samsja@users.noreply.github.com> * feat: apply johannes suggestion Co-authored-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com> Signed-off-by: samsja <55492238+samsja@users.noreply.github.com> * fix: add comments Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: remove classvar Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: rename Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: apply johannes suggestion Co-authored-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com> Signed-off-by: samsja <55492238+samsja@users.noreply.github.com> * fix: rename Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: apply johannes suggestion Co-authored-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com> Signed-off-by: samsja <55492238+samsja@users.noreply.github.com> * fix: rename Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: rename Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: remvoe proprety Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: bring back proprety Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: apply johannes suggestion Co-authored-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com> Signed-off-by: samsja <55492238+samsja@users.noreply.github.com> --------- Signed-off-by: samsja <sami.jaghouar@hotmail.fr> Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> Signed-off-by: samsja <55492238+samsja@users.noreply.github.com> Co-authored-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> Co-authored-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com> Co-authored-by: Joan Fontanals <joan.martinez@jina.ai> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * add equal function Signed-off-by: RStar2022 <risingstar2880@gmail.com> * Add equality function Signed-off-by: RStar2022 <risingstar2880@gmail.com> * Add equal to array and document Signed-off-by: RStar2022 <risingstar2880@gmail.com> * changes a bit Signed-off-by: RStar2022 <risingstar2880@gmail.com> * add __eq__ function in abstract_comp_backend and add test_array Signed-off-by: RStar2022 <risingstar2880@gmail.com> * feat: ellipsis in tensor shape definition (#1228) * feat: ellipsis in tensor shape definition Signed-off-by: jupyterjazz <saba.sturua@jina.ai> * fix: type hint Signed-off-by: jupyterjazz <saba.sturua@jina.ai> * refactor: add tests for other tensors Signed-off-by: jupyterjazz <saba.sturua@jina.ai> * test: fix tensorflow test Signed-off-by: jupyterjazz <saba.sturua@jina.ai> * docs: update docstrings with examples Signed-off-by: jupyterjazz <saba.sturua@jina.ai> * docs: clarify needed dimensions Signed-off-by: jupyterjazz <saba.sturua@jina.ai> --------- Signed-off-by: jupyterjazz <saba.sturua@jina.ai> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * chore: update contributing guideline (#1240) * chore: update contrubing guideline Signed-off-by: samsja <55492238+samsja@users.noreply.github.com> * fix: mark dl test slow Signed-off-by: samsja <sami.jaghouar@hotmail.fr> --------- Signed-off-by: samsja <55492238+samsja@users.noreply.github.com> Signed-off-by: samsja <sami.jaghouar@hotmail.fr> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * fix equality funtion Signed-off-by: RStar2022 <risingstar2880@gmail.com> * refactor: da stack full column wise (#1183) * refactor: wip add storage class Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: remove impl Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * refactor: move some tests Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add storage view Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add storage view Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add storage view Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add document view Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add tests Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * refactor: rename storage to column storage Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * refactor: order of function in da stacked Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * refactor: add inner doc in test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * refactor: rename storage to column Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add from storage Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * refactor: rename file Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * refactor: storage take columns as init Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add back getitiem Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix getatr view problem Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add get array attribute Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add setitem back Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix some tests Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix some tests Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix some tests Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add to Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add ttests Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add unstack Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add validation da set atr Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add da and any Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: remove deleitem overload Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: remove context manager Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: move tests Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: add tidi Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix copy in ndarray mypy Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add indexing sequence mixin Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add list index class Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: add cast from tuple to list Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix indexing test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add back traversle flat Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add to protobuf Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add to protobuf Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add to protobuf Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add to protobuf Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: remove unstack mode Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add proto Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add init mixin Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: add back staticmethod Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: add len to mixin Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: add del to index seq Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy del Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy del Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * docs: change DocumentArrayStacked docstring Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * docs: add docstring dor ColumnStorage Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * refactor: move docs to column storage to da stack Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * refactor: remove useless arg Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy 2 Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy 2 Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy 2 Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix nested array in da stack Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: remove type hint from setitiem Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix getitem Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * merge: merge featrewrite v2 Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: black does nto format pb file Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: remove comment Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix proto3 Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix multimodal dataset tests The initialization of all the subclasses is no longer necessary Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: fix del Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix scalar value Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: remove useless test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: raise error when docs are empty Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix tests Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix tests Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix tests Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix tests Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix type hint Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix settatr Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: apply suggestion Co-authored-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com> Co-authored-by: Joan Fontanals <joan.martinez@jina.ai> Signed-off-by: samsja <55492238+samsja@users.noreply.github.com> * fix: docstring Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: last fix hopefully Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: apply johannes suggestion Co-authored-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com> Signed-off-by: samsja <55492238+samsja@users.noreply.github.com> * feat: apply johannes suggestion Co-authored-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com> Signed-off-by: samsja <55492238+samsja@users.noreply.github.com> * fix: add comments Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: remove classvar Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: rename Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: apply johannes suggestion Co-authored-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com> Signed-off-by: samsja <55492238+samsja@users.noreply.github.com> * fix: rename Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: apply johannes suggestion Co-authored-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com> Signed-off-by: samsja <55492238+samsja@users.noreply.github.com> * fix: rename Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: rename Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: remvoe proprety Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: bring back proprety Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: apply johannes suggestion Co-authored-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com> Signed-off-by: samsja <55492238+samsja@users.noreply.github.com> --------- Signed-off-by: samsja <sami.jaghouar@hotmail.fr> Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> Signed-off-by: samsja <55492238+samsja@users.noreply.github.com> Co-authored-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> Co-authored-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com> Co-authored-by: Joan Fontanals <joan.martinez@jina.ai> * chore: update contribution guideline (#1247) * chore: update contribution guideline Signed-off-by: samsja <55492238+samsja@users.noreply.github.com> * feat: apply alex suggestion Co-authored-by: Alex Cureton-Griffiths <alexcg1@users.noreply.github.com> Signed-off-by: samsja <55492238+samsja@users.noreply.github.com> * feat: add poetry add Signed-off-by: samsja <55492238+samsja@users.noreply.github.com> * feat: apply alex suggestion Co-authored-by: Alex Cureton-Griffiths <alexcg1@users.noreply.github.com> Signed-off-by: samsja <55492238+samsja@users.noreply.github.com> * feat: apply alex suggestion Co-authored-by: Alex Cureton-Griffiths <alexcg1@users.noreply.github.com> Signed-off-by: samsja <55492238+samsja@users.noreply.github.com> --------- Signed-off-by: samsja <55492238+samsja@users.noreply.github.com> Co-authored-by: Alex Cureton-Griffiths <alexcg1@users.noreply.github.com> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * make default search field empty string (#1249) Signed-off-by: azayz <azizbellaweid@hotmail.com> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * delete import numpy Signed-off-by: RStar2022 <risingstar2880@gmail.com> * fix key name Signed-off-by: RStar2022 <risingstar2880@gmail.com> * feat:changes-equality-operation Signed-off-by: RStar2022 <risingstar2880@gmail.com> * feat:add if Signed-off-by: RStar2022 <risingstar2880@gmail.com> * docs: add explanation about id field (#1242) * docs: add explanation about id field Signed-off-by: Johannes Messner <messnerjo@gmail.com> * docs: update docs/tutorials/add_doc_index.md Co-authored-by: Charlotte Gerhaher <charlotte.gerhaher@jina.ai> Signed-off-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com> --------- Signed-off-by: Johannes Messner <messnerjo@gmail.com> Signed-off-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com> Co-authored-by: Charlotte Gerhaher <charlotte.gerhaher@jina.ai> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * feat: shift to mkdocs (#1244) * chore: remove sphinx and mkdocs Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: add DocumentArrayStacked to init py Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add mkdocs Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: apply alex setting Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix ci Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix import Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix import Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix ci Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix ci Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix ci Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: uses sphinx style Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: udpate docs Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add inherited menbers Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add more Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: code block syntax highlight in docstring Signed-off-by: Alex C-G <alexcg@outlook.com> * feat: add typing Signed-off-by: samsja <sami.jaghouar@hotmail.fr> --------- Signed-off-by: samsja <sami.jaghouar@hotmail.fr> Signed-off-by: Alex C-G <alexcg@outlook.com> Co-authored-by: Alex C-G <alexcg@outlook.com> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * fix:equal func Signed-off-by: RStar2022 <risingstar2880@gmail.com> * refactor: rename filter to filter_docs to avoid shadowing of filtern (#1257) Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * feat: ad user defined mapping for python type to db type (#1252) * feat: user defined mapping for python type to db type Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * feat: check if col_type available Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * test: add test for base classes Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: clean up Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: test Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * docs: add documentation for db type and python type Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * docs: add doumentation for runtime config Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: add and test illegal col types Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> --------- Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * fix:tensor type func Signed-off-by: RStar2022 <risingstar2880@gmail.com> * fix: comp_backend code delete Signed-off-by: RStar2022 <risingstar2880@gmail.com> * feat(index): index data with union types (#1220) * refactor: split flattening into separate method Signed-off-by: Johannes Messner <messnerjo@gmail.com> * refactor: don't build column info during schema check Signed-off-by: Johannes Messner <messnerjo@gmail.com> * feat: allos unions and optional in indexed data Signed-off-by: Johannes Messner <messnerjo@gmail.com> * fix: mypy Signed-off-by: Johannes Messner <messnerjo@gmail.com> * fix: mypy Signed-off-by: Johannes Messner <messnerjo@gmail.com> * fix: import from typing inspect instead of typing Signed-off-by: Johannes Messner <messnerjo@gmail.com> * fix: equality and hash for parametrized tensors Signed-off-by: Johannes Messner <messnerjo@gmail.com> * test: add test for flatten docs Signed-off-by: Johannes Messner <messnerjo@gmail.com> * refactor: apply suggestions Signed-off-by: Johannes Messner <messnerjo@gmail.com> * docs: better docstrings Signed-off-by: Johannes Messner <messnerjo@gmail.com> * refactor: use construct to create docarray Signed-off-by: Johannes Messner <messnerjo@gmail.com> * fix: check for nonetype Signed-off-by: Johannes Messner <messnerjo@gmail.com> * fix: none in equals check Signed-off-by: Johannes Messner <messnerjo@gmail.com> --------- Signed-off-by: Johannes Messner <messnerjo@gmail.com> Signed-off-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * fix eq Signed-off-by: RStar2022 <risingstar2880@gmail.com> * fix: move test to integration test (#1260) * fix: move test to integration test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: use a different runner for doc index Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: use a different runner for doc index Signed-off-by: samsja <sami.jaghouar@hotmail.fr> --------- Signed-off-by: samsja <sami.jaghouar@hotmail.fr> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * fix: disable pycharm da property detection (#1262) * fix: disable unresolved attr detection for da in pycharm Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: add docstring Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: call super instead of pass Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: getattr Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: getattribute Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> --------- Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * feat: add minimal logger (#1254) * feat: add minimal logger Signed-off-by: jupyterjazz <saba.sturua@jina.ai> * docs: add an example in contributing Signed-off-by: jupyterjazz <saba.sturua@jina.ai> * docs: try different format Signed-off-by: jupyterjazz <saba.sturua@jina.ai> * refactor: set one logger name Signed-off-by: jupyterjazz <saba.sturua@jina.ai> * docs: add quotes Signed-off-by: jupyterjazz <saba.sturua@jina.ai> * refactor: put logger as a class attr Signed-off-by: jupyterjazz <saba.sturua@jina.ai> * docs: small change Signed-off-by: jupyterjazz <saba.sturua@jina.ai> * fix: typo Signed-off-by: jupyterjazz <saba.sturua@jina.ai> * docs: rephrase text Signed-off-by: jupyterjazz <saba.sturua@jina.ai> * refactor: requested changes Signed-off-by: jupyterjazz <saba.sturua@jina.ai> --------- Signed-off-by: jupyterjazz <saba.sturua@jina.ai> Signed-off-by: Saba Sturua <45267439+jupyterjazz@users.noreply.github.com> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * fix: bytes type in `TextDoc` and `VideoDoc` (#1270) * fix: bytes type is None in predefined documents Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: remove defaults to none Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> --------- Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * refactor: doc index structure (#1266) * refactor: doc index structure Signed-off-by: jupyterjazz <saba.sturua@jina.ai> * fix: import unused error Signed-off-by: jupyterjazz <saba.sturua@jina.ai> * fix: run black Signed-off-by: jupyterjazz <saba.sturua@jina.ai> * refactor: remove noqa Signed-off-by: jupyterjazz <saba.sturua@jina.ai> * refactor: doc_index to index Signed-off-by: jupyterjazz <saba.sturua@jina.ai> * refactor: small changes Signed-off-by: jupyterjazz <saba.sturua@jina.ai> * refactor: readme Signed-off-by: jupyterjazz <saba.sturua@jina.ai> * refactor: add init file Signed-off-by: jupyterjazz <saba.sturua@jina.ai> * refactor: move docindex outside integration tests Signed-off-by: jupyterjazz <saba.sturua@jina.ai> * refactor: adjust config paths Signed-off-by: jupyterjazz <saba.sturua@jina.ai> --------- Signed-off-by: jupyterjazz <saba.sturua@jina.ai> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * refactor(da): remove tensor type from `DocumentArray` init (#1268) * fix: remove tensor type from DocumentArray Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix tensorflow test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: docstrng Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: apply charllote suggestion Co-authored-by: Charlotte Gerhaher <charlotte.gerhaher@jina.ai> Signed-off-by: samsja <55492238+samsja@users.noreply.github.com> * feat: apply saba suggestion Co-authored-by: Saba Sturua <45267439+jupyterjazz@users.noreply.github.com> Signed-off-by: samsja <55492238+samsja@users.noreply.github.com> --------- Signed-off-by: samsja <sami.jaghouar@hotmail.fr> Signed-off-by: samsja <55492238+samsja@users.noreply.github.com> Co-authored-by: Charlotte Gerhaher <charlotte.gerhaher@jina.ai> Co-authored-by: Saba Sturua <45267439+jupyterjazz@users.noreply.github.com> * refactor: bytes to bytes_ in predefined documents (#1273) * refactor: bytes to bytes_ Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * refactor: missed fields Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> --------- Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * fix: doc summary for dict and set attributes (#1279) Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * feat: add `get_paths()` instead of v1 `from_files()` (#1267) * feat: add from_files() Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * feat: add da classmethod from_files() Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * docs: update docstring Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * docs: add example usage Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: add get_paths, rm from_files Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: add print to debug ci Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: test Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: apply suggestions from code review Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> --------- Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * fix: proto ser and deser for nested tuple/dict/list (#1278) * feat: add failing test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * refactor: shorten if else statememt Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * refactor: shorten if else statememt Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix proto and list Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix proto and dict Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add very complex test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: fix pure tensor stuff Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: fix pure tensor stuff Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix from protobuf Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix from protobuf tensorflow Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: add more test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: add more test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: import ndarray Signed-off-by: samsja <sami.jaghouar@hotmail.fr> --------- Signed-off-by: samsja <sami.jaghouar@hotmail.fr> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * feat: create documents from dict (#1283) * feat: create documents from dict Signed-off-by: jupyterjazz <saba.sturua@jina.ai> * fix: ignore type Signed-off-by: jupyterjazz <saba.sturua@jina.ai> * refactor: change fn names Signed-off-by: jupyterjazz <saba.sturua@jina.ai> --------- Signed-off-by: jupyterjazz <saba.sturua@jina.ai> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * docs: fix up english (#1285) Signed-off-by: Alex C-G <alexcg@outlook.com> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * fix: add int, float and others to doc summary (#1287) Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * fix: hnswlib doc index (#1277) * fix: support for torch and tf Signed-off-by: Johannes Messner <messnerjo@gmail.com> * fix: allow arbitrary payloads, including tensors Signed-off-by: Johannes Messner <messnerjo@gmail.com> * test: mark tf tests Signed-off-by: Johannes Messner <messnerjo@gmail.com> * test: another attempt at fixing tf tests Signed-off-by: Johannes Messner <messnerjo@gmail.com> * test: remove parametrization of test Signed-off-by: Johannes Messner <messnerjo@gmail.com> * test: fix test Signed-off-by: Johannes Messner <messnerjo@gmail.com> * fix: add suggestion Signed-off-by: Johannes Messner <messnerjo@gmail.com> * ci: exlude tf tests from index tests Signed-off-by: Johannes Messner <messnerjo@gmail.com> --------- Signed-off-by: Johannes Messner <messnerjo@gmail.com> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * fix Signed-off-by: RStar2022 <risingstar2880@gmail.com> * fix: tensorflow Signed-off-by: RStar2022 <risingstar2880@gmail.com> * feat(test): DocumentArray method tests similar to list methods like reverse, sort, remove, pop (#1291) * feat: isort format fix Signed-off-by: agaraman0 <agaraman0@gmail.com> * refactor: comment fixes Signed-off-by: agaraman0 <agaraman0@gmail.com> * refactor: comment fixes Signed-off-by: agaraman0 <agaraman0@gmail.com> --------- Signed-off-by: agaraman0 <agaraman0@gmail.com> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * fix:doc Signed-off-by: RStar2022 <risingstar2880@gmail.com> * feat: implement push/pull interface from JAC, file and s3 (#1182) * refactor: move streaming serialization into separate method Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * refactor: add binary io like protocol definition Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * feat: ported push pull to JAC Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: protocol is not in 3.7 typing Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: make mypy happy Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: patch missing waterfall Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * refactor: jit import backends Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * feat: implement cache in jinaai pull Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: add hubble dependency to jina group Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * refactor: better division of concerns Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * feat: add concept of namespace Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: ignore missing hubble stubs Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * feat: streaming protocol stubs Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * refactor: make more general buffered caching reader Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * test: add tests for hubble pushpull Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * test: add tests for file backend Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: remove hubble dependency from jina group This reverts commit b3044213d58517becb9d71194af34f3833560ebc. Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * feat: implement push pull for local filesystem Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * test: test concurrent pushes and pulls in file protocol Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: resolve concurrent pushes and pulls correctly Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: rename text to textdoc Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * feat: added some logging Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * test: s3 tests Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * feat: s3 pushpull Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: add smart open dependency Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: add smart opens silly python bound Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * test: update hubble tests (failing) Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: fix delete return in hubble pushpull Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * Revert "fix: add smart open dependency" This reverts commit cf78c6cc6d2b367501d2358c18773a456426a448. This reverts commit eb0e52b4c521f2b638bf5de850701546a4996bc3. Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: add hubble and smart open dependencies Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: mypy fixes Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * ci: allow tests to see jina auth token Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * feat: add progress bars for streaming Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * style: blacken Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * feat: buffer writes to s3 Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: mypy no like sequence Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: make progress bar quieter when disabled Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * test: skip failing tests Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * feat: add tables when listing Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * test: add jina auth token to uncaped test Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * test: mock s3 tests with minio container Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: silly error that cost me 2 hours of life Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * test: use tolerance ratio in file tests Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * feat: add caching to s3 pull Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * feat: add log messages for unused parameters Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * refactor: take out unneeded buffering smart open already buffers Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * feat: pick fastest protocol compression configuration for s3 Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * test: bump tolerance ratio for s3 test Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * refactor: reduce code duplication Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * refactor: put reader chunk size constant at top of file Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * test: reduce reader chunk size for memory tests Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: multipart uploads get stuck frequently lets just do big uploads for now... Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * docs: add docstrings to mixin and file backend Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * docs: add docstring for s3 and hubble backends Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * test: remove unused test Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * refactor: use literal in protocol Co-authored-by: samsja <55492238+samsja@users.noreply.github.com> Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * refactor: protocols dont need to be inherited Co-authored-by: samsja <55492238+samsja@users.noreply.github.com> Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: add make mypy happy with the literals Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: literals not in 3.7 Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * refactor: move mixin out of init file Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * refactor: move cache path resolution to utils Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * feat: cache path is only evaluated once Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * refactor: loading backends makes more sense as debug log Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * tests: add slow and internet marks Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * refactor: pin image tag Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * refactor: use abc instead of protocol for typing backends Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: revert - add hubble and smart open dependencies This reverts commit 1d1d2eeaf2b51be6ef00e6ab6ee5b9fd1bcf1d92. Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: add hubble and aws dependencies Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * refactor: change all push pull mixin methods to class methods Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: misstyped class method self reference Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * refactor: rename pushpull to docstore and use more classmethods Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * refactor: separate remote backend implementations from mixin Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: missed import refactor Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * refactor: change submodule name to store Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * refactor: remove list and delete from mixin Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * tests: clear all the garbage in ci account Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * tests: skip test that is broken on ci Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * refactor: standardize naming to jac Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> --------- Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> Co-authored-by: samsja <55492238+samsja@users.noreply.github.com> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * refactor: rename `Document` to `Doc` (#1293) * refactor: rename document to doc Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * refactor: rename document to doc in da Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * refactor: rename base doc in md files Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * refactor: rename base base document ot base doc Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix(docs): fix docs building Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: ingore hubble test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: ingore hubble test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> --------- Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * chore(docs): add ci and fix docs ui (#1295) * refactor: rename document to doc Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * refactor: rename document to doc in da Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * refactor: rename base doc in md files Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * refactor: rename base base document ot base doc Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix(docs): fix docs building Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: ingore hubble test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: ingore hubble test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * docs: add userguide install Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * docs: add awesome-pages Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * docs: add install Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * docs: rename tutorials to how to Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * chore: add pre commit blacken docs Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * chore: add blacken docs Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * docs: arr warning docarray version Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * docs: repo url Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * docs: add social Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * docs: add logo Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * docs: add first step emtpy page Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * docs: add document docs Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add markdown documentation test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * docs: remove content Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * docs: fix ci Signed-off-by: samsja <sami.jaghouar@hotmail.fr> --------- Signed-off-by: samsja <sami.jaghouar@hotmail.fr> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * chore: add docstring test (#1298) * wip Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: cleanup namespace utils Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add docstring test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> --------- Signed-off-by: samsja <sami.jaghouar@hotmail.fr> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * fix: rename DocArrayProto to DocumentArrayProto (#1297) Signed-off-by: samsja <sami.jaghouar@hotmail.fr> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * fix: docstring polish typing (#1299) * wip Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: cleanup namespace utils Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add docstring test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix video url docstring Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix text url Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix image url Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fic audio url Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: mesh 3d url Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: mesh 3d url Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: remove useless data Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix docstring ndarray and torch tensor Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix docstring ndarray and torch tensor Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix fix audio url and audio ndarray Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix fix audio url and audio ndarray Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix video tensor Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix video tensor Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix audio bytes Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: video and image bytes Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * docs: move typing section Signed-off-by: samsja <sami.jaghouar@hotmail.fr> --------- Signed-off-by: samsja <sami.jaghouar@hotmail.fr> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * fix: remove files (#1305) Signed-off-by: samsja <sami.jaghouar@hotmail.fr> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * fix: flatten schema of abstract index (#1294) * fix: flatten schema of abstract index Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: _convert_dict_to_doc Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: catch exception when flatten schema Signed-off-by: AnneY <evangeline-lun@foxmail.com> * refactor: remove useless assignemnt Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: use Abstractensor as tensor doc_type Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: add AbstractTensor to hnswlib Signed-off-by: AnneY <evangeline-lun@foxmail.com> * docs: AbstractTensor as doc_type Signed-off-by: AnneY <evangeline-lun@foxmail.com> * docs: complete description about AbstracTensor Co-authored-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com> Signed-off-by: Anne Yang <evangeline-lun@foxmail.com> --------- Signed-off-by: AnneY <evangeline-lun@foxmail.com> Signed-off-by: Anne Yang <evangeline-lun@foxmail.com> Co-authored-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * docs: add utils section (#1307) * feat: add utils for map to docs and fix docstring Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add utils for map to docs and fix docstring Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add utils for find and fix docstring Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix video ndaray docstrng Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix video find docstrng Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix map docstring Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix fileter docstring Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix add reduce Signed-off-by: samsja <sami.jaghouar@hotmail.fr> --------- Signed-off-by: samsja <sami.jaghouar@hotmail.fr> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * docs: fix docstring example of find_batched (#1308) Signed-off-by: Johannes Messner <messnerjo@gmail.com> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * docs: fix map docstring (#1311) * fix: fix utils Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix map Signed-off-by: samsja <sami.jaghouar@hotmail.fr> --------- Signed-off-by: samsja <sami.jaghouar@hotmail.fr> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * feat: elasticsearch document index (#1196) * feat: __init__ of ElasticDocumentIndex Signed-off-by: AnneY <evangeline-lun@foxmail.com> * feat: add index func Signed-off-by: AnneY <evangeline-lun@foxmail.com> * feat: get and del funcs Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: init and index creation Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: __init__ and _index Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: _get_items Signed-off-by: AnneY <evangeline-lun@foxmail.com> * feat: add _find Signed-off-by: AnneY <evangeline-lun@foxmail.com> * feat: add filter text and their batch version Signed-off-by: AnneY <evangeline-lun@foxmail.com> * feat: store id and get nested doc Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: vector cannot be all zero Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: __getitem__ raise error Signed-off-by: AnneY <evangeline-lun@foxmail.com> * feat: support more python types Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: mypy Signed-off-by: AnneY <evangeline-lun@foxmail.com> * test: elastic index tests Signed-off-by: AnneY <evangeline-lun@foxmail.com> * test: comment scripts before ci setup Signed-off-by: AnneY <evangeline-lun@foxmail.com> * chore: add elasticsearch dependency to poetry Signed-off-by: AnneY <evangeline-lun@foxmail.com> * test: elastic index ci setup Signed-off-by: AnneY <evangeline-lun@foxmail.com> * feat: add num_candidates to rumtime config Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: let user pass index_settings Signed-off-by: AnneY <evangeline-lun@foxmail.com> * feat: degrade to v7 and add query builder Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: remove elastic_transport Signed-off-by: AnneY <evangeline-lun@foxmail.com> * feat: minor features Signed-off-by: AnneY <evangeline-lun@foxmail.com> * refactor: style fix Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: fix mypy Signed-off-by: AnneY <evangeline-lun@foxmail.com> * feat: add chunk size to runtime config Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: chunk size Signed-off-by: AnneY <evangeline-lun@foxmail.com> * feat: add chunk_size to funcs Signed-off-by: AnneY <evangeline-lun@foxmail.com> * feat: rewrite elastic v7 query builder Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: poetry Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: db_type should be elastic types Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: minor adjustment Signed-off-by: AnneY <evangeline-lun@foxmail.com> * refactor: rename elastic index files Signed-off-by: AnneY <evangeline-lun@foxmail.com> * refactor: remove comments Signed-off-by: AnneY <evangeline-lun@foxmail.com> * feat: rename, batch operations, etc Signed-off-by: AnneY <evangeline-lun@foxmail.com> * test: add test for persistency and col config Signed-off-by: AnneY <evangeline-lun@foxmail.com> * feat: support more field types and subclass Signed-off-by: AnneY <evangeline-lun@foxmail.com> * feat: support more python types Signed-off-by: AnneY <evangeline-lun@foxmail.com> * test: tf, tensor and more elastic field types Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: elastic should be optional in toml Co-authored-by: Charlotte Gerhaher <charlotte.gerhaher@jina.ai> Signed-off-by: Anne Yang <evangeline-lun@foxmail.com> * refactor: rename class Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: change Dict to Mapping Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: add AbstractTensor Signed-off-by: AnneY <evangeline-lun@foxmail.com> * test: rename class and add tests Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: poetry Signed-off-by: AnneY <evangeline-lun@foxmail.com> --------- Signed-off-by: AnneY <evangeline-lun@foxmail.com> Signed-off-by: Anne Yang <evangeline-lun@foxmail.com> Co-authored-by: Charlotte Gerhaher <charlotte.gerhaher@jina.ai> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * refactor: map_docs_batch to map_docs_batched (#1312) Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * refactor: map_docs_batch to map_docs_batched (#1312) Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * feat: torch backend basic operation tests (#1306) Signed-off-by: agaraman0 <agaraman0@gmail.com> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * chore: add instructions to pip installs and group extras (#1281) * chore: group extras and add instructions for pip installs Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: throw runtime error with install instructions for hnswlib Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * feat: add instructions for video imports Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * feat: add instructions for audio imports Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * feat: add instructions for 3d imports Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * feat: add instructions for image imports Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: import only audiosegment from pydub Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: generalize audio and image imports Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: add instructions for web imports Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: add instructions for web imports Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: add instructions for protobuf imports Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: add instructions for lz4 imports Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: fastapi import Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: revert changes in protobuf import Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: add instructions for torch, without raising error Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: add instructions for torch, with raising error Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: add instructions for tensorflow Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: base doc io imports Sign…
samsja
added a commit
that referenced
this pull request
Apr 3, 2023
* refactor: dummy change Signed-off-by: jupyterjazz <saba.sturua@jina.ai> * feat: implement == for document and document array (#1224) * refactor: rename predefined documents (#1208) * refactor: rename Image to ImageDoc Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * refactor: rename Text to TextDoc Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * refactor: rename Audio to AudioDoc Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * refactor: rename Video to VideoDOc Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> --------- Signed-off-by: samsja <sami.jaghouar@hotmail.fr> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * feat: support other text formats (#1207) * feat: add validation function and filetypes feat: add validation function and filetypes fix: formatting Signed-off-by: rik61072@gmail.com <rik61072@gmail.com> * feat: added extension validation tests for TextURL Signed-off-by: rik61072@gmail.com <rik61072@gmail.com> * fix: fix for tests and PR codereview Signed-off-by: rik61072@gmail.com <rik61072@gmail.com> * fix: added internet mark to json dump test for texturl Signed-off-by: rik61072@gmail.com <rik61072@gmail.com> --------- Signed-off-by: rik61072@gmail.com <rik61072@gmail.com> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * refactor: refactor query builder (#1213) * refactor: query builder Signed-off-by: Johannes Messner <messnerjo@gmail.com> * docs: add guidance for query builder Signed-off-by: Johannes Messner <messnerjo@gmail.com> --------- Signed-off-by: Johannes Messner <messnerjo@gmail.com> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * feat(index): automatically convert a dict to Document (#1215) * feat: add _convert_dict_to_doc Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: str cut Signed-off-by: AnneY <evangeline-lun@foxmail.com> * feat: _get_items add return type Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: mypy Signed-off-by: AnneY <evangeline-lun@foxmail.com> * test: add test for _convert_dict_to_doc Signed-off-by: AnneY <evangeline-lun@foxmail.com> * refactor: simplify _convert_to_doc_list Signed-off-by: AnneY <evangeline-lun@foxmail.com> * docs: add docstring for _convert_dict_to_doc Signed-off-by: AnneY <evangeline-lun@foxmail.com> * feat: filter find text add return type Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: type judge Signed-off-by: AnneY <evangeline-lun@foxmail.com> * refactor: change search_field default value Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: always check length Signed-off-by: AnneY <evangeline-lun@foxmail.com> * refactor: create private result class Signed-off-by: AnneY <evangeline-lun@foxmail.com> * refactor: seperate helper method Signed-off-by: AnneY <evangeline-lun@foxmail.com> * refactor: create private result class Signed-off-by: AnneY <evangeline-lun@foxmail.com> --------- Signed-off-by: AnneY <evangeline-lun@foxmail.com> Signed-off-by: Anne Yang <evangeline-lun@foxmail.com> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * refactor: da stack full column wise (#1183) * refactor: wip add storage class Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: remove impl Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * refactor: move some tests Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add storage view Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add storage view Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add storage view Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add document view Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add tests Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * refactor: rename storage to column storage Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * refactor: order of function in da stacked Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * refactor: add inner doc in test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * refactor: rename storage to column Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add from storage Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * refactor: rename file Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * refactor: storage take columns as init Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add back getitiem Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix getatr view problem Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add get array attribute Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add setitem back Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix some tests Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix some tests Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix some tests Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add to Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add ttests Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add unstack Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add validation da set atr Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add da and any Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: remove deleitem overload Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: remove context manager Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: move tests Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: add tidi Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix copy in ndarray mypy Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add indexing sequence mixin Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add list index class Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: add cast from tuple to list Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix indexing test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add back traversle flat Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add to protobuf Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add to protobuf Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add to protobuf Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add to protobuf Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: remove unstack mode Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add proto Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add init mixin Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: add back staticmethod Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: add len to mixin Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: add del to index seq Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy del Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy del Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * docs: change DocumentArrayStacked docstring Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * docs: add docstring dor ColumnStorage Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * refactor: move docs to column storage to da stack Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * refactor: remove useless arg Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy 2 Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy 2 Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy 2 Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix nested array in da stack Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: remove type hint from setitiem Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix getitem Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * merge: merge featrewrite v2 Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: black does nto format pb file Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: remove comment Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix proto3 Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix multimodal dataset tests The initialization of all the subclasses is no longer necessary Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: fix del Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix scalar value Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: remove useless test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: raise error when docs are empty Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix tests Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix tests Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix tests Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix tests Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix type hint Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix settatr Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: apply suggestion Co-authored-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com> Co-authored-by: Joan Fontanals <joan.martinez@jina.ai> Signed-off-by: samsja <55492238+samsja@users.noreply.github.com> * fix: docstring Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: last fix hopefully Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: apply johannes suggestion Co-authored-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com> Signed-off-by: samsja <55492238+samsja@users.noreply.github.com> * feat: apply johannes suggestion Co-authored-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com> Signed-off-by: samsja <55492238+samsja@users.noreply.github.com> * fix: add comments Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: remove classvar Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: rename Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: apply johannes suggestion Co-authored-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com> Signed-off-by: samsja <55492238+samsja@users.noreply.github.com> * fix: rename Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: apply johannes suggestion Co-authored-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com> Signed-off-by: samsja <55492238+samsja@users.noreply.github.com> * fix: rename Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: rename Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: remvoe proprety Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: bring back proprety Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: apply johannes suggestion Co-authored-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com> Signed-off-by: samsja <55492238+samsja@users.noreply.github.com> --------- Signed-off-by: samsja <sami.jaghouar@hotmail.fr> Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> Signed-off-by: samsja <55492238+samsja@users.noreply.github.com> Co-authored-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> Co-authored-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com> Co-authored-by: Joan Fontanals <joan.martinez@jina.ai> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * add equal function Signed-off-by: RStar2022 <risingstar2880@gmail.com> * Add equality function Signed-off-by: RStar2022 <risingstar2880@gmail.com> * Add equal to array and document Signed-off-by: RStar2022 <risingstar2880@gmail.com> * changes a bit Signed-off-by: RStar2022 <risingstar2880@gmail.com> * add __eq__ function in abstract_comp_backend and add test_array Signed-off-by: RStar2022 <risingstar2880@gmail.com> * feat: ellipsis in tensor shape definition (#1228) * feat: ellipsis in tensor shape definition Signed-off-by: jupyterjazz <saba.sturua@jina.ai> * fix: type hint Signed-off-by: jupyterjazz <saba.sturua@jina.ai> * refactor: add tests for other tensors Signed-off-by: jupyterjazz <saba.sturua@jina.ai> * test: fix tensorflow test Signed-off-by: jupyterjazz <saba.sturua@jina.ai> * docs: update docstrings with examples Signed-off-by: jupyterjazz <saba.sturua@jina.ai> * docs: clarify needed dimensions Signed-off-by: jupyterjazz <saba.sturua@jina.ai> --------- Signed-off-by: jupyterjazz <saba.sturua@jina.ai> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * chore: update contributing guideline (#1240) * chore: update contrubing guideline Signed-off-by: samsja <55492238+samsja@users.noreply.github.com> * fix: mark dl test slow Signed-off-by: samsja <sami.jaghouar@hotmail.fr> --------- Signed-off-by: samsja <55492238+samsja@users.noreply.github.com> Signed-off-by: samsja <sami.jaghouar@hotmail.fr> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * fix equality funtion Signed-off-by: RStar2022 <risingstar2880@gmail.com> * refactor: da stack full column wise (#1183) * refactor: wip add storage class Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: remove impl Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * refactor: move some tests Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add storage view Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add storage view Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add storage view Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add document view Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add tests Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * refactor: rename storage to column storage Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * refactor: order of function in da stacked Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * refactor: add inner doc in test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * refactor: rename storage to column Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add from storage Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * refactor: rename file Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * refactor: storage take columns as init Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add back getitiem Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix getatr view problem Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add get array attribute Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add setitem back Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix some tests Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix some tests Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix some tests Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add to Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add ttests Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add unstack Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add validation da set atr Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add da and any Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: remove deleitem overload Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: remove context manager Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: move tests Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: add tidi Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix copy in ndarray mypy Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add indexing sequence mixin Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add list index class Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: add cast from tuple to list Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix indexing test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add back traversle flat Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add to protobuf Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add to protobuf Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add to protobuf Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add to protobuf Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: remove unstack mode Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add proto Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add init mixin Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: add back staticmethod Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: add len to mixin Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: add del to index seq Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy del Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy del Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * docs: change DocumentArrayStacked docstring Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * docs: add docstring dor ColumnStorage Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * refactor: move docs to column storage to da stack Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * refactor: remove useless arg Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy 2 Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy 2 Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy 2 Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix nested array in da stack Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: remove type hint from setitiem Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix getitem Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * merge: merge featrewrite v2 Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: black does nto format pb file Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: remove comment Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix proto3 Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix multimodal dataset tests The initialization of all the subclasses is no longer necessary Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: fix del Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix scalar value Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: remove useless test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: raise error when docs are empty Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix tests Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix tests Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix tests Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix tests Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix type hint Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix settatr Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: apply suggestion Co-authored-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com> Co-authored-by: Joan Fontanals <joan.martinez@jina.ai> Signed-off-by: samsja <55492238+samsja@users.noreply.github.com> * fix: docstring Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: last fix hopefully Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: apply johannes suggestion Co-authored-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com> Signed-off-by: samsja <55492238+samsja@users.noreply.github.com> * feat: apply johannes suggestion Co-authored-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com> Signed-off-by: samsja <55492238+samsja@users.noreply.github.com> * fix: add comments Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: remove classvar Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: rename Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: apply johannes suggestion Co-authored-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com> Signed-off-by: samsja <55492238+samsja@users.noreply.github.com> * fix: rename Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: apply johannes suggestion Co-authored-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com> Signed-off-by: samsja <55492238+samsja@users.noreply.github.com> * fix: rename Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: rename Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: remvoe proprety Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: bring back proprety Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: apply johannes suggestion Co-authored-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com> Signed-off-by: samsja <55492238+samsja@users.noreply.github.com> --------- Signed-off-by: samsja <sami.jaghouar@hotmail.fr> Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> Signed-off-by: samsja <55492238+samsja@users.noreply.github.com> Co-authored-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> Co-authored-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com> Co-authored-by: Joan Fontanals <joan.martinez@jina.ai> * chore: update contribution guideline (#1247) * chore: update contribution guideline Signed-off-by: samsja <55492238+samsja@users.noreply.github.com> * feat: apply alex suggestion Co-authored-by: Alex Cureton-Griffiths <alexcg1@users.noreply.github.com> Signed-off-by: samsja <55492238+samsja@users.noreply.github.com> * feat: add poetry add Signed-off-by: samsja <55492238+samsja@users.noreply.github.com> * feat: apply alex suggestion Co-authored-by: Alex Cureton-Griffiths <alexcg1@users.noreply.github.com> Signed-off-by: samsja <55492238+samsja@users.noreply.github.com> * feat: apply alex suggestion Co-authored-by: Alex Cureton-Griffiths <alexcg1@users.noreply.github.com> Signed-off-by: samsja <55492238+samsja@users.noreply.github.com> --------- Signed-off-by: samsja <55492238+samsja@users.noreply.github.com> Co-authored-by: Alex Cureton-Griffiths <alexcg1@users.noreply.github.com> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * make default search field empty string (#1249) Signed-off-by: azayz <azizbellaweid@hotmail.com> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * delete import numpy Signed-off-by: RStar2022 <risingstar2880@gmail.com> * fix key name Signed-off-by: RStar2022 <risingstar2880@gmail.com> * feat:changes-equality-operation Signed-off-by: RStar2022 <risingstar2880@gmail.com> * feat:add if Signed-off-by: RStar2022 <risingstar2880@gmail.com> * docs: add explanation about id field (#1242) * docs: add explanation about id field Signed-off-by: Johannes Messner <messnerjo@gmail.com> * docs: update docs/tutorials/add_doc_index.md Co-authored-by: Charlotte Gerhaher <charlotte.gerhaher@jina.ai> Signed-off-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com> --------- Signed-off-by: Johannes Messner <messnerjo@gmail.com> Signed-off-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com> Co-authored-by: Charlotte Gerhaher <charlotte.gerhaher@jina.ai> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * feat: shift to mkdocs (#1244) * chore: remove sphinx and mkdocs Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: add DocumentArrayStacked to init py Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add mkdocs Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: apply alex setting Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix ci Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix import Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix import Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix ci Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix ci Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix ci Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: uses sphinx style Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: udpate docs Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add inherited menbers Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add more Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: code block syntax highlight in docstring Signed-off-by: Alex C-G <alexcg@outlook.com> * feat: add typing Signed-off-by: samsja <sami.jaghouar@hotmail.fr> --------- Signed-off-by: samsja <sami.jaghouar@hotmail.fr> Signed-off-by: Alex C-G <alexcg@outlook.com> Co-authored-by: Alex C-G <alexcg@outlook.com> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * fix:equal func Signed-off-by: RStar2022 <risingstar2880@gmail.com> * refactor: rename filter to filter_docs to avoid shadowing of filtern (#1257) Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * feat: ad user defined mapping for python type to db type (#1252) * feat: user defined mapping for python type to db type Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * feat: check if col_type available Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * test: add test for base classes Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: clean up Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: test Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * docs: add documentation for db type and python type Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * docs: add doumentation for runtime config Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: add and test illegal col types Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> --------- Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * fix:tensor type func Signed-off-by: RStar2022 <risingstar2880@gmail.com> * fix: comp_backend code delete Signed-off-by: RStar2022 <risingstar2880@gmail.com> * feat(index): index data with union types (#1220) * refactor: split flattening into separate method Signed-off-by: Johannes Messner <messnerjo@gmail.com> * refactor: don't build column info during schema check Signed-off-by: Johannes Messner <messnerjo@gmail.com> * feat: allos unions and optional in indexed data Signed-off-by: Johannes Messner <messnerjo@gmail.com> * fix: mypy Signed-off-by: Johannes Messner <messnerjo@gmail.com> * fix: mypy Signed-off-by: Johannes Messner <messnerjo@gmail.com> * fix: import from typing inspect instead of typing Signed-off-by: Johannes Messner <messnerjo@gmail.com> * fix: equality and hash for parametrized tensors Signed-off-by: Johannes Messner <messnerjo@gmail.com> * test: add test for flatten docs Signed-off-by: Johannes Messner <messnerjo@gmail.com> * refactor: apply suggestions Signed-off-by: Johannes Messner <messnerjo@gmail.com> * docs: better docstrings Signed-off-by: Johannes Messner <messnerjo@gmail.com> * refactor: use construct to create docarray Signed-off-by: Johannes Messner <messnerjo@gmail.com> * fix: check for nonetype Signed-off-by: Johannes Messner <messnerjo@gmail.com> * fix: none in equals check Signed-off-by: Johannes Messner <messnerjo@gmail.com> --------- Signed-off-by: Johannes Messner <messnerjo@gmail.com> Signed-off-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * fix eq Signed-off-by: RStar2022 <risingstar2880@gmail.com> * fix: move test to integration test (#1260) * fix: move test to integration test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: use a different runner for doc index Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: use a different runner for doc index Signed-off-by: samsja <sami.jaghouar@hotmail.fr> --------- Signed-off-by: samsja <sami.jaghouar@hotmail.fr> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * fix: disable pycharm da property detection (#1262) * fix: disable unresolved attr detection for da in pycharm Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: add docstring Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: call super instead of pass Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: getattr Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: getattribute Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> --------- Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * feat: add minimal logger (#1254) * feat: add minimal logger Signed-off-by: jupyterjazz <saba.sturua@jina.ai> * docs: add an example in contributing Signed-off-by: jupyterjazz <saba.sturua@jina.ai> * docs: try different format Signed-off-by: jupyterjazz <saba.sturua@jina.ai> * refactor: set one logger name Signed-off-by: jupyterjazz <saba.sturua@jina.ai> * docs: add quotes Signed-off-by: jupyterjazz <saba.sturua@jina.ai> * refactor: put logger as a class attr Signed-off-by: jupyterjazz <saba.sturua@jina.ai> * docs: small change Signed-off-by: jupyterjazz <saba.sturua@jina.ai> * fix: typo Signed-off-by: jupyterjazz <saba.sturua@jina.ai> * docs: rephrase text Signed-off-by: jupyterjazz <saba.sturua@jina.ai> * refactor: requested changes Signed-off-by: jupyterjazz <saba.sturua@jina.ai> --------- Signed-off-by: jupyterjazz <saba.sturua@jina.ai> Signed-off-by: Saba Sturua <45267439+jupyterjazz@users.noreply.github.com> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * fix: bytes type in `TextDoc` and `VideoDoc` (#1270) * fix: bytes type is None in predefined documents Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: remove defaults to none Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> --------- Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * refactor: doc index structure (#1266) * refactor: doc index structure Signed-off-by: jupyterjazz <saba.sturua@jina.ai> * fix: import unused error Signed-off-by: jupyterjazz <saba.sturua@jina.ai> * fix: run black Signed-off-by: jupyterjazz <saba.sturua@jina.ai> * refactor: remove noqa Signed-off-by: jupyterjazz <saba.sturua@jina.ai> * refactor: doc_index to index Signed-off-by: jupyterjazz <saba.sturua@jina.ai> * refactor: small changes Signed-off-by: jupyterjazz <saba.sturua@jina.ai> * refactor: readme Signed-off-by: jupyterjazz <saba.sturua@jina.ai> * refactor: add init file Signed-off-by: jupyterjazz <saba.sturua@jina.ai> * refactor: move docindex outside integration tests Signed-off-by: jupyterjazz <saba.sturua@jina.ai> * refactor: adjust config paths Signed-off-by: jupyterjazz <saba.sturua@jina.ai> --------- Signed-off-by: jupyterjazz <saba.sturua@jina.ai> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * refactor(da): remove tensor type from `DocumentArray` init (#1268) * fix: remove tensor type from DocumentArray Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix tensorflow test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: docstrng Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: apply charllote suggestion Co-authored-by: Charlotte Gerhaher <charlotte.gerhaher@jina.ai> Signed-off-by: samsja <55492238+samsja@users.noreply.github.com> * feat: apply saba suggestion Co-authored-by: Saba Sturua <45267439+jupyterjazz@users.noreply.github.com> Signed-off-by: samsja <55492238+samsja@users.noreply.github.com> --------- Signed-off-by: samsja <sami.jaghouar@hotmail.fr> Signed-off-by: samsja <55492238+samsja@users.noreply.github.com> Co-authored-by: Charlotte Gerhaher <charlotte.gerhaher@jina.ai> Co-authored-by: Saba Sturua <45267439+jupyterjazz@users.noreply.github.com> * refactor: bytes to bytes_ in predefined documents (#1273) * refactor: bytes to bytes_ Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * refactor: missed fields Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> --------- Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * fix: doc summary for dict and set attributes (#1279) Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * feat: add `get_paths()` instead of v1 `from_files()` (#1267) * feat: add from_files() Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * feat: add da classmethod from_files() Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * docs: update docstring Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * docs: add example usage Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: add get_paths, rm from_files Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: add print to debug ci Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: test Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: apply suggestions from code review Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> --------- Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * fix: proto ser and deser for nested tuple/dict/list (#1278) * feat: add failing test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * refactor: shorten if else statememt Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * refactor: shorten if else statememt Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix proto and list Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix proto and dict Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add very complex test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: fix pure tensor stuff Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: fix pure tensor stuff Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix from protobuf Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix from protobuf tensorflow Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: add more test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: add more test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: import ndarray Signed-off-by: samsja <sami.jaghouar@hotmail.fr> --------- Signed-off-by: samsja <sami.jaghouar@hotmail.fr> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * feat: create documents from dict (#1283) * feat: create documents from dict Signed-off-by: jupyterjazz <saba.sturua@jina.ai> * fix: ignore type Signed-off-by: jupyterjazz <saba.sturua@jina.ai> * refactor: change fn names Signed-off-by: jupyterjazz <saba.sturua@jina.ai> --------- Signed-off-by: jupyterjazz <saba.sturua@jina.ai> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * docs: fix up english (#1285) Signed-off-by: Alex C-G <alexcg@outlook.com> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * fix: add int, float and others to doc summary (#1287) Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * fix: hnswlib doc index (#1277) * fix: support for torch and tf Signed-off-by: Johannes Messner <messnerjo@gmail.com> * fix: allow arbitrary payloads, including tensors Signed-off-by: Johannes Messner <messnerjo@gmail.com> * test: mark tf tests Signed-off-by: Johannes Messner <messnerjo@gmail.com> * test: another attempt at fixing tf tests Signed-off-by: Johannes Messner <messnerjo@gmail.com> * test: remove parametrization of test Signed-off-by: Johannes Messner <messnerjo@gmail.com> * test: fix test Signed-off-by: Johannes Messner <messnerjo@gmail.com> * fix: add suggestion Signed-off-by: Johannes Messner <messnerjo@gmail.com> * ci: exlude tf tests from index tests Signed-off-by: Johannes Messner <messnerjo@gmail.com> --------- Signed-off-by: Johannes Messner <messnerjo@gmail.com> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * fix Signed-off-by: RStar2022 <risingstar2880@gmail.com> * fix: tensorflow Signed-off-by: RStar2022 <risingstar2880@gmail.com> * feat(test): DocumentArray method tests similar to list methods like reverse, sort, remove, pop (#1291) * feat: isort format fix Signed-off-by: agaraman0 <agaraman0@gmail.com> * refactor: comment fixes Signed-off-by: agaraman0 <agaraman0@gmail.com> * refactor: comment fixes Signed-off-by: agaraman0 <agaraman0@gmail.com> --------- Signed-off-by: agaraman0 <agaraman0@gmail.com> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * fix:doc Signed-off-by: RStar2022 <risingstar2880@gmail.com> * feat: implement push/pull interface from JAC, file and s3 (#1182) * refactor: move streaming serialization into separate method Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * refactor: add binary io like protocol definition Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * feat: ported push pull to JAC Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: protocol is not in 3.7 typing Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: make mypy happy Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: patch missing waterfall Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * refactor: jit import backends Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * feat: implement cache in jinaai pull Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: add hubble dependency to jina group Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * refactor: better division of concerns Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * feat: add concept of namespace Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: ignore missing hubble stubs Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * feat: streaming protocol stubs Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * refactor: make more general buffered caching reader Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * test: add tests for hubble pushpull Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * test: add tests for file backend Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: remove hubble dependency from jina group This reverts commit b3044213d58517becb9d71194af34f3833560ebc. Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * feat: implement push pull for local filesystem Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * test: test concurrent pushes and pulls in file protocol Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: resolve concurrent pushes and pulls correctly Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: rename text to textdoc Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * feat: added some logging Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * test: s3 tests Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * feat: s3 pushpull Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: add smart open dependency Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: add smart opens silly python bound Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * test: update hubble tests (failing) Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: fix delete return in hubble pushpull Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * Revert "fix: add smart open dependency" This reverts commit cf78c6cc6d2b367501d2358c18773a456426a448. This reverts commit eb0e52b4c521f2b638bf5de850701546a4996bc3. Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: add hubble and smart open dependencies Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: mypy fixes Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * ci: allow tests to see jina auth token Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * feat: add progress bars for streaming Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * style: blacken Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * feat: buffer writes to s3 Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: mypy no like sequence Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: make progress bar quieter when disabled Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * test: skip failing tests Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * feat: add tables when listing Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * test: add jina auth token to uncaped test Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * test: mock s3 tests with minio container Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: silly error that cost me 2 hours of life Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * test: use tolerance ratio in file tests Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * feat: add caching to s3 pull Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * feat: add log messages for unused parameters Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * refactor: take out unneeded buffering smart open already buffers Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * feat: pick fastest protocol compression configuration for s3 Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * test: bump tolerance ratio for s3 test Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * refactor: reduce code duplication Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * refactor: put reader chunk size constant at top of file Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * test: reduce reader chunk size for memory tests Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: multipart uploads get stuck frequently lets just do big uploads for now... Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * docs: add docstrings to mixin and file backend Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * docs: add docstring for s3 and hubble backends Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * test: remove unused test Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * refactor: use literal in protocol Co-authored-by: samsja <55492238+samsja@users.noreply.github.com> Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * refactor: protocols dont need to be inherited Co-authored-by: samsja <55492238+samsja@users.noreply.github.com> Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: add make mypy happy with the literals Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: literals not in 3.7 Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * refactor: move mixin out of init file Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * refactor: move cache path resolution to utils Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * feat: cache path is only evaluated once Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * refactor: loading backends makes more sense as debug log Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * tests: add slow and internet marks Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * refactor: pin image tag Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * refactor: use abc instead of protocol for typing backends Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: revert - add hubble and smart open dependencies This reverts commit 1d1d2eeaf2b51be6ef00e6ab6ee5b9fd1bcf1d92. Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: add hubble and aws dependencies Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * refactor: change all push pull mixin methods to class methods Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: misstyped class method self reference Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * refactor: rename pushpull to docstore and use more classmethods Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * refactor: separate remote backend implementations from mixin Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: missed import refactor Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * refactor: change submodule name to store Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * refactor: remove list and delete from mixin Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * tests: clear all the garbage in ci account Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * tests: skip test that is broken on ci Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * refactor: standardize naming to jac Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> --------- Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> Co-authored-by: samsja <55492238+samsja@users.noreply.github.com> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * refactor: rename `Document` to `Doc` (#1293) * refactor: rename document to doc Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * refactor: rename document to doc in da Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * refactor: rename base doc in md files Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * refactor: rename base base document ot base doc Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix(docs): fix docs building Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: ingore hubble test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: ingore hubble test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> --------- Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * chore(docs): add ci and fix docs ui (#1295) * refactor: rename document to doc Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * refactor: rename document to doc in da Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * refactor: rename base doc in md files Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * refactor: rename base base document ot base doc Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix(docs): fix docs building Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: ingore hubble test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: ingore hubble test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * docs: add userguide install Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * docs: add awesome-pages Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * docs: add install Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * docs: rename tutorials to how to Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * chore: add pre commit blacken docs Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * chore: add blacken docs Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * docs: arr warning docarray version Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * docs: repo url Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * docs: add social Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * docs: add logo Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * docs: add first step emtpy page Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * docs: add document docs Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add markdown documentation test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * docs: remove content Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * docs: fix ci Signed-off-by: samsja <sami.jaghouar@hotmail.fr> --------- Signed-off-by: samsja <sami.jaghouar@hotmail.fr> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * chore: add docstring test (#1298) * wip Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: cleanup namespace utils Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add docstring test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> --------- Signed-off-by: samsja <sami.jaghouar@hotmail.fr> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * fix: rename DocArrayProto to DocumentArrayProto (#1297) Signed-off-by: samsja <sami.jaghouar@hotmail.fr> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * fix: docstring polish typing (#1299) * wip Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: cleanup namespace utils Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add docstring test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix video url docstring Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix text url Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix image url Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fic audio url Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: mesh 3d url Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: mesh 3d url Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: remove useless data Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix docstring ndarray and torch tensor Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix docstring ndarray and torch tensor Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix fix audio url and audio ndarray Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix fix audio url and audio ndarray Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix video tensor Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix video tensor Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix audio bytes Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: video and image bytes Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * docs: move typing section Signed-off-by: samsja <sami.jaghouar@hotmail.fr> --------- Signed-off-by: samsja <sami.jaghouar@hotmail.fr> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * fix: remove files (#1305) Signed-off-by: samsja <sami.jaghouar@hotmail.fr> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * fix: flatten schema of abstract index (#1294) * fix: flatten schema of abstract index Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: _convert_dict_to_doc Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: catch exception when flatten schema Signed-off-by: AnneY <evangeline-lun@foxmail.com> * refactor: remove useless assignemnt Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: use Abstractensor as tensor doc_type Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: add AbstractTensor to hnswlib Signed-off-by: AnneY <evangeline-lun@foxmail.com> * docs: AbstractTensor as doc_type Signed-off-by: AnneY <evangeline-lun@foxmail.com> * docs: complete description about AbstracTensor Co-authored-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com> Signed-off-by: Anne Yang <evangeline-lun@foxmail.com> --------- Signed-off-by: AnneY <evangeline-lun@foxmail.com> Signed-off-by: Anne Yang <evangeline-lun@foxmail.com> Co-authored-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * docs: add utils section (#1307) * feat: add utils for map to docs and fix docstring Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add utils for map to docs and fix docstring Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add utils for find and fix docstring Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix video ndaray docstrng Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix video find docstrng Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix map docstring Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix fileter docstring Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix add reduce Signed-off-by: samsja <sami.jaghouar@hotmail.fr> --------- Signed-off-by: samsja <sami.jaghouar@hotmail.fr> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * docs: fix docstring example of find_batched (#1308) Signed-off-by: Johannes Messner <messnerjo@gmail.com> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * docs: fix map docstring (#1311) * fix: fix utils Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix map Signed-off-by: samsja <sami.jaghouar@hotmail.fr> --------- Signed-off-by: samsja <sami.jaghouar@hotmail.fr> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * feat: elasticsearch document index (#1196) * feat: __init__ of ElasticDocumentIndex Signed-off-by: AnneY <evangeline-lun@foxmail.com> * feat: add index func Signed-off-by: AnneY <evangeline-lun@foxmail.com> * feat: get and del funcs Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: init and index creation Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: __init__ and _index Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: _get_items Signed-off-by: AnneY <evangeline-lun@foxmail.com> * feat: add _find Signed-off-by: AnneY <evangeline-lun@foxmail.com> * feat: add filter text and their batch version Signed-off-by: AnneY <evangeline-lun@foxmail.com> * feat: store id and get nested doc Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: vector cannot be all zero Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: __getitem__ raise error Signed-off-by: AnneY <evangeline-lun@foxmail.com> * feat: support more python types Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: mypy Signed-off-by: AnneY <evangeline-lun@foxmail.com> * test: elastic index tests Signed-off-by: AnneY <evangeline-lun@foxmail.com> * test: comment scripts before ci setup Signed-off-by: AnneY <evangeline-lun@foxmail.com> * chore: add elasticsearch dependency to poetry Signed-off-by: AnneY <evangeline-lun@foxmail.com> * test: elastic index ci setup Signed-off-by: AnneY <evangeline-lun@foxmail.com> * feat: add num_candidates to rumtime config Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: let user pass index_settings Signed-off-by: AnneY <evangeline-lun@foxmail.com> * feat: degrade to v7 and add query builder Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: remove elastic_transport Signed-off-by: AnneY <evangeline-lun@foxmail.com> * feat: minor features Signed-off-by: AnneY <evangeline-lun@foxmail.com> * refactor: style fix Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: fix mypy Signed-off-by: AnneY <evangeline-lun@foxmail.com> * feat: add chunk size to runtime config Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: chunk size Signed-off-by: AnneY <evangeline-lun@foxmail.com> * feat: add chunk_size to funcs Signed-off-by: AnneY <evangeline-lun@foxmail.com> * feat: rewrite elastic v7 query builder Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: poetry Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: db_type should be elastic types Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: minor adjustment Signed-off-by: AnneY <evangeline-lun@foxmail.com> * refactor: rename elastic index files Signed-off-by: AnneY <evangeline-lun@foxmail.com> * refactor: remove comments Signed-off-by: AnneY <evangeline-lun@foxmail.com> * feat: rename, batch operations, etc Signed-off-by: AnneY <evangeline-lun@foxmail.com> * test: add test for persistency and col config Signed-off-by: AnneY <evangeline-lun@foxmail.com> * feat: support more field types and subclass Signed-off-by: AnneY <evangeline-lun@foxmail.com> * feat: support more python types Signed-off-by: AnneY <evangeline-lun@foxmail.com> * test: tf, tensor and more elastic field types Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: elastic should be optional in toml Co-authored-by: Charlotte Gerhaher <charlotte.gerhaher@jina.ai> Signed-off-by: Anne Yang <evangeline-lun@foxmail.com> * refactor: rename class Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: change Dict to Mapping Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: add AbstractTensor Signed-off-by: AnneY <evangeline-lun@foxmail.com> * test: rename class and add tests Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: poetry Signed-off-by: AnneY <evangeline-lun@foxmail.com> --------- Signed-off-by: AnneY <evangeline-lun@foxmail.com> Signed-off-by: Anne Yang <evangeline-lun@foxmail.com> Co-authored-by: Charlotte Gerhaher <charlotte.gerhaher@jina.ai> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * refactor: map_docs_batch to map_docs_batched (#1312) Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * refactor: map_docs_batch to map_docs_batched (#1312) Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * feat: torch backend basic operation tests (#1306) Signed-off-by: agaraman0 <agaraman0@gmail.com> Signed-off-by: RStar2022 <risingstar2880@gmail.com> * chore: add instructions to pip installs and group extras (#1281) * chore: group extras and add instructions for pip installs Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: throw runtime error with install instructions for hnswlib Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * feat: add instructions for video imports Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * feat: add instructions for audio imports Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * feat: add instructions for 3d imports Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * feat: add instructions for image imports Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: import only audiosegment from pydub Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: generalize audio and image imports Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: add instructions for web imports Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: add instructions for web imports Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: add instructions for protobuf imports Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: add instructions for lz4 imports Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: fastapi import Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: revert changes in protobuf import Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: add instructions for torch, without raising error Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: add instructions for torch, with raising error Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: add …
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Goals:
Rename the
bytesfields in the predefined documents tobytes_to avoid overwriting built-inbyteswhen setting the default= None, such as inbytes: Optional[bytes] = None.