Skip to content
This repository has been archived by the owner on Aug 31, 2021. It is now read-only.

add postgresql indexer #6897

Merged
merged 72 commits into from
Apr 21, 2021
Merged

add postgresql indexer #6897

merged 72 commits into from
Apr 21, 2021

Conversation

Yongxuanzhang
Copy link
Contributor

@Yongxuanzhang Yongxuanzhang commented Mar 24, 2021

Postgress Indexer.

  • Docker image running with a Dummy Executor -- get PostgreSQL Docker image and install Jina in it
  • Executor performs CRUD ops
  • Integration test.

Benchmark

### indexing 100000 docs takes 1 minute and 4 seconds (64.40s)
### dumping 100000 docs takes 1 second (2.00s)
### dump path size: 112.35556 MBs
### reloading 100000 takes 3 seconds (3.16s)

NOTE: PSQL indexing can be optimized by inserting all in one go (for a future PR)

Compare w BinaryPb and Base here: jina-ai/jina#2307

@CatStark
Copy link
Member

The work on postgresqldbhandler.py will mostly have to be re done after last conversation with Cristian but can serve as a base

@jina-bot jina-bot added size/L and removed size/M labels Apr 15, 2021
Copy link
Contributor

@cristianmtr cristianmtr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

still some things to address

@CatStark CatStark marked this pull request as ready for review April 19, 2021 13:14
cristianmtr
cristianmtr previously approved these changes Apr 21, 2021
@cristianmtr cristianmtr merged commit 063c297 into master Apr 21, 2021
@cristianmtr cristianmtr deleted the postgresql-indexer branch April 21, 2021 10:03
@cristianmtr cristianmtr mentioned this pull request Apr 28, 2021
3 tasks
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants