Skip to content

Commit

Permalink
feat: add mongodb component (#198)
Browse files Browse the repository at this point in the history 
Because

- We need to integrate MongoDB into our data pipeline.
- This will allow us to cover data movement user stories

This commit

- Allow client to connect to their MongoDB database through VDP
- Mongo URI auth
- Create execution function to handle these following tasks

- [x] TASK_INSERT
- [x] TASK_UPDATE
- [x] TASK_FIND
- [x] TASK_DELETE
- [x] TASK_DROP_COLLECTION
- [x] TASK_DROP_DATABASE
- [x] TASK_INSERT_MANY
  • Loading branch information
@zidanehakim
zidanehakim authored Aug 7, 2024
1 parent dabf472 commit 2cb550f
Show file tree
Hide file tree
Showing 12 changed files with 3,142 additions and 0 deletions.
295 changes: 295 additions & 0 deletions data/mongodb/v0/README.mdx
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,295 @@
---
title: "MongoDB"
lang: "en-US"
draft: false
description: "Learn about how to set up a VDP MongoDB component https://github.com/instill-ai/instill-core"
---

The MongoDB component is a data component that allows users to access the MongoDB NoSQL database.
It can carry out the following tasks:

- [Insert](#insert)
- [Insert Many](#insert-many)
- [Find](#find)
- [Update](#update)
- [Delete](#delete)
- [Drop Collection](#drop-collection)
- [Drop Database](#drop-database)
- [Create Search Index](#create-search-index)
- [Drop Search Index](#drop-search-index)
- [Vector Search](#vector-search)



## Release Stage

`Alpha`



## Configuration

The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/data/mongodb/v0/config/definition.json).




## Setup


| Field | Field ID | Type | Note |
| :--- | :--- | :--- | :--- |
| URI (required) | `uri` | string | Fill in your MongoDB URI |




## Supported Tasks

### Insert

Perform an insert operation


| Input | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Task ID (required) | `task` | string | `TASK_INSERT` |
| Database Name (required) | `database-name` | string | The name of the database in MongoDB |
| Collection Name (required) | `collection-name` | string | The name of the collection in MongoDB |
| ID | `id` | string | The ID of the document |
| Data (required) | `data` | object | The data to be inserted |



| Output | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Status | `status` | string | Insert status |






### Insert Many

Perform an insert many operation


| Input | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Task ID (required) | `task` | string | `TASK_INSERT_MANY` |
| Database Name (required) | `database-name` | string | The name of the database in MongoDB |
| Collection Name (required) | `collection-name` | string | The name of the collection in MongoDB |
| Array ID | `array-id` | array[string] | The array of id |
| Data (required) | `array-data` | array[object] | The array data to be inserted |



| Output | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Status | `status` | string | Insert many status |






### Find

Perform a find operation


| Input | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Task ID (required) | `task` | string | `TASK_FIND` |
| Database Name (required) | `database-name` | string | The name of the database in MongoDB |
| Collection Name (required) | `collection-name` | string | The name of the collection in MongoDB |
| ID | `id` | string | The ID of the document |
| Filter | `filter` | object | The filter to find documents, please refer to the documentations https://www.mongodb.com/docs/manual/reference/operator/query/. If empty then all documents will be returned |
| Limit | `limit` | integer | The number of documents to return. If empty then all documents will be returned |
| Fields | `fields` | array[string] | The fields to return in the documents. If empty then all fields will be returned |



| Output | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Result | `result` | object | Result of the find operation |
| Status | `status` | string | Find status |






### Update

Perform an update operation


| Input | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Task ID (required) | `task` | string | `TASK_UPDATE` |
| Database Name (required) | `database-name` | string | The name of the database in MongoDB |
| Collection Name (required) | `collection-name` | string | The name of the collection in MongoDB |
| ID | `id` | string | The ID of the document |
| Filter | `filter` | object | The filter to update documents, please refer to the documentations https://www.mongodb.com/docs/manual/reference/operator/query/. If empty then all documents will be returned |
| Update (required) | `update-data` | object | The updated data to be applied to the documents |



| Output | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Status | `status` | string | Update status |






### Delete

Perform a delete operation


| Input | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Task ID (required) | `task` | string | `TASK_DELETE` |
| Database Name (required) | `database-name` | string | The name of the database in MongoDB |
| Collection Name (required) | `collection-name` | string | The name of the collection in MongoDB |
| ID | `id` | string | The ID of the document |
| Filter | `filter` | object | The filter to delete documents, please refer to the documentations https://www.mongodb.com/docs/manual/reference/operator/query/. If empty then all documents will be returned |



| Output | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Status | `status` | string | Delete status |






### Drop Collection

Delete the collection


| Input | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Task ID (required) | `task` | string | `TASK_DROP_COLLECTION` |
| Database Name (required) | `database-name` | string | The name of the database in MongoDB |
| Collection Name (required) | `collection-name` | string | The name of the collection in MongoDB |



| Output | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Status | `status` | string | Delete collection status |






### Drop Database

Delete the database


| Input | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Task ID (required) | `task` | string | `TASK_DROP_DATABASE` |
| Database Name (required) | `database-name` | string | The name of the database in MongoDB |



| Output | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Status | `status` | string | Delete database status |






### Create Search Index

Create a search index, only works for M10 or larger clusters


| Input | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Task ID (required) | `task` | string | `TASK_CREATE_SEARCH_INDEX` |
| Database Name (required) | `database-name` | string | The name of the database in MongoDB |
| Collection Name (required) | `collection-name` | string | The name of the collection in MongoDB |
| Index Name (required) | `index-name` | string | The name of the index to be created |
| Index Type (required) | `index-type` | string | The type of the index to be created |
| Syntax (required) | `syntax` | object | The syntax structure of the search index, please refer to the MongoDB documentation for more information. search https://www.mongodb.com/docs/atlas/atlas-search/create-index/. vectorSearch https://www.mongodb.com/docs/atlas/atlas-vector-search/vector-search-type/ |



| Output | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Status | `status` | string | Create index status |






### Drop Search Index

Drop a search index, only works for M10 or larger clusters


| Input | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Task ID (required) | `task` | string | `TASK_DROP_SEARCH_INDEX` |
| Database Name (required) | `database-name` | string | The name of the database in MongoDB |
| Collection Name (required) | `collection-name` | string | The name of the collection in MongoDB |
| Index Name (required) | `index-name` | string | The name of the index to be dropped |



| Output | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Status | `status` | string | Delete index status |






### Vector Search

Perform a vector search operation


| Input | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Task ID (required) | `task` | string | `TASK_VECTOR_SEARCH` |
| Database Name (required) | `database-name` | string | The name of the database in MongoDB |
| Collection Name (required) | `collection-name` | string | The name of the collection in MongoDB |
| Index Name (required) | `index-name` | string | The name of the index to be used for vector search |
| Query Vector (required) | `query-vector` | array[number] | The query vector to be used for vector search |
| Exact | `exact` | boolean | The exact value for vector search. If true, then ENN search will be performed, otherwise ANN search will be performed. Default to false |
| Limit (required) | `limit` | integer | Limit the documents to be returned |
| Number of Candidates | `num-candidates` | integer | The number of candidates to the field to be used for vector search. Default to 3 times limit |
| Path (required) | `path` | string | The path to the field to be used for vector search |
| Filter | `filter` | object | The filter to be used for vector search, need to first create filter vectorSearch search index, please refer to the documentations https://www.mongodb.com/docs/manual/reference/operator/query/. If empty then all documents will be returned to be used for vector search |
| Fields | `fields` | array[string] | The fields to return in the documents. If empty then all fields will be returned |



| Output | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Result | `result` | object | Result of the vector search operation |
| Status | `status` | string | Vector search status |







3 changes: 3 additions & 0 deletions data/mongodb/v0/assets/mongodb.svg
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
19 changes: 19 additions & 0 deletions data/mongodb/v0/client.go
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
package mongodb

import (
"context"

"go.mongodb.org/mongo-driver/mongo"
"go.mongodb.org/mongo-driver/mongo/options"
"google.golang.org/protobuf/types/known/structpb"
)

func newClient(ctx context.Context, setup *structpb.Struct) *mongo.Client {
client, _ := mongo.Connect(ctx, options.Client().ApplyURI(getURI(setup)))

return client
}

func getURI(setup *structpb.Struct) string {
return setup.GetFields()["uri"].GetStringValue()
}
Loading

0 comments on commit 2cb550f

Please sign in to comment.