Achieve compatibility with WeKan

[b1ron]

What should be done?

Refs #24, #2501

They use Meteor which often tails the oplog for performance benefits but tailing the oplog is an optional setting for WeKan defined here. It uses Meteor's poll-and-diff method by default which just uses find under-the-hood.

They also use GridFS to store avatars and attachments.

https://github.com/wekan/wekan/tree/master/models

Tasks

Beta Give feedback

1. Support $pull array update operator #826
   code/feature +1
   
2. Support $addToSet array update operator #825
   code/feature +1
   
3. Implement $match aggregation pipeline stage #1428
   code/feature +1
   
4. Implement $group aggregation pipeline stage #1423
   code/feature +1
   
5. Implement $sort aggregation pipeline stage #1439
   code/feature +1
   
6. Implement the $sum aggregation operator #2013
   code/feature +1
   
7. Implement $unset aggregation pipeline stage #1432
   area/aggregations code/feature good first issue +3
   
8. Implement aggregation pipeline accumulators #1467
   area/aggregations code/feature not ready +3
9. Implement $lookup aggregation pipeline stage #1427
   area/aggregations code/feature not ready +3
10. Support OpLog #76
    4 of 5
    area/oplog code/feature +2
11. Support GridFS #1754
    code/feature not ready +2
    
12. Support partial indexes (partialFilterExpression, sparse) #2448
    area/indexes code/feature not ready +3
    

[xet7]

@b1ron @AlekSi

What is missing from WeKan running with FerretDB ? That tasklist has some error.

[AlekSi]

@xet7 oops, sorry, fixed it

[AlekSi]

We just released a new version with OpLog tailing support, which was a major blocker for supporting Meteor apps: https://github.com/FerretDB/FerretDB/releases/tag/v1.18.0

[xet7]

@AlekSi

When I try FerretDB SQLite, I get this exception:

Exception from sub board id wGvX5gYqr2vaFFisK Error: Exception while polling query {"collectionName":"boards","selector":{"_id":"gzST3j8ZiW78FndH7","archived":false,"$or":[{"permission":"public"},{"members":{"$elemMatch":{"userId":"iZYN9KrYfJNG3i9sQ","isActive":true}}},{"orgs.orgId":{"$in":[]}},{"teams.teamId":{"$in":[]}}]},"options":{"limit":1,"sort":{"sort":1}}}: unknown operator: userId
    at PollingObserveDriver._pollMongo (packages/mongo/polling_observe_driver.js:165:11)
    at Object.task (packages/mongo/polling_observe_driver.js:93:12)
    at Meteor._SynchronousQueue.SQp._run (packages/meteor.js:1026:16)
    at packages/meteor.js:1003:12

At browser right click console:

Error invoking SubsManager.subscribe("board", "gzST3j8ZiW78FndH7", false):  Internal server error ab3fce4c494d44ba7eec83cd68639d536a5c1f47.js:746:2638

At docker-compose.yml of https://github.com/wekan/wekan/blob/main/docker-compose.yml#L91-L110 I have changed to FerretDB SQLite like this, then docker compose up -d and registered at https://localhost/sign-up and logged in at https://localhost/sign-in

services:

  wekandb:
    image: ghcr.io/ferretdb/all-in-one
    networks:
      - wekan-tier
    restart: on-failure
    expose:
      - 27017
    environment:
      - FERRETDB_HANDLER=sqlite
      - FERRETDB_SQLITE_URL=file:/data/
      - DOCKER_DEFAULT_PLATFORM=linux/x86_64/v8
    volumes:
      - ./data:/data/
      - /etc/localtime:/etc/localtime:ro
      - wekan-db:/data/db
      - wekan-db-dump:/dump
      #- /etc/timezone:/etc/timezone:ro # Do not use https://github.com/wekan/wekan/issues/5123

[xet7]

Hmm, I think I did not yet add those OpLog settings from https://github.com/FerretDB/FerretDB/releases/tag/v1.18.0

I'll try

[xet7]

https://www.mongodb.com/docs/v6.0/reference/method/db.createCollection/#create-a-capped-collection

[xet7]

In Progress. Just guessing.

I don't have any of these working yet, I'm just guessing how these could work. I don't have any working examples yet.

How to set OpLog URL setting?

MONGO_OPLOG_URL , with MongoDB, was previously made this way https://github.com/wekan/wekan/blob/main/docker-compose.yml#L218 . Does Meteor also need MONGO_OPLOG_URL ?

For FerretDB, it looks like FERRETDB_REPL_SET_NAME should be something like this:

      - MONGO_URL=mongodb://wekandb:27017/wekan
      - MONGO_OPLOG_URL=mongodb://wekandb:27017/local?authSource=admin&replicaSet=oplog.rs

services:

  wekandb:
    image: ghcr.io/ferretdb/all-in-one
    networks:
      - wekan-tier
    restart: on-failure
    #ports:
    #  - 27017:27017
    expose:
      - 27017
    environment:
      - FERRETDB_HANDLER=sqlite
      - FERRETDB_SQLITE_URL=file:/data/
      - FERRETDB_REPL_SET_NAME=oplog.rs
      - DOCKER_DEFAULT_PLATFORM=linux/x86_64/v8
    volumes:
      - ./data:/data/
      - /etc/localtime:/etc/localtime:ro
      - wekan-db:/data/db
      - wekan-db-dump:/dump
      #- /etc/timezone:/etc/timezone:ro # Do not use https://github.com/wekan/wekan/issues/5123

How to create oplog.rs capped collection at Snap?

https://github.com/wekan/wekan/blob/main/snap-src/bin/mongodb-control#L148

For creating oplog.rs capped collection, I guess it would be something like this:

mongosh wekan --eval 'db.createCollection("oplog.rs", { capped : true, size : 5242880, max : 5000 } );' $BIND_OPTIONS

Previous MongoDB OpLog at ToroDB/PostgreSQL

Previously, with ToroDB to PostgreSQL https://github.com/wekan/wekan/tree/main/torodb-postgresql , replicaset was previosly made this way https://github.com/wekan/wekan/blob/main/torodb-postgresql/docker-compose.yml#L129-L142

  mongodb:
    image: mongo:3.2
    networks:
      - wekan-tier
    ports:
      - "27017:27017"
    entrypoint:
      - /bin/bash
      - "-c"
      - mongo --nodb --eval '
            var db;
            while (!db) {
                try {
                  db = new Mongo("mongodb:27017").getDB("local");
                } catch(ex) {}
                sleep(3000);
            };
            rs.initiate({_id:"rs1",members:[{_id:0,host:"mongodb:27017"}]});
        ' 1>/dev/null 2>&1 &
        mongod --replSet rs1

But I don't know does FerretDB all-in-one container have mongosh command? MongoDB 3.x MongoShell command mongo is at MongoDB 6.x commmand mongosh.

Maybe creating oplog.rs with Meteor?

I don't know, would it be possible to create replicaset using Meteor commands? https://docs.meteor.com/api/collections

Maybe creating oplog.rs at models?

In theory, like at wekan/models/ferretdb-oplog.js, maybe something like this, not tested:

if (Meteor.isServer) {
 Meteor.startup(() => {
    // Attachments.collection.createIndex({ 'meta.cardId': 1 });
    "oplog.rs".collection.createCollection({ capped : true, size : 5242880, max : 5000 });
  });
}

[AlekSi]

Does Meteor also need MONGO_OPLOG_URL ?

I think so.

FERRETDB_REPL_SET_NAME=oplog.rs

That should be just a name like rs0, not an OpLog collection name (that is fixed). The same name should be used in MongoDB URI's replicaSet parameter.

But I don't know does FerretDB all-in-one container have mongosh command?

It does. But rs.initiate will not work. Instead, just create a capped collection with db.createCollection.

[wreiske]

Yes, Meteor can use the environment variable MONGO_OPLOG_URL https://docs.meteor.com/environment-variables#MONGO-OPLOG-URL

MONGO_OPLOG_URL=mongodb://localhost:27017/local?replSet=rs01

Excited to see progress on FerretDB and Meteor!

[xet7]

@wreiske @AlekSi

Is there any working code example of Meteor working with FerretDB/SQLite ?

For example, can you make modifications to Dockerfile and docker-compose.yml at https://github.com/wekan/wekan and create GitHub Gist or repo, where WeKan would work with FerretDB/SQLite ? And if possible, also example for FerretDB/PostgreSQL ?

[wreiske]

@wreiske @AlekSi

Is there any working code example of Meteor working with FerretDB/SQLite ?

For example, can you make modifications to Dockerfile and docker-compose.yml at https://github.com/wekan/wekan and create GitHub Gist or repo, where WeKan would work with FerretDB/SQLite ? And if possible, also example for FerretDB/PostgreSQL ?

Absolutely! There's a PostgreSQL example here https://github.com/wreiske/meteor-ferretdb I made around 2 years ago. It looks like FerretDB forked it and updated it a bit https://github.com/FerretDB/meteor-ferretdb

Maybe with small tweaks it can work with SQLite.