Question: back end other than Mongo?

[xet7]

From @romu70 on November 16, 2016 10:11

Hi,
Is there any simple way to setup WeKan with another back end than Mongo? Is this on the roadmap?

Thanks.

Copied from original issue: wefork#9

[xet7]

@romu70

Could you do some Google searching for projects that make interface like Mongo but translate it to another database backend like PostgreSQL or MySQL ? And also find examples how Meteor / Mongo projects have added support for other databases?

[xet7]

@romu70

Somebody else did ask about this before, so I added it to roadmap. I have not had time to add all feature requests from original wekan project to roadmap yet.

[xet7]

From @romu70 on November 16, 2016 10:58

Thanks a lot, so I think you deserve a little explanation ;-)

Self hosting is great but raises some issues, even more if you plan to do self-hosting with Docker:

• the more is the list of apps you want to host, the more is the list of required data back ends, and this makes self hosting more and more difficult, and resources consuming, both human and hardware
• when you self-host an application through Docker, you absolutely need to externalize the data if you want to be able to update the Docker without loosing things.

Personally, I don't need specifically a SQL DB, Redis would be great also, and maybe easier for switching from a NoSQL DB, but I don't know well both.

Thanks a lot.

[xet7]

From @pierreozoux on November 17, 2016 10:18

Meteor is using Mongo.
Wekan uses Meteor as a Framework.
As long as Meteor doesn't allow other db, wekan can't use any other db.

Closing this for now.

Feel free to reopen once meteor add support to other db.

[xet7]

@pierreozoux

There can be some way to use Meteor with other databases, I did see some search results about that, it's not impossible. This is looking for somebody needing other database support to do the research.

Reopening issue.

[xet7]

From @aegrey on December 18, 2016 2:34

Meteor said last year it was a high priority, but with MongoDB's growing popularity I would assume it's become less so.

There's a library that does it (numtel:mysql - haven't used it in a while). Hasn't been updated since mid-2016, but his example project is active (https://github.com/numtel/meteor-mysql-leaderboard). For those interested, you could try integration in your fork (I am just delving into Wekan's code, so couldn't say how heavy of a lift it'd be).

[xet7]

Currently Wekan works like this:

Frontend Javascript <=> Backend Node.js <=> MongoDB

Is there this kind of Database Query converter somewhere?

Frontend Javascript <=> Backend Node.js <=> Database Query converter <=> MongoDB/MariaDB/PostgreSQL etc

When Node.js sends MongoDB query, Database Query converter would then use something like search-replace/regex/parsing to convert it to MariaDB syntax, get results from MariaDB and send them in MongoDB format back to Node.js.

This Database Query converter would have it's own web interface, where database queries can be translated to different database syntax, storing those at converter's own syntax database. If some syntax was not able to be converted, there would be error listed at error log page of converter's own web interface.

This way we could use any database, so if there is already some existing database then it's not required to install MongoDB, just use existing database.

That Database Query converter could be also used for auditing, stats, logging, permission management, limiting to only allowed queries, exposing REST API to other software etc.

[entrptaher]

It's quite possible. IMO, you need to have the following changes,

• Building a DB query converter that plays along meteor.
• Changing the whole codebase that calls MongoDB in any way.
• Making unit tests to make sure nothing falls out and the performance does not drop.

If not from scratch, then we could try to use some other libraries available online to avoid repetition.

• https://www.npmjs.com/package/mongo2mysql
• https://www.npmjs.com/package/mongo-mysql
• https://github.com/doubaokun/MongoDB-to-MySQL
• https://www.npmjs.com/package/mysql-mongo-db
• https://www.npmjs.com/package/fruit
• https://www.npmjs.com/package/db-migrate
• https://www.npmjs.com/package/databridge
• https://www.npmjs.com/package/east

We can try to find if this is possible or not.

[xet7]

I do not mean changing anything in Wekan code.

I mean replacing MongoDB with, for example, single executeable binary:

• It works in same port as MongoDB, port could be changed in environment variable:
• That binary could be programmed with any language, like Go
• The same binary could also include database server that's made with Go
• When receiving MongoDB query, it could use internal database server, or convert query to other database syntax, connect to that database, get results, convert results to MongoDB format, and return to that app that uses MongoDB. I mean any app that uses MongoDB, not just Wekan.

[xet7]

I'm currently looking at ToroDB:

• https://www.torodb.com
• https://github.com/torodb/torodb
• https://twit.tv/shows/floss-weekly/episodes/377

[xet7]

Ok I got ToroDB working on docker-compose, so this it Wekan reporting with new LibreOffice 3.5 native PostgreSQL support, using Wekan <=> MongoDB <=> ToroDB => PostgreSQL read-only mirroring. I will create new repo wekan-postgresql for this. You can use any programming language, Office suite or reporting package to get read-write MongoDB data and read-only PostgreSQL data with SQL queries.

[xet7]

Ok I have now added docker-compose.yml etc code to:
https://github.com/wekan/wekan-postgresql

[xet7]

At ToroDB website was said, that it could be possible to use only ToroDB/PostgreSQL and not write data to MongoDB at all, completely replacing MongoDB, but I did't yet figure out how to do it.

[xet7]

@romu70

Does this help with your use case in any way?

[gortiz]

Hi there,

A ToroDB developer here! As you may know, ToroDB Stampede is on beta right now, in fact, we are looking for real applications to find (and solve!) possible incompatibilities and we would be glad to help if you find problems. On the other hand, Stampede has been designed to read from a MongoDB. We have another product called ToroDB Server which can be used instead of MongoDB (talking the same protocol), but it is on developer preview version and Meteor is not supported (as ToroDB Server does not support replication yet).

\cc @ahachete

[ahachete]

As @gortiz said, ToroDB Stampede (https://www.torodb.com/stampede/) replicates from MongoDB and will provide a read-only, native SQL version of your MongoDB data in real-time.

If you want to completely replace MongoDB, you will need to wait a bit for ToroDB Server (https://www.torodb.com/ "ToroDB Technology") to be available (ETA: Q3 2017).

We would be happy to help out understand your requirements and how ToroDB may fit in your stack. Please let us know :)

[xet7]

@gortiz @ahachete

Some time ago I tried wekan-postgresql and it had errors starting, has something changed in docker-compose file?
https://github.com/wekan/wekan-postgresql

Is it possible to test Wekan with ToroDB server?

[xet7]

@gortiz @ahachete

When working with Node.js applications, have you come across errors like at #1096 ? If yes, do you have ideas how to debug and fix ?

[killua-eu]

Not sure if you guys noticed, mysql has had native support for json documents for over a year now also including a REST api built in into the database. IMO picking up support for mysql could be relatively easy + give anyone the luxury of combining things like joins, stored procedures and the whole sql thing including replication with the flexibility of a document store. Oh and performance is really good too.

[xet7]

@killua-eu

Are there Meteor.js projects that use JSON support or REST API in MySQL ?

What does "relatively easy" mean in practice? How to convert Meteor.js project to use MySQL ?

[killua-eu]

@xet7 , that I won't be able to answer well with respect to Meteor.js - javascript is definitely not my strong point. Nontheless I've been using the mysql xdevapi and the builtin sql functions to manipulate and query json documents since the feature was available with python and php - never looked back. Js has been a first class citizen here with https://devhub.io/repos/mysql-mysql-connector-nodejs appearing right after the feature has been made available.

[rautamiekka]

I made little research: https://stackoverflow.com/a/20616188/460695 recommends https://github.com/mysqljs/mysql (the link in the post is different but doesn't error out, which has to mean the original was renamed) for complete support (looking at the source code it has a wide array of features, so I can see why it's recommended) in pure Node.js but doesn't have reactive SELECT, for which some place recommended https://github.com/numtel/meteor-mysql.

Seems, with those you're most of the time using functions instead of writing MySQL yourself. However, should you have to and need help, I'm available. I'm not awesome at it (or SQLite 3, which is largely similar) but not trash either.

[xet7]

@rautamiekka

Database abstraction layer by using functions would make supporting more databases much easier.

Is there documentation how to port Meteor.js/MongoDB application like Wekan to it?

[xet7]

@rautamiekka

If you have time to try even beginning steps of implementation to see what is actually required, it would be nice.

There is no requirements for spending a long time or being awesome at it.

[rautamiekka]

^ Meant MySQL code that's known to be needed, and is known what it should do.

[petrk94]

Is there any update?
I would like to use Wekan with MariaDB or mysql, due my hosting provider (VPS), doesnt allow a easy way to let it run mongodb :/

[xet7]

This also mentions FerretDB:

• https://www.tigrisdata.com/blog/tigris-mongodb-atlas-alternative/
• https://github.com/tigrisdata-community/go-mongo-quickstart
• https://news.ycombinator.com/item?id=34889051

[xet7]

More about WeKan with FerretDB/PostgreSQL instead of MongoDB:

• WeKan progress from FerretDB developers: Achieve compatibility with WeKan FerretDB/FerretDB#1752 . Note: I only today found these links and issues, I have not known about these before, and I have not yet tried FerretDB.
• Announce of In Progress FerretDB to support Meteor, PostgreSQL and sometime maybe SQLite https://blog.ferretdb.io/ferretdb-1-0-ga-opensource-mongodb-alternative/
• In that announce above, I did not find mention of Open Source MongoDB client GUI https://dbgate.org
• How to try FerretDB Docker image: https://github.com/FerretDB/FerretDB#quickstart
• Hacker News article https://news.ycombinator.com/item?id=35539464
• Meteor related issues at FerretDB https://github.com/FerretDB/FerretDB/issues?q=is%3Aissue+is%3Aopen+meteor

[xet7]

FerretDB High Level Roadmap https://github.com/orgs/FerretDB/projects/2/views/1

[AlekSi]

Hi, FerretDB maintainer there.

In that announce above, I did not find mention of Open Source MongoDB client GUI https://dbgate.org/

Thanks, we added that to our list to investigate.

We are already working or plan to start working soon on all things that are needed for Wekan except GridFS. Is it a hard dependency? Do you plan to continue using it?

[xet7]

@AlekSi

Hi, WeKan maintainer here.

No, I am trying to move from GridFS to Minio https://github.com/wekan/minio-metadata or other external file storage.

There are lots of problems with GridFS. Currently WeKan has migrations from CollectionFS to Meteor-Files that temporarily writes file to disk to WRITABLE_PATH enviroment variable directory that is different on each platform, and back to MongoDB GridFS different structure:

https://github.com/wekan/wekan/blob/master/server/migrations.js#L1248-L1404

Then there is Meteor-Files features to move files from GridFS to filesystem and back. I'm trying to add Minio support by adding minio-js package https://github.com/minio/minio-js because Meteor-Files only supports AWS S3, not custom endpoint like Minio veliovgroup/Meteor-Files#862

MongoDB uses a lot of CPU and RAM on Raspberry Pi etc smaller platforms. SQLite would be better, both on Raspberry Pi, https://sandstorm.io etc.

[xet7]

Currently we are trying to get those attachment and avatar fles migrations working so that those files would stay visible:

#4416 (comment)

It would be better to not write all file chunks to disk and back, instead only rename collections.

[AlekSi]

We are working on the SQLite backend for FerretDB, by the way.

[xet7]

Added FerretDB support to WeKan Roadmap

[AlekSi]

We changed our plans a bit and are going to support OpLog in a few months before Change Streams.

[xet7]

@AlekSi

That's great! I presume many more Meteor apps use OpLog. Not many use Change Streams.

[xet7]

2023-07-31 FerretDB and other database support progress

• I did find a way to use both MongoDB 3.x and MongoDB 6.x in same Javascript file: https://github.com/wekan/wekan-node20 . There in package.json make alias of old mongodb driver name as mongodb-legacy. And then in main.js use both.
• I also used FerretDB in that main.js file.
• FerretDB is MongoDB 6.x wire protocol compatible proxy to PostgreSQL and SQLite.
• PostgreSQL support works better than SQLite support. But not all WeKan MongoDb queries are supported yet by FerretDB.
• I also used newest Node.js 20 in that test, and I'm adding some tests for other databases too.
• I also added some npm packages WeKan already uses to that package.json

Prisma vs TypeORM

• I'm trying some database abstraction DAL/ORM, is there any benefit.
• This article claims Prisma would be better, I don't know is that up-to-date or not https://www.prisma.io/docs/concepts/more/comparisons/prisma-and-typeorm
• While abstracting database queries could be useful, it depends does query hydration (generating NoSQL/SQL queries) take too much time, or is it better to write queries for each database separately? Because for example FerretDB does not support all kinds of queries yet.

[RogerPenna]

I am not an IT administrator or anything.

So I had the problem with MongoDB 6. Wekan had worked previously on my laptop, which obviously is newer than the server I then tried to install it. On my laptop it also would run directly in Brazilian Portuguese, by default.

I changed the docker-compose.yml file to use MongoDB 4.4 or 4.6 then, on the server.

It worked and Wekan run. BUT the language was set to English and it was impossible to set it to any other language. I had two brower tabs open at the same time, one with the local laptop Wekan (at MongoDB 6) and the server Wekan (running on MongoDB 4), and in the laptop I could change between languages, while at the other tab I was stuck to English and when opening the languages dropdown list, it was just impossible to select anything else.

[xet7]

@RogerPenna

That is not a database question. That is translations question, see #4518

WeKan uses newest MongoDB 6.x. Currently it is 6.0.9.

[RogerPenna]

@RogerPenna

That is not a database question. That is translations question, see #4518

WeKan uses newest MongoDB 6.x. Currently it is 6.0.9.

My apologies. As I was running a Wekan on my laptop localhost and on a server where I had to install MongoDB 4.6 (because the server doesn´t run MongoDB 5 and above), and looking at the info from both, it seemed the ONLY difference was the database.

I will make some tests from the thread you linked about language, while running on MongoDB 4. Thank you very much xet7!

[xet7]

@RogerPenna

I don't know does newest WeKan work with MongoDB 4. It's possible that there are some errors and bugs.

[RogerPenna]

Thank you. Right now I am running Wekan 6.22 on MongoDB 4.4.24.

I am using 6.22 because that is the last version that people said they were able to choose language according to the threads you directed me above, by using a correct root URL config.

[xet7]

@RogerPenna

I am using WeKan 7.09 with MongoDB 6.0.9 , and I can change language just fine.

[RogerPenna]

Well, I followed what people on the other thread were talking. I have no clue and debugging is complicated for me. I could not even select different languages. Clicking over them resutled in nothing when using the last version with MongoDB 4.4.

You gave me the tip to check that other thread which was about language, and people suggested using version 6.22 and setting properly the root-url. So I used version 6.22 and I think I set up root-url.

You might ask how the hell I don´t know that? Well, ignoramuses like me use ChatGPT to set up the Docker-Compose YML based on what people said on that thread LMAO.

ChatGPT said it would use 6.22 and set the root url.

"Given the range of issues mentioned in the GitHub threads and the settings in your Docker Compose file, here are some steps you can consider:

Version Compatibility:
MongoDB Version: Since your processor doesn't support MongoDB versions 5.0 and above due to the absence of the VMX feature, you may want to stick with MongoDB 4.4 as per your last working setup. Update the docker-compose.yml file to specify this version."

"Language Issue:
ROOT_URL: According to the thread, setting the ROOT_URL correctly solved the language issue for some users. If you are running Wekan behind a reverse proxy, then ROOT_URL should be set to the domain name your users will use to access Wekan."

"Language Issue:
ROOT_URL: According to the thread, setting the ROOT_URL correctly solved the language issue for some users. If you are running Wekan behind a reverse proxy, then ROOT_URL should be set to the domain name your users will use to access Wekan."

[xet7]

@RogerPenna

VMX? Do you mean AVX? There is MongoDB 6.x without AVX here:

#4321 (comment)

That could be used as MongoDB image in newest docker-compose.yml from https://github.com/wekan/wekan

[xet7]

Hmm. I'm thinking, maybe I should just compile newest compatible MongoDB without AVX and use it at all platforms. Because I already have my compiled Node.js version too.

Or alternatively, migrate to other database.

[RogerPenna]

Ok, went to version 7.09 again, while keeping Mongo 4.4

Again, language selection stopped working. So WeKan 6.22 + Mongo 4.4 = language selection works
Wekan 7.09 + Mogo 4.4 = language doesn´t work

I ckick, click, but it doesnt deselect English and select other languages(it's not other languages that are broken, it's the selection menu that misteriously gets broken)

As for the no AVX version, I was going to try it, but in that thread you linked they informed WeKan 6.21 did not work with the that Mongo no AVX version, so you had to recompile it, etc.

Well, I never tried it. But then, you are not mentioning recompiling it, so maybe it works with WeKan 7.

I will try it. Thank you very much Xet7!

[RogerPenna]

@RogerPenna

VMX? Do you mean AVX? There is MongoDB 6.x without AVX here:

#4321 (comment)

That could be used as MongoDB image in newest docker-compose.yml from https://github.com/wekan/wekan

Ok. I need to build the MongoDB no AVX image, however, right?

[xet7]

@RogerPenna

To this line:

https://github.com/wekan/wekan/blob/master/docker-compose.yml#L94

Change this:

image: vidvidex/mongodb-without-avx:6.0.7

Then in docker-compose.yml change your server IP address, for example like this:

ROOT_URL=http://192.168.0.100

And then start WeKan:

docker-compose up -d

[xet7]

@RogerPenna

Ok. I need to build the MongoDB no AVX image, however, right?

No, that is prebuilt MongoDB without AVX. No need to compile.

[RogerPenna]

@RogerPenna

To this line:

https://github.com/wekan/wekan/blob/master/docker-compose.yml#L94

Change this:

image: vidvidex/mongodb-without-avx:6.0.7

Then in docker-compose.yml change your server IP address, for example like this:

ROOT_URL=http://192.168.0.100

And then start WeKan:

docker-compose up -d

OK... I used the available docker-compose and changed the required lines. This already happened before but I had forgotten.

Mongo Db keeps restarting.

when running the logs
Wekan$ docker logs wekan-db
Invalid command: mongod

ChatGPT suggests the following

The error message "Invalid command: mongod" suggests that MongoDB is not starting because the command specified in the docker-compose.yml file for the MongoDB service is incorrect or not being recognized.

Here are some suggestions to resolve this:

Option 1: Simplify the MongoDB Service Configuration
Try running the MongoDB container without any custom command field. Most MongoDB Docker images have a default entry-point set up to run mongod. Update your docker-compose.yml file to remove the command line from the MongoDB service configuration.

Here's the updated part of your docker-compose.yml:

wekandb:
  image: vidvidex/mongodb-without-avx:6.0.7
  container_name: wekan-db
  restart: always
  networks:
    - wekan-tier
  expose:
    - 27017
  volumes:
    - /etc/localtime:/etc/localtime:ro
    - /etc/timezone:/etc/timezone:ro
    - wekan-db:/data/db
    - wekan-db-dump:/dump

Option 2: Specify Entry Point
Another option is to use entrypoint to specify mongod as the entry point.

Update your docker-compose.yml as follows:

wekandb:
  image: vidvidex/mongodb-without-avx:6.0.7
  container_name: wekan-db
  restart: always
  entrypoint: mongod
  networks:
    - wekan-tier
  expose:
    - 27017
  volumes:
    - /etc/localtime:/etc/localtime:ro
    - /etc/timezone:/etc/timezone:ro
    - wekan-db:/data/db
    - wekan-db-dump:/dump

I am aware that ChatGPT can be quite dumb if you are not surewhat you are asking... so if you can give your input in whats happening... thanks a lot for your patience.

[RogerPenna]

I will try both options above, but I am not very optimistic. Will probably end up returning to Mongo 4.4

[mfilser]

Language settings has nothing to do if you are running Mongo 4.4, Mongo 5 or Mongo 6.
Language settings are part of the application, not the database.

[RogerPenna]

Language settings has nothing to do if you are running Mongo 4.4, Mongo 5 or Mongo 6. Language settings are part of the application, not the database.

Yes, I think Xet7 already established that before.

As I said in a post above

"Ok, went to version 7.09 again, while keeping Mongo 4.4

Again, language selection stopped working. So WeKan 6.22 + Mongo 4.4 = language selection works
Wekan 7.09 + Mogo 4.4 = language doesn´t work"

The problem is that language selection only works with Mongo 4.4 on WeKan 6.22.

And the reason I need to use an old version of WeKan is because I need Mongo 4.4 because the newer Mongo versions have AVX.

So while language is not directly related to database, the problem with the language selection has as a root cause the database Mongo 5 and above requiring AVX processors.