Making imports easier by allowing to upload JSON and not copy-paste large text files

[saschafoerster]

I am trying to migrate manually board by board larger boards. Even one of the smaller exports is 8,5 MB. Larger exports likely will exceed 50MB or maybe more.

Is there a way to import a json directly by uploading it to the server and then using some command? Or maybe a upload-field, where I can directly upload the JSON file, without copy-pasting the content of it (which makes the browser slow, doesn't give feedback to the user for a long time and is simply unreliable).

[xet7]

@saschafoerster

Not yet, maybe some In Progress code. I'm writing code that could import/export using CLI, but it's not ready yet.

It's possible to export all MongoDB to JSON but that's not so useful yet. Better would be to save in per board JSON export format, and separate attachment files.

Related Import/Export of Boards/Swimlands/Lists/Cards etc:

• Bug: Exporting big board timeouts #4603
• Board export times out #2776
• Feature Request: Import/Export from/to any CSV (and maybe also JSON, YAML, XML, Excel, any database) #3775
• Feature Request: CLI tool to convert between Standalone Wekan MongoDB database, and Sandstorm grain .zip file. (Was: Cards and lists disappeared after update in sandstorm.io) #1695
• More Import/Export/Sync #4578
• Feature Request: Export Board anonymise #1705
• https://github.com/wekan/wekan/blob/master/models/import.js
• https://github.com/wekan/wekan/blob/master/models/wekanCreator.js
• https://github.com/wekan/wekan/wiki/CSV
• https://github.com/wekan/wekan/wiki/Jira
• https://github.com/wekan/wekan/wiki/Asana
• Resources to Import from Zenkit #4487
• https://github.com/wekan/hx/tree/main/prototypes/database/MongoDB
• https://github.com/wekan/wekan/wiki/Python
• https://github.com/wekan/wekan-python-api-client
• https://github.com/wekan/python-wekan
• https://github.com/wekan/wekan-gogs
• https://github.com/wekan/wekan-cleanup
• https://github.com/wekan/wekan-stats
• https://www.mongodb.com/docs/drivers/
• https://github.com/wekan/php/blob/main/src/page/mongo.php
• Question: back end other than Mongo? #787
• First steps on migrating to a new version of mongo. sandstorm-io/sandstorm#3651

Attachments:

• Failed to import JSON from Trello #4418
• Missing images after big Import from Trello #3175
• Feature Request: Bidirectional sync Wekan <=> Trello #1322
• https://github.com/wekan/wekan/tree/master/trello
• v5.27 Attachment filenames not URL-decoded #3774
• FeatureRequest: board setting to enable/disable cover / thumbnail of attachments by default #3739
• ostrio-files errors #3110
• Any files drag from file manager, email etc to attachments #2936
• Image attachments get lost randomly // status error 503 // #2635
• [bug] image previews disappear when defined in template  #2331
• Avatars won't display if the app is migrated to a different sub-path #1776
• Feature Request: Use EXIF orientation of uploaded pictures #1758
• Feature Request: Option to disable cover image #1747
• Add Feature to REST-API : attachments #1482
• Feature request: Attachment swipebox handling of non-images like viewing PDF etc #1318
• v5.20/5.22 - Orphaned file report in admin section is running infinite and does not end ... #3716
• Bad Request [400] when programtically accessing Attachment. #3694
• Files not found: Error 502 Bad gateway #3628
• Local file permissions while uploading attachment may make it inaccessible #3583
• How to set max size of attachment file in wekan snap? #3581
• Feature Request: My Attachments #3461
• Access denied [403] on download attachment #3317
• cfs:gridfs deprecated #3272
• Card to Card attachments/links #3257
• Multi File / Attachment Storage #4463
• Images attached to card does not show correctly #4507
• Feature Request: Click attachment thumbnail to show image bigger #4564
• Covers not visible in Cards #4541
• Images attached to card does not show correctly #4507
• wrong attachement link  #4453
• Wekan broken on mongo restore #4369
• Terrible use case leading to lose attachments #4357
• JSON files attached to a card (with .json filename extension) are empty when downloaded #4324
• Attachment names #4144
• Document name changed after download - google chrome #4367
• Feature Request: Click attachment thumbnail to show image bigger #4564
• Board with 865 cards got slow #4061
• Feature Request: Limit uploaded file size and type #4005
• Export JSON Files #3966
• Feature Request: Email with attachments automated responses from Wekan #3948
• Feature Requests: Attachment file preview #3888
• Upcoming upgrades: Meteor 2.2 to 2.3, Node.js 12.x to 14.x, MongoDB 3.2.x-4.x to 5.x, attachments, email sending #3881
• Feature Request: Provide different ways to upload and store attachments, and drag attachment to card to upload it #142

[saschafoerster]

I think that the export/import of single boards is also broken somehow. I exported from Wekan snap 6.09 to another server, Wekan docker 6.09 and I get an error, that the JSON is not valid:

For my exporting and importing single boards is important, as my database seems to be full of old files, that are not used anymore. We have a mongodb of 13 GB, even though it should be only 1-2 GB from the files I can see. I guess over the years a lot stayed there.

I also try it the cleanup-script, but it deleted to much on the other hand, but will keep on checking again and find a way to migrate from snap 6.09 to a newer docker version of Wekan.

[xet7]

@saschafoerster

Yes, I try to create Import/Export tool that could:

• Work with all current and previous WeKan versions
• Import/Export any format like mongodump/mongorestore, Board JSON, CSV, Excel, Attachment files
• Transfer files between all MongoDB versions, filesystem, S3, etc
• WeKan board could have any cards, attachments, etc from different storages and software shown at same board [Feature Request] Master Dashboard similar to Bigboard plugin for Kanboard. #4223
• Compare and store only diffs between backups https://github.com/paulfitz/daff , and with that also free also a lot of my own disk space

[xet7]

For beginnings of that Import/Export CLI tool, I'm following progress of this with great interest:

sandstorm-io/sandstorm#3651

[xet7]

Related to database migrations:

• https://github.com/securitybunker/databunker

Related to attachment migrations:

• https://github.com/rclone/rclone

Related to S3 proxy:

• https://github.com/caddyserver/caddy
• https://github.com/lindenlab/caddy-s3-proxy

Related to repo migrations:

• https://github.com/go-gitea/gitea
• https://github.com/wekan/wekan-gogs
• https://fossil-scm.org

Related to issue migrations:

• https://github.com/MichaelMure/git-bug

Related to calendar migrations:

• https://github.com/paviliondev/omnievent

Related to auth:

• https://github.com/omniauth/omniauth
• https://github.com/omniauth/omniauth/wiki/List-of-Strategies

Related to JMAP standard format Import/Export:

• Adding to WeKan any JMAP features, that do not yet exist at WeKan
• Adding to JMAP any WeKan features, that do not yet exist at JMAP
• https://datatracker.ietf.org/doc/draft-ietf-jmap-tasks/
• https://dapsi.ngi.eu/hall-of-fame/openxport/
• https://gist.github.com/jaudriga/cd323fa75397c76b1dd30861ef590302

[xet7]

From Meteor Slack 2022-07-31, about splitting code to separate parts

From afrokick

I had a working build script to bundle code with ESBuild and install it via meteorInstall. Maybe someone want to try it, just eval the result of build in your meteor project.

build.js

import { exec } from 'child_process';
import path from 'path';
import { build } from 'esbuild';

// isProduction flag for watch mode
const isProduction = process.env.NODE_ENV === 'production';
const MODULE_NAME = process.env.npm_package_name ?? '';
const NAME_PARTS = MODULE_NAME.split('/'); //to supports @org/package
const isPrefixed = NAME_PARTS.length > 1;

const SAFE_NAME = NAME_PARTS.join('_'); //@org/package -> @org_package

const OUTPUT_FILE_PATH = path.resolve(`./dist/${SAFE_NAME}.js`);

(async () => {
  build({
    watch: isProduction
      ? false
      : {
          onRebuild(error) {
            if (!error) {
              console.log('Build succeeded');
            }
          },
        },
    entryPoints: [path.resolve('./src/index.ts')],
    outbase: 'src',
    bundle: true,
    platform: 'node',
    format: 'cjs',
    external: ['mobx', 'react', 'react-dom', 'mobx-react', 'meteor/*'],
    outfile: OUTPUT_FILE_PATH,
    banner: {
      js: `meteorInstall({'node_modules':{${
        isPrefixed ? `'${NAME_PARTS[0]}':{'${NAME_PARTS[1]}'` : '${MODULE_NAME}'
      }(require,exports,module) {`,
    },
    footer: {
      js: `${isPrefixed ? '}' : ''}}}},{extensions:['.js','.jsx','.ts','.tsx','.json','.less','.css','.html','.mjs']})`,
    },
  })
    .then(() => {
    })
    .catch(e => console.error(e.message));
})();

Run with Node 16:

node build.js

From xet7

Can someone explain what this means? Anyway I'll try. Thanks!

From afrokick

You can use any build tools like SWC or ESBuild to build you code and it’ll work with meteor project.

We want to use it to build microfrontends and load it dynamically via import()

Why not meteor itself?

Because now our code base very large (~ 500k loc) and build time ~ 5min. Event if we use dynamic import from meteor it will increase our build time.

So, the separate project will have a fast build time(because swc/esbuild works fast), will have a separate deploy process and will work with current meteor codebase(thanks to meteorInstall aka install)

Check https://github.com/benjamn/install

In the future, i want to create a package live-modules which brings a possibility to update runtime code without redeploy all codebase.

Some cases:

• fix localization (we have a PoC and it works perfectly)
• enable/disable experimental code(also have a small prototype which add new route and page)
• apply critical bug fix without redeploy all app(yes, it is just a temporary solution but sometimes it is necessary to fix a bug ASAP)

I think it will be usefull for mid/large project:

• where CI/CD time is large, but you want to deploy some code fast
• if you want to split the app into microfrontends and load it only if it needs.

Ready https://github.com/itgenio/meteor-live-modules

[xet7]

Moved to here from #3966 (comment)

From @nstoeckigt

What are the plans to fix this?

From @xet7

Plan is to figure out the plan. Help welcome.

[xet7]

https://github.com/wekan/wekan/wiki/Sync