[Bug?]: rw data-migration looks for src folder #8059

razzeee · 2023-04-19T10:01:46Z

What's not working?

I'm trying to run rw data-migration from our CI after copying the files over to the docker container.

It complains about the files not being in /app/api/src/lib/db, which is correct.
As I copied them to /app/api/db and would like to keep it like that, as that's the git path and similar to the schema migrations.

rw data-migrate up

Run any outstanding Data Migrations against the database

Options:
  --help     Show help                                                 [boolean]
  --version  Show version number                                       [boolean]
  --cwd      Working directory to use (where `redwood.toml` is located)

Also see the Redwood CLI Reference
(https://redwoodjs.com/docs/cli-commands#datamigrate-up)

Error: Cannot find module '/app/api/src/lib/db'
Require stack:
- /usr/local/share/.config/yarn/global/node_modules/@redwoodjs/cli/dist/commands/dataMigrate/up.js
- /usr/local/share/.config/yarn/global/node_modules/yargs/build/index.cjs
- /usr/local/share/.config/yarn/global/node_modules/yargs/helpers/index.js
- /usr/local/share/.config/yarn/global/node_modules/@redwoodjs/cli/dist/index.js
    at Function.Module._resolveFilename (node:internal/modules/cjs/loader:1026:15)
    at Function.Module._load (node:internal/modules/cjs/loader:871:27)
    at Module.require (node:internal/modules/cjs/loader:1098:19)
    at require (node:internal/modules/cjs/helpers:108:18)
    at Object.handler (/usr/local/share/.config/yarn/global/node_modules/@redwoodjs/cli/dist/commands/dataMigrate/up.js:126:7)
    at /usr/local/share/.config/yarn/global/node_modules/yargs/build/index.cjs:1:8993
    at /usr/local/share/.config/yarn/global/node_modules/yargs/build/index.cjs:1:4949
    at processTicksAndRejections (node:internal/process/task_queues:96:5) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/usr/local/share/.config/yarn/global/node_modules/@redwoodjs/cli/dist/commands/dataMigrate/up.js',
    '/usr/local/share/.config/yarn/global/node_modules/yargs/build/index.cjs',
    '/usr/local/share/.config/yarn/global/node_modules/yargs/helpers/index.js',
    '/usr/local/share/.config/yarn/global/node_modules/@redwoodjs/cli/dist/index.js'
  ]
}

How do we reproduce the bug?

Build a docker container with both api and web in one container, like in the examples.

Install redwood like this at some point

RUN yarn --cwd "api" --production install && \
  yarn global add @redwoodjs/cli prisma @sentry/cli

Try to run your data migrations, in my case I added this line
rw data-migrate up
to my run.sh

What's your environment? (If it applies)

The affected system is a docker container with node 16 on alipne 3.16

Are you interested in working on this?

I'm interested in working on this

The text was updated successfully, but these errors were encountered:

razzeee · 2023-04-19T11:02:38Z

If I copy the whole api folder (not just dist and needed files) I end up with this error

rw data-migrate up

Run any outstanding Data Migrations against the database

Options:
  --help     Show help                                                 [boolean]
  --version  Show version number                                       [boolean]
  --cwd      Working directory to use (where `redwood.toml` is located)

Also see the Redwood CLI Reference
(https://redwoodjs.com/docs/cli-commands#datamigrate-up)

Error: Cannot find module '@babel/plugin-proposal-class-properties'
Require stack:
- /usr/local/share/.config/yarn/global/node_modules/@babel/core/lib/config/files/plugins.js
- /usr/local/share/.config/yarn/global/node_modules/@babel/core/lib/config/files/index.js
- /usr/local/share/.config/yarn/global/node_modules/@babel/core/lib/index.js
- /usr/local/share/.config/yarn/global/node_modules/@babel/register/lib/worker/babel-core.js
- /usr/local/share/.config/yarn/global/node_modules/@babel/register/lib/worker/handle-message.js
- /usr/local/share/.config/yarn/global/node_modules/@babel/register/lib/worker-client.js
- /usr/local/share/.config/yarn/global/node_modules/@babel/register/lib/node.js
- /usr/local/share/.config/yarn/global/node_modules/@babel/register/lib/nodeWrapper.js
- /usr/local/share/.config/yarn/global/node_modules/@babel/register/lib/index.js
- /usr/local/share/.config/yarn/global/node_modules/@redwoodjs/internal/dist/build/babel/common.js
- /usr/local/share/.config/yarn/global/node_modules/@redwoodjs/internal/dist/build/babel/api.js
- /usr/local/share/.config/yarn/global/node_modules/@redwoodjs/cli/dist/commands/dataMigrate/up.js
- /usr/local/share/.config/yarn/global/node_modules/yargs/build/index.cjs
- /usr/local/share/.config/yarn/global/node_modules/yargs/helpers/index.js
- /usr/local/share/.config/yarn/global/node_modules/@redwoodjs/cli/dist/index.js
    at Function.Module._resolveFilename (node:internal/modules/cjs/loader:1026:15)
    at resolve (node:internal/modules/cjs/helpers:114:19)
    at tryRequireResolve (/usr/local/share/.config/yarn/global/node_modules/@babel/core/lib/config/files/plugins.js:115:9)
    at resolveStandardizedNameForRequire (/usr/local/share/.config/yarn/global/node_modules/@babel/core/lib/config/files/plugins.js:149:19)
    at sync (/usr/local/share/.config/yarn/global/node_modules/@babel/core/lib/config/files/plugins.js:172:12)
    at sync (/usr/local/share/.config/yarn/global/node_modules/gensync/index.js:182:19)
    at /usr/local/share/.config/yarn/global/node_modules/gensync/index.js:210:24
    at Generator.next (<anonymous>)
    at resolvePlugin (/usr/local/share/.config/yarn/global/node_modules/@babel/core/lib/config/files/plugins.js:53:17)
    at resolvePlugin.next (<anonymous>) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/usr/local/share/.config/yarn/global/node_modules/@babel/core/lib/config/files/plugins.js',
    '/usr/local/share/.config/yarn/global/node_modules/@babel/core/lib/config/files/index.js',
    '/usr/local/share/.config/yarn/global/node_modules/@babel/core/lib/index.js',
    '/usr/local/share/.config/yarn/global/node_modules/@babel/register/lib/worker/babel-core.js',
    '/usr/local/share/.config/yarn/global/node_modules/@babel/register/lib/worker/handle-message.js',
    '/usr/local/share/.config/yarn/global/node_modules/@babel/register/lib/worker-client.js',
    '/usr/local/share/.config/yarn/global/node_modules/@babel/register/lib/node.js',
    '/usr/local/share/.config/yarn/global/node_modules/@babel/register/lib/nodeWrapper.js',
    '/usr/local/share/.config/yarn/global/node_modules/@babel/register/lib/index.js',
    '/usr/local/share/.config/yarn/global/node_modules/@redwoodjs/internal/dist/build/babel/common.js',
    '/usr/local/share/.config/yarn/global/node_modules/@redwoodjs/internal/dist/build/babel/api.js',
    '/usr/local/share/.config/yarn/global/node_modules/@redwoodjs/cli/dist/commands/dataMigrate/up.js',
    '/usr/local/share/.config/yarn/global/node_modules/yargs/build/index.cjs',
    '/usr/local/share/.config/yarn/global/node_modules/yargs/helpers/index.js',
    '/usr/local/share/.config/yarn/global/node_modules/@redwoodjs/cli/dist/index.js'
  ]
}

razzeee · 2023-04-19T15:46:53Z

Seems like that's caused by running yarn install with --production only in my deploy container

razzeee · 2023-04-19T15:54:27Z

It would be neat, if this would work more like prisma and be able to be installed and work without having to have a redwood project available.

Right now I can use prisma like this in docker

COPY --from=build /app/api/db/schema.prisma /app/api/db/schema.prisma
COPY --from=build /app/api/db/migrations /app/api/db/migrations

RUN yarn global add @redwoodjs/cli prisma


# this then gets run in the entrypoint
prisma migrate deploy --schema=/app/api/db/schema.prisma

I guess it's also reading env vars.

thedavidprice · 2023-04-26T23:51:46Z

@razzeee I think an easy solution is to add an optional path to the dataMigrate command yarn rw dataMigrate, e.g.:

yarn rw dataMigrate up --path /app/api/db/migration-file.js

Might this work for your case? If so, would you be interested + available to work on implementing it?

razzeee · 2023-04-26T23:56:08Z

It's more complex.

This would need to be standalone. Currently it tries to pull in babel from the dev dependencies
It needs to get the whole migration folder and probably envs for database access

thedavidprice · 2023-04-27T00:19:14Z

If you're installing @redwoodjs/cli, both of those items are happening under the hood. Regardless, understood there's likely a lot more nuance happening with your Docker configuration.

As you know, we're working on resolving cases like this (i.e. deploy steps build, prisma, data, serve) for Docker via the in-motion Container Working Group. Thank you for adding your comment to the post!

Looping in @jtoar and @Josh-Walker-GM as an FYI about this being a good use-case.

razzeee · 2023-04-27T05:37:38Z

I don't think your correct, I did always install this

RUN yarn global add @redwoodjs/cli prisma

and then run rw data-migrate up on container start/entrypoint.

When I was still doing yarn --cwd "api" --production install, it ended up complaining about babel missing

jtoar · 2023-05-19T19:14:51Z

@thedavidprice I met with @razzeee a few weeks ago and he showed me this problem. The issue is that yarn rw data-migrate up uses registerApiSideBabelHook, which is not ideal for Docker at all. I meant to look into this sooner so let me take a quick look at it today

razzeee · 2023-06-09T15:48:17Z

@jtoar do I need to copy the migration folder in addition to the dist folder? I don't think the dist contains them?

(I find this a bit confusing/unexpected)

jtoar · 2023-06-09T18:55:18Z

The command still expects the dataMigrations folder to be there in api/db, it just grabs the db singleton from dist in api/dist/lib/db.js instead of api/src/lib.db.js. I thought that was the main problem but it is also that your api/db folder isn't there as well?

@razzeee I could add another flag for --data-migrations-path fairly easily

razzeee · 2023-06-09T19:04:14Z

I can copy that easily I guess, I just trimmed the code down as much as before and forgot about the migration folder/thought it might be in the dist. As I copied the whole api folder before.

jtoar · 2023-06-09T19:29:59Z

@razzeee let me know if that works for you if not I can totally add the flag, but yeah it's going to have to be copied manually one way or another since it's not in src and build won't move it over

razzeee · 2023-06-12T07:44:56Z

After copying the migrations over I run into this:

[STARTED] 20230426101708-migrate-avatar-images.ts

Error in data migration: Cannot find module '@babel/plugin-proposal-class-properties'

Require stack:

- /usr/local/share/.config/yarn/global/node_modules/@babel/core/lib/config/files/plugins.js

- /usr/local/share/.config/yarn/global/node_modules/@babel/core/lib/config/files/index.js

- /usr/local/share/.config/yarn/global/node_modules/@babel/core/lib/index.js

- /usr/local/share/.config/yarn/global/node_modules/@babel/register/lib/worker/babel-core.js

- /usr/local/share/.config/yarn/global/node_modules/@babel/register/lib/worker/handle-message.js

- /usr/local/share/.config/yarn/global/node_modules/@babel/register/lib/worker-client.js

- /usr/local/share/.config/yarn/global/node_modules/@babel/register/lib/node.js

- /usr/local/share/.config/yarn/global/node_modules/@babel/register/lib/nodeWrapper.js

- /usr/local/share/.config/yarn/global/node_modules/@babel/register/lib/index.js

- /usr/local/share/.config/yarn/global/node_modules/@redwoodjs/internal/dist/build/babel/common.js

- /usr/local/share/.config/yarn/global/node_modules/@redwoodjs/internal/dist/build/babel/api.js

- /usr/local/share/.config/yarn/global/node_modules/@redwoodjs/cli/dist/commands/dataMigrate/upHandler.js

- /usr/local/share/.config/yarn/global/node_modules/@redwoodjs/cli/dist/commands/dataMigrate/up.js

- /usr/local/share/.config/yarn/global/node_modules/@redwoodjs/cli/dist/commands/dataMigrate.js

- /usr/local/share/.config/yarn/global/node_modules/@redwoodjs/cli/dist/index.js

Make sure that all the Babel plugins and presets you are using

are defined as dependencies or devDependencies in your package.json

file. It's possible that the missing plugin is loaded by a preset

you are using that forgot to add the plugin to its dependencies: you

can workaround this problem by explicitly adding the missing package

to your top-level package.json.

[COMPLETED] 20230426101708-migrate-avatar-images.ts

Which I guess I can circumvent, but I'm not sure if that's expected @jtoar

thedavidprice · 2023-06-15T17:19:18Z

@jtoar ^^ I believe this is the same package missing from my Storybook tests, correct?

jtoar · 2023-06-15T22:58:21Z

@thedavidprice it sounds similar but I don't remember the exact package, I still haven't looked into, will try to do that today. The similarity between this issue and that issue is missing packages, but they're missing in this issue because we're in a Dockerized environment where we're deliberately trying to remove packages to make a small image. The case you saw may just be a mistake on our part, but again I need to look into it.

@razzeee here's what I think is going on... the yarn rw data-migrate command depends on @redwoodjs/internal. All the logic for how to build a Redwood project lives in that package. Normally you wouldn't need that package once you've built the framework, but data migration files never get built. They're still ES6 syntax, so the strategy for running those is still to transpile them on the fly, as they're imported. So we need @redwoodjs/internal around for that.

But it's more unfortunate than that... @redwoodjs/internal doesn't list the packages it needs to run Babel, @redwoodjs/core does. Why? I'm not sure, that was a bit before my time.

Anyway none of that is good news. I'm trying to isolate data migrate as a command here; maybe I can fix the issues in that PR by bundling everything: #8572

I'll report back when I've actually run this successfully in a Docker container this time

razzeee added the bug/needs-info More information is needed for reproduction label Apr 19, 2023

jtoar mentioned this issue May 19, 2023

fix(data-migrate): provide option to use db in dist instead of src #8375

Merged

jtoar closed this as completed in #8375 May 24, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug?]: rw data-migration looks for src folder #8059

[Bug?]: rw data-migration looks for src folder #8059

razzeee commented Apr 19, 2023

razzeee commented Apr 19, 2023

razzeee commented Apr 19, 2023

razzeee commented Apr 19, 2023

thedavidprice commented Apr 26, 2023

razzeee commented Apr 26, 2023

thedavidprice commented Apr 27, 2023

razzeee commented Apr 27, 2023

jtoar commented May 19, 2023 •

edited

Loading

razzeee commented Jun 9, 2023

jtoar commented Jun 9, 2023 •

edited

Loading

razzeee commented Jun 9, 2023

jtoar commented Jun 9, 2023

razzeee commented Jun 12, 2023

thedavidprice commented Jun 15, 2023

jtoar commented Jun 15, 2023

[Bug?]: rw data-migration looks for src folder #8059

[Bug?]: rw data-migration looks for src folder #8059

Comments

razzeee commented Apr 19, 2023

What's not working?

How do we reproduce the bug?

What's your environment? (If it applies)

Are you interested in working on this?

razzeee commented Apr 19, 2023

razzeee commented Apr 19, 2023

razzeee commented Apr 19, 2023

thedavidprice commented Apr 26, 2023

razzeee commented Apr 26, 2023

thedavidprice commented Apr 27, 2023

razzeee commented Apr 27, 2023

jtoar commented May 19, 2023 • edited Loading

razzeee commented Jun 9, 2023

jtoar commented Jun 9, 2023 • edited Loading

razzeee commented Jun 9, 2023

jtoar commented Jun 9, 2023

razzeee commented Jun 12, 2023

thedavidprice commented Jun 15, 2023

jtoar commented Jun 15, 2023

jtoar commented May 19, 2023 •

edited

Loading

jtoar commented Jun 9, 2023 •

edited

Loading