Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Greenlight v2 => v3 migration script fails #5742

Open
zulf10 opened this issue Mar 8, 2024 · 25 comments
Open

Greenlight v2 => v3 migration script fails #5742

zulf10 opened this issue Mar 8, 2024 · 25 comments

Comments

@zulf10
Copy link

zulf10 commented Mar 8, 2024

Hello,

I have a system setup with V2 and V3 and I am trying to migrate my roles, users etc.
BBB is on 2.7.

Greenlight (v2 and v3) docker already in last version.

.env V2

SECRET_KEY_BASE=123
BIGBLUEBUTTON_ENDPOINT=https://XXX/bigbluebutton/ 
V3_ENDPOINT=https://XXX/bigbluebutton/ 
V3_SECRET_KEY_BASE=123

.env V3

BIGBLUEBUTTON_ENDPOINT=https://XXX/bigbluebutton/ 
SECRET_KEY_BASE=123

Command to migrate rôle:
docker exec -it greenlight-v2 bundle exec rake migrations:roles

/usr/src/app/vendor/bundle/ruby/2.7.0/gems/net-protocol-0.1.3/lib/net/protocol.rb:68: warning: already initialized constant Net::ProtocRetryError
/usr/local/lib/ruby/2.7.0/net/protocol.rb:66: warning: previous definition of ProtocRetryError was here                                                                                    /usr/src/app/vendor/bundle/ruby/2.7.0/gems/net-protocol-0.1.3/lib/net/protocol.rb:208: warning: already initialized constant Net::BufferedIO::BUFSIZE
/usr/local/lib/ruby/2.7.0/net/protocol.rb:206: warning: previous definition of BUFSIZE was here                                                                                            /usr/src/app/vendor/bundle/ruby/2.7.0/gems/net-protocol-0.1.3/lib/net/protocol.rb:504: warning: already initialized constant Net::NetPrivate::Socket
/usr/local/lib/ruby/2.7.0/net/protocol.rb:503: warning: previous definition of Socket was here
warning: parser/current is loading parser/ruby27, which recognizes2.7.6-compliant syntax, but you are running 2.7.7.
Please see https://github.com/whitequark/parser#compatibility-with-ruby-mri.
Unable to migrate Role:                                                                                                                                                                    
ID: 6
Name: Moderateur
Errors: Provider does not exist
Unable to migrate Role:
ID: 7
Name: Basic
Errors: Provider does not exist
Unable to migrate Role:
ID: 8
Name: Membre niv. 2
Errors: Provider does not exist

Roles migration complete.
In case of an error please retry the process to resolve.

Nginx logs show 400 code

172.24.0.3 - - [08/Mar/2024:23:51:21 +0100] "POST /api/v1/migrations/roles.json HTTP/1.1" 400 47 "-" "Ruby"
172.24.0.3 - - [08/Mar/2024:23:51:21 +0100] "POST /api/v1/migrations/roles.json HTTP/1.1" 400 47 "-" "Ruby"
172.24.0.3 - - [08/Mar/2024:23:51:21 +0100] "POST /api/v1/migrations/roles.json HTTP/1.1" 400 47 "-" "Ruby"

Docker Greenlight-v3 logs same status 400

greenlight-v3    | I, [2024-03-08T22:51:21.636704 #1]  INFO -- : [a2cc9721-705b-41f4-b7aa-06638d3497f4] method=POST path=/api/v1/migrations/roles.json format=json controller=Api::V1::Migrations::ExternalController action=create_role status=400 allocations=1549 duration=8.18 view=0.60 db=4.16 time=2024-03-08 22:51:21 UTC host=
greenlight-v3    | I, [2024-03-08T22:51:21.649238 #1]  INFO -- : [9d593cf2-6ca7-4419-b161-1066a4d6a8ef] method=POST path=/api/v1/migrations/roles.json format=json controller=Api::V1::Migrations::ExternalController action=create_role status=400 allocations=510 duration=1.78 view=0.34 db=0.43 time=2024-03-08 22:51:21 UTC host=
greenlight-v3    | I, [2024-03-08T22:51:21.660275 #1]  INFO -- : [7bc57e43-c72e-4fa3-9e51-6a69ec40eb98] method=POST path=/api/v1/migrations/roles.json format=json controller=Api::V1::Migrations::ExternalController action=create_role status=400 allocations=507 duration=2.58 view=0.50 db=0.62 time=2024-03-08 22:51:21 UTC host=

Is there any special configuration for these errors?

It's as if API V1 is no longer available. Do I need to backtrack to BBB version 2.6 or even 2.5?

Thanks

@farhatahmad
Copy link
Collaborator

What's the command that you ran?

@zulf10
Copy link
Author

zulf10 commented Mar 11, 2024

I used:
docker exec -it greenlight-v2 bundle exec rake migrations:roles

@farhatahmad
Copy link
Collaborator

Can you try updating both v2 and v3 please

@zulf10
Copy link
Author

zulf10 commented Mar 11, 2024

Yes, the containers are in the latest version and so is bbb.

@farhatahmad
Copy link
Collaborator

What's the output of

docker exec -it greenlight-v2 rails c
Role.all.pluck(:provider)

@zulf10
Copy link
Author

zulf10 commented Mar 12, 2024

It seems there is no rails command:

# docker exec -it greenlight-v2 rails c
OCI runtime exec failed: exec failed: unable to start container process: exec: "rails": executable file not found in $PATH: unknown

@farhatahmad
Copy link
Collaborator

Try either bin/rails or bundle exec rails

@zulf10
Copy link
Author

zulf10 commented Mar 12, 2024

Thanks for reply.

with command : # docker exec -it greenlight-v2 bin/rails c
And : irb(main):001:0> Role.all.pluck(:provider)

=> ["greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight"]

@farhatahmad
Copy link
Collaborator

Thats very strange - everything looks good from here. re-running it will give you the same errors?

@zulf10
Copy link
Author

zulf10 commented Mar 13, 2024

Yes migration role command docker exec -it greenlight-v2 bundle exec rake migrations:roles give same errors:

/usr/src/app/vendor/bundle/ruby/2.7.0/gems/net-protocol-0.1.3/lib/net/protocol.rb:68: warning: already initialized constant Net::ProtocRetryError
/usr/local/lib/ruby/2.7.0/net/protocol.rb:66: warning: previous definition of ProtocRetryError was here
/usr/src/app/vendor/bundle/ruby/2.7.0/gems/net-protocol-0.1.3/lib/net/protocol.rb:208: warning: already initialized constant Net::BufferedIO::BUFSIZE
/usr/local/lib/ruby/2.7.0/net/protocol.rb:206: warning: previous definition of BUFSIZE was here
/usr/src/app/vendor/bundle/ruby/2.7.0/gems/net-protocol-0.1.3/lib/net/protocol.rb:504: warning: already initialized constant Net::NetPrivate::Socket
/usr/local/lib/ruby/2.7.0/net/protocol.rb:503: warning: previous definition of Socket was here
warning: parser/current is loading parser/ruby27, which recognizes2.7.6-compliant syntax, but you are running 2.7.7.
Please see https://github.com/whitequark/parser#compatibility-with-ruby-mri.
Unable to migrate Role:
ID: 6
Name: Moderateur
Errors: Provider does not exist
Unable to migrate Role:
ID: 7
Name: Basic
Errors: Provider does not exist
Unable to migrate Role:
ID: 8
Name: Membre niv. 2
Errors: Provider does not exist

Roles migration complete.
In case of an error please retry the process to resolve.

Is normal command Role.all.pluck(:provider) return many greenlight ?

=> ["greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight"]

isn't there a problem with the database that would cause this kind of error?

@farhatahmad
Copy link
Collaborator

Nope that's fine. Can you confirm your v2 and v3 versions please?

@zulf10
Copy link
Author

zulf10 commented Mar 13, 2024

  • For v2

# docker image inspect bigbluebutton/greenlight:v2 | grep VERSION_CODE

"VERSION_CODE=release-2.14.9"

  • For v3

# docker image inspect bigbluebutton/greenlight:v3 | grep VERSION_TAG

"VERSION_TAG=release-3.3.1"

  • BBB package version with command

# dpkg -l | grep bbb

ii  bbb-apps-akka                         1:2.7-61                                     all          BigBlueButton Apps (Akka)
ii  bbb-config                            1:2.7-34                                     amd64        BigBlueButton configuration utilities
ii  bbb-etherpad                          1:2.7-14                                     amd64        The EtherPad Lite components for BigBlueButton
ii  bbb-export-annotations                1:2.7-19                                     amd64        BigBlueButton Export Annotations
ii  bbb-freeswitch-core                   2:2.7-15                                     amd64        BigBlueButton build of FreeSWITCH
ii  bbb-freeswitch-sounds                 1:2.7-3                                      amd64        FreeSWITCH Sounds
ii  bbb-fsesl-akka                        1:2.7-28                                     all          BigBlueButton FS-ESL (Akka)
ii  bbb-html5                             1:2.7-243                                    amd64        The HTML5 components for BigBlueButton
ii  bbb-html5-nodejs                      1:2.7-6                                      amd64        Include a specific NodeJS version for bbb-html5
ii  bbb-learning-dashboard                1:2.7-13                                     amd64        BigBlueButton bbb-learning-dashboard
ii  bbb-libreoffice-docker                1:2.7-7                                      amd64        BigBlueButton setup for LibreOffice running in docker
ii  bbb-mkclean                           1:2.7-3                                      amd64        Clean and optimize Matroska and WebM files
ii  bbb-pads                              1:2.7-12                                     amd64        BigBlueButton Pads
ii  bbb-playback                          1:2.7-7                                      amd64        Player for BigBlueButton presentation format recordings
ii  bbb-playback-presentation             1:2.7-5                                      amd64        BigBluebutton playback of presentation
ii  bbb-record-core                       1:2.7-13                                     amd64        BigBlueButton record and playback
ii  bbb-web                               1:2.7-47                                     amd64        BigBlueButton API
ii  bbb-webrtc-recorder                   1:2.7-7                                      amd64        BigBlueButton WebRTC Recorder
ii  bbb-webrtc-sfu                        1:2.7-33                                     amd64        BigBlueButton WebRTC SFU
ii  bigbluebutton                         1:2.7-2                                      amd64        Open source web conferencing platform (bbb)

isn't there a problem with the BBB version ?

@farhatahmad
Copy link
Collaborator

Nope, this is unrelated to BBB. One final thing, maybe this will work

docker exec -it greenlight-v3 bundle exec rails c
Tenant.create(name: "greenlight", client_secret: 'test')

Try that and then re-run the commands

@zulf10
Copy link
Author

zulf10 commented Mar 13, 2024

Return :

docker exec -it greenlight-v3 bundle exec rails c
Loading production environment (Rails 7.1.3.1)
irb(main):001> Tenant.create(name: "greenlight", client_secret: 'test')
=> 
#<Tenant:0x00007f786213e2a0
 name: "greenlight",
 client_secret: "[FILTERED]",
 created_at: Wed, 13 Mar 2024 18:31:08.914868000 UTC +00:00,
 updated_at: Wed, 13 Mar 2024 18:31:08.914868000 UTC +00:00,
 id: "b40fe1e9-fb8a-4467-8661-8e981a8abb1b">

And run again commands migrate role do same things:

# docker exec -it greenlight-v2 bundle exec rake migrations:roles
/usr/src/app/vendor/bundle/ruby/2.7.0/gems/net-protocol-0.1.3/lib/net/protocol.rb:68: warning: already initialized constant Net::ProtocRetryError
/usr/local/lib/ruby/2.7.0/net/protocol.rb:66: warning: previous definition of ProtocRetryError was here
/usr/src/app/vendor/bundle/ruby/2.7.0/gems/net-protocol-0.1.3/lib/net/protocol.rb:208: warning: already initialized constant Net::BufferedIO::BUFSIZE
/usr/local/lib/ruby/2.7.0/net/protocol.rb:206: warning: previous definition of BUFSIZE was here
/usr/src/app/vendor/bundle/ruby/2.7.0/gems/net-protocol-0.1.3/lib/net/protocol.rb:504: warning: already initialized constant Net::NetPrivate::Socket
/usr/local/lib/ruby/2.7.0/net/protocol.rb:503: warning: previous definition of Socket was here
warning: parser/current is loading parser/ruby27, which recognizes2.7.6-compliant syntax, but you are running 2.7.7.
Please see https://github.com/whitequark/parser#compatibility-with-ruby-mri.
Unable to migrate Role:
ID: 6
Name: Moderateur
Errors: Provider does not exist
Unable to migrate Role:
ID: 7
Name: Basic
Errors: Provider does not exist
Unable to migrate Role:
ID: 8
Name: Membre niv. 2
Errors: Provider does not exist

Roles migration complete.
In case of an error please retry the process to resolve.

@farhatahmad
Copy link
Collaborator

Oh - I just looked through your env variables

V3_ENDPOINT=https://XXX/bigbluebutton/ 

This should be pointing at wherever your v3 server is deployed, not at your BigBlueButton server. How did you install v3?

@zulf10
Copy link
Author

zulf10 commented Mar 13, 2024

BigBlueButton server, v2 and v3 are on same server.

Web interface v3 on https://XXX/
Web interface v2 on https://XXX/b

But endpoint already same on v2 and v3 env

@psy
Copy link

psy commented Mar 15, 2024

Web interface v3 on https://XXX/

try V3_ENDPOINT=https://XXX/

@zulf10
Copy link
Author

zulf10 commented Mar 20, 2024

Same issue with V3_ENDPOINT=https://XXX/

do the containers have to be on the same docker network ?

@psy
Copy link

psy commented Mar 21, 2024

No, v2 only needs to be able to reach the url specified via V3_ENDPOINT. I just recently migrated a greenlight installation to a completely different server. Do you run both versions under the same domain (and use greenlights nginx config)? If yes, your api routes mights be forwarded to the old instance.

Try curl -H 'Accept: application/json' https://XXX/api/v1/env.json and look for VERSION_TAG.

@zulf10
Copy link
Author

zulf10 commented Mar 25, 2024

Yes both versions run under the same domain and use greenlights nginx config.

Command curl -H 'Accept: application/json' https://XXX/api/v1/env.json return "VERSION_TAG":"release-3.3.1"

@psy
Copy link

psy commented Mar 25, 2024

lgtm. i'm out of ideas.

@caos30
Copy link

caos30 commented Apr 17, 2024

Hello, I've been suffering from the "same" problem, and I have read on other threads that this issue is at least 1 year old. In my case, I have the Greenlight v3 running in a different VM than the v2. I followed the same instructions and checks as above with similar results.

Indeed, nobody above commented on this, but I suppose that the OP did it: put the same SECRET KEY in both .env files, for SECRET_KEY_BASE (v2 & v3) and for V3_SECRET_KEY_BASE (v2).

In my case, the return of the command:

sudo docker exec -it greenlight-v2 bundle exec rake migrations:roles

is this:

/usr/src/app/vendor/bundle/ruby/2.7.0/gems/net-protocol-0.1.3/lib/net/protocol.rb:68: warning: already initialized constant Net::ProtocRetryError
/usr/local/lib/ruby/2.7.0/net/protocol.rb:66: warning: previous definition of ProtocRetryError was here
/usr/src/app/vendor/bundle/ruby/2.7.0/gems/net-protocol-0.1.3/lib/net/protocol.rb:208: warning: already initialized constant Net::BufferedIO::BUFSIZE
/usr/local/lib/ruby/2.7.0/net/protocol.rb:206: warning: previous definition of BUFSIZE was here
/usr/src/app/vendor/bundle/ruby/2.7.0/gems/net-protocol-0.1.3/lib/net/protocol.rb:504: warning: already initialized constant Net::NetPrivate::Socket
/usr/local/lib/ruby/2.7.0/net/protocol.rb:503: warning: previous definition of Socket was here
warning: parser/current is loading parser/ruby27, which recognizes2.7.6-compliant syntax, but you are running 2.7.7.
Please see https://github.com/whitequark/parser#compatibility-with-ruby-mri.

Roles migration complete.

I think the message "Roles migration complete" is wrong, isn't it? I think that if the migrations were successful, I should see a list of the roles migrated. Am I wrong?

When i run the next command (to migrate users):

sudo docker exec -it greenlight-v2 bundle exec rake migrations:users

then i get this:

/usr/src/app/vendor/bundle/ruby/2.7.0/gems/net-protocol-0.1.3/lib/net/protocol.rb:68: warning: already initialized constant Net::ProtocRetryError
/usr/local/lib/ruby/2.7.0/net/protocol.rb:66: warning: previous definition of ProtocRetryError was here
/usr/src/app/vendor/bundle/ruby/2.7.0/gems/net-protocol-0.1.3/lib/net/protocol.rb:208: warning: already initialized constant Net::BufferedIO::BUFSIZE
/usr/local/lib/ruby/2.7.0/net/protocol.rb:206: warning: previous definition of BUFSIZE was here
/usr/src/app/vendor/bundle/ruby/2.7.0/gems/net-protocol-0.1.3/lib/net/protocol.rb:504: warning: already initialized constant Net::NetPrivate::Socket
/usr/local/lib/ruby/2.7.0/net/protocol.rb:503: warning: previous definition of Socket was here
warning: parser/current is loading parser/ruby27, which recognizes2.7.6-compliant syntax, but you are running 2.7.7.
Please see https://github.com/whitequark/parser#compatibility-with-ruby-mri.

Unable to migrate User:
UID: gl-mzyagccgkozf
Name: personal_user_name_1
Provider: greenlight
Errors: ["InvalidParams"]

Unable to migrate User:
UID: gl-wymjnjolxued
Name: personal_user_name_2
Provider: greenlight
Errors: ["InvalidParams"]

Unable to migrate User:
UID: gl-bnbdsokhbqbl
Name: personal_user_name_3
Provider: greenlight
Errors: ["InvalidParams"]

Unable to migrate User:
UID: gl-kctsrjtrpejx
Name: personal_user_name_4
Provider: greenlight
Errors: ["InvalidParams"]

Unable to migrate User:
UID: gl-elkertszzprw
Name: personal_user_name_5
Provider: greenlight
Errors: ["InvalidParams"]

Unable to migrate User:
UID: gl-jvwfwvxiusmp
Name: personal_user_name_6
Provider: greenlight
Errors: ["InvalidParams"]

Users migration completed.
In case of an error please retry the process to resolve.
If you have not migrated your roles, kindly run 'rake migrations:roles' first and then retry.

So, I understand that the migrations didn't complete, and I suppose that this is the cause of the "unable to migrate user" error messages.

I'm really not sure about what the expected result of each command is because this is my first migration from v2 to v3. Although the official guide for it (https://docs.bigbluebutton.org/greenlight/v3/migration/) mentions the results for a successful migration, it doesn't cover a "strange mid-successful migration". I'm not very clear on how to understand these messages.

So, any help will be very welcomed.

@caos30
Copy link

caos30 commented Apr 21, 2024

Here's the grammatically corrected version:

Corrected Version

Please, can at least someone guide me on a way to migrate recordings done in v2 to the v3 server MANUALLY and enable users on v3 to manage and share those recordings from Greenlight v3?

I've manually recreated the roles and users of v2 in v3. But we need now to copy the recordings of the last 2 years from v2 to the v3 server and make them accessible for users on v3. How can I do that? Taking into account that the official "migration script" is failing and nobody is able to help us make it run. 🤗

Any help is welcomed. Thanks in advance.

Note: just after install the v3 server i set the same SECRET_KEY (editing greenligh-v3/.env file) existing at greenlight/.env file on v2 server. I suspect that this key would be important to some "crypted file names".

@zulf10
Copy link
Author

zulf10 commented Jun 6, 2024

Hello,

We migrated again. We removed the roles.
We had to modify the script migrations.rake in /lib/tasks/migrations

We added instructions to the script like this:

@@ -68,7 +68,7 @@
         .where.not(roles: { name: COMMON[:filtered_user_roles] }, deleted: true)
         .find_each(start: start, finish: stop, batch_size: COMMON[:batch_size]) do |u|
       role_name = infer_role_name(u.role.name)
-      params = { user: { name: u.name, email: u.email, external_id: u.social_uid, language: u.language, role: role_name } }
+      params = { user: { name: u.name, email: u.email, external_id: u.social_uid, language: u.language, role: role_name, provider: 'greenlight' } }
 
       response = Net::HTTP.post(uri('users'), payload(params), COMMON[:headers])

@@ -133,7 +133,8 @@
                          last_session: r.last_session&.to_datetime,
                          owner_email: r.owner.email,
                          room_settings: room_settings,
-                         shared_users_emails: shared_users_emails } }
+                         shared_users_emails: shared_users_emails,
+                         provider: 'greenlight' } }
 
       response = Net::HTTP.post(uri('rooms'), payload(params), COMMON[:headers])
 
@@ -188,7 +189,7 @@
       glRequireAuthentication: infer_room_config_value(setting.get_value('Room Authentication'))
     }.compact
 
-    params = { settings: { site_settings: site_settings, room_configurations: room_configurations } }
+    params = { settings: { site_settings: site_settings, room_configurations: room_configurations, provider: 'greenlight' } }
 
     response = Net::HTTP.post(uri('settings'), payload(params), COMMON[:headers])

We didn't migrate settings or records

We check again but it seem solved for us

@CrossFire10100
Copy link

CrossFire10100 commented Jun 15, 2024

Same issue here.. I wonder why they changed the whole DB structure making it virtually impossible to migrate correctly any data at all.

If anyone wants to migrate recordings just create the old rooms with the same room name and add to the docker compose file pgadmin on both sides login to both postgres db and just clone the old meeting id to the new room and resync recordings.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants