Avatar migration still not working properly

[lucasvanhalst]

The improved avatar migration (#7352) still isn't working properly, and the error messages aren't really saying much. See the log here: https://pastebin.com/kN7F0Npz
Those avatars are currently stored in the database so I assume it's trying to migrate them to place them on disk but it's not working. I'm using the docker image from dockerhub if that makes a difference. The migration is also marked as being executed successfully while it's clearly not.

Rocket.Chat Version: 010a60a
Running Instances: 1
DB Replicaset OpLog: true
Node Version: 4.6

docker-compose.yml:

services:
  mongo:
    image: mongo
    restart: unless-stopped
    volumes:
      - ./data/runtime/db:/data/db
    command: mongod --smallfiles --oplogSize 128 --replSet rs0
  rocketchat:
    image: rocketchat/rocket.chat:develop
    restart: unless-stopped
    volumes:
      - ./uploads:/app/uploads
    environment:
      - PORT=3000
      - ROOT_URL=https://example.com
      - MONGO_URL=mongodb://mongo:27017/rocketchat
      - MONGO_OPLOG_URL=mongodb://mongo:27017/local
      - DDP_DEFAULT_CONNECTION_URL=https://example.com
    ports:
      - 3000:3000
    depends_on:
      - mongo

[MartinSchoeler]

Probably related to #7172

[khobbits]

I'm using the docker rocketchat image, and and i just had a failed migration.
I tried to rerun the migration by lowering the migration version in the db, but it skipped it saying the migration had already ran.

Newly uploaded avatars appear to work, but none of the old ones do.

My log file looked similar to the above one.

My avatar storage was gridfs, my upload storage s3.

[geekgonecrazy]

@lucasvanhalst / @khobbits had you guys tried any release candidate prior to rc.3 ? If so your migrations are in a partially functioning state. Please see #7172 where trying to walk through and get instances stuck in a limbo state repaired.

[khobbits]

No, no rc builds here. Upgraded from .56 to .57. I'll look though that thread tomorrow.

…

On 4 Jul 2017 22:44, "Aaron Ogle" ***@***.***> wrote: @lucasvanhalst <https://github.com/lucasvanhalst> / @khobbits <https://github.com/khobbits> had you guys tried any release candidate prior to rc.3 ? If so your migrations are in a partially functioning state. Please see #7172 <#7172> where trying to walk through and get instances stuck repaired. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#7385 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAaLC9lgedFPUPYzMJ0dMOni9xBOdKN6ks5sKrJFgaJpZM4OLrc2> .

[KervyN]

We don't use docker and hat 5 instances set up on a single 6 core server.
After upgrading from 0.56.0 to 0.57.0 we encountered the problem that people are not able to login via LDAP when the avatars will be synced. We disabled the avatar sync but now some, not all, avatars wen't missing.

In version 0.56.0 we had the problem that avatar went missing random too. We just clean the the avatarchunks from the database (db.avatars.chunks.remove({})) and resynced.
Now the avatars seem to be in another collection (db.rocketchat_avatars) with only 79 entires (we got 104 user total).

[sampaiodiego]

guys we just released 0.57.1 which should fix any avatar migration issue.

if you were using file system as your avatar storage previous the update, you'll need to set a new environmente variable on first restart to be able to fix the issue. set an env var: AVATARS_PATH with the path used for avatar storage.

[KervyN]

Does anyone know what will happen if multiple instances have different versions?

Edit: comparing the 0.57.1 and 0.57.2 milestones I think I will wait a bit. o.o

[khobbits]

I can confirm that upgrading from 0.57.0 to 0.57.1 does fix my avatars.
(after previously having a broken 0.56.0 -> 0.57.0 migration using gridfs->s3)

[lucasvanhalst]

I can also confirm 0.57.1 sort off works, but not completely. I have 58 users, it migrated 48 avatars. Those remaining 10 users now have a blank avatar. Requests for the avatar return status code 200 but with no content.

Whatever, not gonna waste any more time on this, they'll just have to upload a new one.

[TheReal1604]

@sampaiodiego 0.57.1 had also resolved my issues with migrating avatars. But its @lucasvanhalst mentioned, 2 of 12 users are now missing their avatar. There are no errors in the log after first start of the instance and running the migration.

[TheReal1604]

@sampaiodiego upgraded another env, it seems that there is an issue with jpegs?

2017-07-11T07:09:02.750535783Z { [Error: Command failed: convert: no decode delegate for this image format `' @ error/constitute.c/ReadImage/501.
2017-07-11T07:09:02.750535783Z convert: no images defined `jpeg:/tmp/ufs/EdcrMCcim46yB9L5v' @ error/convert.c/ConvertImageCommand/3210.
2017-07-11T07:09:02.750535783Z ] code: 1, signal: null }

[sampaiodiego]

@TheReal1604 are you getting this on regular uploads or just during the migration?

[TheReal1604]

Doesn't tested uploads, this one was during the migration. @sampaiodiego

[herste]

we have a fresh install of 0.57.1 and the login problem persists - meaning: as soon as we have the avatar sync option turned on, users cannot log in. users that are already logged in get all avatars displayed as they are stored in the LDAP directory. But logging out such a client and trying to re-login fails.
From the logs:

LDAPSync ➔ info Syncing user avatar

After that, no more info on this login process.

Any ideas on that?

[sampaiodiego]

@herste please see #7472

[sampaiodiego]

@TheReal1604 which was the avatar file storage and the file upload storage?

[danpospisil]

@sampaiodiego @TheReal1604 - we have the same problem while upgrading from 0.56.0 to 0.57.1 - we see the same errors like
Command failed: convert: no decode delegate for this image format
for some users. We use GridFS

I have reverted back to 0.56.0 until this is solved.

[rodrigok]

@danpospisil The problem seams to be related with your imagemagick

Which OS are you using?

Can you run identify -list format and check if JPEG is in the list?

[danpospisil]

@rodrigok - I am using the official Docker image for running Rocket.Chat and I see JPEG there:

     JPEG* JPEG      rw-   Joint Photographic Experts Group JFIF format (62)
      JPG* JPEG      rw-   Joint Photographic Experts Group JFIF format (62)

[Horgix]

Can confirm, using the official Docker image in 0.57.1 (rocketchat/rocket.chat:0.57.1), and after having upgraded from 0.56.0 to 0.57.0 to 0.57.1, the problem of avatars migration is not solved.

{"line":"147","file":"rocketchat_migrations.js","message":"Migrations: Migrating from version 98 -> 99","time":{"$date":1500904312651},"level":"info"}
{"line":"147","file":"rocketchat_migrations.js","message":"Migrations: Running up() on version 99","time":{"$date":1500904312659},"level":"info"}
[AVATAR] Migrating avatars. This might take a while.
{"line":"147","file":"rocketchat_migrations.js","message":"Migrations: Finished migrating.","time":{"$date":1500904312700},"level":"info"}
Not migrating, control is locked. Attempt 1/30. Trying again in 10 seconds.
{"line":"147","file":"rocketchat_migrations.js","message":"Migrations: Not migrating, already at version 99","time":{"$date":1500904322721},"level":"info"}
Updating process.env.MAIL_URL
Using FileSystem for custom sounds storage
Using FileSystem for custom emoji storage
ufs: temp directory created at "/tmp/ufs"
➔ System ➔ startup
➔ +----------------------------------------------------+
➔ |                   SERVER RUNNING                   |
➔ +----------------------------------------------------+
➔ |                                                    |
➔ |  Rocket.Chat Version: 0.57.1                       |
➔ |       NodeJS Version: 4.8.1 - x64                  |
➔ |             Platform: linux                        |
[...]
➔ |          Commit Hash: 0c0fa2a8af                   |
➔ |        Commit Branch: HEAD                         |
[...]
[AVATAR] Total users to migrate avatars -> 1
[AVATAR] Migrating 1 of 1

[geekgonecrazy]

@Horgix please try 0.57.2 as it contains an additional bug fix specifically for avatars

[Horgix]

@geekgonecrazy : just updated to 0.57.2, and it looks like it actually made it worse:

{"line":"147","file":"rocketchat_migrations.js","message":"Migrations: Not migrating, already at version 99","time":{"$date":1500965564043},"level":"info"}

So nothing has been migrated anyway, but now it's not even possible to update avatars:

 [Error: FileNotFound: no file with id yG2Q9ZmgvFmF4yz64 found]
 ufs: cannot write file "yG2Q9ZmgvFmF4yz64" (EACCES: permission denied, open '/yG2Q9ZmgvFmF4yz64') { [Error: EACCES: permission denied, open '/yG2Q9ZmgvFmF4yz64']
   errno: -13,
   code: 'EACCES',
   syscall: 'open',
   path: '/yG2Q9ZmgvFmF4yz64' }
 ufs: cannot delete temp file "/tmp/ufs/yG2Q9ZmgvFmF4yz64" (ENOENT: no such file or directory, unlink '/tmp/ufs/yG2Q9ZmgvFmF4yz64')

No idea why it's not trying to look for files directly in /, but it didn't do that before and I have trouble understanding why user avatars would ever be stored in the root directory

[geekgonecrazy]

@sampaiodiego @rodrigok you guys have ideas here?

[rodrigok]

@Horgix are you using filesystem to store avatars? Can you show me the configuration?

[Horgix]

@rodrigok yes I am!

I'm using the official Docker image with the default configuration that is provided inside it. For reference, here is the docker-compose part that I'm using to run the container:

services:
  rocketchat:
    image:          rocketchat/rocket.chat:0.57.2
    restart:        unless-stopped
    volumes:
      - /srv/docker_volumes/rocketchat-uploads:/app/uploads
    environment:
      - PORT=3000
      - ROOT_URL=<my_root_url>
      - MONGO_URL=mongodb://mongo:27017/rocketchat
      - MONGO_OPLOG_URL=mongodb://mongo:27017/local
    depends_on:
      - mongo
    labels:
      traefik.docker.network: "traefik_default"
      traefik.port: "3000"
      traefik.backend: "rocketchat"
      traefik.frontend.rule: "Host:<my_root_url>"
    networks:
      - default
      - traefik

[bbrauns]

We just updated our instance from 0.56 to 0.57.2. I can see some migration steps taking place in the logs. But afterwards for users who had an uploaded avatar it shows a blank image:

Startup Logs: https://pastebin.com/raw/vagNu3sH

[danpospisil]

@rodrigok - any update on this?

[herste]

using avatars in LDAP seems to be still broken on 0.58.2. Whenever the avatar sync option is enabled, users cant login anymore. Anything we can do to help fixing this?

[KervyN]

@herste For this particular issue check #7773

[ghost]

It looks like that it works already.
There was a problem upgrading from 0.56 to 0.58.2, but upgrading from 0.58.2 to 0.58.4 did not give any error.

[maxdwit]

Just to be sure, it isn't fixed in 0.59.1 but moved to 0.60 ? @rodrigok

[Horgix]

Updated to 0.62.2 today and the issue seems to be fixed: avatars are back!

[geekgonecrazy]

Going to go ahead and close. Should be resolved for everyone