Migration from 64 bits to 32 bits.

[tobogdan]

Detailed description of the problem

I was moving my installation from one system to another, and I followed the instructions located here: https://github.com/LycheeOrg/Lychee/wiki/FAQ#how-can-i-move-my-installation

I'm moving from a 4.0.0 installation to another 4.0.0 installation, though from MySQL to MariaDB, which I understand is functionally identical.

When logging in to the session on the new location, I notice that none of the albums or subalbums exist, nor do any pictures exist from the old installation. I get an error in the log:
2019-12-13 21:01:15 -- error -- App\Http\Middleware\ReadCheck::handle -- 45 -- Could not find specified album
2019-12-13 21:01:12 -- error -- App\Http\Middleware\ReadCheck::handle -- 45 -- Could not find specified album

Creating a new album/uploading a new image works as expected.

I've verified the DB instance and all the data in there from the old albums and photos looks correct and is populated. Artisan migration showed no issues, either.

Steps to reproduce the issue

Steps to reproduce the behavior:

1. Install the prerequisites as normal, but instead, follow the instructions from here: https://github.com/LycheeOrg/Lychee/wiki/FAQ#how-can-i-move-my-installation; instead of initializing a new DB.
2. Start the Apache session and log in using the permissions from the old system
3. Observe that none of the albums are populated, and none of the pictures from the old location are there

Output of the diagnostics

Diagnostics
-----------
Info: Latest version of PHP is 7.4
Warning: Using 32 bit PHP, recommended upgrade to 64 bit
Warning: Dropbox import not working. dropbox_key is empty.

System Information
------------------
Lychee-front Version:      3.2.16
Lychee Version (git):      6867dd4 (master) - Data not in Cache
DB Version:                040000
System:                    Linux
PHP Version:               7.3
MySQL Version:             10.3.17-MariaDB-0+deb10u1

Lychee total space:        2.00 GB
Upload folder space:       2.00 GB
System total space:        57.31 GB
System free space:         38.56 GB (67%)

Imagick:                   1
Imagick Active:            1
Imagick Version:           1690
GD Version:                2.2.5

Config Information
------------------
version:                   040000
check_for_updates:         1
sorting_Photos_col:        title
sorting_Photos_order:      ASC
sorting_Albums_col:        description
sorting_Albums_order:      ASC
imagick:                   1
skip_duplicates:           0
small_max_width:           0
small_max_height:          360
medium_max_width:          1920
medium_max_height:         1080
lang:                      en
layout:                    1
image_overlay:             0
image_overlay_type:        takedate
default_license:           reserved
compression_quality:       90
full_photo:                1
delete_imported:           1
Mod_Frame:                 0
Mod_Frame_refresh:         30
landing_page_enable:       1
landing_owner:             someowners
landing_title:             sometitle
landing_subtitle:          somesubtitle
landing_facebook:          
landing_flickr:            
landing_twitter:           
landing_instagram:         
landing_youtube:           
landing_background:        dist/landing.jpg
thumb_2x:                  1
small_2x:                  0
medium_2x:                 0
site_title:                sometitle
site_copyright_enable:     1
site_copyright_begin:      2018
site_copyright_end:        2019
additional_footer_text:    
display_social_in_gallery: 0
public_search:             0
public_recent:             0
recent_age:                1
public_starred:            0
downloadable:              0
photos_wraparound:         1
map_display:               0
zip64:                     1
map_display_public:        0
map_provider:              Wikimedia
force_32bit_ids:           1
map_include_subalbums:     0
update_check_every_days:   3

Browser and system

N/A, all show the same issue

[tobogdan]

The statement below is likely related to the issue:
Lychee total space: 2.00 GB
Upload folder space: 2.00 GB
System total space: 57.31 GB
System free space: 38.56 GB (67%)

Upload folder space should be 6.8GB at the moment, so I'm wondering if it's trying to locate data from a different folder that isn't public/uploads/*

EDIT:

After verifying the image/album I created after-the-fact, I noticed it was using the same directory as the rest of the images correctly. Permissions look OK too, as far as I can tell, as they match the newly created file too.
Excerpt:
2019-12-13 20:51:23 -- notice -- App\ModelFunctions\PhotoFunctions::resizePhoto -- 499 -- No resize (image is too small: 1920x1080)!
2019-12-13 20:51:23 -- notice -- App\Image\ImagickHandler::crop -- 78 -- Saving thumb to /opt/LycheeLaravel/public/uploads/thumb/6865179a9a0815dceb70b387ba959aef@2x.jpeg
2019-12-13 20:51:22 -- notice -- App\Image\ImagickHandler::crop -- 78 -- Saving thumb to /opt/LycheeLaravel/public/uploads/thumb/6865179a9a0815dceb70b387ba959aef.jpeg
2019-12-13 20:51:22 -- notice -- App\ModelFunctions\PhotoFunctions::createThumb -- 115 -- Photo URL is 6865179a9a0815dceb70b387ba959aef.jpg

[d7415]

The folder space stats are taken from the file system. If the usage of your uploads folder is bigger than what's reported, and permissions aren't an issue, that's probably a bug. Edit to add: Check this, as there may have been an incomplete copy - it claims to be using 2GB after all.

That said, it looks like the error you're experiencing is before that step, which would imply the DB. If the tables there really are populated, and were populated using the same user Lychee is, then I'm running low on ideas. Perhaps you could looks for any differences between your test upload and the pre-existing ones? Particularly anywhere they're not listed.

My only other thought would be Lychee versions. So long as the "new" instance is the same or newer that should probably be ok.

[tobogdan]

The transfer is a complete rsync of the public uploads folder. The only difference is that on the older system, I was running on a very early pre-release of the 4.0.0 version - from about a year ago, vs. a fresh pull from a few days ago for this new system. du -h lists the folders taking up the full size as expected and match the source.

DB-wise, the major difference is the old system used MySQL vs. the new one using MariaDB. When importing the DB, and logging in to the website itself, I use my old u/p combination and it works, and the Top-Level album names are there and correct. Just no photos.
When logging in to the MariaDB instance itself, I can see the tables are filled and point to the filenames that do exist in the folder structure - no absolute path, just relative path. The structure seems to match what my source system has.

I may need to torpedo my current DB and re-import, and try just a full rsync of the old lychee install and see if I can match behaviour.

Oh, one other major change, though I really hope this isn't it, is that I went from an x86-based arch to armhf (basically, using an old raspi). I don't think that should be an issue, though, since new photos seem to update and populate in the DB just fine. It feels like the old files just need to be 'rescanned' or something.

[d7415]

The only difference is that on the older system, I was running on a very early pre-release of the 4.0.0 version - from about a year ago, vs. a fresh pull from a few days ago for this new system.

The migration should have handled that

du -h lists the folders taking up the full size as expected and match the source.

This is concerning. It seems too much of a difference to be a difference between real filesize and on-disk. Please double check the permissions. The command we're using is

ls -ltrR {$dir} |awk '{print $5}'|awk 'BEGIN{sum=0} {sum=sum+$1} END {print sum}' 2>&1

If you replace {$dir} with your uploads directory and run it as the Lychee user you should get the answer from the diagnostics page.

DB-wise, the major difference is the old system used MySQL vs. the new one using MariaDB.

That shouldn't be a problem.

When importing the DB, and logging in to the website itself, I use my old u/p combination and it works, and the Top-Level album names are there and correct. Just no photos.

Ah, ok. When I read "none of the albums or subalbums exist" that's quite different to empty from a debugging point of view (but I'm a little time limited right now, and nothing's leaping out at me)

When logging in to the MariaDB instance itself, I can see the tables are filled and point to the filenames that do exist in the folder structure - no absolute path, just relative path. The structure seems to match what my source system has.

That's promising (though not for debugging). The IDs and permissions should carry across too, so that shouldn't be it...

Oh, one other major change, though I really hope this isn't it, is that I went from an x86-based arch to armhf (basically, using an old raspi). I don't think that should be an issue, though, since new photos seem to update and populate in the DB just fine.

I'd be surprised.

One other thought - is Lychee at the same path on both installations?

[tobogdan]

Hey, thanks for replying! Following up on some questions.

If you replace {$dir} with your uploads directory and run it as the Lychee user you should get the answer from the diagnostics page.

I actually don't, which is really weird. I get:

sudo -u www-data ls -ltrR /opt/LycheeLaravel/public/uploads |awk '{print $5}'|awk 'BEGIN{sum=0} {sum=sum+$1} END {print sum}' 2>&1
7383318038

Which is a heck of a lot more than the 2GB it says it's using.

One other thought - is Lychee at the same path on both installations?

Yes, the exact same path.

I had another thought that maybe I was running Apache2 as another user in the old server, and it was enumerating correctly there but not on this, but that's not the case either - same user on both.

Please double check the permissions.

All files are using the following permissions. I assume since Apache's running as that user, we should be getting rwx across the board.

-rwxrwxr-x 1 www-data www-data 7380177 Aug 18 2018 ff0929d1aaa0cf51759a5574c79e3429.jpg

Nothing obvious jumps out at me either. I'm going to hopefully have some time later today to just do a full copy of the original site's tree and retry with that to see if it's some weird stuff happening with my local pull.

[tobogdan]

Did a clone of the original site, using the same DB, and I'm getting a "Server Error or API not found" error. Strange.

[tobogdan]

@d7415, is there any known issue with using a DB primarily populated by 64-bit keys when you move to a 32-bit PHP in a 32-bit OS environment? Should there be some migration step that simplifies the keys or something to that effect?
Also, some sad bit of news, I've done a full reset of the local repository today in hopes it may resolve something, but I saw that instead, I can no longer even upload a new photo (Edits: started with git pull, then escalated to full reset, symptoms are the same throughout). I'm getting an internal error (500) and it's listing it as "Unknown" so I can't even get meaningful error data out of it.
I'll keep investigating, but would appreciate any other ideas.

One other question - the new .env.example file contains LYCHEE_DIST, LYCHEE_UPLOADS, and the URL counterparts, commented out by default. My install is actually entirely located in /opt/LycheeLaravel/ and I've symlinked the public folder over to my /var/www/html/subdir folder, as I did in the prior installation. Should I be uncommenting/changing these values to anything?

[tobogdan]

I've had some time recently to play around again. I got uploads working, and compared a prior-existing photo to the one I uploaded:

|     1577475738 | NAME1(NEWUPLOAD)                         |             | eceb809ca15a175cffd1b30130f121be.jpg |      |      0 |        0 | image/jpeg |   512 |    512 | 55.5 KB  |      |          |                   |                       |                             |           |        |     NULL |      NULL |     NULL |         NULL | NULL                |    0 | eceb809ca15a175cffd1b30130f121be.jpeg | NULL         |           NULL | 5ba730315b51d1c907809adc8f67b034b8a54222 | NULL              | none    | 2019-12-27 19:42:20 | 2019-12-27 19:42:20 |           |          | 360x360 |         |       1 | NULL               |
| 15559814384463 | NAME2(IMPORTED)                          |             | f9d64445c97b4f777a37d1fe1c9d454f.jpg |      |      0 |        0 | image/jpeg |  5760 |   3840 | 7.7 MB   | 800  | f/3.5    | Canon             | Canon EOS 5D Mark III | EF50mm f/1.2L USM           | 1/2500 s  | 50 mm  |     NULL |      NULL |     NULL |         NULL | 2018-08-18 10:55:51 |    0 | f9d64445c97b4f777a37d1fe1c9d454f.jpeg | NULL         | 15559813947835 | 20986d6a900af7ef54c0857bc1534a1d9e089189 | NULL              | none    | 2019-04-23 01:04:04 | 2019-05-31 23:49:56 | 1620x1080 |          | 540x360 |         |       1 | NULL               |

You'll note that the ALBUMID of the old file is 15559813947835 whereas the new upload is NULL.
I noticed that this 4.0.0 pull is actually not even populating the sub-albums that I had existing in my prior configuration.

EDIT:
I played with sub-album generation a bunch and definitely do see sub-albums getting created properly if I do it manually.
It seems the import step doesn't work with sub-albums. Perhaps a scrape needs to happen to re-populate nested albums.
Has anyone else tried this simple exercise?
1: Set up an instance, create test album and test subalbum, put a picture in it
2: Backup the DB, create a new instance, copy over the imported picture, and point the new instance to the backed up album
3: Run the new instance and observe whether the sub-album copied over correctly

EDIT2:
Sorry about the spam, but playing a bit more inside albums.
Noticed some issues right away that may actually point to bitness more than anything.
When I enter an album that was imported from before, I noticed that the title and any metadata also didn't show up. It just shows the album as Untitled.
I created a new album and called it TEST, and the TEST title shows up prominently.
I went into the DB and noticed the IDs are obviously truncated to 32-bit for this album, and any subalbums I make in it. 64-bit values for everything else.

So... maybe it is a bitness issue. See below:

|     1577477957 | TEST                       |        0 |           NULL |                 | NULL          | NULL          |      0 |          1 |              1 |            0 |                    0 | NULL     | none    | 2019-12-27 20:19:17 | 2019-12-27 20:19:17 |
| 15559813803382 | ImportedAlbum              |        0 |           NULL | 20180818        | NULL          | NULL          |      1 |          1 |              1 |            1 |                    1 | NULL     | none    | 2019-04-23 01:03:00 | 2019-12-27 19:36:05 |

[d7415]

@d7415, is there any known issue with using a DB primarily populated by 64-bit keys when you move to a 32-bit PHP in a 32-bit OS environment?

That seems likely...

One other question - the new .env.example file contains LYCHEE_DIST, LYCHEE_UPLOADS, and the URL counterparts, commented out by default. My install is actually entirely located in /opt/LycheeLaravel/ and I've symlinked the public folder over to my /var/www/html/subdir folder, as I did in the prior installation. Should I be uncommenting/changing these values to anything?

If the symlinks are in the appropriate places, that should be sufficient.

Yeah, so your testing indicates that the old album had an ID > 2^32, which will break everything. The 32-bit code will be expecting a 32-bit installation from the start, and so I'm not surprised it doesn't work here. It might even (I haven't checked) make things worse.

If I get a chance later I'll look at adding a note to the migration instructions and see how best to convert.

[d7415]

After a quick look, the migration should handle this. You did run php artisan migrate, right? (Note that this needs to be run for the first time after the old data is in place)

I also only just noticed that the FAQ you linked was from version 3...

[tobogdan]

Yes, I did run php artisan migrate, it did some other stuff but didn't address this piece, at least not in a meaningful way.

The FAQ I linked is off of this Github page - did you move to a new instruction set somewhere else specifically for migration on 4.0.0?

All that I can find by going through here is that link above, which you can get to via going to the Wiki here, then going to the FAQs.

The installation instructions proper (https://github.com/LycheeOrg/Lychee-Laravel/wiki/Install) don't really have much in the way of migrating to a new host as it's very high level instructions. The assumption was that I had to dump the lychee DB from one host, import it on the other, then migrate, after matching my file structure and permissions.

[d7415]

Right. So. When you copied the database, you'll have copied the migrations table. try delete from migrations where migration='2019_04_07_193345_fix_32bit'; and then run php artisan migrate again. Hopefully that will trigger it.

The migration step was added as a one-off fix when upgrading from v3, but migrations only run once per installation. That step should trick it into running again.

The FAQ you linked is from the old repo. But essentially the procedure should work, if you're not changing between 32- and 64-bit architectures.

[tobogdan]

Ah, ok, I'll give that a try on the weekend and report back. Honestly, I'm not that familiar with this side of things, and that was along the lines of what I was thinking was needed. Thanks for the attention and help, nonetheless!

[tobogdan]

Partially there! That seemed to have gotten my first-level stuff all sorted, but all my sub-albums are still missing. It seems the sub-album is still referencing a 64-bit album ID (parent_id), so the 32-bit migration step is only partially addressing it all.

I don't suppose there's another migration I can reset like that?

EDIT
Here's a sample of what I'm talking about:

+------------+----------------------------+----------+----------------+-----------------+---------------+---------------+--------+------------+----------------+--------------+----------------------+----------+---------+---------------------+---------------------+
| id         | title                      | owner_id | parent_id      | description     | min_takestamp | max_takestamp | public | full_photo | visible_hidden | downloadable | share_button_visible | password | license | created_at          | updated_at          |
+------------+----------------------------+----------+----------------+-----------------+---------------+---------------+--------+------------+----------------+--------------+----------------------+----------+---------+---------------------+---------------------+
| 1555981380 | MainAlbum                  |        0 |           NULL | 20180818        | NULL          | NULL          |      1 |          1 |              1 |            1 |                    1 | NULL     | none    | 2019-04-23 01:03:00 | 2019-12-27 19:36:05 |
| 1555981394 | SubAlbum                   |        0 | 15559813803382 |                 | NULL          | NULL          |      1 |          1 |              1 |            1 |                    1 | NULL     | none    | 2019-04-23 01:03:14 | 2019-12-27 19:36:05 |

Note that the parent_id has the extra digits from the original main album id.

EDIT2:
I did manually just reassign the values - it seemed like you're just truncating the last 4 digits of parent_id for the 32-bit variant (insofar as my album setup was concerned, at least), and I've got all the albums populated correctly now!

So it seems like your migration step just missed the parent_id adjustment - a hopefully easy change to implement, and a documentation step to include in the wiki for when migrating across architectures!

[d7415]

Added to FAQ