Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Flickr2Piwigo sees only 190,168 photos and presents errors. It chops up long album titles. #12

Closed
ElliotMcGucken opened this issue Mar 8, 2018 · 52 comments
Labels

Comments

@ElliotMcGucken
Copy link

ElliotMcGucken commented Mar 8, 2018

Thank you! I have just set piwigo up on a brand new Centos server! I hope to help you debug Flickr2Piwigo. At the moment, it is transferring files, with some issues:

2018-03-08_11-39-39
Here is a screenshot!

Some things I've noticed:

  1. It says "Selection 190168 elements ready for impotration". I actually have over 300,000 images total! Most of those are in albums too!

Here is the site:
http://66.45.225.82/pw/piwigo/

  1. Flickr2Piwigo is taking long album titles and chopping them up, so I am getting multiple album titles.

For instance, it takes an album title from flickr which has 85 photos, chops it up, and creates five albums, each which has 85 photos:

http://66.45.225.82/pw/piwigo/

Albums:
**- and Beautiful! Athena! [85]

  • Fit [85]
  • Thin [85]
  • and a Gold 45 Revolver Gun! Short Shorts Denim Daisy Dukes Jeans Shorts! Tall [85]
  • Cowboy Boots [85]
  • Pretty Cowgirl Model Goddess with Cowboy Hat [85]**

The good news is that it is transferring photos! Thanks!

Also, a question:

Is there a way to run Flickr2Piwigo from the linux shell or command line, rather than from a browser?

I have a dedicated server where I would like to leave it running so as to transfer all my images.

A related question to my above question is that if one closes the browser running Flickr2Piwigo and then opens a new browser on a new computer and runs Flickr2Piwigo, will Flickr2Piwigo continue where it last left off when the first browser was closed? Or will Flickr2Piwigo begin all over again, and download every gallery starting from the very first gallery again?

Thanks! :)

@samwilson samwilson added the bug label Mar 8, 2018
@samwilson
Copy link
Collaborator

Thanks for reporting this; I'll hopefully get to examining things this weekend.

And no, if you start again it'll do it all again (let's aim to fix that though! and maybe add a cronjob possibility, for on-going syncing).

@ElliotMcGucken
Copy link
Author

Thanks Sam!

So if I start it again, will it overwrite the images and albums, or will it add the same images again under different names and in different albums? Will it take up new disk space with the exact same images, as it downloads them again?

@ElliotMcGucken ElliotMcGucken changed the title Flickr2Piwigo sees only 190,168 photos and presents errors. It chops up log album titles. Flickr2Piwigo sees only 190,168 photos and presents errors. It chops up long album titles. Mar 8, 2018
@ElliotMcGucken
Copy link
Author

It seems that if it doesn't download every single image in one go, then one must delete all the photos and databases and reinstall and start over.

I guess this means that each time one needs to update it with some new flickr photos, one must delete the installation, database, and all the images downloaded so far, and reinstall and start over?

Thanks!

@samwilson
Copy link
Collaborator

No, if it's not working correctly now, we'll fix it so it does work correctly. There will be no need to delete the database! :)

@ElliotMcGucken
Copy link
Author

ElliotMcGucken commented Mar 9, 2018

Thanks! After downloading 8,000 photos and closing one browser, I opened a new one and logged in. Flickr2Piwigo has put me in an endless loop for the last few hours. I press "Import All my Pictures," it says "processing" for ten or fifteen minutes, and then takes me back to the beginning, and no photos are added. I have done this a couple dozen times over the last few hours, so I will probably just reinstall and start the whole thing over.

@ElliotMcGucken
Copy link
Author

ElliotMcGucken commented Mar 9, 2018

Greetings! So I deleted the old installation last night and started with a brand new one. I left the browser running overnight, and after it transferred a few thousand photos from flickr to piwigo, the browser crashed.

So now I am trying to start the transfer again, and again I am in a loop. I press "Import All my Pictures," it says "processing" for ten or fifteen minutes, and then takes me back to the beginning, and no photos are added. I can list all my albums, but there is no way to transfer photos.

It seems that if Flickr2Piwigo does not successfully transfer every photo in one single browser-based run, then there is no way to transfer the photos it failed to transfer?

Thanks!

P.S. Each time I get to the "Begin Transfer" screen, I hit the button to begin the transfer, and it takes me back to "Import all my Pictures" screen. I hit the "Import all my Pictures" and after ten minutes it takes me to the "Begin Transfer" screen. I hit the button to begin the transfer, and it takes me back to "Import all my Pictures" screen and again and again. This is the loop i have been stuck in in two different installations.

@ElliotMcGucken
Copy link
Author

A simple, and great, solution would be a simple linux shell command to transfer all the photos for flickr.

It would likely simplify your work, and solve 90%+ of the transfer needs! :)

And if the transfer died, the script could just pick up where it last left off?

Thanks! :)

@samwilson
Copy link
Collaborator

Yes, I'm also thinking that a shell command would be a great addition, and perhaps will work on such a thing before getting some of these other bugs ironed out. You're right, it wouldn't be too hard.

@ElliotMcGucken
Copy link
Author

Thanks! Yes the shell command would make Flickr2Piwigo even more popular, and after it works, it could easily become a browser button used during the setup of Piwigo too. :)

Regarding learning about object-oriented programming for php and the Flickr API, what resources/books might you recommend? Thanks! :)

@ElliotMcGucken
Copy link
Author

Hello Sam!

Might you have any idea when a command-line script for Flickr2Piwigo would be available?

It would be great for new installations! Simple functionality would rock:

  1. Import all flickr photos & data into fresh Piwigo installation.
  2. Pick up where it left off, on case the import dies during import.

And too, the above functionality would also be able to update the Piwigo installation, via a regular chron job or just running the script now and then.

Any idea on how long that might take?

Thanks!

@samwilson
Copy link
Collaborator

The CLI tool is being tracked in #13.

As for books, I'm not really sure. Just start coding something you need! And go from there. Developing a library is a good learning experience, because you're thinking from the point of view of programmers, rather than users, and so you make nicer APIs. :-)

samwilson added a commit that referenced this issue Mar 15, 2018
This switches the import-all function to work with any number of
images and without timing out, by paging through a user's photos
and importing each photo. The API endpoints are renamed (as a
new one is introduced).

DO NOT MERGE YET.

Refs: #12
@samwilson
Copy link
Collaborator

@ElliotMcGucken I have started work on this, and would love any feedback. The PR is not yet complete, but is working for the "import all" function (it has broken some other things, that I'll fix tomorrow). If you could test it it'd be great! You just need to checkout the import-all branch. The album names should be working too.

@ElliotMcGucken
Copy link
Author

ElliotMcGucken commented Mar 15, 2018

Thanks Sam!

I installed your import-all branch in a brand new installation of Piwigo!

It now sees all my photos, but it doesn't seem to be transferring them. After about 15 minutes of "processing," it still shows this:

image

There is no sign of the new Piwigo database being populated, nor of the photos actually being transferred.

Should I just wait longer for the transfer to begin?

Also, if I stop it and restart it, should it still work just the same?

Thanks! Elliot

@ElliotMcGucken
Copy link
Author

Hello Sam!

I did another Fresh Install of your import-all branch in a brand new installation of Piwigo!!

It is able to list all the albums after a couple tries/reloads and some errors.

But then, when I try to import the photos, it doesn't seem to do anything--again it just hangs on this page, saying that it is "Processing:"

image

Thanks!

@samwilson
Copy link
Collaborator

Hm, bother. :)

Yep you should be able to stop and start all you like.

Could you try opening the javascript console (pressing F12 usually does it) and reloading the page and seeing if there are any errors output there?

I'll keep working on it.

@ElliotMcGucken
Copy link
Author

No Javascript errors seen!

image

image

@samwilson
Copy link
Collaborator

I think this is because of the deprecated function warning you're getting.

  • What version of PHP are you using?
  • What database type?
  • Can you set $conf['show_php_errors'] = 0; in your local/config/config.inc.php to hide the notices, and see if anything changes.
  • In the 'network' tab of the developer tools, can you click on the last request that starts with ws.php and have a look at the response that it's getting?

Thanks!

@ElliotMcGucken
Copy link
Author

Thanks! There is no config.inc.php file in that directory:

image

Do I need to create config.inc.php in that directory? What all does it need?

Where might I find the "developer tools"? Are they in Piwigo?

Here is my PHP version:

image

Here is my MYSQL version:

image

Thanks!

@samwilson
Copy link
Collaborator

The local config stuff is explained here: http://piwigo.org/doc/doku.php?id=user_documentation:use:features:conf_locale — basically it involves creating that file if it doesn't exist and putting the following in it:

<?php

$conf['show_php_errors'] = 0;

(Or you can do the equivalent setting in your php.ini file by setting the error_reporting setting.)

The problem is that the create_function() function has been deprecated in PHP 7.2. The fix for this seems to be being tracked in Piwigo/Piwigo#834

Oh and "Developer tools" is the name of that sidebar that appears when you hit F12. It's jolly useful, if a little confusing!

samwilson added a commit that referenced this issue Mar 15, 2018
This switches the import-all function to work with any number of
images and without timing out, by paging through a user's photos
and importing each photo. The API endpoints are renamed (as a
new one is introduced).

DO NOT MERGE YET.

Refs: #12
@ElliotMcGucken
Copy link
Author

Thanks Sam!

Adding the file config.inc.php makes it work now:

It is importing! Will let you know how it goes!

Thanks,

Elliot

@ElliotMcGucken
Copy link
Author

ElliotMcGucken commented Mar 16, 2018

Update! It froze after importing 275 photos and hung for ten minutes or so.

So I restarted it, and it looks like it is starting the import all over from the very beginning?

Is it actually re-downloading all the files again? Or is it skipping the ones it already has?

image

If I stop it and hit refresh again, it starts all over again.

Does this look right?

P.S. It DOES look like it is re-downloading everything again, as it is eating up new disk space.

@samwilson
Copy link
Collaborator

it shouldn't re-download: it checks the file name of each photo before downloading. The Flickr ones are given names like flickr-<username>-<id> and if it finds a match it'll say "Already imported: " in the little popup message. That's the idea anyway!

I've updated the patch; you should redownload it.

It will start the download from the beginning each time, note. But it'll go a lot quicker for the ones it's already got.

Also, make sure to purge your browser cache when reloading the import page (ctrl-shift-r) to make sure it's running with the latest javascript.

@samwilson
Copy link
Collaborator

Oh, just to clarify, if you're running this from a git checkout, you'll need to:

git fetch
git reset origin/import-all --hard

because I'm doing a single-commit patch.

samwilson added a commit that referenced this issue Mar 16, 2018
This switches the import-all function to work with any number of
images and without timing out by paging through a user's photos
and importing each photo. The API endpoints are renamed (as a
new one is introduced).

Tag importing is fixed to use the 'raw' value rather than the
spaceless normalized one.

Refs: #12
@ElliotMcGucken
Copy link
Author

Thanks Sam!

Just to be clear, just now I made a brand new, fresh installation of Piwigo and your Flickr2Piwigo import-all branch. In order to install your Flickr2Piwigo import-all branch, I just now entered:

git clone -b import-all https://github.com/mistic100/Flickr2Piwigo.git

Is that enough to give me the latest version, on a brand new installation made just now?

Or must I also enter:

git fetch
git reset origin/import-all --hard

And if I need to enter these commands too, can you please provide the full text of these commands for this case, without abstraction?

Thanks!

Elliot

samwilson added a commit that referenced this issue Mar 20, 2018
This switches the import-all function to work with any number of
images and without timing out by paging through a user's photos
and importing each photo. The API endpoints are renamed (as a
new one is introduced).

Tag importing is fixed to use the 'raw' value rather than the
spaceless normalized one.

Checksums of files are also checked before import, along with
filenames.

Refs: #12
@samwilson
Copy link
Collaborator

Right, I think I've found and fixed a few issues. Your username has an ampersand in it, which was confusing the checking; the reporting back of checksum-matched already-imported photos was not working; and there was a chance that long album names were being truncated prior to creation, and so the creation was failing.

The other thing I'd like you to check is the 'orphan' photos that get imported. If there's no album on Flickr, then the photo will be imported as an orphan. This is fine, but I think you won't be able to view it in the front end till you've given it an album.

Could you have another crack with the latest code, and see how you go? :-) Thanks!!

@ElliotMcGucken
Copy link
Author

ElliotMcGucken commented Mar 20, 2018

Thanks Sam!

Here are the log files for today's install: http://66.45.225.82/20logs/

I created a brand new fresh install of everything today, and I'm seeing some of the same errors as yesterday (I think the errors are due to characters such as apostrophes and/or periods in the photo titles). Again, when it gets to the album:

Irish Lass St. Patty's Day! Cowgirl Swimuit Bikini Model! Beautiful Golden Ratio Composition Photography Surf Goddess! Athletic Action Portraits of St. Patrick's Day Bikini Model! Irish Athena, Artemis, and Aphrodite! Sexy Hot dx4/dt=ic! Cowboy Boots!

It fails to create the album again. Just like yesterday, it creates the thumbnails:

http://66.45.225.82/20pw/piwigo/index.php?/recent_pics/start-480

But when one clicks on a thumbnail, one gets the same error again:

image

The titles of the photos are the same as the album title:

Irish Lass St. Patty's Day! Cowgirl Swimuit Bikini Model! Beautiful Golden Ratio Composition Photography Surf Goddess! Athletic Action Portraits of St. Patrick's Day Bikini Model! Irish Athena, Artemis, and Aphrodite! Sexy Hot dx4/dt=ic! Cowboy Boots!

This character string has 251 characters in it. Might that be a problem after addslashes and all?

Are the apostrophes ' in "St. Patty's" and "St. Patrick's Day" problems perhaps? Or are the periods in "St." a problem?

I have long titles, but once your code is debugged for the long titles, it should be great!

Orphan Photo Test
To test the orphan photo importation, I uploaded 274 new photos to Flickr without placing them in an album. It appears they were imported by Flickr2Piwigo as the browser reported Success! with each photo title "Nikon D810 Beach Portraits Santa Barbara! White Summer Dress! 45Epic 45SURF Swimsuit Bikini Model! Beautiful Golden Ratio Composition Photography Surf Goddesses! Athletic Action Portraits of Swimsuit Bikini Models! High Res Venus! Sexy Hot dx4/dt=ic !", but they weren't visible. Your code said "Success: Already imported:" when I refreshed the browser to restart the importation:

image

but there is still no way to view these photos in Piwigo--they show up nowhere.

ONLY photos in albums are seen anywhere:

http://66.45.225.82/20pw/piwigo/index.php?/categories

The orphan photos appear nowhere in "recent_pics" either:
http://66.45.225.82/20pw/piwigo/index.php?/recent_pics

I then added the photos to a brand new album on Flickr, and I refreshed Flickr2Piwigo, but Flickr2Piwigo failed to create the new album. It still stated, "already imported" for all the photos, skipping them, but again, there is no way to view these photos at my Piwigo install.

Here are the log files for today's install: http://66.45.225.82/20logs/

Hope this helps!

Thanks, Elliot :)

@ElliotMcGucken
Copy link
Author

ElliotMcGucken commented Mar 20, 2018

Greetings Sam!

Flickr2Piwigo has been running fairly well for a few hours now! Here are the logfiles for up until this moment:
http://66.45.225.82/20logs.2/

It has been skipping many albums. Here are the Piwigo albums:
http://66.45.225.82/20pw/piwigo/

And here are the flickr albums:
https://www.flickr.com/photos/herosjourneymythology45surf/albums

It seems that many of the albums that are skipped are albums that were created some time after the photos were uploaded to flickr? It seems to do better with photos that are uploaded into albums, and/or at the same time that the albums are created.

And too, it still gives errors:
image

Here are the most recent logfiles which are up to date:
http://66.45.225.82/20logs.2/

I just stopped Flickr2Piwigo for the day.

We're getting close!

Thanks Sam!

Elliot :)

samwilson added a commit that referenced this issue Mar 22, 2018
This switches the import-all function to work with any number of
images and without timing out by paging through a user's photos
and importing each photo. The API endpoints are renamed (as a
new one is introduced).

Tag importing is fixed to use the 'raw' value rather than the
spaceless normalized one.

Checksums of files are also checked before import, along with
filenames.

Logging is added throughout the import process.

Refs: #12
samwilson added a commit that referenced this issue Mar 22, 2018
This switches the import-all function to work with any number of
images and without timing out by paging through a user's photos
and importing each photo. The API endpoints are renamed (as a
new one is introduced).

Tag importing is fixed to use the 'raw' value rather than the
spaceless normalized one.

Checksums of files are also checked before import, along with
filenames.

Logging is added throughout the import process.

Refs: #12
@samwilson
Copy link
Collaborator

You can see orphans at /admin.php?page=batch_manager&filter=prefilter-no_album

I've added some more logging. I'm really at a bit of a loss as to what is going wrong! :-(

It should be logging the titles of the albums it's creating, could you check to see if they're what you expect? It doesn't do anything about the creation time of the album, so that's not the problem. It only creates the albums as it encounters the photos in them, so it may not do them in any particular order.

Sorry I'm not very helpful this evening. :-) I'll keep thinking and debugging.

@ElliotMcGucken
Copy link
Author

ElliotMcGucken commented Mar 22, 2018

Thanks Sam!

So basically Flickr2Piwigo imports photos in chronological order, as opposed to albums in chronological order? That means that if I uploaded photos a year ago, but just put them in an album created today, the album won't be created by Flickr2Piwigo until the photos are encountered during the chronological import?

Were you able to fix this bug from the previous install:

Here are the log files for today's install: http://66.45.225.82/20logs/

I created a brand new fresh install of everything today, and I'm seeing some of the same errors as yesterday (I think the errors are due to characters such as apostrophes and/or periods in the photo titles). Again, when it gets to the album:

Irish Lass St. Patty's Day! Cowgirl Swimuit Bikini Model! Beautiful Golden Ratio Composition Photography Surf Goddess! Athletic Action Portraits of St. Patrick's Day Bikini Model! Irish Athena, Artemis, and Aphrodite! Sexy Hot dx4/dt=ic! Cowboy Boots!

It fails to create the album again. Just like yesterday, it creates the thumbnails:

http://66.45.225.82/20pw/piwigo/index.php?/recent_pics/start-480

But when one clicks on a thumbnail, one gets the same error again:

image

The titles of the photos are the same as the album title:

Irish Lass St. Patty's Day! Cowgirl Swimuit Bikini Model! Beautiful Golden Ratio Composition Photography Surf Goddess! Athletic Action Portraits of St. Patrick's Day Bikini Model! Irish Athena, Artemis, and Aphrodite! Sexy Hot dx4/dt=ic! Cowboy Boots!

This character string has 251 characters in it. Might that be a problem after addslashes and all?

Are the apostrophes ' in "St. Patty's" and "St. Patrick's Day" problems perhaps? Or are the periods in "St." a problem?

What do you think about this question--"Are the apostrophes ' in "St. Patty's" and "St. Patrick's Day" problems perhaps? Or are the periods in "St." a problem?"

I know you said the ampersand posed a problem in names/titles. What about the ' apostrophes and . periods?

Thanks Sam!

@samwilson
Copy link
Collaborator

Yep, it goes in chronological order (of upload date), or at least it goes in whatever order Flickr gives us for the "people.getPhotos" API call. So albums will not be populated in order, but will be by the end.

And no, it's a good thought, but slashes, apostrophes, quotation marks, periods, etc. are all fine in photo and album titles. I've tested with https://www.flickr.com/photos/153362204@N03/39172591000/ with no problems.

Maybe there's some rate-limiting or something going on? :(

@samwilson
Copy link
Collaborator

Tags! Maybe it's tags with funny characters. :-)

[mysql error 1064] You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '' quotes and \slashes\'' at line 3

SELECT id
  FROM tags
  WHERE name = 'tags 'with' quotes and \slashes\'
;

samwilson added a commit that referenced this issue Mar 24, 2018
This switches the import-all function to work with any number of
images and without timing out by paging through a user's photos
and importing each photo. The API endpoints are renamed (as a
new one is introduced).

Tag importing is fixed to use the 'raw' value rather than the
spaceless normalized one.

Checksums of files are also checked before import, along with
filenames.

Logging is added throughout the import process.

Refs: #12
@samwilson
Copy link
Collaborator

Yep, seems to have been one problem at any rate. Could you please test again with the latest patch? :-)

@ElliotMcGucken
Copy link
Author

ElliotMcGucken commented Mar 24, 2018

Greetings Sam! Thanks!

I just installed a brand new fresh install of everything, and here is the result:
http://66.45.225.82/24pw/piwigo/
http://66.45.225.82/24logs/ (latest log files for today's install)

I didn't see any glaring errors, but when Flickr2Piwigo showed 500 photos imported in the "Processing 500," it hung and stopped importing.

So I refreshed the browser, and it seemed to work fine, skipping all the photos that it had already imported.

But then when it hit "Processing 500" again, it stopped again, so it hangs here at "Processing 500:"

image

I refreshed the browser one more time, and again, it stopped at "Processing 500."

I just copied the latest log files here:
http://66.45.225.82/24logs/

I feel we're getting close!

Thanks so much! Elliot :)

P.S. Have you seen these funny characters in your Flickr2Piwigo install? I've noticed them before too:
image

samwilson added a commit that referenced this issue Mar 25, 2018
This switches the import-all function to work with any number of
images and without timing out by paging through a user's photos
and importing each photo. The API endpoints are renamed (as a
new one is introduced).

Tag importing is fixed to use the 'raw' value rather than the
spaceless normalized one.

Checksums of files are also checked before import, along with
filenames.

Logging is added throughout the import process.

A button is added to stop the import before it's finished.

Refs: #12
@samwilson
Copy link
Collaborator

Oops, I was checking "totalPages < currentPage" rather than "currentPage < totalPages"! :) Fixed now. Try again.

I also made the stop button work a bit better (so it gets its own little popup message saying "stopped").

No idea about the funny characters I'm afraid. Do they appear when no plugins are turned on? If so, I reckon you should raise a bug against core Piwigo.

@ElliotMcGucken
Copy link
Author

ElliotMcGucken commented Mar 25, 2018

Thanks Sam!

I did a fresh install of everything, and it seems to be working well!
http://66.45.225.82/25pw/piwigo/

I did notice just a few errors, but it keeps on going and keeps transferring images and all, despite these errors which seem isolated so far:
image

Here are the log files after transferring around 2500 photos, and as it is still running:
http://66.45.225.82/25logs/

Flickr2Piwigo is still running & transferring--just one issue here:

The Home/Recent Photos pages seems stuck at 2248 photos:
http://66.45.225.82/25pw/piwigo/index.php?/recent_pics/start-2235
image

Even though the photos are continuing to load and it says 2633 under the albums:

image
(this number under the albums list is growing as I write this, while the 2248 number above in Home/Recent photos is stuck at 2248)

Here are the logs:
http://66.45.225.82/25logs/

Thanks!

P.S.: Now and then the counter at the top of the page jumps to 2249 photos, but then it quickly jumps back down to 2248 photos: http://66.45.225.82/25pw/piwigo/index.php?/recent_pics/start-2235
image

Meanwhile, the number under the albums keeps growing as photos are imported:
image

@ElliotMcGucken
Copy link
Author

ElliotMcGucken commented Mar 25, 2018

Hello Sam!

It seems that the above problem is still centered about the same problematic album from earlier attempts, as Flickr2Piwigo is still unable to create this album: "Irish Lass St. Patty's Day! Cowgirl Swimuit Bikini Model! Beautiful Golden Ratio Composition Photography Surf Goddess! Athletic Action Portraits of St. Patrick's Day Bikini Model! Irish Athena, Artemis, and Aphrodite! Sexy Hot dx4/dt=ic! Cowboy Boots!"

Note that Flickr2Piwigo imports the albums just before and after this problematic album circrled below at my flickr page:

image

Note how the circled album above is missing here, while the other albums around it are present:
http://66.45.225.82/25pw/piwigo/index.php?/categories

And this time, after it hits the album during import, it won't add any more photos to the Home / Recent photos [2248] which is now stuck at 2248, as described above.

Here is the album on flickr:
https://www.flickr.com/photos/herosjourneymythology45surf/albums/72157664755226537

Here is one of the photos:
https://www.flickr.com/photos/herosjourneymythology45surf/39957959485/in/album-72157664755226537/

Are there any characters in the album title, or photo title, descriptions, or keywords which could cause a bug? Is the album or photo title or description too long? When I run the title through addslashes (https://www.tools4noobs.com/online_php_functions/addslashes/), I get:

Irish Lass St. Patty's Day! Cowgirl Swimuit Bikini Model! Beautiful Golden Ratio Composition Photography Surf Goddess! Athletic Action Portraits of St. Patrick's Day Bikini Model! Irish Athena, Artemis, and Aphrodite! Sexy Hot dx4/dt=ic! Cowboy Boots!

So are the apostrophes a problem? Does the addslashes make the string too long? Could it be a problem that there are two apostrophes to handle in the above string?

Is the title too long, or does it have too many apostrophes, periods, or exclamation marks? Are the apostrophes the problem somewhere in the code?

Irish Lass St. Patty's Day! Cowgirl Swimuit Bikini Model! Beautiful Golden Ratio Composition Photography Surf Goddess! Athletic Action Portraits of St. Patrick's Day Bikini Model! Irish Athena, Artemis, and Aphrodite! Sexy Hot dx4/dt=ic! Cowboy Boots!

Is the slash a problem in the keywords in dx4/dt=ic?

Is the album title too long perhaps?
image

Irish Lass St. Patty's Day! Cowgirl Swimuit Bikini Model! Beautiful Golden Ratio Composition Photography Surf Goddess! Athletic Action Portraits of St. Patrick's Day Bikini Model! Irish Athena, Artemis, and Aphrodite! Sexy Hot dx4/dt=ic! Cowboy Boots!

This same album seems to keep breaking Flickr2Piwigo, and now it doesn't even look like it creates the thumbnails anymore for the album? So it seems like now the album and all its photos are skipped?

Other than this album, things are looking great! :)

Here are the latest logs:
http://66.45.225.82/25logs.2/

Thanks Sam!

Elliot :)

@samwilson
Copy link
Collaborator

I think the errors being displayed might be something to do with being rate limited from Flickr's end. I'll add some better checking for that.

The recent photos thing only shows photos posted within a day before the lastest-posted photo. So if you're importing posted dates from Flickr, the recent-photos view should max out at however many you uploaded within a day... but of course, that will vary as the latest-posted date changes (as other photos are uploaded).

I'll look into the album naming a bit more. I think it shouldn't mind the length or funny characters, but will do some better checking.

Thanks so much for helping debug all this!

@biffhero
Copy link

I wanted to test what Flickr did when I deleted an album (need to make sure that it does not delete what is in it, even though they tell you that in the help).

  1. Uploaded one image.
  2. Created an album.
  3. Put that image in the new album.
  4. Deleted the album.
    -- At this point, my test with Flickr was finished. So I had a new photo in my Flickr photostream that I needed to import with the import-all branch of Flickr2Piwigo.

When I went to Flickr2Piwigo, the plugin says "No photo in the current set."

Thank you,
Rob

@samwilson
Copy link
Collaborator

samwilson commented Mar 26, 2018 via email

@ElliotMcGucken
Copy link
Author

ElliotMcGucken commented Mar 26, 2018

Thanks Sam!

You write:

I'll look into the album naming a bit more. I think it shouldn't mind the length or funny characters, but will do some better checking.

It is interesting to note that the album title is the same as the photo titles in this case, and that Flickr2Piwigo is both unable to create the album and unable to successfully display the full-size photos.

Album name & photo title:

Irish Lass St. Patty's Day! Cowgirl Swimuit Bikini Model! Beautiful Golden Ratio Composition Photography Surf Goddess! Athletic Action Portraits of St. Patrick's Day Bikini Model! Irish Athena, Artemis, and Aphrodite! Sexy Hot dx4/dt=ic! Cowboy Boots!

In order to see the thumbnails, do a search on "Irish" here:
http://66.45.225.82/25pw/piwigo/index.php?/search/1

This is the page of thumbnails one sees:
image

Now when one clicks on any one thumbnail, one gets this error:
image

Hope this helps!

Here are the latest logs again from the most recent import:
http://66.45.225.82/25logs.2/

Thanks,

Elliot

P.S. I searched for the word "Irish" in the log file (http://66.45.225.82/25logs.2/), and it does look like the album is being created?

image

But then there is no such album here:
http://66.45.225.82/25pw/piwigo/

I think that the album name is giving Flickr2Piwigo a problem somewhere:

Irish Lass St. Patty's Day! Cowgirl Swimuit Bikini Model! Beautiful Golden Ratio Composition Photography Surf Goddess! Athletic Action Portraits of St. Patrick's Day Bikini Model! Irish Athena, Artemis, and Aphrodite! Sexy Hot dx4/dt=ic! Cowboy Boots!

Perhaps if you created an album and photos with this same title at your flickr, you'd see the error too?

Here is the album on flickr:
https://www.flickr.com/photos/herosjourneymythology45surf/albums/72157664755226537

Here is one of the photos:
https://www.flickr.com/photos/herosjourneymythology45surf/39957959485/in/album-72157664755226537/

Thanks! Elliot :)

samwilson added a commit that referenced this issue Mar 28, 2018
This switches the import-all function to work with any number of
images and without timing out by paging through a user's photos
and importing each photo. The API endpoints are renamed (as a
new one is introduced).

Tag importing is fixed to use the 'raw' value rather than the
spaceless normalized one.

Checksums of files are also checked before import, along with
filenames.

Logging is added throughout the import process.

A button is added to stop the import before it's finished.

Refs: #12
@samwilson
Copy link
Collaborator

Ah! Yes, indeed, the apostrophes (or rather, my lack of understanding of various bits of the Piwigo codebase) were to blame. Fixed now, and I've tested with your erroring album title and it's worked.

So, perhaps we're closer now! :-) Give it a test and see how it goes.

@ElliotMcGucken
Copy link
Author

Thanks Sam!

I ran a brand new install today, and it now works for this "Irish" gallery too!

image

http://66.45.225.82/28pw/piwigo/index.php?/category/21

It's working great!

I have seen some random errors, which may be flickr timing out or something, but they seem fairly minor--here are today's logs:

http://66.45.225.82/28pw/piwigo/_data/logs/

And here is what the minor errors look like:

image

Looking great!

One small request would be to put any photo which is not in an album into an album titled, "Not in an official album."! It's not a huge deal, but it seems it may be simple and useful to people on down the line, as sometimes flickr won't add every photo to an album, even though it makes it look like it did.

Also, if two albums have the same name, can we please make sure it creates both albums? Perhaps it already does this?

Thanks Sam!

I am going to stop Flickr2Piwigo now and restart it from home, and let it run overnight.

Will let you know how it goes.

Thanks for all your work on this! It's looking great.

Best,

Elliot

@ElliotMcGucken
Copy link
Author

Hello Sam!

Would it be possible to import and store the safety status of the photo from flickr, such as safe, moderate, or restricted?

If this isn't possible or easy, would it be possible to import moderate and restricted photos as private, while importing the safe photos as public?

Thanks!

@samwilson
Copy link
Collaborator

One small request would be to put any photo which is not in an album into an album titled, "Not in an official album."! It's not a huge deal, but it seems it may be simple and useful to people on down the line, as sometimes flickr won't add every photo to an album, even though it makes it look like it did.

Tracked in #18.

Also, if two albums have the same name, can we please make sure it creates both albums? Perhaps it already does this?

No, it doesn't, and actually it's sort of tricky to do that because it checks to see if the album already exists by looking for its name. I think the easiest solution would be to rename one of the albums on Flickr prior to import. Do you have many like this?

Would it be possible to import and store the safety status of the photo from flickr, such as safe, moderate, or restricted?

Tracked as #17.

Let's carry on discussion over on #15, given that we're pretty close to solving the problem at hand. :-)

samwilson added a commit that referenced this issue Mar 30, 2018
This switches the import-all function to work with any number of
images and without timing out by paging through a user's photos
and importing each photo. The API endpoints are renamed (as a
new one is introduced).

Tag importing is fixed to use the 'raw' value rather than the
spaceless normalized one.

Checksums of files are also checked before import, along with
filenames.

Logging is added throughout the import process.

A button is added to stop the import before it's finished.

Refs: #12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants