Downsync fails with Node 10.22

[glowrocks]

Ubuntu 12.04 32 bit on digital ocean.

node app --downsync fails with the following error:

---------------------------------------------------------------------------] 1/5913
events.js:72
throw er; // Unhandled 'error' event
^
Error: spawn ENOENT
at errnoException (child_process.js:980:11)
at Process.ChildProcess._handle.onexit (child_process.js:771:34)
root@riverside-cafe:/usr/local/src/flickrmirror-master#

This is after it's "done fetching photo information from Flickr and during the 1st image during the "downloading photos and metadata from Flickr."

Though I just now noticed it didn't complete the download apparently:

==============================================================-------------] 5900/5913
done fetching photo information from Flickr.

Using node 0.8.9 the downsync completes w/out errors.

[Pomax]

I'm using it with 10.21 myself, and don't see any errors. It's possible it's trying to use imagemagick despite it not existing, in which case it should simply pass over instead of error out. Checking the code now.

[Pomax]

should now work, node-flickrapi was updated and 0.2.17 should no longer cause a crash when the imagemagick program is not installed

[Pomax]

after an npm update, that is.

[glowrocks]

I did an "npm update" but didn't actually confirm the code was updated.

Still saw the error. Installed imagemagick (how is it used if installed?)
and it seems to be working now.

I say seems to be, because it's been running for over 10 mins and usually
failed w/in one.

Can't scroll back far enough to see what it's doing, but I suspect it
wasn't/hadn't downloaded the complete set of data from flickr before.

if that theory is correct it would explain why the main page isn't working;
we'll find out later.

thanks again,

mike

On Wed, Nov 20, 2013 at 9:55 PM, Mike Kamermans notifications@github.comwrote:

should now work, node-flickrapi was updated and 0.2.17 should no longer
cause a crash when the imagemagick program is not installed

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/10#issuecomment-28956324
.

[Pomax]

it should still load a main page if it fails to download anything; even without a single photo, it should still generate a skeleton page =/

[Pomax]

I also just ran node-flickrapi (v02.17) with imagemagick uninstalled, and it doesn't crash, so the errors your seeing may not be caused by imagemagick in this case.

[glowrocks]

I've now broken the display of secondary pages, likely due to the error I
introduced when I manually edited the description fields.

I'll be tied up most of the day, but what I plan to do is remove the old
data, and start over w/the new code.

I'll let you know once I've done that and we can see where things stand at
that time.

thanks!

Mike

On Thu, Nov 21, 2013 at 12:10 AM, Mike Kamermans
notifications@github.comwrote:

I also just ran node-flickrapi (v02.17) with imagemagick uninstalled, and
it doesn't crash, so the errors your seeing may not be caused by
imagemagick in this case.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/10#issuecomment-28960392
.

[Pomax]

If you've comfortable with it, you could zip your ia dir (no need for the actual images. broken image links don't prevent the pages from loading, they just result in lots of 404s after it loads and it tries to load in the images) and mail it to me, and then I'd be happy to run as many tests as I can come up with against that, to see what's going on.

[glowrocks]

no problem. I'll wait til the current downsync completes, see if there's
still an issue, and if so get you the zip

On Thu, Nov 21, 2013 at 12:03 PM, Mike Kamermans
notifications@github.comwrote:

If you've comfortable with it, you could zip your ia dir (no need for the
actual images. broken image links don't prevent the pages from loading,
they just result in lots of 404s after it loads and it tries to load in the
images) and mail it to me, and then I'd be happy to run as many tests as I
can come up with against that, to see what's going on.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/10#issuecomment-29007597
.

[glowrocks]

well, that didn't work, file too large

you can pick up the zip, if you don't mind, at
http://www.allthepages.org/ia.zip

---------- Forwarded message ----------
From: Mail Delivery Subsystem mailer-daemon@googlemail.com
Date: Thu, Nov 21, 2013 at 5:30 PM
Subject: Delivery Status Notification (Failure)
To: mike.rodriquez@gmail.com

Delivery to the following recipient failed permanently:

reply@reply.github.com

Technical details of permanent failure:
Google tried to deliver your message, but it was rejected by the server for
the recipient domainreply.github.com by github-smtp2-ext1.iad.github.net.
[192.30.252.192].

The error that the other server returned was:
552 5.3.4 Error: message file too big

----- Original message -----

DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=gmail.com; s=20120113;
h=mime-version:sender:in-reply-to:references:date:message-id:subject
:from:to:content-type;
bh=9zhjbzr6JHeemzAWCdZ+jhChRt0Kkq/7WZ9fgJ2hqTg=;
b=DEZcrastmB8fZwpxQfCyQmTbAXOFBgRGOdtJV59/Nl4uCT8ATo1ZUoElmJyMyZB4eq

oCXxpuIKZ6crvsYxjJkdLKZmPauwhpdEKrwE8Ti7wcQOq2Ppv9SnG/NZ3qkQhFGZIIt+

4bAuFnN3OXqZ1bSJc8EqzRmYZIu7Npm2XEqwHWRMN9LnJ2KsaOE5JSGNixG35jXH7sC0

pQVrY/G78lkxtbn0hoNKBsUnhkeh1bmMU9qE57+7t2GHpvHAKB/0DeLG7IGQ40ag7mky

M/FjjwYeP1tCw4EAEhojn63sIq3Bm6kdrKsGyg0cCb6p5P+gqqhcSFPXFIO8bM71Hzd8
Pjog==
MIME-Version: 1.0
X-Received: by 10.194.109.167 with SMTP id ht7mr7715487wjb.5.1385076620541;
Thu, 21 Nov 2013 15:30:20 -0800 (PST)
Sender: mike.rodriquez@gmail.com
Received: by 10.217.161.6 with HTTP; Thu, 21 Nov 2013 15:30:19 -0800 (PST)
In-Reply-To: Pomax/flickrmirror/issues/10/29007597@github.com
References: Pomax/flickrmirror/issues/10@github.com
Pomax/flickrmirror/issues/10/29007597@github.com
Date: Thu, 21 Nov 2013 17:30:19 -0600
X-Google-Sender-Auth: a0pY-K7XYVLq3JYBEvGgD4ZAAbo
Message-ID: <
CAGTM4LqGDD9b38eVwFUNimM+PTOyU+2RbhBVr9y0P_L_06LZsw@mail.gmail.com>
Subject: Re: [flickrmirror] Downsync fails with Node 10.22 (#10)
From: Mike Rodriquez mike@mike-sheryl.com
To: "Pomax/flickrmirror" <
reply@reply.github.com

Content-Type: multipart/mixed; boundary=047d7bf10b1af94d4704ebb847b6

Here you go. I'm having a bad day, and thought I made a backup before
editing the descriptions, but I didn't. So this file has the blank
descriptions replaced w/"test1" I'm pretty sure there are no typos this
time, but am still seeing problems when viewing the main page (2ndary pages
still work).

arg. got this from gmail: ia.zip contains an executable file. For security
reasons, Gmail does not allow you to send this type of file.

since when is json an executable?

anyway, I renamed the file and it seems to work

thanks,

Mike

On Thu, Nov 21, 2013 at 12:03 PM, Mike Kamermans
notifications@github.comwrote:

If you've comfortable with it, you could zip your ia dir (no need for the
actual images. broken image links don't prevent the pages from loading,
they just result in lots of 404s after it loads and it tries to load in the
images) and mail it to me, and then I'd be happy to run as many tests as I
can come up with against that, to see what's going on.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/10#issuecomment-29007597
.

[Pomax]

that's just gmail being a twit (which, frankly, is pretty par for the course with it). I downloaded the zip file, and will be seeing what's making the code not generate pages for you.

[Pomax]

errors reproduced with your .json data, so I should be able to find out what's wrong with some poking and coffee. Thanks for packing up the ia dir!

[glowrocks]

Wonderful, glad to hear it!

On Thu, Nov 21, 2013 at 6:11 PM, Mike Kamermans notifications@github.comwrote:

errors reproduced with your .json data, so I should be able to find out
what's wrong with some poking and coffee. Thanks for packing up the ia dir!

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/10#issuecomment-29037869
.

[Pomax]

Turns out one of your collections is a collection-of-collections. I had no idea Flickr allowed you to build one of those, but that's a) pretty cool and b) definitely broke the code. I rewrote it a bit to cope with arbitrarily nested collections, so you should now be able to see your main page (I can, although I see broken image links of course =) on version v0.1.11

[glowrocks]

Thank you very much! It's now working on my site as well. Whoo hoo!

On Thu, Nov 21, 2013 at 6:30 PM, Mike Kamermans notifications@github.comwrote:

Turns out one of your collections is a collection-of-collections. I had no
idea Flickr allowed you to build one of those, but that's a) pretty cool
and b) definitely broke the code. I rewrote it a bit to cope with
arbitrarily nested collections, so you should now be able to see your main
page (I can, although I see broken image links of course =) on version
v0.1.11

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/10#issuecomment-29038982
.

[glowrocks]

Also, since I don't really manage my collections (guess I should, I didn't
realize I could nest "folders" either and that could be very handy!) I
changed the code to show sets instead.

diff views/index.orig.html views/index.html

10,15d9
< {% if collection_keys.length > 0 %}
< {% for key in collection_keys %}
< {% set collection = collections[key] %}
< {% include "stream_collection.html" %}
< {% endfor %}
< {% else %}
23d16
< {% endif %}

On Thu, Nov 21, 2013 at 6:30 PM, Mike Kamermans notifications@github.comwrote:

Turns out one of your collections is a collection-of-collections. I had no
idea Flickr allowed you to build one of those, but that's a) pretty cool
and b) definitely broke the code. I rewrote it a bit to cope with
arbitrarily nested collections, so you should now be able to see your main
page (I can, although I see broken image links of course =) on version
v0.1.11

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/10#issuecomment-29038982
.

[Pomax]

I had a look at the code some more and imagemagick is honestly just now worth it... so I updated node-flickrapi and flickrmirror to just get rid of it entirely. The collection thumbnail is simply copied from flickr.com now as well, based on the collection.iconsmall/.iconlarge properties. It actually brings down main page load time a lot too, as it's no longer loading all the tiny thumbnails. So I guess you get two solutions to the problem =P

[glowrocks]

Hi,

I shared the link to my mirror and then went like I would on any other
server I manage ... I went to look at the logs. D'oh, logging via node
isn't the same as apache/nginx.

Do you have any plans for logging capabilities, or any pointers on how I
might consider adding my own?

I'm a novice javascript programmer, but who knows, I might be able to
figure it out.

And if this is something you've thought about doing, I'll be glad to post
an enhancement request.

thanks,

Mike

On Thu, Nov 21, 2013 at 8:41 PM, Mike Kamermans notifications@github.comwrote:

I had a look at the code some more and imagemagick is honestly just now
worth it... so I updated node-flickrapi and flickrmirror to just get rid of
it entirely. The collection thumbnail is simply copied from flickr.comnow as well, based on the collection.iconsmall/.iconlarge properties. It
actually brings down main page load time a lot too, as it's no longer
loading all the tiny thumbnails. So I guess you get two solutions to the
problem =P

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/10#issuecomment-29044967
.

[Pomax]

It's trivial to add logging to the app by adding a app.use(express.logger()); line in app.js, but we might be able to tailor it to something better - want to file a new issue for that and then we can tackle it there?

[Pomax]

v0.1.13 now logs server events to the logs directory, using one file per 24 hour period, marked on when the application is (re)started. You can control the log verbosity through the .env file, and there are two new vars: LOGCYCLE for the interval between two log files, and LOGFORMAT, which can be any that connect.logger() allows (http://www.senchalabs.org/connect/logger.html)

I'm going to keep this ticket closed, so if you have any other feature request, please file a new issue and I'll be happy to look at it there.