Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FTP Issue #74

Closed
spacemanbobby opened this issue Apr 10, 2019 · 53 comments
Closed

FTP Issue #74

spacemanbobby opened this issue Apr 10, 2019 · 53 comments

Comments

@spacemanbobby
Copy link

So I have put the tags into the scripts to upload the files with that has been mentioned... Image-resize.jpg, startrails, keograms and the daily mp4 all get uploaded to the website.... but it is as if it only puts them in the base directory that I set the FTP account to log into.

lets say...
Allsky@website.com logs into ftp.website.com
On my site I set this account to access public_html\allsky
I have the other folders under allsky.... images, keograms, startrails, videos.... all lower caps and they are 755 under the allsky folder.
When the scripts fire off they all just put them in the base allsky folder even though I have them as \allsky\ for images, \allsky\videos\ for videos... etc...

I have tried all sorts of combo's on the paths but it always just gets put into the base allsky directory that the ftp account points to for signing into a certain directory.
I even tried setting the ftp account to sign into \public_html\ and then let the upload scripts try to get to allsky, but it just dumped it in public_html.

Any ideas.
At this point I am going into ftp from my computer and moving the files to the respective folders.

I even bought a second respberry pi board and installed the OS and the Allsky software onto it. Same results once I get it to upload to the ftp account.

@thomasjacquin
Copy link
Collaborator

I believe this is an issue with some web hosts. However you can fix the issue by adding a mv command to the lftp command.

lftp "$PROTOCOL"://"$USER":"$PASSWORD"@"$HOST":"$IMGDIR" "set net:max-retries 1; set net:timeout 20; put $FILENAME-resize.$EXTENSION; mv $FILENAME-resize.$EXTENSION allsky/$FILENAME-resize.$EXTENSION; bye"

This will move the uploaded image to the allsky directory

You will have to update the lftp command in a few scripts in order to get it to work for startrails, keograms and timelapse as well.

@gremlinforce007
Copy link

Thanks Thomas this was super helpful :)

@spacemanbobby
Copy link
Author

spacemanbobby commented Apr 13, 2019 via email

@gremlinforce007
Copy link

@spacemanbobby can you please post the script edits you've done for timelapses, keograms and startrails please ?

@thomasjacquin
Copy link
Collaborator

These should work but I haven't tested them. Let me know if they don't and I'll update them.

SaveImageDay.sh and saveImageNight.sh:
lftp "$PROTOCOL"://"$USER":"$PASSWORD"@"$HOST":"$IMGDIR" -e "set net:max-retries 1; set net:timeout 20; put $FILENAME-resize.$EXTENSION; mv $FILENAME-resize.$EXTENSION; $IMGDIR/$FILENAME-resize.$EXTENSION; bye"

Timelapse:
lftp "$PROTOCOL"://"$USER":"$PASSWORD"@"$HOST" -e "set net:max-retries 1; put images/$1/allsky-$1.mp4; mv allsky-$1.mp4 $MP4DIR/allsky-$1.mp4; bye"

Keogram (in endOfNight.sh):
lftp "$PROTOCOL"://"$USER":"$PASSWORD"@"$HOST" -e "set net:max-retries 1; put $OUTPUT; mv $OUTPUT $KEOGRAM_DIR/$OUTPUT; bye"

Startrails (in endOfNight.sh):
lftp "$PROTOCOL"://"$USER":"$PASSWORD"@"$HOST" -e "set net:max-retries 1; put $OUTPUT; mv $OUTPUT $STARTRAILS_DIR/$OUTPUT; bye"

@gremlinforce007
Copy link

Thanks i made an edit i'll let you know tonight :)

@gremlinforce007
Copy link

Hi @thomasjacquin

Just tried your post: Still went to root folder:

See code below:

Timelapse:
lftp "$PROTOCOL"://"$USER":"$PASSWORD"@"$HOST":"$MP4DIR" -e "set net:max-retries 1; set net:timeout 20; set ftp:ssl-allow no;put images/$1/allsky-$1.mp4; mv allsky-$1.mp4 $allsky/videos/allsky-$1.mp4; bye"

KeoGram:
lftp "$PROTOCOL"://"$USER":"$PASSWORD"@"$HOST":"$KEOGRAM_DIR" \ -e "set net:max-retries 1;set net:timeout 20; set ftp:ssl-allow no; put $OUTPUT; mv $OUTPUT $allsky/keogram/$OUTPUT; bye"

StarTrails:
lftp "$PROTOCOL"://"$USER":"$PASSWORD"@"$HOST":"$STARTRAILS_DIR" \ -e "set net:max-retries 1; set net:timeout 20; set ftp:ssl-allow no; put $OUTPUT; mv $OUTPUT $allsky/startrails/$OUTPUT; bye"

Question should there be a $ near >> $allsky/startrails/$OUTPUT; ?
Screen Shot 2019-04-15 at 8 23 21 am 2

@spacemanbobby
Copy link
Author

spacemanbobby commented Apr 15, 2019 via email

@thomasjacquin
Copy link
Collaborator

@gremlinforce007 Looking at your Cpanel structure, I think you'll need to remove that dollar sign in front of allsky and it should just work.

@gremlinforce007
Copy link

gremlinforce007 commented Apr 16, 2019

So removing the dollar sign worked for timelapse .. not startrails or keogram any ideas ?

KeoGram:
lftp "$PROTOCOL"://"$USER":"$PASSWORD"@"$HOST":"$KEOGRAM_DIR" \ -e "set net:max-retries 1;set net:timeout 20; set ftp:ssl-allow no; put $OUTPUT; mv $OUTPUT allsky/keogram/$OUTPUT; bye"

StarTrails:
lftp "$PROTOCOL"://"$USER":"$PASSWORD"@"$HOST":"$STARTRAILS_DIR" \ -e "set net:max-retries 1; set net:timeout 20; set ftp:ssl-allow no; put $OUTPUT; mv $OUTPUT allsky/startrails/$OUTPUT; bye"

@thomasjacquin
Copy link
Collaborator

Can you take a screenshot of your cpanel with the content of the allsky directory?
Also I think the directory is called keograms on my server. If these directories don’t exist, you’ll have to create them.

@gremlinforce007
Copy link

Yep added the s to keogram

Screen Shot 2019-04-16 at 5 02 22 pm

@thomasjacquin
Copy link
Collaborator

If it still doesn't send, you can try calling scripts/endOfNight.sh manually from the terminal and look for errors.

@spacemanbobby
Copy link
Author

spacemanbobby commented Apr 16, 2019 via email

@thomasjacquin
Copy link
Collaborator

Ah yes, I see, it's because $OUTPUT evaluates to home/pi/allsky/images/20190416/keogram/keogram-20190416.jpg instead of keogram-20190416.jpg instead of keogram-20190416.jpg.

Adding a variable like this should work I think.
OUTPUT="/home/pi/allsky/images/$LAST_NIGHT/keogram/keogram-$LAST_NIGHT.jpg"
KEOGRAM="keogram-$LAST_NIGHT.jpg"

lftp "$PROTOCOL"://"$USER":"$PASSWORD"@"$HOST":"$KEOGRAM_DIR"
-e "set net:max-retries 1; put $OUTPUT; mv $KEOGRAM
$KEOGRAM_DIR/$KEOGRAM; bye"

@spacemanbobby
Copy link
Author

spacemanbobby commented Apr 16, 2019 via email

@thomasjacquin
Copy link
Collaborator

Yes same idea for startrails, just replace KEOGRAM by STARTRAILS

@spacemanbobby
Copy link
Author

spacemanbobby commented Apr 16, 2019 via email

@gremlinforce007
Copy link

@spacemanbobby Thanks for your assistance as well, can you kindly post your scripts as i'm still having issues :)

@spacemanbobby
Copy link
Author

spacemanbobby commented Apr 17, 2019 via email

@spacemanbobby
Copy link
Author

spacemanbobby commented Apr 17, 2019 via email

@gremlinforce007
Copy link

Hi Team,

My Startrails and keograms are still not working:

FTP/SFTP settings

PROTOCOL='ftp'
USER='allsky@webpage.com'
PASSWORD='*******'
HOST='ftp.webpage.com'
IMGDIR='/images'
MP4DIR='/videos'
KEOGRAM_DIR='/keograms'
STARTRAILS_DIR='/startrails'

> scripts/endOfNight.sh: line 22: keogram-20190417.jpg: command not found

endOfNight.txt

Can you let me know whats wrong ?

@thomasjacquin
Copy link
Collaborator

thomasjacquin commented Apr 18, 2019

Try getting rid of the space in KEOGRAM= "keogram-$LAST_NIGHT.jpg"

@gremlinforce007
Copy link

gremlinforce007 commented Apr 18, 2019

@ZebrasHaveBigEars
Copy link

ONLY IF YOU HAVE MAJOR FTP PROBLEMS, test your ftp using this simple code saved into test.sh and then run from terminal on your pi from within the directory using ./test.sh. Subcommand "debug" provides lots of help for errors and "dir" tells you the starting path for your allsky directory.

#!/bin/bash

lftp "
set ssl:verify-certificate no
set net:max-retries 1
set net:timeout 20
open ftp://website.com
user 'username' 'password'
debug
cd
dir
lpwd
bye"
echo "FTP DONE"
date

In fact, I substituted this lftp code into the allsky scripts and simply added the transfer code provided by Thomas into it. After a week of trying to get the original code to work or using more complex code, this simple code was the only way I managed to get it to work and it was very easy to do. The subcommand "set ssl:verify-certificate/ftp.website.com no" did not work for me for some reason.

It should be easy to substitute in the new code for Thomas' slicker code but which is harder to troubleshoot:

  1. Make a copy of the original code
  2. Put # in front of the original lftp command line
  3. Once tested copy in the new lftp code below the original lftp line (not the first line)
  4. Put in new code using "cd folder/folder" etc to move to the correct folder in your remote website
  5. Check Thomas' original lftp code and put in the appropriate "put" command into the script
  6. Make sure you indent the new code to be positioned in the same place as the original
  7. Make sure you don't forget the fi command!
  8. Save. Test using ./allsky.sh

If anyone wants examples then let me know.

@elineberry75
Copy link

Forgive my ignorance but while I'm pretty good at most "techy" things, I'm pretty new to Pi's and Linux. I have the pi up and running with Allsky and an ASI20MC-S and it seems to work great! I can access it via GUI and see the live images but I'm having problems with getting my FTP to work correctly as NOTHING is uploading. I've ran the scripts manually to look for errors and I get the following error: Fatal error: Certificate verification: certificate common name doesn't match requested host name......
I've edited the scripts to use "set ftp:ssl-allow no" like I've read in some of your examples on these posts but it still does not work. I'm using GoDaddy hosting and configured FTP in the cPanel. I can get it to connect without problems in Filezilla using the same credentials I have configured in ftp-setting.sh. Any suggestions?? I think I could figure out other small issues to get it working like it should if I could ever get the FTP to communicate.

Also, on a side note, does sameImageDay(or Night) upload every image as a separate file or does it overwrite it each time to the filename I designated in settings.json? My image name in settings.json is simply "image.jpg" and it is in the allsky main directory which is where my FTP settings are setup to deposit the image. In other words, I do not have a separate directory for images like I do for startrails, MP4, keograms, etc. Will that work or do I need a "images" directory?

Thanks in advance for any help!!

@spacemanbobby
Copy link
Author

spacemanbobby commented Oct 27, 2019 via email

@elineberry75
Copy link

Thank you. That would be great! I really appreciate it.

@elineberry75
Copy link

So, I've been working on it some and although I'm still getting the "Certificate not matching" error, it is uploading the files. The image and video files are being deposited in the correct directories, however the keograms and startrails are not. Like you said, they are going into the allsky root directory. My ftp_settings.sh file is below. Permissions for all of the folders are the same (0755).

#!/bin/bash

FTP/SFTP settings

PROTOCOL='ftp'
USER='xxxxxxxxx'
PASSWORD='xxxxxxxx'
HOST='xxxxxxxxxxxxxxxxxx'
IMGDIR='/'
MP4DIR='/videos'
KEOGRAM_DIR='/keograms'
STARTRAILS_DIR='/startrails'

@spacemanbobby
Copy link
Author

spacemanbobby commented Oct 27, 2019 via email

@elineberry75
Copy link

For some reason, the images and videos are correct but the keograms and startrails are still going in the root directory. My files are below. I will let it run through the night and check it tomorrow but when I manually run the scripts, it still puts the startrails and keograms in the root directory. Not sure what is different for those and not the images and videos which seems to be working correctly.

saveImageDay.sh
lftp "$PROTOCOL"://"$USER":"$PASSWORD"@"$HOST":"$IMGDIR" -e "set net:max-retries 1; set net:timeout 20; set ftp:ssl-allow no; put $FILENAME-resize.$EXTENSION; mv $FILENAME-resize.$EXTENSION images/$FILENAME-resize.$EXTENSION; bye" &

saveImageNight.sh
lftp "$PROTOCOL"://"$USER":"$PASSWORD"@"$HOST":"$IMGDIR" -e "set net:max-retries 1; set net:timeout 20; set ftp:ssl-allow no; put $FILENAME-resize.$EXTENSION; mv $FILENAME-resize.$EXTENSION images/$FILENAME-resize.$EXTENSION; bye" &

timelapse.sh
lftp "$PROTOCOL"://"$USER":"$PASSWORD"@"$HOST":"$MP4DIR" -e "set net:max-retries 1; set ftp:ssl-allow no; put images/$1/allsky-$1.mp4; mv allsky-$1.mp4 $MP4DIR/allsky-$1.mp4; bye"

endOfNight.sh
lftp "$PROTOCOL"://"$USER":"$PASSWORD"@"$HOST":"$KEOGRAM_DIR" -e "set net:max-retries 5; set ftp:ssl-allow no; put $OUTPUT; mv $KEOGRAM $KEOGRAM_DIR/$KEOGRAM; bye"

lftp "$PROTOCOL"://"$USER":"$PASSWORD"@"$HOST":"$STARTRAILS_DIR" -e "set net:max-retries 5; set ftp:ssl-allow no; put $OUTPUT; mv $STARTRAILS $STARTRAILS_DIR/$STARTRAILS; bye"

@elineberry75
Copy link

So, I figured it out with the help of spacemanbobby. Thanks! The problem was with the "mv" command in endOfNight.sh. Here's what I ended up with that worked:

endOfNight.sh
lftp "$PROTOCOL"://"$USER":"$PASSWORD"@"$HOST":"$KEOGRAM_DIR" -e "set net:max-retries 5; set ftp:ssl-allow no; put $OUTPUT; mv keogram-$LAST_NIGHT.jpg $KEOGRAM_DIR/keogram-$LAST_NIGHT.jpg; bye"

lftp "$PROTOCOL"://"$USER":"$PASSWORD"@"$HOST":"$STARTRAILS_DIR" -e "set net:max-retries 5; set ftp:ssl-allow no; put $OUTPUT; mv startrails-$LAST_NIGHT.jpg $STARTRAILS_DIR/startrails-$LAST_NIGHT.jpg; bye"

The only thing I see having an error at this point and maybe it doesn't matter is the postData.sh script that is called on at the very beginning of endOfNight.sh. For whatever reason, it does not send the data.json file like it is supposed to.

@spacemanbobby
Copy link
Author

spacemanbobby commented Oct 29, 2019 via email

@maserowik
Copy link

Sorry to bring up a old thread but

The only thing I see having an error at this point and maybe it doesn't matter is the postData.sh script that is called on at the very beginning of endOfNight.sh. For whatever reason, it does not send the data.json file like it is supposed to.

on my postData.sh I added to following and the data.json appears to get updated.

lftp "$PROTOCOL"://"$USER":"$PASSWORD"@"$HOST":"$IMGDIR" -e "set net:max-retries 1; set net:timeout 20; set ftp:ssl-allow no; put data.json; mv data.json data.json; bye"

@spacemanbobby
Copy link
Author

spacemanbobby commented Aug 7, 2020 via email

@maserowik
Copy link

When you installed the ne version, did you need to modify the .SH files to get the FTP to work. Other than entering your FTP information into the FTP Upload file. Let me know when you can, thanks.

On Thu, Aug 6, 2020 at 11:22 AM maserowik @.***> wrote: Sorry to bring up a old thread but The only thing I see having an error at this point and maybe it doesn't matter is the postData.sh script that is called on at the very beginning of endOfNight.sh. For whatever reason, it does not send the data.json file like it is supposed to. on my postData.sh I added to following and the data.json appears to get updated. lftp "$PROTOCOL"://"$USER":"$PASSWORD"@"$HOST":"$IMGDIR" -e "set net:max-retries 1; set net:timeout 20; set ftp:ssl-allow no; put data.json; mv data.json data.json; bye" — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#74 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AK2MJOUDKEPBIZHMVVMIJH3R7LYFPANCNFSM4HE7XGBA .

Still fighting with setting up the FTP. I followed what was post here but no luck. I can see the files uploaded to my hosting site but they are not moving to the correct directory. The same for the resized image the files are there but not showing up on my website.

If you like to go off list I cam be reached at mserowik@gmail.com

Mike

@thomasjacquin
Copy link
Collaborator

@maserowik What's your website address?

@spacemanbobby
Copy link
Author

spacemanbobby commented Aug 7, 2020 via email

@maserowik
Copy link

@maserowik What's your website address?

n3bsq.com/allsky

@maserowik
Copy link

n3bsq.com.docx

@thomasjacquin
Copy link
Collaborator

@maserowik
In config.js change this line
imageName: "domains/n3bsq.com/public_html/allsky/image-resize.jpg"
for this one
imageName: "image-resize.jpg"

@thomasjacquin
Copy link
Collaborator

@spacemanbobby What's the error in the terminal when you launch ./allsky.sh? Did you set UPLOAD_IMG=true in config.sh?

@maserowik
Copy link

maserowik commented Aug 8, 2020

With much support for the forum and other user, There are too many to list. The allsky website at n3bsq.com/allsky appears to be working with a few issues.
when hitting the n3bsq.com/allsky, I get a aurora forecast stating it is not dark yet come back in 00 minutes. not sure how to fix that. Also on the time lapse i see the files there but for some reason unplayable on two of my laptops and one of my desktop.

also for the ssh/ssl issue i tried to add the command to the *.sh files but still not luck. I followed the following

https://www.librebyte.net/en/ftp/lftp-fatal-error-certificate-verification-not-trusted/

I added it to the /etc/lftp.conf file.

one step closer.

@spacemanbobby
Copy link
Author

spacemanbobby commented Aug 9, 2020 via email

@spacemanbobby
Copy link
Author

spacemanbobby commented Aug 9, 2020 via email

@maserowik
Copy link

maserowik commented Aug 10, 2020 via email

@maserowik
Copy link

OK I want to thank again all who assisted me with my allsky project. I done so many changes i am not quite sure i can duplicate what I did. One more was completed today. I was stuck on the loading image on my live view. I changed the path in the config.js and nothing worked. I finally went and cleared my cache on my machine and went to my webhoster and cleared the website cache. again not sure if that did anything but now i can see my live image on the website.

n3bsq.com/allsky now appears to be working correctly.

@spacemanbobby
Copy link
Author

spacemanbobby commented Aug 10, 2020 via email

@spacemanbobby
Copy link
Author

spacemanbobby commented Aug 18, 2020 via email

@thomasjacquin
Copy link
Collaborator

thomasjacquin commented Aug 18, 2020

I have had reports about this kind of behaviour. I believe it is caused by the auto-exposure calculation. It is not an allsky thing and it seems to affect only certain models. I think it's coming from the ZWO library. Since this is not related to FTP issues, can you submit a new issue report for this? Thanks

Try using an older version of allsky/lib/armv7/libASICamera2.a
Try replacing it with this one and see if it makes a difference: https://github.com/thomasjacquin/allsky/raw/bd03b5b09f5bac4eaac75a93ddf1c79b489a3b71/lib/armv7/libASICamera2.a

@weatherTai
Copy link

weatherTai commented Aug 25, 2020

I made the changes and ran the endofnight script.. Everything uploads to the correct directories now.

Hi, I am having the same problem, can not move the files (images, timelapse..) to my /home/weatherv/public_html/allsky/images ....
I tried to folliow this topic but not be successful yet. Could you please share your steps to fix? Big thanks

@ebeyonder
Copy link

I would be most grateful for any help, as I don't know much about Linux, Pi or FTP...

I managed to set up my own allsky camera here in Singapore. I can see the live image over VNC very well.

I next tried to set up a website using Thomas' template. The problem is that the FTP uploads do not work.

This is what I have done:

  1. Bought a webhosting plan and domain,
  2. I loaded all the files from the allsky website repo on Github into a 'allsky' directory on my site,
  3. Using cPanel, I set up a FTP account,
  4. I went to the ftp-settings.sh script in allsky/scripts and edited the file to include all my FTP details.
  5. I also changed the config.sh file to specify uploads to website as TRUE.

Then... nothing happens. There is no 'live' picture on the website.

I tried to execute the script by clicking on it and selecting 'execute' and also 'execute in terminal'... but nothing happens. Am I supposed to see anything? There is not even a message saying it failed.

Any advice much appreciated.

@EricClaeys
Copy link
Collaborator

This issue is from an older release of the AllSky software and the issues appear to have been resolved. If you think it might still be relevant, please test it with the newest version and submit a new issue if needed. Thanks..

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

9 participants