Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MUPX: files lost on re-deploy and restart #72

Closed
LazerJesus opened this issue Apr 21, 2016 · 34 comments
Closed

MUPX: files lost on re-deploy and restart #72

LazerJesus opened this issue Apr 21, 2016 · 34 comments

Comments

@LazerJesus
Copy link

i have set the storagePath relative to user like suggested here
Images = new Meteor.Files({ collectionName: 'Images', storagePath: '~/tmp/meteor/imgs' });
but when i restart meteor in development or redeploy to the project with mupx, all imgs are gone.

@dr-dimitru
Copy link
Member

Make sure you aren't overwriting folder where files is stored on deploy.
I recommend to use rsync with -azuiP flags.

And currently I won't recommend to use relative path to user. I'm emphatically recommend to always use absolute path. Or at least left default value for storagePath to see if it's working properly.

@LazerJesus
Copy link
Author

then i would have to rsync to my local machine everytime I push code to the server, and when I forget once, the data would be overwritten?
or do i missunderstand?

@dr-dimitru
Copy link
Member

Can't clearly answer as I don't now your server's structure and how it's configured, and I've never worked with MUP/MUPX

  • You don't need to sync your server with local machine (unless you want to keep all uploaded files locally)
  • If storagePath is out of your Meteor/NodeJS directory uploaded files shouldn't be overwritten
  • Have you tried to use package's default storagePath?
  • Have you tried to use absolute value for storagePath?

@LazerJesus
Copy link
Author

i have tried storagePath: '/tmp/UploadedFiles', no storagePath and
storagePath: '~/tmp/meteor/imgs'.

I am running an Ubuntu 14.04.4 x64 droplet on digitalOcean

@dr-dimitru
Copy link
Member

  • Do you able to obtain uploaded files in those directories?
  • Do you upload compiled app (NodeJS) or as "Meteor App"?

@LazerJesus
Copy link
Author

  • what do you mean by this?
  • mupx writes this: Deploy app 'anna_app' (linux) to the console when they start to deploy. so i assume its "Meteor App". but any images that i have locally do not get uploaded

@dr-dimitru
Copy link
Member

  1. Can you go to /tmp/UploadedFiles on you server and ls -la to see uploaded files
  2. Yes, MUPX won't deploy anything what is not related to application codebase - this is expected behaviour

Deploy you application, and then upload files

@LazerJesus
Copy link
Author

  • the tmp folder at /opt/anna_app/tmp contains nothing but bundle.tar.gz. and /tmp is empty.

@dr-dimitru
Copy link
Member

Do you have any error/warning at server's console?
Meteor-Files should warn you if storagePath is not writable/accessible

@LazerJesus
Copy link
Author

LazerJesus commented Apr 21, 2016

i can upload. see the portfolio page
the image you see is uploaded via the package.
the current storagePath is '/tmp/UploadedFiles'
i have no warnings.

@dr-dimitru
Copy link
Member

I guess your server just wiping out /tmp folder frequently, as it's basically temporary folder

@LazerJesus
Copy link
Author

but the files are still online.

@dr-dimitru
Copy link
Member

Sorry, but I did't get you.

  1. What is not working right now?

@LazerJesus
Copy link
Author

when i do mupx deploy again, all the files are gone.
so everytime I make a change to the code, i have to reupload all the projects.

@dr-dimitru
Copy link
Member

probably mupx builds app inside /tmp/ and probably wipes out tmp folder after build, try to use /data/UploadedFiles

@LazerJesus
Copy link
Author

with storagePath: '/data/UploadedFiles' the files still go missing after deploy.

and another thing happened a couple of times now - when uploading an image, onUploaded never gets called. the upload just stops at a 100%.
most of the times onUploaded gets called and everything works. but when this happens. i have to reset the whole project.

terminal
console

@dr-dimitru
Copy link
Member

  1. Could you please post your server's logs, as this error clearly says what path must be a string
  2. Could you please post more of your code, where this package is used?

@LazerJesus
Copy link
Author

LazerJesus commented Apr 21, 2016

    'change #file-input': function (e, t) {
        var project = this._id;
        var advance = true;
        var r = 0;
        var files = e.currentTarget.files;

        function insert(file, rank) {
            log('insert');
            t.currentFile.set(Images.insert({
                file: file,
                meta: {rank: rank, project: project, big: false},
                onUploaded: function (error, fileObj) {
                    log('onUploaded');
                    if (error) {
                        console.log(error);
                        return false;
                    } else {
                        Projects.update({_id: project}, {$push: {imgs: fileObj._id}}, function (err, s) {
                            if (err) {
                                console.log('Projects.update Error: ' + err);
                                // return false;
                            } else {
                                console.log('Projects updated: ' + s);
                                t.currentFile.set(false);
                                advance = true;
                            }
                        });
                    }
                }
            }));
        }

        function loop() {
            if (advance == false) {
                setTimeout(function () {
                    loop()
                }, 500);
            } else {
                advance = false;
                insert(files[r], r);
                r++;
                if (r < files.length) {
                    loop()
                }
            }
        }

        loop();
    }

i had to set it up this way to allow for multiple imgs to be uploaded with one action. when I had just passed the images in without this loop(), advance and timeout architecture, it called Projects.update just once and added only one the last image to the project.

what logs do you need?

@dr-dimitru
Copy link
Member

Logs from your server's console

@LazerJesus
Copy link
Author

the screenshot that i shared, those where all.

@dr-dimitru
Copy link
Member

To insert multiple images use _.each:

'change #file-input': function (e, t) {
   _.each(e.currentTarget.files, function (file) {
    Images.insert({/* ... */});
  });
}

@dr-dimitru
Copy link
Member

BTW which version of Meteor and Package are you using?

@LazerJesus
Copy link
Author

ostrio:files@1.4.3
meteor1.3.1

when I use _.each, onUploaded only gets called once for the last image. and thus only the last image gets added to the projects.imgs array

@dr-dimitru
Copy link
Member

  1. Unfortunately I can't answer why directory gets whipped out after you have deployed app via MUPX. As this question is not related to Meteor-Files package directly, please address this question to Meteor-UP team
  2. About multiple file upload - I'll provide an expressive example soon
  3. About server-side error - The path must be a string means storagePath isn't set or not a string, to debug this:
    • Post here your server-side code related to this package
    • Run package in debug mode (pass debug: true to Constructor) and post here server's logs

@LazerJesus
Copy link
Author

LazerJesus commented Apr 21, 2016

  1. ok i'll ask them and when we find out what it is, i will tell you.
Images = new Meteor.Files({
    collectionName: 'Images',
    storagePath: '/data/UploadedFiles',
    debug: true
});

uploading the code right now. logs follow

@LazerJesus
Copy link
Author

ok, so i looked at the logs and saw that this issue always occurred after i had deleted a file from the Images collection.
I rewrote the .remove function to be a serverside method. now the issue stopped occurring. one question about that. Images.remove vs Images.collection.remove - whats the difference?

about the "images go missing after using mupx deploy - there are others with the same issue.
but i need a quick solution. how do you deploy your applications? i'll just use your way…

@dr-dimitru
Copy link
Member

  1. MeteorFiles.remove({}) removes record from MongoDB and unlink file from FS (removes file physically)
  2. MeteorFiles.collection.remove({}) just removes record from MongoDB
  3. You can allow .remove() from client via allowClientCode flag passed into Constructor
  4. Our deploy system is very simple, but server's structure is complicated
    • The nodejs apps is powered by Phusion Passenger + Nginx and behind proxy gate (for load balancing an durability)

Deploy scheme:

On your local machine (assuming you app called: myapp)
# in meteor's app folder
meteor build ../ --architecture os.linux.x86_64 
rsync -auhiP -e 'ssh -p 22' ../myapp.tar.gz ostrio@SERVER_IP:/home/user/
On server:
# go to directory with transferred archive
cd /home/user

# Unpack archive
tar -xvf myapp.tar.gz
rm myapp.tar.gz

# Block below will allow to serve static files via nginx
# we set server's root in nginx config to /var/www/myapp/public
mkdir -p ./bundle/public
cp ./bundle/programs/web.browser/*.css ./bundle/public/
cp ./bundle/programs/web.browser/*.js ./bundle/public/
cp -R ./bundle/programs/web.browser/app/* ./bundle/public/

# change to your user
chown -R user:usergroup ./bundle

cd ./bundle/programs/server
# install npm dependencies as your user
su -s /bin/bash -c 'npm install --production' user
# install extra dependencies if needed
# su -s /bin/bash -c 'npm install package@0.27.0 package2@0.2.1 --production --save' user
cd ../../../

# Remove old compiled JSs and CSSs
rm -Rf /var/www/myapp/programs/web.browser/*
rm /var/www/myapp/public/*.js /var/www/myapp/public/*.css

# Update project's files
rsync -azuhiP ./bundle/ /var/www/myapp

# Fix permissions
chown -R user:usergroup /var/www/myapp
chmod -R 744 /var/www/myapp
chmod 755 /var/www/myapp
chmod -R 755 /var/www/myapp/public

passenger-config restart-app /var/www/myapp
# Show status
passenger-status -v
# Show logs
tail -n 100 -f /var/log/nginx/error.log
  • Do not forget to change SERVER_IP, user:usergroup, myapp and path to project on server (we used /var/www) to your values
  • You can omit/change part with passenger, for example to god, forever, etc.

@LazerJesus
Copy link
Author

ok so i just spent the last three days to get your package to work in my configuration and i cant get it done.

i teach this stuff to myself and this is too much.

i will need to find a different package to host images. sry

@dr-dimitru
Copy link
Member

this thread clearly explains why your FS is changing on every deploy

@Aur0r
Copy link

Aur0r commented Apr 30, 2016

This is clearly a mup/mupx deploy issue since there is no volume mounted that persists your data in the docker container, so this has nothing to do with this package. I am copying a backup of all the files uploaded by ostrio:files to the newly deployed instance after deployment and that solves the issue for now.
BUT for CI reasons if you directly want to deploy from Jenkins/Travis to your server then I can see the issue. For this, you would have to write your own script at the moment, and also the copying of the files takes time.
I think currently you would have to fork the meteor-up project and add the volume to the container creation routine and then if it is good make a pull request to have this functionality. Because currently I can't see a ticket for it (https://github.com/kadirahq/meteor-up/issues) in the active repository of meteor-up on github (https://github.com/kadirahq/meteor-up).

@dr-dimitru
Copy link
Member

@FinnFrotscher could we close this issue?

@LazerJesus
Copy link
Author

yes. i am now using S3.

@dr-dimitru
Copy link
Member

@FinnFrotscher with this package of CFS?

@LazerJesus
Copy link
Author

@dr-dimitru no, directly via their API

@dr-dimitru dr-dimitru changed the title files lost on re-deploy and restart MupX: files lost on re-deploy and restart May 19, 2016
@dr-dimitru dr-dimitru changed the title MupX: files lost on re-deploy and restart MUPX: files lost on re-deploy and restart May 19, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants