ZipSplit Implementation

[ndoulgeridis]

Do you think adding zipsplit as option a good idea? For Dropbox at least will help a lot to bypass single size limit.

If yes let me know to do it and PR.

Thanks a lot.

[dizda]

Hello @Crash21,

I don't know, but it could be interesting.

Are you sure there is a size limit for Dropbox? How much?

[ndoulgeridis]

Hello,

Yes there is I think about 300mb (a bit more). (also class has constant DropboxUploader::DROPBOX_UPLOAD_LIMIT_IN_BYTES)
I have done a fast and a bit dirty implementation for my needs but is working.

<?php
namespace My\MyBundle\Command;


 use Dizda\CloudBackupBundle\Clients\DropboxUploader;
 use Symfony\Bundle\FrameworkBundle\Command\ContainerAwareCommand;
 use Symfony\Component\Console\Input\InputArgument;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Input\InputOption;
use Symfony\Component\Console\Output\OutputInterface;
use Symfony\Component\Finder\Finder;
use Symfony\Component\Finder\SplFileInfo;
use Symfony\Component\Process\Process;

class MyBackUpCommand extends ContainerAwareCommand
{
private $mongoActive;
private $mysqlActive;
private $postgresqlActive;
private $dropboxActive;
private $googleDriveActive;
private $cloudappActive;
private $gaufretteActive;
private $output;

private $processors = array('tar', 'zip', '7z');

//TODO: Improve and use Symfony Filesystem class. Maybe add more config option

/**
 * {@inheritdoc}
 */
protected function configure()
{
    $this
        ->addArgument(
            'processor',
            InputArgument::OPTIONAL,
            'Which processor use? (' . implode(', ', $this->processors) .')'
        )
        ->addOption(
            'folders',
            'F',
            InputOption::VALUE_NONE,
            'Do you want to export also folders?'
        )
        ->setName('cbm:backup:start')
        ->setDescription('Upload a backup of your database to cloud services (use -F option for backup folders)');
}

protected function initialize(InputInterface $input, OutputInterface $output)
{
    $this->mongoActive = $this->getContainer()->getParameter('dizda_cloud_backup.databases.mongodb.active');
    $this->mysqlActive = $this->getContainer()->getParameter('dizda_cloud_backup.databases.mysql.active');
    $this->postgresqlActive = $this->getContainer()->getParameter('dizda_cloud_backup.databases.postgresql.active');

    $this->dropboxActive   = $this->getContainer()->getParameter('dizda_cloud_backup.cloud_storages.dropbox.active');
    $this->googleDriveActive   = $this->getContainer()->getParameter('dizda_cloud_backup.cloud_storages.google_drive.active');
    $this->cloudappActive  = $this->getContainer()->getParameter('dizda_cloud_backup.cloud_storages.cloudapp.active');
    $this->gaufretteActive = $this->getContainer()->getParameter('dizda_cloud_backup.cloud_storages.gaufrette.active');
}

/**
 * {@inheritdoc}
 */
protected function execute(InputInterface $input, OutputInterface $output)
{
    $this->output = $output;

    if ($input->getArgument('processor')) {
        $processorArgument = $input->getArgument('processor');
        if (!in_array($processorArgument, $this->processors)) {
            $this->output->writeln("<error>Incorrect processor $processorArgument</error>");
            $this->output->writeln("<comment>Need one of ". implode(', ', $this->processors) ."</comment>");
            return;
        }
        $this->getContainer()->setParameter('dizda_cloud_backup.processor.service', 'dizda.cloudbackup.processor.' . $processorArgument);
    }

    $processorType = $this->getContainer()->getParameter('dizda_cloud_backup.processor.service');
    $processor = $this->getContainer()->get($processorType);

    if ($this->mongoActive) {
        $this->output->write('- <comment>Dumping MongoDB database... </comment>');

        $database = $this->getContainer()->get('dizda.cloudbackup.database.mongodb');
        $database->dump();

        $this->output->writeln('<info>OK</info>');
    }

    if ($this->mysqlActive) {
        $this->output->write('- <comment>Dumping MySQL database... </comment>');

        $database = $this->getContainer()->get('dizda.cloudbackup.database.mysql');
        $database->dump();

        $this->output->writeln('<info>OK</info>');
    }

    if ($this->postgresqlActive) {
        $this->output->write('- <comment>Dumping PostgreSQL database... </comment> ');

        $database = $this->getContainer()->get('dizda.cloudbackup.database.postgresql');
        $database->dump();

        $this->output->writeln('<info>OK</info>');
    }

    if ($input->getOption('folders')){
        $this->output->write('- <comment>Copying folders... </comment> ');
        $processor->copyFolders();
        $this->output->writeln('<info>OK</info>');
    }

    $this->output->write('- <comment>Compressing archive... </comment> ');
    $processor->compress();
    $this->output->writeln('<info>OK</info>');

    if ($this->dropboxActive)
    {
        if(filesize($processor->getArchivePath()) > DropboxUploader::DROPBOX_UPLOAD_LIMIT_IN_BYTES / 2)
        {
            $this->zipSplit($processor->getArchivePath());
            $finder = new Finder();
            $finder->files()->in(dirname(realpath($processor->getArchivePath())));

            foreach ($finder as $file /* @var $file SplFileInfo */)
            {
                $new_filename = sprintf("%s/%s_%s", $file->getPath(), date('d_m_Y_H_i_s'), $file->getFilename());
                rename($file->getRealPath(), $new_filename);
                $output->writeln("<info>Uploading file: ".basename($new_filename)."</info>");
                $this->getContainer()->get('dizda.cloudbackup.client.dropbox')->upload($new_filename);
            }
        }
        else
        {
            $this->getContainer()->get('dizda.cloudbackup.client.dropbox')->upload($processor->getArchivePath());
        }
    }

    if ($this->googleDriveActive) {
        $this->getContainer()->get('dizda.cloudbackup.client.google_drive')->upload($processor->getArchivePath());
    }

    if ($this->cloudappActive) {
        $this->getContainer()->get('dizda.cloudbackup.client.cloudapp')->upload($processor->getArchivePath());
    }

    if ($this->gaufretteActive) {
        $filesystemName = $this->getContainer()->getParameter('dizda_cloud_backup.cloud_storages.gaufrette.service_name');

        $gaufrette = $this->getContainer()->get('dizda.cloudbackup.client.gaufrette');
        $gaufrette->setFilesystem($this->getContainer()->get($filesystemName));
        $gaufrette->upload($processor->getArchivePath());
    }

    $processor->cleanUp();
    $this->output->writeln('- <comment>Temporary files have been cleared</comment>.');
}

private function zipSplit($getArchivePath)
{
    $command = sprintf("zipsplit -n %s -b %s %s", DropboxUploader::DROPBOX_UPLOAD_LIMIT_IN_BYTES / 2, dirname(realpath($getArchivePath)), $getArchivePath);
    $process = new Process($command);
    $process->setTimeout(null);
    $process->run();

    $process = new Process(sprintf("rm %s", $getArchivePath));
    $process->setTimeout(null);
    $process->run();
}

 }

[dizda]

I think that it should be activated manually in the config.yml

dizda_cloud_backup:
    #...
    processor:
        type: zip
        options:
            #...
            split:
                enable: true
                split_size: some_value_in_bytes
                # and maybe specify for which storage ? To be sure to split only for Dropbox, or GDrive..
                storages: [Dropbox]

Something like this ?

[ndoulgeridis]

Indeed, I will try do it later.

[dizda]

Great, thanks! ;)

[ndoulgeridis]

Hey @dizda

It's done #43
Please if you or someone else can test other Clients except Dropbox as I can't test them but for Dropbox seems bugs free.

[dizda]

Ok thanks, I'll also test with the Gaufrette Adapter.
Thanks for the PR.
However, I think the bundle need some refactoring, because with all theses functionalities, it's a real mess now.

[ndoulgeridis]

You can create some issues and can help with refactoring.

[dizda]

Cool, thanks :)

[dizda]

Fixed with #43