New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use Amazon Cloud Drive as external storage #17012

Closed
wdenx opened this Issue Jun 18, 2015 · 48 comments

Comments

Projects
None yet
@wdenx

wdenx commented Jun 18, 2015

It would be nice if we could use Amazon Cloud Drive (ACD) as external storage,
for obvious reasons ideally with automatic encryption of the data.
ACD looks interesting as it offers really cheap storage (unlimited storage for < $60 / year).
Note that ACD is not compatible with S3, it uses a different API.

Code to access this API exists (see for example https://github.com/yadayada/acd_cli )

Thanks in advance,

Wolfgang

@DeepDiver1975

This comment has been minimized.

Show comment
Hide comment
@DeepDiver1975

DeepDiver1975 Jun 18, 2015

Member

NOTE: starting with 8.2 we will open up the files_external APIs even more to allow any developer to implement external filesystems as an app - anybody will then be in the position to implement external storages like this.

Member

DeepDiver1975 commented Jun 18, 2015

NOTE: starting with 8.2 we will open up the files_external APIs even more to allow any developer to implement external filesystems as an app - anybody will then be in the position to implement external storages like this.

@ronoliverclarin

This comment has been minimized.

Show comment
Hide comment
@ronoliverclarin

ronoliverclarin Aug 6, 2015

Any updates on this one?
Or Should I say any developer working on this one?

Any updates on this one?
Or Should I say any developer working on this one?

@LukasReschke

This comment has been minimized.

Show comment
Hide comment
@LukasReschke

LukasReschke Aug 6, 2015

Member

Any updates on this one?
Or Should I say any developer working on this one?

No.

Member

LukasReschke commented Aug 6, 2015

Any updates on this one?
Or Should I say any developer working on this one?

No.

@masteranalyze

This comment has been minimized.

Show comment
Hide comment
@masteranalyze

masteranalyze Aug 11, 2015

👍 it`s an good ideea,and i hope this gets implemented.

Amazon Cloud Drive (ACD) as external storage is definetely worth to be implemented into owncloud!

And like deepdiver said,it will be also nice,that if others appears to be able to easily implement them,into owncloud,to have alternatives storages.

1)I did not manage to test this : automatic encryption of the data,but tell me something,if encryption of the data is used,practically if we have 10.000 mp3 files song files,amazon,will not be able to playback that music?It will not know that it`s an mp3 file ?And the file will be able to be playback in OWNCLOUD?

2)Or it`s just an encryption on sending the data between amazon cloud drive and owncloud?

If somebody can clear me out,about this feature,it will be real nice.Most good,it will be 1st,so you can have an total control of the privacy of the files that you store into the 3rd party clouds.

👍 it`s an good ideea,and i hope this gets implemented.

Amazon Cloud Drive (ACD) as external storage is definetely worth to be implemented into owncloud!

And like deepdiver said,it will be also nice,that if others appears to be able to easily implement them,into owncloud,to have alternatives storages.

1)I did not manage to test this : automatic encryption of the data,but tell me something,if encryption of the data is used,practically if we have 10.000 mp3 files song files,amazon,will not be able to playback that music?It will not know that it`s an mp3 file ?And the file will be able to be playback in OWNCLOUD?

2)Or it`s just an encryption on sending the data between amazon cloud drive and owncloud?

If somebody can clear me out,about this feature,it will be real nice.Most good,it will be 1st,so you can have an total control of the privacy of the files that you store into the 3rd party clouds.

@Xenopathic

This comment has been minimized.

Show comment
Hide comment
@Xenopathic

Xenopathic Aug 30, 2015

Member

This can be implemented as an app, especially with the new Backend API

Member

Xenopathic commented Aug 30, 2015

This can be implemented as an app, especially with the new Backend API

@Xenopathic Xenopathic closed this Aug 30, 2015

@MorrisJobke MorrisJobke removed this from the backlog milestone Sep 13, 2015

@AndrewBucklin

This comment has been minimized.

Show comment
Hide comment
@AndrewBucklin

AndrewBucklin Dec 23, 2015

Am I to assume that this was closed because it won't be implemented as a storage option for owncloud?

Am I to assume that this was closed because it won't be implemented as a storage option for owncloud?

@Xenopathic

This comment has been minimized.

Show comment
Hide comment
@Xenopathic

Xenopathic Dec 24, 2015

Member

@AndrewBucklin While it won't be implemented in the near future within ownCloud core, you are free to make an app that implements the required interfaces 😄

Member

Xenopathic commented Dec 24, 2015

@AndrewBucklin While it won't be implemented in the near future within ownCloud core, you are free to make an app that implements the required interfaces 😄

@Storycrafter

This comment has been minimized.

Show comment
Hide comment
@Storycrafter

Storycrafter Jan 18, 2016

High personal interest = I'll be investigating this and gathering an idea of the effort required for me to implement as an app. I'll comment back one way or another in a day or three.

High personal interest = I'll be investigating this and gathering an idea of the effort required for me to implement as an app. I'll comment back one way or another in a day or three.

@dellipse

This comment has been minimized.

Show comment
Hide comment
@dellipse

dellipse Jan 31, 2016

What did you find out, @Storycrafter ? Any luck?

What did you find out, @Storycrafter ? Any luck?

@Storycrafter

This comment has been minimized.

Show comment
Hide comment
@Storycrafter

Storycrafter Feb 1, 2016

It's not as trivial as I had hoped. The documentation around the ownCloud storage mechanism is limited, and I'm by no means an expert at this point. I'm still keenly interested. Storage Made Easy has become a horrible company to work with for some of my projects so I need a solution I can control now more than ever.

At this point I can report that I see no reason it can not be done, but it also would not be a weekend project for me. I'll put together a task list and some tech spec type docs and post them somewhere.

It's not as trivial as I had hoped. The documentation around the ownCloud storage mechanism is limited, and I'm by no means an expert at this point. I'm still keenly interested. Storage Made Easy has become a horrible company to work with for some of my projects so I need a solution I can control now more than ever.

At this point I can report that I see no reason it can not be done, but it also would not be a weekend project for me. I'll put together a task list and some tech spec type docs and post them somewhere.

@masteranalyze

This comment has been minimized.

Show comment
Hide comment
@masteranalyze

masteranalyze Feb 1, 2016

Maybe @Xenopathic or @DeepDiver1975 can help with some proper documentation,and make things easy at least if the core team does not want to do this,at least leave others to do it and provide proper documentation so people with skills like storycrafter to be able to help the whole community.

At least to easily the job of storycrafter,storage made easy is crap and you must get rid of that my friend.

Maybe @Xenopathic or @DeepDiver1975 can help with some proper documentation,and make things easy at least if the core team does not want to do this,at least leave others to do it and provide proper documentation so people with skills like storycrafter to be able to help the whole community.

At least to easily the job of storycrafter,storage made easy is crap and you must get rid of that my friend.

@wdenx

This comment has been minimized.

Show comment
Hide comment
@wdenx

wdenx Feb 1, 2016

Maybe it would help to have a look at "A command line interface and FUSE filesystem for Amazon Cloud Drive" (acd_cli) implementation at https://github.com/yadayada/acd_cli ?
These tools work really well for me, and response to problems/questions has been excellent.

wdenx commented Feb 1, 2016

Maybe it would help to have a look at "A command line interface and FUSE filesystem for Amazon Cloud Drive" (acd_cli) implementation at https://github.com/yadayada/acd_cli ?
These tools work really well for me, and response to problems/questions has been excellent.

@Storycrafter

This comment has been minimized.

Show comment
Hide comment
@Storycrafter

Storycrafter Feb 2, 2016

Thanks @wdenx, I'm aware of that plus I'd looked at the Cloud API prior to looking around for ways to integrate. I'm actually more concerned about the OC side of things, although after posting a little earlier and digging at the code again today with fresh eyes, I feel like I picked up a few clues. I'm going to set up a dev environment sometime this week and start hacking.

Thanks @wdenx, I'm aware of that plus I'd looked at the Cloud API prior to looking around for ways to integrate. I'm actually more concerned about the OC side of things, although after posting a little earlier and digging at the code again today with fresh eyes, I feel like I picked up a few clues. I'm going to set up a dev environment sometime this week and start hacking.

@JohnnyCalavera

This comment has been minimized.

Show comment
Hide comment
@JohnnyCalavera

JohnnyCalavera Feb 4, 2016

It could be an even better idea to create an app doing "rsync for cloud storage[s]" in general (not limited to acd).
So to speak a rclone app (http://rclone.org/)
You'd get:

  • Google Drive
  • Amazon S3
  • Openstack Swift / Rackspace cloud files / Memset Memstore
  • Dropbox
  • Google Cloud Storage
  • Amazon Cloud Drive
  • Microsoft One Drive
  • Hubic
  • Backblaze B2
  • Yandex

It could be an even better idea to create an app doing "rsync for cloud storage[s]" in general (not limited to acd).
So to speak a rclone app (http://rclone.org/)
You'd get:

  • Google Drive
  • Amazon S3
  • Openstack Swift / Rackspace cloud files / Memset Memstore
  • Dropbox
  • Google Cloud Storage
  • Amazon Cloud Drive
  • Microsoft One Drive
  • Hubic
  • Backblaze B2
  • Yandex
@wdenx

This comment has been minimized.

Show comment
Hide comment
@wdenx

wdenx Feb 4, 2016

Dear Johnny,

In message owncloud/core/issues/17012/179955518@github.com you wrote:

It could be an even better idea to create an app doing "rsync for cloud storage[s]" in general (not limited to acd).

Well, the acd_cli based approach to mount the ACD as a FUSE file
system under Linux is working pretty well for me; but you probably
don't think of Linux based environments only, right?

Best regards,

Wolfgang Denk

DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
Two wrongs don't make a right, but three rights make a left.

wdenx commented Feb 4, 2016

Dear Johnny,

In message owncloud/core/issues/17012/179955518@github.com you wrote:

It could be an even better idea to create an app doing "rsync for cloud storage[s]" in general (not limited to acd).

Well, the acd_cli based approach to mount the ACD as a FUSE file
system under Linux is working pretty well for me; but you probably
don't think of Linux based environments only, right?

Best regards,

Wolfgang Denk

DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
Two wrongs don't make a right, but three rights make a left.

@JohnnyCalavera

This comment has been minimized.

Show comment
Hide comment
@JohnnyCalavera

JohnnyCalavera Feb 19, 2016

but you probably don't think of Linux based environments only, right?

Dear Wolfgang,

as a matter of fact I do think of Linux based environments - especially the Rockstor NAS OS (CentOS-based). Nonetheless the rclone approach would give us more options in terms of cloud service providers and since it is still actively developed there will be even more possibilities in the future.
So the owncloud app could end up being a rather simple one and it's code doesn't need to be touched that often because the underlying rclone software will perform the syncing tasks while the owncloud app is barely more than a GUI for rclone.

Well, the acd_cli based approach to mount the ACD as a FUSE file system under Linux is working pretty well for me

I did test acd_cli a few weeks ago and it seemed to work pretty well. My only reason to bring rclone to the table is the fact that we would get way more possibilities without having to code a lot from scratch.

but you probably don't think of Linux based environments only, right?

Dear Wolfgang,

as a matter of fact I do think of Linux based environments - especially the Rockstor NAS OS (CentOS-based). Nonetheless the rclone approach would give us more options in terms of cloud service providers and since it is still actively developed there will be even more possibilities in the future.
So the owncloud app could end up being a rather simple one and it's code doesn't need to be touched that often because the underlying rclone software will perform the syncing tasks while the owncloud app is barely more than a GUI for rclone.

Well, the acd_cli based approach to mount the ACD as a FUSE file system under Linux is working pretty well for me

I did test acd_cli a few weeks ago and it seemed to work pretty well. My only reason to bring rclone to the table is the fact that we would get way more possibilities without having to code a lot from scratch.

@Storycrafter

This comment has been minimized.

Show comment
Hide comment
@Storycrafter

Storycrafter Feb 19, 2016

@girvo

This comment has been minimized.

Show comment
Hide comment
@girvo

girvo Mar 10, 2016

@Storycrafter I'm about to re-setup my OwnCloud instance and am looking at backends for storage, so I'm likely to use ACD. Do you mind contacting me via email? I can provide an API key/account details/dev env perhaps :)

girvo commented Mar 10, 2016

@Storycrafter I'm about to re-setup my OwnCloud instance and am looking at backends for storage, so I'm likely to use ACD. Do you mind contacting me via email? I can provide an API key/account details/dev env perhaps :)

@Storycrafter

This comment has been minimized.

Show comment
Hide comment
@Storycrafter

Storycrafter Mar 10, 2016

@girvo ,thanks for the offer, but I was able to get a dev environment started at openshift. I plan to start taking a whack at things this weekend. I'll ping you via email as well.

@girvo ,thanks for the offer, but I was able to get a dev environment started at openshift. I plan to start taking a whack at things this weekend. I'll ping you via email as well.

@blubitv

This comment has been minimized.

Show comment
Hide comment
@blubitv

blubitv Mar 21, 2016

Hello guys,

did anyone manage to get a working solution and use ACD as external storage? (with rclone or acdfuse or some other connector)

blubitv commented Mar 21, 2016

Hello guys,

did anyone manage to get a working solution and use ACD as external storage? (with rclone or acdfuse or some other connector)

@Orderchaos

This comment has been minimized.

Show comment
Hide comment
@Orderchaos

Orderchaos Apr 20, 2016

@blubitv Hey, not sure if you're even still interested in this, but I saw your question so figured better late than never.

I'm not a developer (although I'm considering looking into this just out of personal interest/hobby), but I do have an owncloud instance setup on a VPS with Amazon Cloud Drive mounted using acd_cli and it's been working well for several months now.

I used the acd_cli to mount it as a drive and then setup the external storage in owncloud (connecting to the ACD as 'local'). I've even shared access to owncloud with a few people and limited their storage to specific folders ('group' permission) to subfolders on a single amazon cloud drive account and it works exactly as expected. Here's a pic of my owncloud settings after getting acd_cli installed and mounted (see the github link in my previous paragraph for instructions on that):
image

@blubitv Hey, not sure if you're even still interested in this, but I saw your question so figured better late than never.

I'm not a developer (although I'm considering looking into this just out of personal interest/hobby), but I do have an owncloud instance setup on a VPS with Amazon Cloud Drive mounted using acd_cli and it's been working well for several months now.

I used the acd_cli to mount it as a drive and then setup the external storage in owncloud (connecting to the ACD as 'local'). I've even shared access to owncloud with a few people and limited their storage to specific folders ('group' permission) to subfolders on a single amazon cloud drive account and it works exactly as expected. Here's a pic of my owncloud settings after getting acd_cli installed and mounted (see the github link in my previous paragraph for instructions on that):
image

@masteranalyze

This comment has been minimized.

Show comment
Hide comment
@masteranalyze

masteranalyze Apr 20, 2016

@Orderchaos did you done this in owncloud 9.01?

Can you replicate the steps for it?So i can test this.

In amazon cloud drive,i noticed,that when i wish to get my backup of arhives of an website,witch is basically over 10gb,so let`s say an backup file of 15gb,the shared link from amazon cloud drive,does not contain that /temp / path ,and the link is basically from amazon s3,and files of over 10gb mainly cannot be Wget directly into the linux server.
Somebody else did encountered this error ?It basically gives 403 error,because the link is from Amazon S3,not from amazon cloud drive,but the files are on amazon cloud drive.

I talked 100 times with the Amazonians,and i explained to them this problem,but they all seems to not understand,that the files are on Amazon cloud drive,they are asking for Amazon S3 account,but i don`t have no amazon S3,i have amazon cloud drive.I did even make them video,and they did not understand,again they asked for my amazon s3.

How the heck they hire people on Amazon?!?!I just wonder!

@Orderchaos did you done this in owncloud 9.01?

Can you replicate the steps for it?So i can test this.

In amazon cloud drive,i noticed,that when i wish to get my backup of arhives of an website,witch is basically over 10gb,so let`s say an backup file of 15gb,the shared link from amazon cloud drive,does not contain that /temp / path ,and the link is basically from amazon s3,and files of over 10gb mainly cannot be Wget directly into the linux server.
Somebody else did encountered this error ?It basically gives 403 error,because the link is from Amazon S3,not from amazon cloud drive,but the files are on amazon cloud drive.

I talked 100 times with the Amazonians,and i explained to them this problem,but they all seems to not understand,that the files are on Amazon cloud drive,they are asking for Amazon S3 account,but i don`t have no amazon S3,i have amazon cloud drive.I did even make them video,and they did not understand,again they asked for my amazon s3.

How the heck they hire people on Amazon?!?!I just wonder!

@masteranalyze

This comment has been minimized.

Show comment
Hide comment
@masteranalyze

masteranalyze Apr 20, 2016

@Orderchaos in acd_cli i saw : workaround for download of files larger than 10 GiB ,did you try to upload an 15gb file,and then download ?And can you tell us if this works or not ?I don`t understand what they are trying to mean with "WORKAROUND For files larger then 10gb",if anybody nows what this means,feel free to reply.

I know amazon haves limit for upload of 50gb for an file,but on download file limit,they are saying there is no limit,then why the heck files of over 10gb cannot be download in linux,centos or any other.

It can only be downloaded via Windows,with their Download Manager,but on linux seems to not work to download files of over 10gb.

And basically,if you can`t download files of over 10gb,why they let you upload files of up to 50gb?Because it does not make no sense.

You upload an 20gb file,and then if you need it,what you do?Your stuck on Windows ?!

@Orderchaos in acd_cli i saw : workaround for download of files larger than 10 GiB ,did you try to upload an 15gb file,and then download ?And can you tell us if this works or not ?I don`t understand what they are trying to mean with "WORKAROUND For files larger then 10gb",if anybody nows what this means,feel free to reply.

I know amazon haves limit for upload of 50gb for an file,but on download file limit,they are saying there is no limit,then why the heck files of over 10gb cannot be download in linux,centos or any other.

It can only be downloaded via Windows,with their Download Manager,but on linux seems to not work to download files of over 10gb.

And basically,if you can`t download files of over 10gb,why they let you upload files of up to 50gb?Because it does not make no sense.

You upload an 20gb file,and then if you need it,what you do?Your stuck on Windows ?!

@Orderchaos

This comment has been minimized.

Show comment
Hide comment
@Orderchaos

Orderchaos Apr 20, 2016

@masteranalyze larger files can timeout and I've had some issues with them causing sync problems when they do. I haven't tried a lot of large files with owncloud, but on a different server (same cloud drive) I've been able to use rsync to copy directly from a local drive to a mounted cloud drive with large media files (up to ~30GB). Just make sure that the acd_cli sync command doesn't run while the file is being transferred as that seems to be what causes the connection issues. Also don't have more than one transfer per server going at a time (can work, but can also cause more sync issues).

That's all for uploading though. Downloading I haven't had any problems with as I'm mostly using this for backup. Let me get a download started and see how it fares and I'll let you know.

Oh and this is on owncloud 8.2.2. I haven't updated in a while.

Orderchaos commented Apr 20, 2016

@masteranalyze larger files can timeout and I've had some issues with them causing sync problems when they do. I haven't tried a lot of large files with owncloud, but on a different server (same cloud drive) I've been able to use rsync to copy directly from a local drive to a mounted cloud drive with large media files (up to ~30GB). Just make sure that the acd_cli sync command doesn't run while the file is being transferred as that seems to be what causes the connection issues. Also don't have more than one transfer per server going at a time (can work, but can also cause more sync issues).

That's all for uploading though. Downloading I haven't had any problems with as I'm mostly using this for backup. Let me get a download started and see how it fares and I'll let you know.

Oh and this is on owncloud 8.2.2. I haven't updated in a while.

@masteranalyze

This comment has been minimized.

Show comment
Hide comment
@masteranalyze

masteranalyze Apr 20, 2016

@Orderchaos on upload on my side,i did not have no issues,because basically,i did not upload no file larger then 50gb,but the problem on my side,is on downloading files of over 10gb to linux.

There is no problem to download on windows with their download manager,an file of 15gb,but if you want to upload it directly into an webserver with linux with wget command,it gives error 403,i checked why is giving 403,because 403 means ACCESS denied,and when i used an firefox extension to give me the exactly link for Wget on Windows,the link of larger files of over 10gb,are links from AMAZON S3.

From what i know,amazon cloud drive is build up on amazon s3,as an retail version of the amazon s3,but i don`t know what the heck they are doing and how is their arhitecture exactly,for files of over 10gb.

Because if you link of the link shared for files of over 10gb,you will notice,is not anymore that /TMP/ link,witch is provided for all files of under 10gb,its an S3 link basically,and when you try to get that link witch is basically from Amazon S3,with wget on centos 6.7,the server gives access denied,and is kind of logical to give access denied,because i dont put no user or password on it or access key,basically i dont have no Amazon S3,but the files of over 10gb,seems to be there,and cant be donwnload in Linux.

Please try out with an single file of 15gb,and let me know.

Your using 64 bit system or 32bit system?Just checked out the documentation,and seems owncloud,also haves some restriction on uploading files larger then 2gb.I guess because of the browser.

I did not find out,what is the maximum file upload limit in owncloud,if anybody knows this?Please write your comment,in 64bit systems.

@Orderchaos on upload on my side,i did not have no issues,because basically,i did not upload no file larger then 50gb,but the problem on my side,is on downloading files of over 10gb to linux.

There is no problem to download on windows with their download manager,an file of 15gb,but if you want to upload it directly into an webserver with linux with wget command,it gives error 403,i checked why is giving 403,because 403 means ACCESS denied,and when i used an firefox extension to give me the exactly link for Wget on Windows,the link of larger files of over 10gb,are links from AMAZON S3.

From what i know,amazon cloud drive is build up on amazon s3,as an retail version of the amazon s3,but i don`t know what the heck they are doing and how is their arhitecture exactly,for files of over 10gb.

Because if you link of the link shared for files of over 10gb,you will notice,is not anymore that /TMP/ link,witch is provided for all files of under 10gb,its an S3 link basically,and when you try to get that link witch is basically from Amazon S3,with wget on centos 6.7,the server gives access denied,and is kind of logical to give access denied,because i dont put no user or password on it or access key,basically i dont have no Amazon S3,but the files of over 10gb,seems to be there,and cant be donwnload in Linux.

Please try out with an single file of 15gb,and let me know.

Your using 64 bit system or 32bit system?Just checked out the documentation,and seems owncloud,also haves some restriction on uploading files larger then 2gb.I guess because of the browser.

I did not find out,what is the maximum file upload limit in owncloud,if anybody knows this?Please write your comment,in 64bit systems.

@Orderchaos

This comment has been minimized.

Show comment
Hide comment
@Orderchaos

Orderchaos Apr 20, 2016

@masteranalyze

Oh I see what you mean. The issue you're having is because you want a direct download link which Amazon doesn't provide (not in a straightforward manner anyways). There are ways to get it, but really you'd be better off doing it the way I mentioned which is mounting it as a network drive and using rsync or another copy method to copy the file from your amazon cloud drive to a local drive.

I just tested that and it worked fine for > 20GB file.

Also owncloud can be configured with custom upload limits. Mine is currently 10GB/file but I haven't tested anything quite that large yet.

Orderchaos commented Apr 20, 2016

@masteranalyze

Oh I see what you mean. The issue you're having is because you want a direct download link which Amazon doesn't provide (not in a straightforward manner anyways). There are ways to get it, but really you'd be better off doing it the way I mentioned which is mounting it as a network drive and using rsync or another copy method to copy the file from your amazon cloud drive to a local drive.

I just tested that and it worked fine for > 20GB file.

Also owncloud can be configured with custom upload limits. Mine is currently 10GB/file but I haven't tested anything quite that large yet.

@masteranalyze

This comment has been minimized.

Show comment
Hide comment
@masteranalyze

masteranalyze Apr 20, 2016

@Orderchaos thanks for testing,is good if it`s working like that,i will try it out,hopefully via owncloud,i will be able to download my backup directly into the linux server,and not downloading back in windows,and then re-uploading back in the linux server,because that is waste of time.

Any ideea about the file limits upload in owncloud as setup?On 64 bits,i cannot find nothing exactly into their documentation.

masteranalyze commented Apr 20, 2016

@Orderchaos thanks for testing,is good if it`s working like that,i will try it out,hopefully via owncloud,i will be able to download my backup directly into the linux server,and not downloading back in windows,and then re-uploading back in the linux server,because that is waste of time.

Any ideea about the file limits upload in owncloud as setup?On 64 bits,i cannot find nothing exactly into their documentation.

@masteranalyze

This comment has been minimized.

Show comment
Hide comment
@masteranalyze

masteranalyze Apr 20, 2016

@Orderchaos i just saw now ,that you mentioned that your is 10gb/file,10gb is fine.

What is the largest file gb you uploaded into owncloud till now?!via browser

masteranalyze commented Apr 20, 2016

@Orderchaos i just saw now ,that you mentioned that your is 10gb/file,10gb is fine.

What is the largest file gb you uploaded into owncloud till now?!via browser

@Orderchaos

This comment has been minimized.

Show comment
Hide comment
@Orderchaos

Orderchaos Apr 20, 2016

@masteranalyze here's the owncloud documentation on it: https://doc.owncloud.org/server/9.0/admin_manual/configuration_files/big_file_upload_configuration.html

I'll start an upload of a large file via owncloud and let you know how it handles.

@masteranalyze here's the owncloud documentation on it: https://doc.owncloud.org/server/9.0/admin_manual/configuration_files/big_file_upload_configuration.html

I'll start an upload of a large file via owncloud and let you know how it handles.

@masteranalyze

This comment has been minimized.

Show comment
Hide comment
@masteranalyze

masteranalyze Apr 20, 2016

@Orderchaos i saw that,but it`s only mentioning this : " 64-bit filesystems have much higher limits; consult the documentation for your filesystem."

This are for 32bits systems not for 64bits systems:

< 2GB on 32Bit OS-architecture
< 2GB on Windows (32Bit and 64Bit)
< 2GB with Server Version 4.5 or older
< 2GB with IE6 - IE8
< 4GB with IE9 - IE11

"consult the documentation for your filesystem" this reffers to the filesystem on witch owncloud is run?So in my case it will be centos 6.7?!Or the filesystem of the external storage used?

Because what is in here : https://access.redhat.com/solutions/1532 ,i think as centos 6.x copies practically red hat,that means in ext4,16tb is the file upload limit?

From what i know till now from my research,only google drive can handle the largest upload size,witch is 500gb/per file.

Anyhow i dont think somebody needs 500gb per file,but 20-30gb if its working fine,it will be allright!

As for example upload limit in amazon cloud drive we know it s 50gb,so even if you put an limit of 60gb per file,is no use,because you wont get it into your amazon cloud drive.

Please let me know the filesize and how it`s handled!

@Orderchaos i saw that,but it`s only mentioning this : " 64-bit filesystems have much higher limits; consult the documentation for your filesystem."

This are for 32bits systems not for 64bits systems:

< 2GB on 32Bit OS-architecture
< 2GB on Windows (32Bit and 64Bit)
< 2GB with Server Version 4.5 or older
< 2GB with IE6 - IE8
< 4GB with IE9 - IE11

"consult the documentation for your filesystem" this reffers to the filesystem on witch owncloud is run?So in my case it will be centos 6.7?!Or the filesystem of the external storage used?

Because what is in here : https://access.redhat.com/solutions/1532 ,i think as centos 6.x copies practically red hat,that means in ext4,16tb is the file upload limit?

From what i know till now from my research,only google drive can handle the largest upload size,witch is 500gb/per file.

Anyhow i dont think somebody needs 500gb per file,but 20-30gb if its working fine,it will be allright!

As for example upload limit in amazon cloud drive we know it s 50gb,so even if you put an limit of 60gb per file,is no use,because you wont get it into your amazon cloud drive.

Please let me know the filesize and how it`s handled!

@Orderchaos

This comment has been minimized.

Show comment
Hide comment
@Orderchaos

Orderchaos Apr 20, 2016

@masteranalyze those are theoretical limits based on the math behind 32 and 64bit addressing...Actual hard limits are dependent on the OS, but are still very likely higher than you will actually need.

If you've already got the server, why don't you just get it setup and try it out? Amazon has a free trial for the cloud drive and owncloud and the acd_cli are both free.

I'm currently uploading a ~5GB file via the owncloud web interface, but my upload speed isn't very fast so it may be a while. I will let you know if it has any problems.

@masteranalyze those are theoretical limits based on the math behind 32 and 64bit addressing...Actual hard limits are dependent on the OS, but are still very likely higher than you will actually need.

If you've already got the server, why don't you just get it setup and try it out? Amazon has a free trial for the cloud drive and owncloud and the acd_cli are both free.

I'm currently uploading a ~5GB file via the owncloud web interface, but my upload speed isn't very fast so it may be a while. I will let you know if it has any problems.

@masteranalyze

This comment has been minimized.

Show comment
Hide comment
@masteranalyze

masteranalyze Apr 20, 2016

@Orderchaos i will deploy owncloud 9.0.2,in 5 may-6may,when they will release that update,i expect some fixes,witch should be in this release,and i will definetly try out everything you said,and share results here.

They are dependent on Os,not on the external file share you use or browser limits? Because if the Os permits 2TB file upload,and browser cannot handle that file,it will fail,same if amazon cloud drive limit is 50gb per file,is no use to set that limit to 50gb per file,maybe 20-30gb,to not be problems.

Did you tryed out if you already have setup owncloud on your side,for safety and privacy,the encryption,i mean if you have conected your amazon cloud drive into your owncloud with encryption enable,and basically you upload your pcbackup.zip file witch is 1gb your personal files,with encryption on,amazon will know what you have in your archive or how will that file appear to them?

Let`s say photo.jpg ,encrypted when upload via owncloud,stored in amazon cloud drive.

@Orderchaos i will deploy owncloud 9.0.2,in 5 may-6may,when they will release that update,i expect some fixes,witch should be in this release,and i will definetly try out everything you said,and share results here.

They are dependent on Os,not on the external file share you use or browser limits? Because if the Os permits 2TB file upload,and browser cannot handle that file,it will fail,same if amazon cloud drive limit is 50gb per file,is no use to set that limit to 50gb per file,maybe 20-30gb,to not be problems.

Did you tryed out if you already have setup owncloud on your side,for safety and privacy,the encryption,i mean if you have conected your amazon cloud drive into your owncloud with encryption enable,and basically you upload your pcbackup.zip file witch is 1gb your personal files,with encryption on,amazon will know what you have in your archive or how will that file appear to them?

Let`s say photo.jpg ,encrypted when upload via owncloud,stored in amazon cloud drive.

@Storycrafter

This comment has been minimized.

Show comment
Hide comment
@Storycrafter

Storycrafter Jun 28, 2016

I have a development environment setup and have identified the integration point in code. I've also submitted my signed agreement for contributor licensing -- no response on that (and not expected any time soon, honestly). I will track progress at http://mark-martin.com starting tomorrow.

I have a development environment setup and have identified the integration point in code. I've also submitted my signed agreement for contributor licensing -- no response on that (and not expected any time soon, honestly). I will track progress at http://mark-martin.com starting tomorrow.

@DeepDiver1975

This comment has been minimized.

Show comment
Hide comment
@DeepDiver1975

DeepDiver1975 Jun 28, 2016

Member

Hi Martin. Following your blog you are working on a fork of core. Please don't walk this path but better implement an app which provides the acd storage.
Please reach out on IRC in case you have questions.

Regarding your cla: i was answering you last night.

Member

DeepDiver1975 commented Jun 28, 2016

Hi Martin. Following your blog you are working on a fork of core. Please don't walk this path but better implement an app which provides the acd storage.
Please reach out on IRC in case you have questions.

Regarding your cla: i was answering you last night.

@Storycrafter

This comment has been minimized.

Show comment
Hide comment
@Storycrafter

Storycrafter Jun 28, 2016

@DeepDiver1975

Thanks for taking a look. When you suggest not to fork, are you suggesting to wait for commit privs and commit to a branch on the original report?

Others also mentioned going with an app instead of via files_external before. What is the benefit to that approach? Distribution? Are you suggesting that making it part of core isn't desirable?

@DeepDiver1975

Thanks for taking a look. When you suggest not to fork, are you suggesting to wait for commit privs and commit to a branch on the original report?

Others also mentioned going with an app instead of via files_external before. What is the benefit to that approach? Distribution? Are you suggesting that making it part of core isn't desirable?

@DeepDiver1975

This comment has been minimized.

Show comment
Hide comment
@DeepDiver1975

DeepDiver1975 Jun 28, 2016

Member

The core repo already holds a hell of code and especially all the external file systems put some burden on us with respect to unit testing. When implementing an external filesystem as an app on it's own it can live isolated in a repo of it's own where unit testing an integration testing can be setup much easier using travis ci.

As an example: https://github.com/icewind1991/files_external_ftp

Member

DeepDiver1975 commented Jun 28, 2016

The core repo already holds a hell of code and especially all the external file systems put some burden on us with respect to unit testing. When implementing an external filesystem as an app on it's own it can live isolated in a repo of it's own where unit testing an integration testing can be setup much easier using travis ci.

As an example: https://github.com/icewind1991/files_external_ftp

@Storycrafter

This comment has been minimized.

Show comment
Hide comment
@Storycrafter

Storycrafter Jun 28, 2016

I think I understand now. The full API is available externally as an app, but as you suggest -- it can be isolated for testing and management purposes.

Would certainly make keeping the repo in synch with the upstream a ton easier if there is no upstream.

I'll modify my approach. Thanks for adding clarity.

I think I understand now. The full API is available externally as an app, but as you suggest -- it can be isolated for testing and management purposes.

Would certainly make keeping the repo in synch with the upstream a ton easier if there is no upstream.

I'll modify my approach. Thanks for adding clarity.

@Storycrafter

This comment has been minimized.

Show comment
Hide comment
@Storycrafter

Storycrafter Jun 28, 2016

OK, I've restarted the bootstrapping and have taken the files_external_ftp as a starting template.

I'm encountering an issue on travis-ci with the core before_install.sh script. It fails to install oc core due to a cryptic error. If anyone has insights here, I'd appreciate hearing them.

https://travis-ci.org/Storycrafter/files_external_acd/jobs/140862133

Storycrafter commented Jun 28, 2016

OK, I've restarted the bootstrapping and have taken the files_external_ftp as a starting template.

I'm encountering an issue on travis-ci with the core before_install.sh script. It fails to install oc core due to a cryptic error. If anyone has insights here, I'd appreciate hearing them.

https://travis-ci.org/Storycrafter/files_external_acd/jobs/140862133

@Flux3PO

This comment has been minimized.

Show comment
Hide comment
@Flux3PO

Flux3PO Sep 11, 2016

I don't own a VPS, can I run this on a Webspace only Install ?

I really want ACD to work as external Storage Provider.

Flux3PO commented Sep 11, 2016

I don't own a VPS, can I run this on a Webspace only Install ?

I really want ACD to work as external Storage Provider.

@masteranalyze

This comment has been minimized.

Show comment
Hide comment
@masteranalyze

masteranalyze Nov 2, 2016

It seems people from owncloud ,are now NEXT CLOUD.

@Storycrafter check this link maybe it does help you to do the amazon cloud drive implementation 👍
https://help.nextcloud.com/t/amazon-cloud-drive-joomla-nextcloud-how-to/4607/6

It seems people from owncloud ,are now NEXT CLOUD.

@Storycrafter check this link maybe it does help you to do the amazon cloud drive implementation 👍
https://help.nextcloud.com/t/amazon-cloud-drive-joomla-nextcloud-how-to/4607/6

@WLCD

This comment has been minimized.

Show comment
Hide comment
@WLCD

WLCD Dec 14, 2016

Any new development on this matter ?

WLCD commented Dec 14, 2016

Any new development on this matter ?

@Flux3PO

This comment has been minimized.

Show comment
Hide comment
@Flux3PO

Flux3PO Dec 14, 2016

Yeah, would love to see this implemented as well

Flux3PO commented Dec 14, 2016

Yeah, would love to see this implemented as well

@philjak

This comment has been minimized.

Show comment
Hide comment

philjak commented Dec 18, 2016

+1

@masteranalyze

This comment has been minimized.

Show comment
Hide comment
@masteranalyze

masteranalyze Dec 20, 2016

It seems owncloud it`s splited between old owncloud and nextcloud,i think this should be post in nextcloud 12 overview : nextcloud/server#2672 ,as nextcloud is FOSS complectly and owncloud is not FOSS on everything witch is an must at least on the security side.

I will ask there if there is any interest,and you can follow up,the more will be,maybe this gets implemented,as right now it seems dead.

It seems owncloud it`s splited between old owncloud and nextcloud,i think this should be post in nextcloud 12 overview : nextcloud/server#2672 ,as nextcloud is FOSS complectly and owncloud is not FOSS on everything witch is an must at least on the security side.

I will ask there if there is any interest,and you can follow up,the more will be,maybe this gets implemented,as right now it seems dead.

@Flux3PO

This comment has been minimized.

Show comment
Hide comment
@Flux3PO

Flux3PO Dec 21, 2016

@masteranalyze
can you share the Link ? I've already ditched CM over NC and I only can advise everyone else doing the same.

Flux3PO commented Dec 21, 2016

@masteranalyze
can you share the Link ? I've already ditched CM over NC and I only can advise everyone else doing the same.

@masteranalyze

This comment has been minimized.

Show comment
Hide comment
@masteranalyze

masteranalyze Jan 22, 2017

The link was in my prevision post,anyhow this is the link to NC 12 overview 👍
nextcloud/server#2672

But from what i see,nobody wants to implement this either if we talk about the old group of owncloud or the new cloud members,but maybe if we post more there,and ask for this,maybe things happens.

The link was in my prevision post,anyhow this is the link to NC 12 overview 👍
nextcloud/server#2672

But from what i see,nobody wants to implement this either if we talk about the old group of owncloud or the new cloud members,but maybe if we post more there,and ask for this,maybe things happens.

@rafaelsaru

This comment has been minimized.

Show comment
Hide comment
@rafaelsaru

rafaelsaru Mar 3, 2017

@Orderchaos I'm getting permissions errors when trying to add the Local storage to Owncloud.
Maybe the folder permissions is for root only and I can't change it by chown -R

@Orderchaos I'm getting permissions errors when trying to add the Local storage to Owncloud.
Maybe the folder permissions is for root only and I can't change it by chown -R

@rafaelsaru

This comment has been minimized.

Show comment
Hide comment
@rafaelsaru

rafaelsaru Mar 3, 2017

I'm using acd_cli mount --allow-other -i 0 /mnt/

I'm using acd_cli mount --allow-other -i 0 /mnt/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment