update code for 2gb filesize limit #88

Open
indian4646 opened this Issue Jul 10, 2012 · 7 comments

Comments

Projects
None yet
2 participants

Hi
I have updated grive in my fork grive. I have checked uploading large file successfully and spend over 3600 second for refresh token.

https://github.com/indian4646/grive

1.I modified support more 2gb filesize,
2.Checking "exprie in" time for long-lived access token

   modified:   grive/src/main.cc
   modified:   libgrive/CMakeLists.txt
   modified:   libgrive/src/drive/Drive.cc
   modified:   libgrive/src/drive/Drive.hh
   modified:   libgrive/src/drive/Resource.cc
   modified:   libgrive/src/drive/Resource.hh
   modified:   libgrive/src/drive/State.cc
   modified:   libgrive/src/drive/State.hh
   modified:   libgrive/src/http/Agent.hh
   modified:   libgrive/src/http/CurlAgent.cc
   modified:   libgrive/src/http/CurlAgent.hh
   modified:   libgrive/src/protocol/OAuth2.cc
   modified:   libgrive/src/protocol/OAuth2.hh
   modified:   libgrive/src/util/OS.cc
   modified:   libgrive/src/util/StdioFile.cc
   modified:   libgrive/src/util/StdioFile.hh
Owner

nestal commented Jul 11, 2012

Which commit contain your change? I can't find it in github.

On Tue, Jul 10, 2012 at 6:21 PM, indian4646
reply@reply.github.com
wrote:

Hi
I have updated grive in my fork grive. I have checked uploading large file successfully and spend over 3600 second for refresh token.

https://github.com/indian4646/grive

1.I modified support more 2gb filesize,
2.Checking "exprie in" time for long-lived access token

   modified:   grive/src/main.cc
   modified:   libgrive/CMakeLists.txt
   modified:   libgrive/src/drive/Drive.cc
   modified:   libgrive/src/drive/Drive.hh
   modified:   libgrive/src/drive/Resource.cc
   modified:   libgrive/src/drive/Resource.hh
   modified:   libgrive/src/drive/State.cc
   modified:   libgrive/src/drive/State.hh
   modified:   libgrive/src/http/Agent.hh
   modified:   libgrive/src/http/CurlAgent.cc
   modified:   libgrive/src/http/CurlAgent.hh
   modified:   libgrive/src/protocol/OAuth2.cc
   modified:   libgrive/src/protocol/OAuth2.hh
   modified:   libgrive/src/util/OS.cc
   modified:   libgrive/src/util/StdioFile.cc
   modified:   libgrive/src/util/StdioFile.hh

Reply to this email directly or view it on GitHub:
#88

I am sorry for this, could you check again, https://github.com/indian4646/grive

Owner

nestal commented Jul 13, 2012

Thanks.

Your patch is very interesting.

I see you passed the OAuth to Resource and Refresh() it. Why is that needed?

Please dont cast OAuth to http::Header. It may not work.

Please change CStdioFile to fopen64() instead of calling it directly.

This code maybe for your reference, because I have tested this code can work to support 2G limit.

  1. Refresh(), The token have expire time 3600 second, we need to refresh older token in expire time in
    https://developers.google.com/accounts/docs/OAuth2ForDevices#obtainingatoken
  2. Http::Header includes access token which should refresh when uploaded file is too large for spending lots of time.
  3. Would you more explain change CStdioFile to fopen64() instead of calling it directly. Thanks
Owner

nestal commented Jul 16, 2012

Sorry, I mean util/StdioFile instead of CStdioFile. You can see the fopen() is actually encapsulated in a class called StdioFile. If we use fopen64() instead of fopen(), we should modify StdioFile instead.

@nestal nestal added a commit that referenced this issue Jul 16, 2012

@nestal nestal added _FILE_OFFSET_BITS=64 (#88) 394982d

@nestal nestal added a commit that referenced this issue Jul 16, 2012

@nestal nestal use StdioFile for uploading (#88) ce24557

Hi nestal
Today I try to upload a file 4.4G to Google drive, it happened http 401 error in next file, I spend a over night, I think Grive should refresh access token, because I have try my code to refresh token, it never respond http 401. Do you think? in issue
#83 I have reported.

sync ./4g doesn't exist in server, uploading
HTTP POST "https://docs.google.com/feeds/upload/create-session/default/private/full?convert=false" with "
<title>4g</title>"
HTTP response 200
HTTP PUT "https://docs.google.com/feeds/upload/create-session/default/private/full?convert=false&upload_id=AEnB2UrvlBAYMXcs-9OHEVM3rpMihDdHe0SKlZCoxT9xsYrfcvjaewA7FZOTtrfj0ZGIKYo9TglG-SEzJg6exktYeXg5f2u7AQ"
HTTP response 201
sync ./test already in sync
sync ./log_123.txt already in sync
sync ./新資料夾 already in sync
sync ./AAAA already in sync
sync ./AAAA/187924_348265731911787_2043134911_n.jpg already in sync
HTTP GET "https://docs.google.com/feeds/default/private/changes?start-index=129"
HTTP response 401
exception: /root/host/NasX86/SysUtil/Grive-grive-v0.2.0-13-g7cc4984/libgrive/src/http/CurlAgent.cc(168): Throw in function long int gr::http::CurlAgent::ExecCurl(const std::string&, gr::http::Receivable*, const gr::http::Header&)
Dynamic exception type: N5boost16exception_detail10clone_implIN2gr4http5ErrorEEE
std::exception::what: std::exception
[PN2gr4expt6MsgTagE] =
[PN2gr4http9HeaderTagE] = Authorization: Bearer ya29.AHES6ZSnxOXnJQHqy07BUwUqfa0-k2F7m9zHCg5-L8FQBhqzpMIw0CI
GData-Version: 3.0

[PN2gr4http6UrlTagE] = https://docs.google.com/feeds/default/private/changes?start-index=129
[PN2gr4http15HttpResponseTagE] = 401
[PN2gr4http11CurlCodeTagE] = 0
[PN2gr4expt12BacktraceTagE] = #0 0x8105166 :0 gr::Exception::Exception()
#1 0x80f6c71 :0 gr::http::Error::Error()
#2 0x80f2c84 :0 gr::http::CurlAgent::ExecCurl(std::string const&, gr::http::Receivable_, gr::http::Header const&)
#3 0x80f35c7 :0 gr::http::CurlAgent::Get(std::string const&, gr::http::Receivable_, gr::http::Header const&)
#4 0x80da76f :0 gr::Drive::UpdateChangeStamp(gr::http::Agent_)
#5 0x80dab5c :0 gr::Drive::Update()
#6 0x80c6aea :0 Main(int, char_*)
#7 0x80c6d85 :0 main
#8 0xf737d050 /lib/libc.so.6 __libc_start_main
#9 0x80c4721 :0 _start

Owner

nestal commented Jul 26, 2012

@indian4646 Yes, you are right. I have just added the protocol/AuthAgent to refresh the token. I didn't check the expire time like you did, just refresh the token whenever we get a 401.

Can you please take a look? If it doesn't work at your side, can you please modify protocol/AuthAgent to add whatever you need? You can also add things to protocol/OAuth2. I don't expect we need to touch other places for this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment