Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Copying an object fails when connecting via https [JIRA: RCS-358] #1025

Closed
iby opened this issue Nov 25, 2014 · 10 comments
Closed

Copying an object fails when connecting via https [JIRA: RCS-358] #1025

iby opened this issue Nov 25, 2014 · 10 comments
Assignees
@tburghart
Labels
Bug JIRA: Closed
Milestone
2.1.2

Comments

@iby
Copy link

iby commented Nov 25, 2014

Both AWS SDK (PHP) and s3cmd fail when I try to do copy an object within the same bucket. AWS SDK says The specified key does not exist., while s3cmd says AttributeError: 'NoneType' object has no attribute 'getchildren. Here's the full output from s3cmd.

$ s3cmd cp s3://tst/1234.jpg s3://tst/5678.jpg

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
    An unexpected error has occurred.
  Please try reproducing the error using
  the latest s3cmd code from the git master
  branch found at:
    https://github.com/s3tools/s3cmd
  If the error persists, please report the
  following lines (removing any private
  info as necessary) to:
   s3tools-bugs@lists.sourceforge.net

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Invoked as: /usr/local/bin/s3cmd cp s3://tst/1234.jpg s3://tst/5678.jpg
Problem: AttributeError: 'NoneType' object has no attribute 'getchildren'
S3cmd:   1.5.0-rc1
python:   2.7.6 (default, Sep  9 2014, 15:04:36) 
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)]
environment LANG=en_GB.UTF-8

Traceback (most recent call last):
  File "/usr/local/bin/s3cmd", line 2523, in <module>
    rc = main()
  File "/usr/local/bin/s3cmd", line 2441, in main
    rc = cmd_func(args)
  File "/usr/local/bin/s3cmd", line 752, in cmd_cp
    return subcmd_cp_mv(args, s3.object_copy, "copy", u"File %(src)s copied to %(dst)s")
  File "/usr/local/bin/s3cmd", line 739, in subcmd_cp_mv
    response = process_fce(src_uri, dst_uri, extra_headers)
  File "/usr/local/Cellar/s3cmd/1.5.0-rc1/libexec/S3/S3.py", line 628, in object_copy
    response = self.send_request(request)
  File "/usr/local/Cellar/s3cmd/1.5.0-rc1/libexec/S3/S3.py", line 883, in send_request
    e = S3Error(response)
  File "/usr/local/Cellar/s3cmd/1.5.0-rc1/libexec/S3/Exceptions.py", line 54, in __init__
    self.info.update(self.parse_error_xml(tree))
  File "/usr/local/Cellar/s3cmd/1.5.0-rc1/libexec/S3/Exceptions.py", line 73, in parse_error_xml
    for child in error_node.getchildren():
AttributeError: 'NoneType' object has no attribute 'getchildren'

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
    An unexpected error has occurred.
  Please try reproducing the error using
  the latest s3cmd code from the git master
  branch found at:
    https://github.com/s3tools/s3cmd
  If the error persists, please report the
  above lines (removing any private
  info as necessary) to:
   s3tools-bugs@lists.sourceforge.net
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

It doesn't seem like a client related issue, can you confirm if it's a bug or not? Any other information I can provide?
@bsparrow435
Copy link

Hello,

We are actually investigating this exact issue at another customer site. It is not reproducible in all cases but we've made some progress. Can you please run the test again with the --debug flag for s3cmd and attach the riak-debug and riak-cs-debug from the RiakCS node and Riak node which you sent the request to.

Thanks,
Brian

@bsparrow435
Copy link

Additionally, from you can activate debug logging by running the following code from riak-cs attach on the node you are sending the test request to:

  1. Attach to the Erlang console on a RiakCS node with riak-cs attach.
  2. Paste lager:set_loglevel(lager_file_backend, "/var/log/riak-cs/console.log", debug). and hit ENTER. This should return ok.
  3. Test a copy operation sent to this node.
  4. Reset the log level to info from the attach console by pasting lager:set_loglevel(lager_file_backend, "/var/log/riak-cs/console.log", info). and hitting ENTER. This should return ok.
  5. Detach from the Erlang console with CTRL-C CTRL-C.

This assumes default log location based on a package install. Please replace "/var/log/riak-cs/console.log" with the patch to your console.log noted in the riak-cs app.config.

@iby
Copy link
Author

iby commented Nov 25, 2014

Ok, there's definitely somethings goes wrong: <html><head><title>500 Internal Server Error</title></head><body><h1>Internal Server Error</h1>The server encountered an error while processing this request</body></html>

$ s3cmd cp s3://tst/1234.jpg s3://tst/5678.jpg --debug
https://gist.github.com/ianbytchek/a5edd14af6803f6cb8cd

$ cat /var/log/riak-cs/console.log
https://gist.github.com/ianbytchek/6325014f3306005f2a66

$ riak-debug
http://cl.ly/2h1c1H2O3o0s

$ riak-cs-debug
http://cl.ly/122A131M3m2O

Let me know if I can do anything else.

@kuenishi
Copy link
Contributor

Thank you for reporting. The crash.log told me the answer! This is likely a bug when Riak CS is directly serving SSL, actually copy does not work in that case. Here when the Socket is ssl then this kludge should use ssl module instead of inet. I think you can avoid this problem by letting load balancer serve SSL and connect with CS without SSL. If you are using some SSL-based or source-ip based bucket policy (was:SourceIp or was:SecureTransport), please consult our documentation.

@kuenishi kuenishi added the Bug label Nov 25, 2014
@iby
Copy link
Author

iby commented Nov 25, 2014

Thanks @kuenishi, I do indeed use SSL directly with Riak. Taking the SSL handling outside Riak CS solves the problem. I'll use a workaround for now. Any ideas when you might release a fixed version, though?

@shino shino changed the title Copying an object fails Copying an object fails when connecting via https Jul 27, 2015
@kuenishi kuenishi added this to the 2.1.2 milestone Feb 10, 2016
@kuenishi
Copy link
Contributor

kuenishi commented Mar 4, 2016

create jira issue

@Basho-JIRA Basho-JIRA changed the title Copying an object fails when connecting via https Copying an object fails when connecting via https [JIRA: RCS-358] Mar 4, 2016
@kuenishi
Copy link
Contributor

My finding was ibrowse doesn't allow proxy setting and SSL at the same time so automating a test on fix for this with erlcloud needs some workaround...

do_connect(Host, Port, Options, #state{is_ssl      = true,
                                       use_proxy   = false,
                                       ssl_options = SSLOptions},
           Timeout) ->
    ssl:connect(Host, Port, get_sock_options(Host, Options, SSLOptions), Timeout);

@Basho-JIRA Basho-JIRA assigned kuenishi and unassigned tburghart Mar 11, 2016
@Basho-JIRA
Copy link

[~kota] assigned this to you for now since Ted is moving on to another. Looks like you are doing some work on it.

_[posted via JIRA by Patricia Brewer]_

@kuenishi
Copy link
Contributor

Contradiction of proxy and ssl causes problem at testing as no client library allows both coexisting. But as we already has simple stupid API call , given only for PUT copy API, it won't be such hard to automate it.

@kuenishi kuenishi mentioned this issue Mar 15, 2016
Closed
@Basho-JIRA Basho-JIRA assigned tburghart and unassigned kuenishi Mar 17, 2016
@Basho-JIRA
Copy link

PR #1305 merges #1300 with the 2.1 branch (for full testing) and gets host/port values from configuration in https_tests

_[posted via JIRA by Ted Burghart]_

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tburghart tburghart

Labels
Bug JIRA: Closed
Projects
None yet
Milestone
2.1.2
Development

No branches or pull requests
5 participants
@kuenishi @iby @bsparrow435 @tburghart @Basho-JIRA