Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Release v6.1.0 #58

Merged
merged 45 commits into from

2 participants

@xushiwei
Owner

No description provided.

SunRunAway and others added some commits
@SunRunAway SunRunAway add test ed897f4
@SunRunAway SunRunAway new put api & new multipart_form 865b63a
@SunRunAway SunRunAway crc32 9be4c2e
@SunRunAway SunRunAway fix 4bfb993
@SunRunAway SunRunAway README.md bcb418f
@SunRunAway SunRunAway random FileName when key is None 440ee65
@SunRunAway SunRunAway randomize the test keys 8a3fc5c
@SunRunAway SunRunAway workaround a bug with urllib.quote(unicode) b8688a7
@SunRunAway SunRunAway test code ef30b8e
@SunRunAway SunRunAway rpc.py/httplib can post chunked data f5d1e0e
@SunRunAway SunRunAway changelog & py docstring ac4d35f
@SunRunAway SunRunAway add unittest info in README.md b3cafae
@SunRunAway SunRunAway remove unused code d6bb2fe
@SunRunAway SunRunAway nosetests with coverage ef7b745
@SunRunAway SunRunAway Merge branch 'feature/multipart_reader' into feature/unittest_doc 0bf513f
@SunRunAway SunRunAway fix 6b45d10
@SunRunAway SunRunAway fix ebad5f7
@SunRunAway SunRunAway update demo & docs ace4a75
@SunRunAway SunRunAway fix a bug in io.put, when key is None and extra.params is not empty b989d5d
@SunRunAway SunRunAway Merge branch 'feature/multipart_reader' into feature/unittest_doc ca68739
@SunRunAway SunRunAway fix test 81c93b7
@SunRunAway SunRunAway fix test 2c81a6f
@SunRunAway SunRunAway Merge branch 'feature/multipart_reader' into feature/unittest_doc ae87f05
@SunRunAway SunRunAway version=6.1.0 cba796c
@SunRunAway SunRunAway qiniu_escape a73b91c
@SunRunAway SunRunAway Merge branch 'feature/multipart_reader' into feature/unittest_doc 241b9e2
@SunRunAway SunRunAway x 011c858
@SunRunAway SunRunAway test-env.sh fa0b457
@SunRunAway SunRunAway Merge branch 'feature/multipart_reader' of github.com:SunRunAway/pyth…
…on-sdk into feature/multipart_reader
94e9162
@SunRunAway SunRunAway Merge branch 'feature/multipart_reader' into feature/workaround_urllib 0ff4001
@SunRunAway SunRunAway Merge branch 'feature/multipart_reader' into feature/unittest_doc f418045
@SunRunAway SunRunAway test-env.sh in README.md 9a4bd59
@SunRunAway SunRunAway travis 187be56
@SunRunAway SunRunAway Merge branch 'feature/multipart_reader' into feature/workaround_urllib 39d4eca
@SunRunAway SunRunAway Merge branch 'feature/multipart_reader' into feature/unittest_doc aa26c17
@SunRunAway SunRunAway fix README.md bbed4fe
@SunRunAway SunRunAway fix README.md c5f8498
@SunRunAway SunRunAway fix README.md 3eb366e
@SunRunAway SunRunAway add test 049e0cb
@SunRunAway SunRunAway fix test-env.sh according to impl.md daf2566
@SunRunAway SunRunAway .travis.yml b246a33
@xushiwei xushiwei Merge pull request #53 from SunRunAway/feature/multipart_reader
new put api & new multipart form
2b34296
@xushiwei xushiwei Merge pull request #55 from SunRunAway/feature/workaround_urllib
Feature/workaround urllib
8786716
@xushiwei xushiwei Merge pull request #56 from SunRunAway/feature/unittest_doc
travis CI & README.md
fb5c1b0
@xushiwei xushiwei Update CHANGELOG.md e027dc9
@xushiwei xushiwei merged commit 7ea957a into master

1 check was pending

Details default The Travis CI build is in progress
@SunRunAway SunRunAway referenced this pull request
Closed

NoneType has no len() #54

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Jun 30, 2013
  1. @SunRunAway

    add test

    SunRunAway authored
Commits on Jul 1, 2013
  1. @SunRunAway
  2. @SunRunAway

    crc32

    SunRunAway authored
  3. @SunRunAway

    fix

    SunRunAway authored
  4. @SunRunAway

    README.md

    SunRunAway authored
  5. @SunRunAway
  6. @SunRunAway

    randomize the test keys

    SunRunAway authored
Commits on Jul 2, 2013
  1. @SunRunAway
Commits on Jul 3, 2013
  1. @SunRunAway

    test code

    SunRunAway authored
  2. @SunRunAway
  3. @SunRunAway
  4. @SunRunAway
  5. @SunRunAway

    remove unused code

    SunRunAway authored
  6. @SunRunAway

    nosetests with coverage

    SunRunAway authored
  7. @SunRunAway
  8. @SunRunAway

    fix

    SunRunAway authored
  9. @SunRunAway

    fix

    SunRunAway authored
  10. @SunRunAway

    update demo & docs

    SunRunAway authored
  11. @SunRunAway
  12. @SunRunAway
  13. @SunRunAway

    fix test

    SunRunAway authored
  14. @SunRunAway

    fix test

    SunRunAway authored
  15. @SunRunAway
  16. @SunRunAway

    version=6.1.0

    SunRunAway authored
Commits on Jul 4, 2013
  1. @SunRunAway

    qiniu_escape

    SunRunAway authored
  2. @SunRunAway
  3. @SunRunAway

    x

    SunRunAway authored
Commits on Jul 5, 2013
  1. @SunRunAway

    test-env.sh

    SunRunAway authored
  2. @SunRunAway

    Merge branch 'feature/multipart_reader' of github.com:SunRunAway/pyth…

    SunRunAway authored
    …on-sdk into feature/multipart_reader
  3. @SunRunAway
  4. @SunRunAway
  5. @SunRunAway

    test-env.sh in README.md

    SunRunAway authored
  6. @SunRunAway

    travis

    SunRunAway authored
  7. @SunRunAway
  8. @SunRunAway
  9. @SunRunAway

    fix README.md

    SunRunAway authored
  10. @SunRunAway

    fix README.md

    SunRunAway authored
  11. @SunRunAway

    fix README.md

    SunRunAway authored
  12. @SunRunAway

    add test

    SunRunAway authored
  13. @SunRunAway
  14. @SunRunAway

    .travis.yml

    SunRunAway authored
  15. @xushiwei

    Merge pull request #53 from SunRunAway/feature/multipart_reader

    xushiwei authored
    new put api & new multipart form
  16. @xushiwei

    Merge pull request #55 from SunRunAway/feature/workaround_urllib

    xushiwei authored
    Feature/workaround urllib
  17. @xushiwei

    Merge pull request #56 from SunRunAway/feature/unittest_doc

    xushiwei authored
    travis CI & README.md
  18. @xushiwei

    Update CHANGELOG.md

    xushiwei authored
This page is out of date. Refresh to see the latest.
View
1  .gitignore
@@ -4,6 +4,7 @@
*.py[cod]
+my-test-env.sh
##
## from https://github.com/github/gitignore/blob/master/Python.gitignore
View
16 .travis.yml
@@ -2,13 +2,15 @@ language: python
python:
- "2.6"
- "2.7"
+install:
+ - "pip install coverage --use-mirrors"
before_script:
- export QINIU_ACCESS_KEY="X0XpjFmLMTJpHB_ESHjeolCtipk-1U3Ok7LVTdoN"
- export QINIU_SECRET_KEY="wenlwkU1AYwNBf7Q9cCoG4VT_GYyrHE9AS_R2u81"
- - export QINIU_PIC_KEY="hello_jpg"
- - export QINIU_NOEXIST_PIC_KEY="no_exist"
- - export QINIU_BUCKET_NAME="pysdk"
- - export QINIU_DOMAIN="pysdk.qiniudn.com"
-script:
- - python setup.py nosetests
- - export PYTHONPATH="$PYTHONPATH:." && python docs/demo.py
+ - export QINIU_TEST_BUCKET="pysdk"
+ - export QINIU_TEST_DOMAIN="pysdk.qiniudn.com"
+ - export PYTHONPATH="$PYTHONPATH:."
+script:
+ - nosetests --with-coverage --cover-package=qiniu
+ - python docs/demo.py
+ - python docs/gist/conf.py
View
12 CHANGELOG.md
@@ -1,5 +1,16 @@
## CHANGE LOG
+### v6.1.0
+
+2013-07-03 issue [#58](https://github.com/qiniu/python-sdk/pull/58)
+
+- 实现最新版的上传API,<http://docs.qiniu.com/api/put.html>
+ - io.PutExtra更新,废弃callback_params,bucket,和custom_meta,新增params
+- 修复[#16](https://github.com/qiniu/python-sdk/issues/16)
+ - put接口可以传入类文件对象(file-like object)
+- 修复[#52](https://github.com/qiniu/python-sdk/issues/52)
+
+
### v6.0.1
2013-06-27 issue [#43](https://github.com/qiniu/python-sdk/pull/43)
@@ -7,6 +18,7 @@
- 遵循 [sdkspec v6.0.2](https://github.com/qiniu/sdkspec/tree/v6.0.2)
- 现在,rsf.list_prefix在没有更多数据时,err 会返回 rsf.EOF
+
### v6.0.0
2013-06-26 issue [#42](https://github.com/qiniu/python-sdk/pull/42)
View
17 README.md
@@ -9,6 +9,23 @@ Qiniu Resource Storage SDK for Python
参考文档:[七牛云存储 Python SDK 使用指南](https://github.com/qiniu/python-sdk/blob/develop/docs/README.md)
+## 单元测试
+
+1. 测试环境
+
+ 1. [开通七牛开发者帐号](https://portal.qiniu.com/signup)
+ 2. [登录七牛开发者自助平台,查看 Access Key 和 Secret Key](https://portal.qiniu.com/setting/key) 。
+ 3. 在开发者后台新建一个空间
+
+ 然后将在`test-env.sh`中填入相关信息。
+
+2. 需安装[nosetests](https://nose.readthedocs.org/en/latest/)测试工具。
+
+运行测试:
+
+ source test-env.sh
+ nosetests
+
## 贡献代码
1. Fork
View
9 docs/README.md
@@ -259,10 +259,12 @@ qiniu.conf.SECRET_KEY = "<YOUR_APP_SECRET_KEY>"
import qiniu.io
-extra = qiniu.io.PutExtra(bucket_name)
+extra = qiniu.io.PutExtra()
extra.mime_type = "text/plain"
-ret, err = qiniu.io.put(uptoken, key, "hello!", extra)
+# data 可以是str或read()able对象
+data = StringIO.StringIO("hello!")
+ret, err = qiniu.io.put(uptoken, key, data, extra)
if err is not None:
error(err)
return
@@ -279,9 +281,8 @@ qiniu.conf.SECRET_KEY = "<YOUR_APP_SECRET_KEY>"
import qiniu.io
localfile = "%s" % __file__
-extra = qiniu.io.PutExtra(bucket_name)
-ret, err = qiniu.io.put_file(uptoken, key, localfile, extra)
+ret, err = qiniu.io.put_file(uptoken, key, localfile)
if err is not None:
error(err)
return
View
18 docs/demo.py
@@ -1,6 +1,7 @@
# -*- coding: utf-8 -*-
import os
import sys
+import StringIO
# @gist import_io
import qiniu.io
@@ -49,9 +50,9 @@ def _setup():
if access_key is None:
exit("请配置环境变量 QINIU_ACCESS_KEY")
secret_key = getenv("QINIU_SECRET_KEY")
- bucket_name = getenv("QINIU_BUCKET_NAME")
- domain = getenv("QINIU_DOMAIN")
- pickey = getenv("QINIU_PIC_KEY")
+ bucket_name = getenv("QINIU_TEST_BUCKET")
+ domain = getenv("QINIU_TEST_DOMAIN")
+ pickey = 'QINIU_UNIT_TEST_PIC'
setup(access_key, secret_key, bucket_name, domain, pickey)
def getenv(name):
@@ -95,9 +96,8 @@ def put_file():
# @gist put_file
localfile = "%s" % __file__
- extra = qiniu.io.PutExtra(bucket_name)
-
- ret, err = qiniu.io.put_file(uptoken, key, localfile, extra)
+
+ ret, err = qiniu.io.put_file(uptoken, key, localfile)
if err is not None:
error(err)
return
@@ -110,10 +110,12 @@ def put_binary():
qiniu.rs.Client().delete(bucket_name, key)
# @gist put
- extra = qiniu.io.PutExtra(bucket_name)
+ extra = qiniu.io.PutExtra()
extra.mime_type = "text/plain"
- ret, err = qiniu.io.put(uptoken, key, "hello!", extra)
+ # data 可以是str或read()able对象
+ data = StringIO.StringIO("hello!")
+ ret, err = qiniu.io.put(uptoken, key, data, extra)
if err is not None:
error(err)
return
View
4 qiniu/__init__.py
@@ -1,4 +1,3 @@
-# -*- coding: utf-8 -*-
'''
Qiniu Resource Storage SDK for Python
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -7,4 +6,5 @@
<https://github.com/qiniu/python-sdk/blob/develop/docs/README.md>
'''
-__version__ = '6.0.1'
+# -*- coding: utf-8 -*-
+__version__ = '6.1.0'
View
123 qiniu/httplib_chunk.py
@@ -0,0 +1,123 @@
+"""
+Modified from standard httplib
+
+1. HTTPConnection can send trunked data.
+2. Remove httplib's automatic Content-Length insertion when data is a file-like object.
+"""
+
+# -*- coding: utf-8 -*-
+
+import httplib
+from httplib import _CS_REQ_STARTED, _CS_REQ_SENT
+import string
+import os
+from array import array
+
+class HTTPConnection(httplib.HTTPConnection):
+
+ def send(self, data, is_chunked=False):
+ """Send `data' to the server."""
+ if self.sock is None:
+ if self.auto_open:
+ self.connect()
+ else:
+ raise NotConnected()
+
+ if self.debuglevel > 0:
+ print "send:", repr(data)
+ blocksize = 8192
+ if hasattr(data,'read') and not isinstance(data, array):
+ if self.debuglevel > 0: print "sendIng a read()able"
+ datablock = data.read(blocksize)
+ while datablock:
+ if self.debuglevel > 0:
+ print 'chunked:', is_chunked
+ if is_chunked:
+ if self.debuglevel > 0: print 'send: with trunked data'
+ lenstr = string.upper(hex(len(datablock))[2:])
+ self.sock.sendall('%s\r\n%s\r\n' % (lenstr, datablock))
+ else:
+ self.sock.sendall(datablock)
+ datablock = data.read(blocksize)
+ if is_chunked:
+ self.sock.sendall('0\r\n\r\n')
+ else:
+ self.sock.sendall(data)
+
+
+ def _set_content_length(self, body):
+ # Set the content-length based on the body.
+ thelen = None
+ try:
+ thelen = str(len(body))
+ except (TypeError, AttributeError), te:
+ # Don't send a length if this failed
+ if self.debuglevel > 0: print "Cannot stat!!"
+
+ if thelen is not None:
+ self.putheader('Content-Length', thelen)
+ return True
+ return False
+
+
+ def _send_request(self, method, url, body, headers):
+ # Honor explicitly requested Host: and Accept-Encoding: headers.
+ header_names = dict.fromkeys([k.lower() for k in headers])
+ skips = {}
+ if 'host' in header_names:
+ skips['skip_host'] = 1
+ if 'accept-encoding' in header_names:
+ skips['skip_accept_encoding'] = 1
+
+ self.putrequest(method, url, **skips)
+
+ is_chunked = False
+ if body and header_names.get('Transfer-Encoding') == 'chunked':
+ is_chunked = True
+ elif body and ('content-length' not in header_names):
+ is_chunked = not self._set_content_length(body)
+ if is_chunked:
+ self.putheader('Transfer-Encoding', 'chunked')
+ for hdr, value in headers.iteritems():
+ self.putheader(hdr, value)
+
+ self.endheaders(body, is_chunked=is_chunked)
+
+
+ def endheaders(self, message_body=None, is_chunked=False):
+ """Indicate that the last header line has been sent to the server.
+
+ This method sends the request to the server. The optional
+ message_body argument can be used to pass a message body
+ associated with the request. The message body will be sent in
+ the same packet as the message headers if it is string, otherwise it is
+ sent as a separate packet.
+ """
+ if self.__state == _CS_REQ_STARTED:
+ self.__state = _CS_REQ_SENT
+ else:
+ raise CannotSendHeader()
+ self._send_output(message_body, is_chunked=is_chunked)
+
+
+ def _send_output(self, message_body=None, is_chunked=False):
+ """Send the currently buffered request and clear the buffer.
+
+ Appends an extra \\r\\n to the buffer.
+ A message_body may be specified, to be appended to the request.
+ """
+ self._buffer.extend(("", ""))
+ msg = "\r\n".join(self._buffer)
+ del self._buffer[:]
+ # If msg and message_body are sent in a single send() call,
+ # it will avoid performance problems caused by the interaction
+ # between delayed ack and the Nagle algorithm.
+ if isinstance(message_body, str):
+ msg += message_body
+ message_body = None
+ self.send(msg)
+ if message_body is not None:
+ #message_body was not a string (i.e. it is a file) and
+ #we must run the risk of Nagle
+ self.send(message_body, is_chunked=is_chunked)
+
View
93 qiniu/io.py
@@ -2,52 +2,79 @@
from base64 import urlsafe_b64encode
import rpc
import conf
-import zlib
+import random
+import string
+try:
+ import zlib as binascii
+except ImportError:
+ import binascii
-UNDEFINED_KEY = "?"
class PutExtra(object):
- callback_params = None
- bucket = None
- custom_meta = None
- mime_type = None
+ params = {}
+ mime_type = 'application/octet-stream'
crc32 = ""
check_crc = 0
- def __init__(self, bucket):
- self.bucket = bucket
-def put(uptoken, key, data, extra):
- action = ["/rs-put"]
- action.append(urlsafe_b64encode("%s:%s" % (extra.bucket, key)))
- if extra.mime_type is not None:
- action.append("mimeType/%s" % urlsafe_b64encode(extra.mime_type))
- if extra.custom_meta is not None:
- action.append("meta/%s" % urlsafe_b64encode(extra.custom_meta))
+def put(uptoken, key, data, extra=None):
+ """ put your data to Qiniu
+
+ If key is None, the server will generate one.
+ data may be str or read()able object.
+ """
+ fields = {
+ }
+
+ if not extra:
+ extra = PutExtra()
+
+ if extra.params:
+ for k in extra.params:
+ fields[k] = str(extra.params[k])
if extra.check_crc:
- action.append("crc32/%s" % extra.crc32)
+ fields["crc32"] = str(extra.crc32)
- fields = [
- ("action", '/'.join(action)),
- ("auth", uptoken),
- ]
- if extra.callback_params is not None:
- fields.append(("params", extra.callback_params))
+ if key is not None:
+ fields['key'] = key
+
+ fields["token"] = uptoken
+ fname = key
+ if fname is None:
+ fname = _random_str(9)
+ elif fname is '':
+ fname = 'index.html'
files = [
- ("file", key, data)
+ {'filename': fname, 'data': data, 'mime_type': extra.mime_type},
]
- return rpc.Client(conf.UP_HOST).call_with_multipart("/upload", fields, files)
+ return rpc.Client(conf.UP_HOST).call_with_multipart("/", fields, files)
+
+
+def put_file(uptoken, key, localfile, extra=None):
+ """ put a file to Qiniu
+
+ If key is None, the server will generate one.
+ """
+ if extra is not None and extra.check_crc == 1:
+ extra.crc32 = _get_file_crc32(localfile)
+ with open(localfile) as f:
+ return put(uptoken, key, f, extra)
+
+
+_BLOCK_SIZE = 1024 * 1024 * 4
-def put_file(uptoken, key, localfile, extra):
- f = open(localfile)
- data = f.read()
- f.close()
- if extra.check_crc == 1:
- extra.crc32 = zlib.crc32(data) & 0xFFFFFFFF
- return put(uptoken, key, data, extra)
+def _get_file_crc32(filepath):
+ with open(filepath) as f:
+ block = f.read(_BLOCK_SIZE)
+ crc = 0
+ while len(block) != 0:
+ crc = binascii.crc32(block, crc) & 0xFFFFFFFF
+ block = f.read(_BLOCK_SIZE)
+ return crc
-def get_url(domain, key, dntoken):
- return "%s/%s?token=%s" % (domain, key, dntoken)
+def _random_str(length):
+ lib = string.ascii_lowercase
+ return ''.join([random.choice(lib) for i in range(0, length)])
View
9 qiniu/resumable_io.py
@@ -1,13 +1,14 @@
# -*- coding: utf-8 -*-
import os
-import zlib
+try:
+ import zlib as binascii
+except ImportError:
+ import binascii
from base64 import urlsafe_b64encode
import auth.up
import conf
-UNDEFINED_KEY = "?"
-
_workers = 1
_task_queue_size = _workers * 4
_chunk_size = 256 * 1024
@@ -42,7 +43,7 @@ def setup(chunk_size=0, try_times=0):
# ----------------------------------------------------------
def gen_crc32(data):
- return zlib.crc32(data) & 0xffffffff
+ return binascii.crc32(data) & 0xffffffff
class PutExtra(object):
callback_params = None # 当 uptoken 指定了 CallbackUrl,则 CallbackParams 必须非空
View
148 qiniu/rpc.py
@@ -1,11 +1,14 @@
# -*- coding: utf-8 -*-
-import httplib
+import httplib_chunk as httplib
import json
+import cStringIO
import conf
+
class Client(object):
_conn = None
_header = None
+
def __init__(self, host):
self._conn = httplib.HTTPConnection(host)
self._header = {}
@@ -24,7 +27,7 @@ def call_with(self, path, body, content_type=None, content_length=None):
self.set_header("User-Agent", conf.USER_AGENT)
if content_type is not None:
self.set_header("Content-Type", content_type)
-
+
if content_length is not None:
self.set_header("Content-Length", content_length)
@@ -36,40 +39,40 @@ def call_with(self, path, body, content_type=None, content_length=None):
return None, e
except ValueError:
pass
-
+
if resp.status / 100 != 2:
err_msg = ret if "error" not in ret else ret["error"]
detail = resp.getheader("x-log", None)
if detail is not None:
err_msg += ", detail:%s" % detail
-
+
return None, err_msg
-
+
return ret, None
def call_with_multipart(self, path, fields=None, files=None):
"""
- * fields => [(key, value)]
- * files => [(key, filename, value)]
+ * fields => {key}
+ * files => [{filename, data, content_type}]
"""
- content_type, body = self.encode_multipart_formdata(fields, files)
- return self.call_with(path, body, content_type, len(body))
+ content_type, mr = self.encode_multipart_formdata(fields, files)
+ return self.call_with(path, mr, content_type, mr.length())
def call_with_form(self, path, ops):
"""
* ops => {"key": value/list()}
"""
-
+
body = []
for i in ops:
if isinstance(ops[i], (list, tuple)):
data = ('&%s=' % i).join(ops[i])
else:
data = ops[i]
-
+
body.append('%s=%s' % (i, data))
body = '&'.join(body)
-
+
content_type = "application/x-www-form-urlencoded"
return self.call_with(path, body, content_type, len(body))
@@ -81,32 +84,119 @@ def set_headers(self, headers):
def encode_multipart_formdata(self, fields, files):
"""
- * fields => [(key, value)]
- * files => [(key, filename, value)]
- * return content_type, body
+ * fields => {key}
+ * files => [{filename, data, content_type}]
+ * return content_type, content_length, body
"""
if files is None:
files = []
if fields is None:
- fields = []
+ fields = {}
+ readers = []
BOUNDARY = '----------ThIs_Is_tHe_bouNdaRY_$'
CRLF = '\r\n'
- L = []
- for (key, value) in fields:
- L.append('--' + BOUNDARY)
- L.append('Content-Disposition: form-data; name="%s"' % key)
+ L1 = []
+ for key in fields:
+ L1.append('--' + BOUNDARY)
+ L1.append('Content-Disposition: form-data; name="%s"' % key)
+ L1.append('')
+ L1.append(fields[key])
+ b1 = CRLF.join(L1)
+ readers.append(b1)
+
+ for file_info in files:
+ L = []
L.append('')
- L.append(value)
- for (key, filename, value) in files:
L.append('--' + BOUNDARY)
disposition = "Content-Disposition: form-data;"
- L.append('%s name="%s"; filename="%s"' % (disposition, key, filename))
- L.append('Content-Type: application/octet-stream')
+ filename = _qiniu_escape(file_info.get('filename'))
+ L.append('%s name="file"; filename="%s"' % (disposition, filename))
+ L.append('Content-Type: %s' % file_info.get('content_type', 'application/octet-stream'))
L.append('')
- L.append(value)
- L.append('--' + BOUNDARY + '--')
- L.append('')
- body = CRLF.join(L)
+ L.append('')
+ b2 = CRLF.join(L)
+ readers.append(b2)
+
+ data = file_info.get('data')
+ readers.append(data)
+
+ L3 = ['', '--' + BOUNDARY + '--', '']
+ b3 = CRLF.join(L3)
+ readers.append(b3)
+
content_type = 'multipart/form-data; boundary=%s' % BOUNDARY
- return content_type, body
+ return content_type, MultiReader(readers)
+
+def _qiniu_escape(s):
+ edits = [('\\', '\\\\'), ('\"', '\\\"')]
+ for (search, replace) in edits:
+ s = s.replace(search, replace)
+ return s
+
+
+class MultiReader(object):
+ """ class MultiReader([readers...])
+
+ MultiReader returns a read()able object that's the logical concatenation of
+ the provided input readers. They're read sequentially.
+ """
+
+ def __init__(self, readers):
+ self.readers = []
+ self.content_length = 0
+ self.valid_content_length = True
+ for r in readers:
+ if hasattr(r, 'read'):
+ if self.valid_content_length:
+ length = self._get_content_length(r)
+ if length is not None:
+ self.content_length += length
+ else:
+ self.valid_content_length = False
+ else:
+ buf = r
+ if not isinstance(buf, basestring):
+ buf = str(buf)
+ buf = encode_unicode(buf)
+ r = cStringIO.StringIO(buf)
+ self.content_length += len(buf)
+ self.readers.append(r)
+
+
+ # don't name it __len__, because the length of MultiReader is not alway valid.
+ def length(self):
+ return self.content_length if self.valid_content_length else None
+
+
+ def _get_content_length(self, reader):
+ data_len = None
+ if hasattr(reader, 'seek') and hasattr(reader, 'tell'):
+ try:
+ reader.seek(0, 2)
+ data_len= reader.tell()
+ reader.seek(0, 0)
+ except OSError:
+ # Don't send a length if this failed
+ data_len = None
+ return data_len
+
+ def read(self, n=-1):
+ if n is None or n == -1:
+ return ''.join([encode_unicode(r.read()) for r in self.readers])
+ else:
+ L = []
+ while len(self.readers) > 0 and n > 0:
+ b = self.readers[0].read(n)
+ if len(b) == 0:
+ self.readers = self.readers[1:]
+ else:
+ L.append(encode_unicode(b))
+ n -= len(b)
+ return ''.join(L)
+
+
+def encode_unicode(u):
+ if isinstance(u, unicode):
+ u = u.encode('utf8')
+ return u
View
2  qiniu/rs/rs_token.py
@@ -4,6 +4,7 @@
import urllib
from ..auth import digest
+from ..import rpc
class PutPolicy(object):
scope = None # 可以是 bucketName 或者 bucketName:key
@@ -76,4 +77,5 @@ def make_base_url(domain, key):
* key => str
* return base_url
'''
+ key = rpc.encode_unicode(key)
return 'http://%s/%s' % (domain, urllib.quote(key))
View
25 qiniu/rs/test/__init__.py
@@ -0,0 +1,25 @@
+# -*- coding: utf-8 -*-
+import os
+import urllib
+
+import qiniu.io
+import qiniu.rs
+import qiniu.conf
+
+pic = "http://cheneya.qiniudn.com/hello_jpg"
+key = 'QINIU_UNIT_TEST_PIC'
+
+def setUp():
+ qiniu.conf.ACCESS_KEY = os.getenv("QINIU_ACCESS_KEY")
+ qiniu.conf.SECRET_KEY = os.getenv("QINIU_SECRET_KEY")
+ bucket_name = os.getenv("QINIU_TEST_BUCKET")
+
+ policy = qiniu.rs.PutPolicy(bucket_name)
+ uptoken = policy.token()
+
+ f = urllib.urlopen(pic)
+ _, err = qiniu.io.put(uptoken, key, f)
+ f.close()
+ if err is None or err.startswith('file exists'):
+ print err
+ assert err is None or err.startswith('file exists')
View
6 qiniu/rs/test/rs_test.py
@@ -13,9 +13,9 @@ def r(length):
conf.ACCESS_KEY = os.getenv("QINIU_ACCESS_KEY")
conf.SECRET_KEY = os.getenv("QINIU_SECRET_KEY")
-key = os.getenv("QINIU_PIC_KEY")
-bucket_name = os.getenv("QINIU_BUCKET_NAME")
-noexist_key = os.getenv("QINIU_NOEXIST_PIC_KEY")
+key = 'QINIU_UNIT_TEST_PIC'
+bucket_name = os.getenv("QINIU_TEST_BUCKET")
+noexist_key = 'QINIU_UNIT_TEST_NOEXIST' + r(30)
key2 = "rs_demo_test_key_1_" + r(5)
key3 = "rs_demo_test_key_2_" + r(5)
key4 = "rs_demo_test_key_3_" + r(5)
View
12 qiniu/rs/test/rs_token_test.py
@@ -14,9 +14,9 @@
conf.ACCESS_KEY = os.getenv("QINIU_ACCESS_KEY")
conf.SECRET_KEY = os.getenv("QINIU_SECRET_KEY")
-bucket_name = os.getenv("QINIU_BUCKET_NAME")
-domain = os.getenv("QINIU_DOMAIN")
-key = os.getenv("QINIU_PIC_KEY")
+bucket_name = os.getenv("QINIU_TEST_BUCKET")
+domain = os.getenv("QINIU_TEST_DOMAIN")
+key = 'QINIU_UNIT_TEST_PIC'
class TestToken(unittest.TestCase):
def test_put_policy(self):
@@ -42,5 +42,11 @@ def test_get_policy(self):
self.assertEqual(len(body)>100, True)
+class Test_make_base_url(unittest.TestCase):
+ def test_unicode(self):
+ url1 = rs.make_base_url('1.com', '你好')
+ url2 = rs.make_base_url('1.com', u'你好')
+ assert url1 == url2
+
if __name__ == "__main__":
unittest.main()
View
130 qiniu/test/io_test.py
@@ -3,8 +3,12 @@
import unittest
import string
import random
-import zlib
-from base64 import urlsafe_b64encode as encode
+import urllib
+try:
+ import zlib as binascii
+except ImportError:
+ import binascii
+import cStringIO
from qiniu import conf
from qiniu import rs
@@ -12,11 +16,12 @@
conf.ACCESS_KEY = os.getenv("QINIU_ACCESS_KEY")
conf.SECRET_KEY = os.getenv("QINIU_SECRET_KEY")
-bucket_name = os.getenv("QINIU_BUCKET_NAME")
+bucket_name = os.getenv("QINIU_TEST_BUCKET")
policy = rs.PutPolicy(bucket_name)
-extra = io.PutExtra(bucket_name)
+extra = io.PutExtra()
extra.mime_type = "text/plain"
+extra.params = {'x:a':'a'}
def r(length):
lib = string.ascii_uppercase
@@ -29,20 +34,119 @@ def test_put():
params = "op=3"
data = "hello bubby!"
extra.check_crc = 2
- extra.crc32 = zlib.crc32(data) & 0xFFFFFFFF
+ extra.crc32 = binascii.crc32(data) & 0xFFFFFFFF
ret, err = io.put(policy.token(), key, data, extra)
assert err is None
+ assert ret['key'] == key
def test_put_same_crc():
key = "test_%s" % r(9)
- params = "op=3"
data = "hello bubby!"
extra.check_crc = 2
ret, err = io.put(policy.token(), key, data, extra)
assert err is None
+ assert ret['key'] == key
+
+ def test_put_no_key():
+ data = r(100)
+ extra.check_crc = 0
+ ret, err = io.put(policy.token(), key=None, data=data, extra=extra)
+ assert err is None
+ assert ret['hash'] == ret['key']
+
+ def test_put_quote_key():
+ data = r(100)
+ key = 'a\\b\\c"你好' + r(9)
+ ret, err = io.put(policy.token(), key, data)
+ print err
+ assert err is None
+ assert ret['key'].encode('utf8') == key
+
+ data = r(100)
+ key = u'a\\b\\c"你好' + r(9)
+ ret, err = io.put(policy.token(), key, data)
+ assert err is None
+ assert ret['key'] == key
+
+ def test_put_unicode1():
+ key = "test_%s" % r(9) + '你好'
+ data = key
+ ret, err = io.put(policy.token(), key, data, extra)
+ assert err is None
+ assert ret[u'key'].endswith(u'你好')
+
+ def test_put_unicode2():
+ key = "test_%s" % r(9) + '你好'
+ data = key
+ data = data.decode('utf8')
+ ret, err = io.put(policy.token(), key, data)
+ assert err is None
+ assert ret[u'key'].endswith(u'你好')
+
+ def test_put_unicode3():
+ key = "test_%s" % r(9) + '你好'
+ data = key
+ key = key.decode('utf8')
+ ret, err = io.put(policy.token(), key, data)
+ assert err is None
+ assert ret[u'key'].endswith(u'你好')
+
+ def test_put_unicode4():
+ key = "test_%s" % r(9) + '你好'
+ data = key
+ key = key.decode('utf8')
+ data = data.decode('utf8')
+ ret, err = io.put(policy.token(), key, data)
+ assert err is None
+ assert ret[u'key'].endswith(u'你好')
+
+ def test_put_StringIO():
+ key = "test_%s" % r(9)
+ data = cStringIO.StringIO('hello buddy!')
+ ret, err = io.put(policy.token(), key, data)
+ assert err is None
+ assert ret['key'] == key
+
+ def test_put_urlopen():
+ key = "test_%s" % r(9)
+ data = urllib.urlopen('http://cheneya.qiniudn.com/hello_jpg')
+ ret, err = io.put(policy.token(), key, data)
+ assert err is None
+ assert ret['key'] == key
+
+ def test_put_no_length():
+ class test_reader(object):
+ def __init__(self):
+ self.data = 'abc'
+ self.pos = 0
+ def read(self, n=None):
+ if n is None or n < 0:
+ newpos = len(self.data)
+ else:
+ newpos = min(self.pos+n, len(self.data))
+ r = self.data[self.pos: newpos]
+ self.pos = newpos
+ return r
+ key = "test_%s" % r(9)
+ data = test_reader()
+
+ extra.check_crc = 2
+ extra.crc32 = binascii.crc32('abc') & 0xFFFFFFFF
+ ret, err = io.put(policy.token(), key, data, extra)
+ assert err is None
+ assert ret['key'] == key
test_put()
test_put_same_crc()
+ test_put_no_key()
+ test_put_quote_key()
+ test_put_unicode1()
+ test_put_unicode2()
+ test_put_unicode3()
+ test_put_unicode4()
+ test_put_StringIO()
+ test_put_urlopen()
+ test_put_no_length()
def test_put_file(self):
localfile = "%s" % __file__
@@ -51,11 +155,10 @@ def test_put_file(self):
extra.check_crc = 1
ret, err = io.put_file(policy.token(), key, localfile, extra)
assert err is None
- assert ret is not None
+ assert ret['key'] == key
def test_put_crc_fail(self):
key = "test_%s" % r(9)
- params = "op=3"
data = "hello bubby!"
extra.check_crc = 2
extra.crc32 = "wrong crc32"
@@ -63,5 +166,16 @@ def test_put_crc_fail(self):
assert err is not None
+class Test_get_file_crc32(unittest.TestCase):
+ def test_get_file_crc32(self):
+ file_path = '%s' % __file__
+
+ data = None
+ with open(file_path) as f:
+ data = f.read()
+ io._BLOCK_SIZE = 4
+ assert binascii.crc32(data) & 0xFFFFFFFF == io._get_file_crc32(file_path)
+
+
if __name__ == "__main__":
unittest.main()
View
20 qiniu/test/resumable_io_test.py
@@ -1,7 +1,12 @@
# -*- coding: utf-8 -*-
import os
import unittest
-import zlib
+import string
+import random
+try:
+ import zlib as binascii
+except ImportError:
+ import binascii
import urllib
import tempfile
import shutil
@@ -11,10 +16,15 @@
from qiniu import resumable_io
from qiniu import rs
-bucket = os.getenv("QINIU_BUCKET_NAME")
+bucket = os.getenv("QINIU_TEST_BUCKET")
conf.ACCESS_KEY = os.getenv("QINIU_ACCESS_KEY")
conf.SECRET_KEY = os.getenv("QINIU_SECRET_KEY")
+
+def r(length):
+ lib = string.ascii_uppercase
+ return ''.join([random.choice(lib) for i in range(0, length)])
+
class TestBlock(unittest.TestCase):
def test_block(self):
policy = rs.PutPolicy(bucket)
@@ -25,7 +35,7 @@ def test_block(self):
data_slice_2 = "\nbye!"
ret, err = resumable_io.mkblock(client, len(data_slice_2), data_slice_2)
assert err is None, err
- self.assertEqual(ret["crc32"], zlib.crc32(data_slice_2))
+ self.assertEqual(ret["crc32"], binascii.crc32(data_slice_2))
extra = resumable_io.PutExtra(bucket)
extra.mimetype = "text/plain"
@@ -34,7 +44,7 @@ def test_block(self):
for i in xrange(0, len(extra.progresses)):
lens += extra.progresses[i]["offset"]
- key = "sdk_py_resumable_block_4"
+ key = u"sdk_py_resumable_block_4_%s" % r(9)
ret, err = resumable_io.mkfile(client, key, lens, extra)
assert err is None, err
self.assertEqual(ret["hash"], "FtCFo0mQugW98uaPYgr54Vb1QsO0", "hash not match")
@@ -49,7 +59,7 @@ def test_put(self):
policy = rs.PutPolicy(bucket)
extra = resumable_io.PutExtra(bucket)
extra.bucket = bucket
- key = "sdk_py_resumable_block_5"
+ key = "sdk_py_resumable_block_5_%s" % r(9)
localfile = dst.name
ret, err = resumable_io.put_file(policy.token(), key, localfile, extra)
dst.close()
View
110 qiniu/test/rpc_test.py
@@ -1,4 +1,5 @@
# -*- coding: utf-8 -*-
+import StringIO
import unittest
from qiniu import rpc
@@ -17,11 +18,11 @@ def round_tripper(self, method, path, body):
class TestClient(unittest.TestCase):
def test_call(self):
global round_tripper
-
+
def tripper(client, method, path, body):
self.assertEqual(path, "/hello")
assert body is None
-
+
round_tripper = tripper
client.call("/hello")
@@ -29,14 +30,13 @@ def test_call_with(self):
global round_tripper
def tripper(client, method, path, body):
self.assertEqual(body, "body")
-
+
round_tripper = tripper
client.call_with("/hello", "body")
def test_call_with_multipart(self):
global round_tripper
def tripper(client, method, path, body):
- self.assertEqual(len(body), client._header["Content-Length"])
target_type = "multipart/form-data"
self.assertTrue(client._header["Content-Type"].startswith(target_type))
start_index = client._header["Content-Type"].find("boundary")
@@ -44,10 +44,11 @@ def tripper(client, method, path, body):
dispostion = 'Content-Disposition: form-data; name="auth"'
tpl = "--%s\r\n%s\r\n\r\n%s\r\n--%s--\r\n" % (boundary, dispostion,
"auth_string", boundary)
- self.assertEqual(tpl, body)
-
+ self.assertEqual(len(tpl), client._header["Content-Length"])
+ self.assertEqual(len(tpl), body.length())
+
round_tripper = tripper
- client.call_with_multipart("/hello", fields=[("auth", "auth_string")])
+ client.call_with_multipart("/hello", fields={"auth": "auth_string"})
def test_call_with_form(self):
global round_tripper
@@ -56,9 +57,102 @@ def tripper(client, method, path, body):
target_type = "application/x-www-form-urlencoded"
self.assertEqual(client._header["Content-Type"], target_type)
self.assertEqual(client._header["Content-Length"], len(body))
-
+
round_tripper = tripper
client.call_with_form("/hello", dict(op=["a", "b"], action="a"))
+
+class TestMultiReader(unittest.TestCase):
+ def test_multi_reader1(self):
+ a = StringIO.StringIO('你好')
+ b = StringIO.StringIO('abcdefg')
+ c = StringIO.StringIO(u'悲剧')
+ mr = rpc.MultiReader([a, b, c])
+ data = mr.read()
+ assert data.index('悲剧') > data.index('abcdefg')
+
+ def test_multi_reader2(self):
+ a = StringIO.StringIO('你好')
+ b = StringIO.StringIO('abcdefg')
+ c = StringIO.StringIO(u'悲剧')
+ mr = rpc.MultiReader([a, b, c])
+ data = mr.read(8)
+ assert len(data) is 8
+
+
+def encode_multipart_formdata2(fields, files):
+ if files is None:
+ files = []
+ if fields is None:
+ fields = []
+
+ BOUNDARY = '----------ThIs_Is_tHe_bouNdaRY_$'
+ CRLF = '\r\n'
+ L = []
+ for (key, value) in fields:
+ L.append('--' + BOUNDARY)
+ L.append('Content-Disposition: form-data; name="%s"' % key)
+ L.append('')
+ L.append(value)
+ for (key, filename, value) in files:
+ L.append('--' + BOUNDARY)
+ disposition = "Content-Disposition: form-data;"
+ L.append('%s name="%s"; filename="%s"' % (disposition, key, filename))
+ L.append('Content-Type: application/octet-stream')
+ L.append('')
+ L.append(value)
+ L.append('--' + BOUNDARY + '--')
+ L.append('')
+ body = CRLF.join(L)
+ content_type = 'multipart/form-data; boundary=%s' % BOUNDARY
+ return content_type, body
+
+
+class TestEncodeMultipartFormdata(unittest.TestCase):
+ def test_encode(self):
+ fields = {'a': '1', 'b': '2'}
+ files = [
+ {
+ 'filename': 'key1',
+ 'data': 'data1',
+ 'mime_type': 'application/octet-stream',
+ },
+ {
+ 'filename': 'key2',
+ 'data': 'data2',
+ 'mime_type': 'application/octet-stream',
+ }
+ ]
+ content_type, mr = rpc.Client('localhost').encode_multipart_formdata(fields, files)
+ t, b = encode_multipart_formdata2(
+ [('a', '1'), ('b', '2')],
+ [('file', 'key1', 'data1'), ('file', 'key2', 'data2')]
+ )
+ assert t == content_type
+ assert len(b) == mr.length()
+
+ def test_unicode(self):
+ def test1():
+ files = [{'filename': '你好', 'data': '你好', 'mime_type': ''}]
+ _, body = rpc.Client('localhost').encode_multipart_formdata(None, files)
+ return len(body.read())
+ def test2():
+ files = [{'filename': u'你好', 'data': '你好', 'mime_type': ''}]
+ _, body = rpc.Client('localhost').encode_multipart_formdata(None, files)
+ return len(body.read())
+ def test3():
+ files = [{'filename': '你好', 'data': u'你好', 'mime_type': ''}]
+ _, body = rpc.Client('localhost').encode_multipart_formdata(None, files)
+ return len(body.read())
+ def test4():
+ files = [{'filename': u'你好', 'data': u'你好', 'mime_type': ''}]
+ _, body = rpc.Client('localhost').encode_multipart_formdata(None, files)
+ return len(body.read())
+
+ assert test1() == test2()
+ assert test2() == test3()
+ assert test3() == test4()
+
+
if __name__ == "__main__":
unittest.main()
View
8 qiniu/test/rsf_test.py
@@ -6,14 +6,14 @@
import os
conf.ACCESS_KEY = os.getenv("QINIU_ACCESS_KEY")
conf.SECRET_KEY = os.getenv("QINIU_SECRET_KEY")
-bucket_name = os.getenv("QINIU_BUCKET_NAME")
+bucket_name = os.getenv("QINIU_TEST_BUCKET")
class TestRsf(unittest.TestCase):
def test_list_prefix(self):
c = rsf.Client()
- ret, err = c.list_prefix(bucket_name)
- assert err is rsf.EOF
- self.assertEqual(len(ret.get('items'))>0, True)
+ ret, err = c.list_prefix(bucket_name, limit = 1)
+ self.assertEqual(err is rsf.EOF or err is None, True)
+ assert len(ret.get('items')) == 1
if __name__ == "__main__":
View
4 test-env.sh
@@ -0,0 +1,4 @@
+export QINIU_ACCESS_KEY="<access_key>"
+export QINIU_SECRET_KEY="<secret_key>"
+export QINIU_TEST_BUCKET="<bucket_name>"
+export QINIU_TEST_DOMAIN="<bucket_domain>"
Something went wrong with that request. Please try again.