fallback to v2 signing for non AWS endpoints #46

jindov · 2015-11-27T11:40:36Z

It's would be an awesome feature if you support to customize endpoint or host (s3.mystorage.com), I need to connect to my ceph-s3 system

gaul · 2015-11-27T18:18:08Z

Try running goofys --endpoint http://s3.mystorage.com/.

kahing · 2015-12-01T23:40:52Z

closing, this is already supported

jindov · 2015-12-04T03:06:54Z

This is result when I run mount command:
[root@jin ~]# goofys --endpoint http://s3.mystorage.net -o passwd_file=/opt/s3fs-fuse/.passwd --debug_fuse --debug_s3 -f logstash /mnt/logstore

2015/12/04 09:56:53.418869 s3.DEBUG DEBUG: Request s3/GetBucketLocation Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET http://s3.mystorage.net/logstash?location= HTTP/1.1
Host: s3.mystorage.net
User-Agent: aws-sdk-go/1.0.2 (go1.5.1; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=SXGI5OWV77Q865H027VZ/20151204/us-west-2/s3/aws4_request, SignedHeaders=host;x-amz-date, Signature=88d1accd4fbd7e2b792d766553dfe692e770a8130d1ec6ba7ae131c95a08b5f9
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20151204T025653Z
Accept-Encoding: gzip

2015/12/04 09:56:53.425540 s3.DEBUG DEBUG: Response s3/GetBucketLocation Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 400 Bad Request
Connection: close
Content-Length: 81
Accept-Ranges: bytes
Content-Type: application/xml
Date: Fri, 04 Dec 2015 02:58:17 GMT
Server: Apache/2.4.16 (Ubuntu)

2015/12/04 09:56:53.425785 s3.DEBUG DEBUG: Validate Response s3/GetBucketLocation failed, not retrying, error InvalidArgument:
status code: 400, request id:
2015/12/04 09:56:53.425822 s3.ERROR code= msg=400 request=

2015/12/04 09:56:53.425835 s3.ERROR code= msg=400 request=

2015/12/04 09:56:53.425844 s3.INFO Unable to detect bucket region, staying at 'us-west-2'
2015/12/04 09:56:53 mount helper error: fusermount: mount failed: Invalid argument
2015/12/04 09:56:53.450993 main.FATAL Mounting file system: Mount: mount: fusermount: exit status 1

kahing · 2015-12-04T03:27:27Z

try goofys --use-path-request --endpoint http://s3.mystorage.net --debug_fuse --debug_s3 -f logstash /mnt/logstore (add --use-path-request)

also, goofys doesn't take -o passwd_file option. Instead you need to use the aws command from Amazon to configure your credential or create the ~/.aws/credentials file as explained in README.

kahing · 2015-12-04T03:33:53Z

2015/12/04 09:56:53 mount helper error: fusermount: mount failed: Invalid argument
2015/12/04 09:56:53.450993 main.FATAL Mounting file system: Mount: mount: fusermount: exit status 1

I think this is what's causing your issue, goofys doesn't recognize -o passwd_file so we pass that down to fuse, which doesn't understand this option.

jindov · 2015-12-04T04:24:55Z

Thank you for your reply. I tried to run your command, it's mounted successfully but still get 400 error with s3

[root@jin ~]# goofys --use-path-request --endpoint http://s3.mystorage.net --debug_fuse --debug_s3 -f logstash /mnt/logstore
2015/12/04 11:25:46.281467 s3.DEBUG DEBUG: Request s3/GetBucketLocation Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET http://s3.mystorage.net/logstash?location= HTTP/1.1
Host: s3.mystorage.net
User-Agent: aws-sdk-go/1.0.2 (go1.5.1; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=J8VFMLKZ8QTL75W0LZOD/20151204/us-west-2/s3/aws4_request, SignedHeaders=host;x-amz-date, Signature=052252bf2fc9caa9f5e05c91df3afb3dcb697ac8457c2ed939493f0be1dffd90
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20151204T042546Z
Accept-Encoding: gzip


-----------------------------------------------------
2015/12/04 11:25:46.285671 s3.DEBUG DEBUG: Response s3/GetBucketLocation Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 400 Bad Request
Connection: close
Content-Length: 81
Accept-Ranges: bytes
Content-Type: application/xml
Date: Fri, 04 Dec 2015 04:27:10 GMT
Server: Apache/2.4.16 (Ubuntu)


-----------------------------------------------------
2015/12/04 11:25:46.285887 s3.DEBUG DEBUG: Validate Response s3/GetBucketLocation failed, not retrying, error InvalidArgument:
        status code: 400, request id:
2015/12/04 11:25:46.285912 s3.ERROR code= msg=400 request=

2015/12/04 11:25:46.285924 s3.ERROR code= msg=400 request=

2015/12/04 11:25:46.285933 s3.INFO Unable to detect bucket region, staying at 'us-west-2'
2015/12/04 11:25:46.289276 fuse.DEBUG Op 0x00000001        connection.go:395] <- init
2015/12/04 11:25:46.289338 fuse.DEBUG Op 0x00000001        connection.go:474] -> OK
2015/12/04 11:25:46.289365 main.INFO File system has been successfully mounted.

I use aws cli to create credential but still "Unable to detect bucket region"

Does goofys support s3-compatible API like Ceph Storage?

kahing · 2015-12-04T04:27:18Z

Are you able to access the mount point afterwards? The 400 error you see above is not important

jindov · 2015-12-04T07:30:55Z

This is log when I try to ls the mount point:

ls: reading directory logstore/: Input/output error

cannot create or do anything

2015/12/04 14:21:26.041557 fuse.DEBUG Op 0x00000003        connection.go:395] <- GetInodeAttributes (inode 1)
2015/12/04 14:21:26.042109 fuse.DEBUG GetAttributes 1  []
2015/12/04 14:21:26.042164 fuse.DEBUG Op 0x00000003        connection.go:474] -> OK
2015/12/04 14:21:28.990944 fuse.DEBUG Op 0x00000004        connection.go:395] <- OpenDir (inode 1)
2015/12/04 14:21:28.991221 fuse.DEBUG OpenDir 1  []
2015/12/04 14:21:28.991297 fuse.DEBUG Op 0x00000004        connection.go:474] -> OK
2015/12/04 14:21:28.991435 fuse.DEBUG Op 0x00000005        connection.go:395] <- ReadDir (inode 1)
2015/12/04 14:21:28.991637 fuse.DEBUG ReadDir 1  [0]
2015/12/04 14:21:28.991680 fuse.DEBUG <-- ReadDir 1  [. 1]
2015/12/04 14:21:28.991695 fuse.DEBUG <-- ReadDir 1  [.. 2]
2015/12/04 14:21:28.992525 s3.DEBUG DEBUG: Request s3/ListObjects Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET http://s3.mystorage.net/logstash?delimiter=%2F&prefix= HTTP/1.1
Host: s3.mystorage.net
User-Agent: aws-sdk-go/1.0.2 (go1.5.1; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=J8VFMLKZ8QTL75W0LZOD/20151204/us-west-2/s3/aws4_request, SignedHeaders=host;x-amz-date, Signature=9c7779be6dc9d8197e4d8f806f52680274e3a50007fc51e8df5b284ee4f326d1
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20151204T072128Z
Accept-Encoding: gzip


-----------------------------------------------------
2015/12/04 14:21:28.996166 s3.DEBUG DEBUG: Response s3/ListObjects Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 400 Bad Request
Connection: close
Content-Length: 81
Accept-Ranges: bytes
Content-Type: application/xml
Date: Fri, 04 Dec 2015 07:22:53 GMT
Server: Apache/2.4.16 (Ubuntu)


-----------------------------------------------------
2015/12/04 14:21:28.996371 s3.DEBUG DEBUG: Validate Response s3/ListObjects failed, not retrying, error InvalidArgument:
        status code: 400, request id:
2015/12/04 14:21:28.996391 s3.ERROR code= msg=400 request=

2015/12/04 14:21:28.996429 fuse.DEBUG Op 0x00000005        connection.go:476] -> Error: "InvalidArgument: \n\tstatus code: 400, request id: "
2015/12/04 14:21:28.996805 fuse.DEBUG Op 0x00000006        connection.go:395] <- ReleaseDirHandle
2015/12/04 14:21:28.997003 fuse.DEBUG ReleaseDirHandle
2015/12/04 14:21:28.997048 fuse.DEBUG Op 0x00000006        connection.go:474] -> O

2015/12/04 14:24:11.430400 s3.DEBUG DEBUG: Validate Response s3/ListObjects failed, not retrying, error InvalidArgument:
        status code: 400, request id:
2015/12/04 14:24:11.430484 s3.ERROR code= msg=400 request=

2015/12/04 14:24:11.430552 s3.DEBUG DEBUG: Validate Response s3/HeadObject failed, not retrying, error :
        status code: 400, request id:
2015/12/04 14:24:11.430586 s3.ERROR code= msg=400 request=

kahing · 2015-12-04T18:52:41Z

On the surface this seems to be a problem with ceph not supporting v4 signing and somehow aws-sdk-go is not falling back to v2. Could you open a new issue? I am trying to get an account on dreamobject to test myself but their beta is already full.

kahing · 2015-12-04T19:26:07Z

aws/aws-sdk-go#400 tracks the v2 signing issue

jindov · 2015-12-07T01:31:22Z

Thanks for your supporting

kahing · 2015-12-14T19:35:58Z

reopening until aws/aws-sdk-go#400 is resolved

hack in a copy of v2 signer into goofys and adapting it for s3. refs #46, aws/aws-sdk-go#400

kahing · 2015-12-15T01:31:05Z

@jindov I've merged a fix for this to master but don't have a ceph setup to play with. Could you re-test?

jindov · 2015-12-15T02:16:40Z

I remove and do:
go get github.com/kahing/goofys
go install github.com/kahing/goofys
this is result:
goofys --use-path-request --endpoint http://s3.mystorage.net --debug_fuse --debug_s3 -f logstash /mnt/logstore

2015/12/15 09:12:06.040946 main.INFO --use-path-request is deprecated, it's always on
2015/12/15 09:12:06.042322 s3.DEBUG DEBUG: Request s3/GetBucketLocation Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET http://s3.mystorage.net/logstash?location= HTTP/1.1
Host: s3.mystorage.net
User-Agent: aws-sdk-go/1.0.4 (go1.5.1; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=J8VFMLKZ8QTL75W0LZOD/20151215/us-west-2/s3/aws4_request, SignedHeaders=host;x-amz-date, Signature=bc550d95792f5f23973367cc4ac63ceee46cd1d4070ab2aedc21e2fbfc6a442d
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20151215T021206Z
Accept-Encoding: gzip


-----------------------------------------------------
2015/12/15 09:12:06.047501 s3.DEBUG DEBUG: Response s3/GetBucketLocation Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 400 Bad Request
Connection: close
Content-Length: 81
Accept-Ranges: bytes
Content-Type: application/xml
Date: Tue, 15 Dec 2015 02:18:14 GMT
Server: Apache/2.4.16 (Ubuntu)


-----------------------------------------------------
2015/12/15 09:12:06.047768 s3.DEBUG DEBUG: Validate Response s3/GetBucketLocation failed, not retrying, error InvalidArgument:
        status code: 400, request id:
2015/12/15 09:12:06.047867 s3.INFO Falling back to v2 signer
2015/12/15 09:12:06.048587 s3.DEBUG DEBUG: Request s3/GetBucketLocation Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET http://s3.mystorage.net/logstash?location= HTTP/1.1
Host: s3.mystorage.net
User-Agent: aws-sdk-go/1.0.4 (go1.5.1; linux; amd64)
Authorization: AWS J8VFMLKZ8QTL75W0LZOD:+RRvySSeb4KBXVPDdfCLr5GERzg=
Date: Tue, 15 Dec 2015 02:12:06 UTC
Accept-Encoding: gzip


-----------------------------------------------------
2015/12/15 09:12:06.052853 s3.DEBUG DEBUG: Response s3/GetBucketLocation Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 403 Forbidden
Content-Length: 87
Accept-Ranges: bytes
Content-Type: application/xml
Date: Tue, 15 Dec 2015 02:18:14 GMT
Server: Apache/2.4.16 (Ubuntu)


-----------------------------------------------------
2015/12/15 09:12:06.052942 s3.DEBUG DEBUG: Validate Response s3/GetBucketLocation failed, not retrying, error SignatureDoesNotMatch:
        status code: 403, request id:
2015/12/15 09:12:06.052967 s3.ERROR code= msg=403 request=

2015/12/15 09:12:06.052978 s3.ERROR code= msg=403 request=

2015/12/15 09:12:06.053005 s3.INFO Unable to detect bucket region, staying at 'us-west-2'
2015/12/15 09:12:06.057698 fuse.DEBUG Op 0x00000001        connection.go:395] <- init
2015/12/15 09:12:06.057841 fuse.DEBUG Op 0x00000001        connection.go:474] -> OK
2015/12/15 09:12:06.057882 main.INFO File system has been successfully mounted.

and when I run a ls /mnt/logstore:

ls: reading directory logstore/: Input/output error

2015/12/15 09:15:37.023847 fuse.DEBUG Op 0x0000000b        connection.go:395] <- GetInodeAttributes (inode 1)
2015/12/15 09:15:37.024219 fuse.DEBUG GetAttributes 1  []
2015/12/15 09:15:37.024260 fuse.DEBUG Op 0x0000000b        connection.go:474] -> OK
2015/12/15 09:15:37.024704 fuse.DEBUG Op 0x0000000c        connection.go:395] <- OpenDir (inode 1)
2015/12/15 09:15:37.024993 fuse.DEBUG OpenDir 1  []
2015/12/15 09:15:37.025022 fuse.DEBUG Op 0x0000000c        connection.go:474] -> OK
2015/12/15 09:15:37.025146 fuse.DEBUG Op 0x0000000d        connection.go:395] <- ReadDir (inode 1)
2015/12/15 09:15:37.025408 fuse.DEBUG ReadDir 1  [0]
2015/12/15 09:15:37.025448 fuse.DEBUG <-- ReadDir 1  [. 1]
2015/12/15 09:15:37.025463 fuse.DEBUG <-- ReadDir 1  [.. 2]
2015/12/15 09:15:37.025873 s3.DEBUG DEBUG: Request s3/ListObjects Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET http://s3.fptplay.net/logstash?delimiter=%2F&prefix= HTTP/1.1
Host: s3.fptplay.net
User-Agent: aws-sdk-go/1.0.4 (go1.5.1; linux; amd64)
Authorization: AWS J8VFMLKZ8QTL75W0LZOD:XqZemMh26y3Br45g2zUNr3HEa3Y=
Date: Tue, 15 Dec 2015 02:15:37 UTC
Accept-Encoding: gzip


-----------------------------------------------------
2015/12/15 09:15:37.030279 s3.DEBUG DEBUG: Response s3/ListObjects Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 403 Forbidden
Content-Length: 87
Accept-Ranges: bytes
Content-Type: application/xml
Date: Tue, 15 Dec 2015 02:21:45 GMT
Server: Apache/2.4.16 (Ubuntu)


-----------------------------------------------------
2015/12/15 09:15:37.030419 s3.DEBUG DEBUG: Validate Response s3/ListObjects failed, not retrying, error SignatureDoesNotMatch:
        status code: 403, request id:
2015/12/15 09:15:37.030439 s3.ERROR code= msg=403 request=

2015/12/15 09:15:37.030495 fuse.DEBUG Op 0x0000000d        connection.go:476] -> Error: "SignatureDoesNotMatch: \n\tstatus code: 403, request id: "
2015/12/15 09:15:37.030702 fuse.DEBUG Op 0x0000000e        connection.go:395] <- ReleaseDirHandle
2015/12/15 09:15:37.030902 fuse.DEBUG ReleaseDirHandle
2015/12/15 09:15:37.030928 fuse.DEBUG Op 0x0000000e        connection.go:474] -> OK

I use exactly key and secret key

Thank you

kahing · 2015-12-15T02:30:16Z

That's odd, is it possible for me to get a credential to play with?

jindov · 2015-12-15T02:37:57Z

My credential file
[root@jin ~]# cat .aws/credentials
[default]
aws_access_key_id = key
aws_secret_access_key = secret-key

kahing · 2015-12-15T02:49:57Z

Looks like you already edited the credential out. Just want to let you know that people who watch this project would have had an email notification with the original comment.

jindov · 2015-12-15T02:57:18Z

my mistake and I need to reduce risky to my setup. can u debug the above error when connecting?

kahing · 2015-12-15T02:59:39Z

Looking into it right now

goofys' v2 signer was sending the full URL as the request URI (GET http://s3-us-west-2.amazonaws.com/bucket/object instead of (GET /bucket/object) which works with AWS-S3 but not ceph-s3 ceph-s3 also doesn't seem to send content-length when size is 0 refs #46

kahing · 2015-12-15T05:10:15Z

@jindov I've pushed some changes and basic operations seem to work now

jindov · 2015-12-15T07:34:04Z

2015/12/15 14:32:08.634649 fuse.DEBUG Op 0x0000009c        connection.go:395] <- LookUpInode (parent 1, name "test")
2015/12/15 14:32:08.635003 fuse.DEBUG Ref 123 test [2]
2015/12/15 14:32:08.635050 fuse.DEBUG <-- LookUpInode 123 test []
2015/12/15 14:32:08.635138 fuse.DEBUG Op 0x0000009c        connection.go:474] -> OK
2015/12/15 14:32:08.635306 fuse.DEBUG Op 0x0000009d        connection.go:395] <- OpenFile (inode 123)
2015/12/15 14:32:08.635664 fuse.DEBUG OpenFile 123 test []
2015/12/15 14:32:08.635709 fuse.DEBUG Op 0x0000009d        connection.go:474] -> OK
2015/12/15 14:32:08.635860 fuse.DEBUG Op 0x0000009e        connection.go:395] <- SetInodeAttributes (inode 123, size 0)
2015/12/15 14:32:08.636102 fuse.DEBUG GetAttributes 123 test []
2015/12/15 14:32:08.636152 fuse.DEBUG Op 0x0000009e        connection.go:474] -> OK
2015/12/15 14:32:08.636299 fuse.DEBUG Op 0x0000009f        connection.go:395] <- FlushFile (inode 123)
2015/12/15 14:32:08.636563 fuse.DEBUG FlushFile 123 test []
2015/12/15 14:32:08.636611 fuse.DEBUG Op 0x0000009f        connection.go:474] -> OK
2015/12/15 14:32:08.636828 fuse.DEBUG Op 0x000000a0        connection.go:395] <- WriteFile (inode 123, handle 5, offset 0, 4 bytes)
2015/12/15 14:32:08.637100 fuse.DEBUG WriteFile 123 test [0 4]
2015/12/15 14:32:08.639357 fuse.DEBUG Op 0x000000a0        connection.go:474] -> OK
2015/12/15 14:32:08.639738 fuse.DEBUG Op 0x000000a1        connection.go:395] <- FlushFile (inode 123)
2015/12/15 14:32:08.639922 fuse.DEBUG FlushFile 123 test []
2015/12/15 14:32:08.641009 s3.DEBUG DEBUG: Request s3/PutObject Details:
---[ REQUEST POST-SIGN ]-----------------------------
PUT /logstash/test HTTP/1.1
Host: s3.mystorage.net
User-Agent: aws-sdk-go/1.0.4 (go1.5.1; linux; amd64)
Content-Length: 4
Authorization: AWS J8VFMLKZ8QTL75W0LZOD:waaFMyyGa4Ysq5NC+ZsBcfFXlZ8=
X-Amz-Date: Tue, 15 Dec 2015 07:32:08 +0000
X-Amz-Storage-Class: STANDARD
Accept-Encoding: gzip


-----------------------------------------------------
2015/12/15 14:32:08.722311 s3.DEBUG DEBUG: Response s3/PutObject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Accept-Ranges: bytes
Content-Type: application/xml
Date: Tue, 15 Dec 2015 07:38:17 GMT
Etag: "0bee89b07a248e27c83fc3d5951213c1"
Server: Apache/2.4.16 (Ubuntu)
Vary: Accept-Encoding


-----------------------------------------------------
2015/12/15 14:32:08.722504 fuse.DEBUG Op 0x000000a1        connection.go:474] -> OK
2015/12/15 14:32:08.722688 fuse.DEBUG Op 0x000000a2        connection.go:395] <- ReleaseFileHandle
2015/12/15 14:32:08.723057 fuse.DEBUG ReleaseFileHandle test
2015/12/15 14:32:08.723100 fuse.DEBUG Op 0x000000a2        connection.go:474] -> OK

It seem work ok, I will continue testing and report
Thank for your supporting

kahing · 2015-12-15T07:36:59Z

Thanks for assisting in testing! Please file a new issue if you encounter other problems. Note that you may want to expire your credential since 22+ people have it in their mailbox.

mSys-mislav · 2016-03-23T09:45:56Z

Is there an option signature v2 available? I'm not able to connect with goofys to my S3 share, however I can with s3fs only with using option sigv2.

Is there any flag to add to goofys?

kahing closed this as completed Dec 1, 2015

kahing reopened this Dec 14, 2015

kahing changed the title ~~Support custom endpoint/host~~ fallback to v2 signing for non AWS endpoints Dec 14, 2015

kahing added a commit that referenced this issue Dec 14, 2015

v2 signing

a4d97cf

hack in a copy of v2 signer into goofys and adapting it for s3. refs #46, aws/aws-sdk-go#400

kahing added a commit that referenced this issue Dec 15, 2015

v2 signing

56188a1

hack in a copy of v2 signer into goofys and adapting it for s3. refs #46, aws/aws-sdk-go#400

kahing closed this as completed Dec 15, 2015

spmfox mentioned this issue Jul 21, 2021

S3 storage - renaming not working drakkan/sftpgo#483

Closed

dannycjones mentioned this issue May 31, 2022

How to force goofys to authorize bucket with signv2 #701

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fallback to v2 signing for non AWS endpoints #46

fallback to v2 signing for non AWS endpoints #46

jindov commented Nov 27, 2015

gaul commented Nov 27, 2015

kahing commented Dec 1, 2015

jindov commented Dec 4, 2015

kahing commented Dec 4, 2015

kahing commented Dec 4, 2015

jindov commented Dec 4, 2015

kahing commented Dec 4, 2015

jindov commented Dec 4, 2015

kahing commented Dec 4, 2015

kahing commented Dec 4, 2015

jindov commented Dec 7, 2015

kahing commented Dec 14, 2015

kahing commented Dec 15, 2015

jindov commented Dec 15, 2015

kahing commented Dec 15, 2015

jindov commented Dec 15, 2015

kahing commented Dec 15, 2015

jindov commented Dec 15, 2015

kahing commented Dec 15, 2015

kahing commented Dec 15, 2015

jindov commented Dec 15, 2015

kahing commented Dec 15, 2015

mSys-mislav commented Mar 23, 2016

fallback to v2 signing for non AWS endpoints #46

fallback to v2 signing for non AWS endpoints #46

Comments

jindov commented Nov 27, 2015

gaul commented Nov 27, 2015

kahing commented Dec 1, 2015

jindov commented Dec 4, 2015

kahing commented Dec 4, 2015

kahing commented Dec 4, 2015

jindov commented Dec 4, 2015

kahing commented Dec 4, 2015

jindov commented Dec 4, 2015

kahing commented Dec 4, 2015

kahing commented Dec 4, 2015

jindov commented Dec 7, 2015

kahing commented Dec 14, 2015

kahing commented Dec 15, 2015

jindov commented Dec 15, 2015

kahing commented Dec 15, 2015

jindov commented Dec 15, 2015

kahing commented Dec 15, 2015

jindov commented Dec 15, 2015

kahing commented Dec 15, 2015

kahing commented Dec 15, 2015

jindov commented Dec 15, 2015

kahing commented Dec 15, 2015

mSys-mislav commented Mar 23, 2016