-
Notifications
You must be signed in to change notification settings - Fork 14
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
s3 nippy example doesn't work #6
Comments
Again, thanks for taking the time to report this, and to point out the low VS high level api of nippy. Have you tried something along these lines as the :read fn? #(with-open [dis (DataInputStream. %)
bos (ByteArrayOutputStream. (.available dis))]
(io/copy dis bos)
(nippy/thaw (.toByteArray bos))) If that (or something similar) works for you let me know, and I will update the example asap. As to your other 404 problem, I'm not sure I can help here as Kind regards |
|
OK, so it turns out that I wasn't calling Line 131 in afec9fd
That could be the source of your issue, if duratom wasn't able to initialise the bucket. Can you try with duratom 0.3.9 which I've just cut fixing this? Thanks in advance...
|
I've just cut 0.4.0 which improves streaming the bytes from the s3 bucket (via |
Hey, I tried it out, and I'm still getting it error out with a 404 not found instead of creating the file initially (using s3 and nippy with the instructions in the readme) |
Are you able to create buckets manually (e.g. using |
The bucket already exists, it's the object creation where it's falling down
(and if I manually create the object then it isn't the right format for
nippy so will fall over when it tries to initialise the duratom from it).
As a workaround I've been running the following code for edn, haven't yet
written a version for nippy - hoping the library will deal with it for me:
```
(defn get-or-create-s3 [init-schema]
(log/debug "Attempting to get or set the edn file")
(try
(s3/get-object :bucket-name "my-existing-bucket" :key
"my-duratom-db.edn")
(log/debug "Succesfully found the bucket")
(catch Exception e
(log/error "Couldn't find the bucket, creating it myself")
(let [tmp (File/createTempFile "tmp-edn" ".edn")]
(spit tmp init-schema)
(s3/put-object :bucket-name "my-existing-bucket" :key
"my-duratom-db.edn") :file tmp)))))
```
Just to clarify - can you on your machine go from empty bucket to
initialising a duratom and reloading it using the nippy + s3 example from
the readme? Is this untested, or is it just me having this problem?
…On Thu, Jun 14, 2018 at 1:22 PM Dimitrios Piliouras < ***@***.***> wrote:
Are you able to create buckets manually (e.g. using aws/create-bucket)
outside of duratom? I'm not sure if you've looked in utils.clj, but the
three S3 relevant functions are all wrappers around amazonica (i.e.
aws/create-bucket, aws/aws/does-bucket-exist, aws/delete-object &
aws/put-object). I suggest that you try these functions on your REPL
without involving duratom. Once you establish that they work as expected,
I'd appreciate some code snippets showcasing each of those calls, and
hopefully I will be able to figure out what I 'm doing wrong. Thanks in
advance...
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#6 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AA3140imBcSV0G37gKDYAg1vYs0GLVndks5t8mOqgaJpZM4TLi83>
.
|
The S3 backend is the only backend that is completely untested - I believe I mentioned that in some previous comment. Moreover, you seem to be the first person even remotely trying it out it! Now, in all fairness, the only difference between the various backends are really the different underlying helpers, so if something works on one backend but not on another, it can really only be some something relating to these backend-specific helpers. Ok, so you say the bucket exists. Can you confirm that In your code snippet, I don't see any credentials anywhere, so you must be using some of that |
Revisiting this ticket because I was finally able to create an AWS account and test this myself. The good news is that the 5 S3-related helpers work as expected - observe the following: (def creds {:access-key "..." ;; <= replace with yours
:secret-key "..." ;; <= replace with yours
:endpoint "eu-west-1"}) ;; <= replace with yours
(def dummy-value (pr-str {:a 1 :b 2}))
;; check a bucket I know exists
(bucket-exists? creds "jimpil-test") ;; => true
;; create a brand new one (with public access!)
(create-s3-bucket creds "jimpil-test-delete-me") ;; => {:name "jimpil-test-delete-me"}
(store-value-to-s3 creds "jimpil-test" "dummy.edn" {} dummy-value) ;; => a big response
(get-value-from-s3 creds "jimpil-test" "dummy.edn" {} (partial read-edn-object {})) ;; => {:a 1, :b 2}
(delete-object-from-s3 creds "jimpil-test" "dummy.edn") ;; => nil (but succeeded) The credentials I used were for a user belonging to a user-group with full S3 access, and as you can see, he can still access a bucket which has full BlockPublicAccess enabled. In any case, that's how these functions are supposed to be called - with the credentials (which you pass when you create the duratom) as the first argument. Now, the nippy part of the issue, is somewhat separate, and can be addressed/tweaked using the In any case, I'm (finally) closing this 😌 |
Nippy has low level and high level apis, as such the data saved by freeze can't actually be read by thaw-from-in! because freeze/thaw is a high level api supporting encryption, compression and including the nippy file header, where thaw-from-in! and freeze-from-in! are low level and don't support those, so when it is trying to read the type id and instead finds the first letter of the NPY header, it complains that it doesn't recognise the file.
I also have an issue in that I get a 404 from s3 if the key doesn't exist yet and I don't know what to put in it to manually create a blank nippy file as a starting point.
The text was updated successfully, but these errors were encountered: