Working with API

[r4sas]

Hi again.
I try to write CLI for PrivateBin, but I stucked on checking paste.
My sources: here (I continue work with it, in trunk u can see other version, but it works via proxy).
I really dont know why, but I cant open sended paste, site always asks password.
Also in wiki page this isn't clear described how to correctly encode data... Can you help me?

[elrido]

I'll be glad to try to help. The format of the SJCL JSON isn't described is this is on the JS side simply the full output of the SJCL library, so there never was a need for that (yet). I guess the problem is that you are trying to replicate the format and something does not match up.

Could you provide me with a sample paste (the JSON file stored on the servers filesystem or database) that you created using your CLI tool plus the randomly generated key (and password if one is set), so I can replicate the issue and debug it in a browsers console? Of course please generate a new one using some sample text so that there is no sensitive information stored in it.

Edit: typo

[r4sas]

@elrido thanks!
One for example:

{"meta":{"expire_date":1487672299,"formatter":"plaintext","postdate":1487585899,"salt":"690b9f8503b31725ca41d46c86bd54635db7000558bff82ff4013eefd62d47f83f127522d6e2cafc8c71608406f04f1f1643511bba5236e69d3867ee80fbc4f68393f35602716f6036fe6b7c813f2230737c69de2aa6e3fa064516de895448e985273125fc108a0090162fe63629ce72e3644de3c473b1a48458503a994cf0810aeedde001e5716a370f435832a0432c4a8e0d322d243d726e20e2d9a33ce71b8abe973a4742a4e482513785e70f667efd2bcbb421ebd713e462d1cc742ed77af3ea22ac43e22c114355e2d7906385b869e3710ce5056ec78455007f8285c6cc4fe79d5b8d2e674804094beb01de13c561d7233f5e90d9a22e2819e36c2755e4"},"data":"{\"ks\": 256, \"cipher\": \"aes\", \"mode\": \"gcm\", \"v\": 1, \"adata\": \"\", \"ct\": \"9CxSfv67S7DWNEW1GWuUHuSie05B\", \"salt\": \"P5mduOChbeE=\", \"ts\": 128, \"iter\": 10000, \"iv\": \"czpKzdgtlokZ48ssTRD8hQ==\"}"}

Passphrase: 76dACY4Q44tnNkQbyDFBuegHuQ/kHpqYCQsad8BAQeQ=
It contain string Test!

Yes, It can be replicated.
In browser console debug I see only that:

TypeError: this.on is not a function
...unction(a,c){return arguments.length>0?this.on(b,null,a,c):this.trigger(b)}}),r....
--------------------------------------------^

[r4sas]

@sviceman found that problem in rawde/inflate modules, used by PrivateBin.
For example, that issue: dankogai/js-deflate#15
We wrote fully CLI, and deleted this.compress and this.decompress from privatebin.js code - all started working...

Please watch on Stuk/jszip#102. Maybe you will find an alternative here, which is compatible with standard zlib and can work with PrivateBin.

[cryptomilk]

Now I know why client doesn't work. However if I remote this.decompress and this.compress the site doesn't work anymore, hmm.

[r4sas]

@cryptomilk because you must recalculate integrity hash after changes in js files here and here (I'm used that: https://github.com/markcoker/HTML-Script-Integrity)

[cryptomilk]

+++ privatebin.js       2017-02-21 14:05:51.736505846 +0100
@@ -567,21 +567,25 @@
          */
         decipher: function(key, password, data)
         {
-            if (data !== undefined)
-            {
-                try
-                {
-                    return this.decompress(sjcl.decrypt(key, data));
-                }
-                catch(err)
-                {
-                    try
-                    {
-                        return this.decompress(sjcl.decrypt(key + sjcl.codec.hex.fromBits(sjcl.hash.sha256.hash(password)), data));
+            var plaintext = '';
+            var compressed = '';
+
+            if (data !== undefined) {
+                try {
+                    compressed = sjcl.decrypt(key, data);
+                } catch (err1) {
+                    try {
+                        compressed = sjcl.decrypt(key + sjcl.codec.hex.fromBits(sjcl.hash.sha256.hash(password)), data);
+                    } catch (err2) {
+                        return '';
                     }
-                    catch(e)
-                    {}
                 }
+
+                plaintext = this.decompress(compressed);
+                if (plaintext == '') {
+                    return Base64.fromBase64(compressed);
+                }
+                return plaintext;
             }
             return '';
         }```

This is what I have now. However the Base64.fromBase64() seems to have issues with UTF-8.

[r4sas]

@cryptomilk please format your message correctly (use "```patch")

diff --git a/js/privatebin.js b/js/privatebin.js
index 4fd0e99..a3eb2ed 100644
--- a/js/privatebin.js
+++ b/js/privatebin.js
@@ -550,9 +550,9 @@ jQuery.PrivateBin = function($, sjcl, Base64, RawDeflate) {
             var options = {mode: 'gcm', ks: 256, ts: 128};
             if ((password || '').trim().length === 0)
             {
-                return sjcl.encrypt(key, this.compress(message), options);
+                return sjcl.encrypt(key, message, options);
             }
-            return sjcl.encrypt(key + sjcl.codec.hex.fromBits(sjcl.hash.sha256.hash(password)), this.compress(message), options);
+            return sjcl.encrypt(key + sjcl.codec.hex.fromBits(sjcl.hash.sha256.hash(password)), message, options);
         },

         /**
@@ -571,13 +571,13 @@ jQuery.PrivateBin = function($, sjcl, Base64, RawDeflate) {
             {
                 try
                 {
-                    return this.decompress(sjcl.decrypt(key, data));
+                    return sjcl.decrypt(key, data);
                 }
                 catch(err)
                 {
                     try
                     {
-                        return this.decompress(sjcl.decrypt(key + sjcl.codec.hex.fromBits(sjcl.hash.sha256.hash(password)), data));
+                        return sjcl.decrypt(key + sjcl.codec.hex.fromBits(sjcl.hash.sha256.hash(password)), data);
                     }
                     catch(e)
                     {}
diff --git a/tpl/bootstrap.php b/tpl/bootstrap.php
index 4bf3ca1..df290d6 100644
--- a/tpl/bootstrap.php
+++ b/tpl/bootstrap.php
@@ -69,7 +69,7 @@ if ($MARKDOWN):
 <?php
 endif;
 ?>
-               <script type="text/javascript" src="js/privatebin.js?<?php echo rawurlencode($VERSION); ?>" integrity="sha512-vYYJYraxQNOf41XtehLBU2JbIQ2Uffe+n8TjHyWkpqoZdZX4aL5zyABrUNvRUP02+AxoRsmNJkpvIbmeQqcIXg==" crossorigin="anonymous"></script>
+               <script type="text/javascript" src="js/privatebin.js?<?php echo rawurlencode($VERSION); ?>" integrity="sha512-ilQ2VRPbgSGc819kvmE61FH8o3WAtjHwzOr41BFBxQauDCypz9Xz9tyvDjJVsGnGzQQ7n1sSIowmSvkyi99kqQ==" crossorigin="anonymous"></script>
                <!--[if lt IE 10]>
                <style type="text/css">body {padding-left:60px;padding-right:60px;} #ienotice {display:block;} #oldienotice {display:block;}</style>
                <![endif]-->
diff --git a/tpl/page.php b/tpl/page.php
index 4ae0b6a..db877e9 100644
--- a/tpl/page.php
+++ b/tpl/page.php
@@ -47,7 +47,7 @@ if ($MARKDOWN):
 <?php
 endif;
 ?>
-               <script type="text/javascript" src="js/privatebin.js?<?php echo rawurlencode($VERSION); ?>" integrity="sha512-vYYJYraxQNOf41XtehLBU2JbIQ2Uffe+n8TjHyWkpqoZdZX4aL5zyABrUNvRUP02+AxoRsmNJkpvIbmeQqcIXg==" crossorigin="anonymous"></script>
+               <script type="text/javascript" src="js/privatebin.js?<?php echo rawurlencode($VERSION); ?>" integrity="sha512-ilQ2VRPbgSGc819kvmE61FH8o3WAtjHwzOr41BFBxQauDCypz9Xz9tyvDjJVsGnGzQQ7n1sSIowmSvkyi99kqQ==" crossorigin="anonymous"></script>
                <!--[if lt IE 10]>
                <style type="text/css">body {padding-left:60px;padding-right:60px;} #ienotice {display:block;} #oldienotice {display:block;}</style>
                <![endif]-->

[cryptomilk]

My version still works with compressed data blobs stored in Privatebin.

[sviceman]

When you select library for compressing maybe need to check if compatible with RFC? I try to use zlib.compress, zlib.compressobj, zlib deflate (remove first 2 byte and last 4 bytes)... result without success.

[r4sas]

@cryptomilk why you doing Base64.fromBase64() from compressed message? You not** need it.

upd; not**

[cryptomilk]

Ah, true. We can avoid that. However compression would be nice.

[r4sas]

I see it like that:

@@ -567,21 +567,31 @@ jQuery.PrivateBin = function($, sjcl, Base64, RawDeflate) {
          */
         decipher: function(key, password, data)
         {
+            var plaintext = '';
+            var compressed = '';
+
             if (data !== undefined)
             {
                 try
                 {
-                    return this.decompress(sjcl.decrypt(key, data));
+                    compressed = sjcl.decrypt(key, data);
                 }
-                catch(err)
+                catch(err1)
                 {
                     try
                     {
-                        return this.decompress(sjcl.decrypt(key + sjcl.codec.hex.fromBits(sjcl.hash.sha256.hash(password)), data));
+                        compressed = sjcl.decrypt(key + sjcl.codec.hex.fromBits(sjcl.hash.sha256.hash(password)), data);
                     }
-                    catch(e)
-                    {}
+                    catch(err2)
+                    {
+                        return '';
+                    }
+                }
+                plaintext = this.decompress(compressed);
+                if (plaintext == '') {
+                    return compressed;
                 }
+                return plaintext;
             }
             return '';
         }

[cryptomilk]

The compress function does base64encode after compressing it. If you want compression working when pasting in the web interface you need it or this.decompress will fail. However compression should be fixed.

[cryptomilk]

this.decompress() will not work if it is not b64 encoded ...

[r4sas]

we nowhere here decoding message with base64

-                    return this.decompress(sjcl.decrypt(key, data));

If here be string like

-                    return this.decompress(Base64.fromBase64(sjcl.decrypt(key, data)));

then I will agree with you.

[cryptomilk]

Ah you're right ... :)

[elrido]

Thank you guys so much for investigating and sharing your findings!

So there is still the chance (still to be checked) if SJCL does return standard conforming AES content, but the problem was the non-default deflate compression of the libraries used for that all along.

This and other issues (#74) with the original ZeroBin encryption format really proove that we need to implement a new format that is generic and can be de- and encrypted with standard tools in any language. Using the already existing version field we can do this in a way that allows the PrivateBin web-client to still open existing old pastes and the to be designed new format.

As an intermediate solution until we can provide this, I have implemented a minimalistic CLI client for the "classic" format using node JS to allow the use of the current libraries. You find it over at:
https://github.com/PrivateBin/PrivateBin-Cli

[r4sas]

@elrido by the way, I found that API not works correctly.
For example, I try delete existing paste:

# python pbincli.py delete -d -p 8461f33905cd38e0 -t 35ad9317dd1a57d5092f78c907ee383b0ea404c33ba20b694fe5e8fc61030073
PasteID:        8461f33905cd38e0
Token:          35ad9317dd1a57d5092f78c907ee383b0ea404c33ba20b694fe5e8fc61030073
Response:

PBinCLI Error: No JSON object could be decoded

That's POST with {"pasteid":p, "deletetoken":t} data in it.
Server response 200 without any text. But, when I use delete on BurnAfterReading pastes, I create POST with data {"pasteid":p, "deletetoken":"burnafterreading"} and it works (response status 0 with pasteid).

What about SJCL: it works correcly, I can encode and decode pastes on website and CLI, but only if I disable this.decompress and this.compress on server.

[cryptomilk]

@elrido It isn't bad what SJCL is doing, it is just how it is doing it! I think @r4sas and I struggled understanding it and we needed to debug the current Javascript to understand what it is doing.

We should design a new way to do the Crypto together.

[cryptomilk]

Paste Encryption

Just a suggestion

Data passed in

The following data is what we pass in

paste_password: UTF-8 string
paste_plaintext: UTF-8 text

Process data

Start with generating a one time password.

paste_otp = random(32) # 32 bytes
paste_passphrase = paste_otp

If paste_password is set:

paste_passphrase = paste_otp + bytes(paste_password)

Processing of the paste_plaintext:

If we want compression:

paste_datablob = zlib.compress(paste_data)
paste_compressed = 1

else

paste_datablob = paste_plaintext
paste_compressed = 0

Key generation for encryption (PBKDF2)

kdf_salt = random(16) - 16 bytes
kdf_iterations = 256000
kdf_keysize = 256 # bits of resulting kdf_key

kdf_key = PBKDF2HMAC(SHA256, kdf_keysize, kdf_salt, paste_passphrase)

Encryption

cipher_algo = "aes"
cipher_mode = "gcm"

cipher_iv = random(16) # 128 bit
cipher_associated_data = ""
cipher_tag_size = 16

cipher = Cipher(AES(kdf_key), GCM(iv, cipher_associated_data), paste_datablob)

cipher_text = cipher.text
cipher_tag = cipher.tag


json_data = {"v": 2,
             "kdf": "pbkdf2",
             "hash": "sha256",
             "salt": base64(kdf_salt),
             "iter": kdf_iterations,
             "ks": kdf_keysize,
             "cipher": cipher_algo,
             "mode": cipher_mode,
             "iv": base64(cipher_iv),
             "adata": base64(cipher_associated_data),
             "ct": base64(cipher_text),
             "ts": cipher_tag_size,
             "tag": base64(cipher_tag),
             "compressed": paste_compressed}

URL

url = "<paste_url>/?<paste_id>#<base64(paste_otp)>

[rugk]

Maybe you want to post this in the other issue about Channing the crypto.

[elrido]

@r4sas Thanks for pointing out the incorrect response. When I tested with your client, the paste was deleted as requested, but the response was always empty. This was addressed in db307c3.

One note on your client: I think that on line 52 of actions.py it should be if args.file and cipherfile and cipherfilename: as cipherfile might not be set when just sending text via argument. Would you consider moving your repo under the PrivateBin project?

[r4sas]

@elrido Thanks! I checked it, now works.

About chipherfile: it added only when we add file argument. So here no reason to add one more check. When we sending only text via comment variable, we mustn't add attachmentfile and attachmentfilename into data of POST request.

And about moving repo: I think, firstly I must work out all possible variants of incorrect using.

Upd:
Oh... I broke sending comment only... I'll fix it. Yes, your variant is one of possible methods to fix it.
Upd2: fixed.

[r4sas]

@elrido is here a way to add to API variable or function, which will check size of sended data to server to avoid harmful network load with big pastes to privatebin instances with low limits (512 Kb for example)?
Or for example request, that will get that limit from server to check with size of data that we will send.

API POST request to / with data, like: {'get':'sizelimit'}
RESPONSE {'status':0,'sizelimit':512000} (sizelimit in bytes)

[rugk]

In #95 we discussed/are discussing (feel free to leave a comment there) whether such a thing would be good. The issue was about showing it in the UI, but an API would be basically the same…

[r4sas]

I created PR to replace current raw[de/in]flate libraries with pako: #193
Now I will continue developing of PBinCLI using zlib compression.

[r4sas]

Next question:
How I must work with files, that will be sended to instance?
What mean's variable e.target.result here? And what it contain?
Does it like that?

[elrido]

The JS FileReader object would return you the file contents in the result property of the event target (yes, that means the file gets loaded completely into RAM), so if I understand the python script correctly, you did indeed do the same.

[elrido]

This stackoverflow.com answer provides a lot of background on the differences of the "raw" deflate, zlib, gzip and zip formats. While the compression algo is always the same, zlib and gzip wrap these differently, using different checksum algorithms.

[otaku]

@elrido I found that I could decompress / compress the text with rawdeflate / rawinflate in both Python and Go. You have to however note that rawdeflate / rawinflate converts the output to a string. The pako library supports output to a string as well so you can switch to that for performance gains without losing compatibility: https://jsfiddle.net/548Lcqyw/

After decoding the base64 string, you must convert the string into a byte array. Once you decompress that byte array everything works. To compress, you do the reverse, compress into a byte array and turn each byte into a character, before you base64 encode that string.

[fterrag]

Is compression required when using the PrivateBin API?

[otaku]

The frontend and other tools will only display the data in the following format: btou(rawinflate(Buffer.from(data, 'base64')))

The data should be sent to Privatebin API in the following format: Buffer.from(rawdeflate(utob(message))).toString('base64')

Do note that rawdeflate and rawinflate must output a string.
Javascript: String.prototype.charCodeAt() / String.fromCharCode()

@fterrag compression is required if you want compatibility with other tools / frontend. Technically speaking, as it is currently, it just stores the data parameter that's provided. It'd be nice to be able to specify the format however.

[fterrag]

@otaku thank you for clarifying. I appreciate the help.

[elrido]

API and encryption format pages in the wiki got updated.

[r4sas]

Little question about workflow:
How PBin acts when paste sent with both burnafterreading and opendiscussion flags?
Must I just reset opendiscussion to zero before sending or leave it as is?

[elrido]

In the master logic it should get rejected as invalid by the server, so enabling one of these two should disable the other:
https://github.com/PrivateBin/PrivateBin/blob/master/lib/Model/Paste.php#L242

This is harsh, but should prevent unintended behaviour. If it's rejected you instantly know that you triggered a corner case, while having a burn after reading paste that invites to discussions only to be rejected if you comment, because the paste is already deleted, is not. Better to fail early.

[r4sas]

Ok, will do same.