Plugin encrypt #385

dbw9580 · 2020-03-30T17:19:25Z

This plugin adds support for password-protected galleries. Images are encrypted using a password during the build, and then decrypted in browser when they are accessed by viewers if they can provide the correct password.

Summaries:

Pure Javascript implementaion thanks to Web Crypto, no server application or database is needed;
Images are encrypted with AES-128-GCM, keys are generated using PBKDF2 with SHA1;
Support incremental builds;
decrypt.js is tested to work with all 3 themes bundled with Sigal, on Chrome and Firefox;

dbw9580 · 2020-03-30T17:29:37Z

Demo: https://dbw9580.github.io/sigal-encrypt-demo/
The password is password.

saimn · 2020-04-01T01:29:46Z

Looks cool, thanks!
The code seems good, and Ihave not much to say about the encryption part, maybe just a few questions.
I wonder if we could simplify a bit your implementation by adding some signals in sigal's code, for instance when writing an image, which could be used to write the encrypted version without having to check if it was already encrypted or not. It would also be useful to have a way to inject the HTML code in the header when writing HTML files, I think templates have an extra_head tag (or similar) so if there was a signal before writing HTML fies which you could use to inject your code it would also simplify things I guess.

sigal/plugins/encrypt/encrypt.py

tests/sample/sigal.conf.py

saimn · 2020-04-01T01:41:44Z

Do you think you could add some tests, building the gallery with the example settings and checking that images are indeed encrypted?

dbw9580 · 2020-04-01T09:10:17Z

simplify a bit your implementation by adding some signals in sigal's code

Definitely a good idea. I'll work on that.

add some tests, building the gallery with the example settings and checking that images are indeed encrypted

So I was trying to create some tests, but not sure about how fine granularity it is suitable for such a plugin. So basically what the test should look like is first build the sample gallery, encrypt several pictures and decrypt them in-place to see if decryption works out with no error, right? As for the javascript parts of code, I am not sure how to test them properly.

setup.cfg

saimn · 2020-04-02T02:27:40Z

So I was trying to create some tests, but not sure about how fine granularity it is suitable for such a plugin. So basically what the test should look like is first build the sample gallery, encrypt several pictures and decrypt them in-place to see if decryption works out with no error, right?

Yes, something like this. Testing that some images are indeed encrypted, and maybe that they can be descrypted with the Python decrypt code that you included ?
There is an example of how to build the test gallery and modify the settings here:

sigal/tests/test_plugins.py

Lines 9 to 19 in 0d8a3c8

    
           def test_plugins(settings, tmpdir, disconnect_signals): 
        
               settings['destination'] = str(tmpdir) 
        
               if "sigal.plugins.nomedia" not in settings["plugins"]: 
        
                   settings['plugins'] += ["sigal.plugins.nomedia"] 
        
               if "sigal.plugins.media_page" not in settings["plugins"]: 
        
                   settings['plugins'] += ["sigal.plugins.media_page"] 
        
               init_plugins(settings) 
        
               gal = Gallery(settings) 
        
               gal.build()

As for the javascript parts of code, I am not sure how to test them properly.

Yeah that's doable but it's a lot of work so I never went into this.

codecov · 2020-04-04T18:14:16Z

Codecov Report

Merging #385 into master will decrease coverage by 1.20%.
The diff coverage is 78.07%.

@@            Coverage Diff             @@
##           master     #385      +/-   ##
==========================================
- Coverage   87.49%   86.28%   -1.21%     
==========================================
  Files          19       22       +3     
  Lines        1503     1750     +247     
==========================================
+ Hits         1315     1510     +195     
- Misses        188      240      +52

Impacted Files	Coverage Δ
sigal/plugins/zip_gallery.py	`90.24% <ø> (ø)`
sigal/plugins/encrypt/endec.py	`70.00% <70.00%> (ø)`
sigal/plugins/encrypt/encrypt.py	`80.24% <80.24%> (ø)`
sigal/plugins/encrypt/__init__.py	`100.00% <100.00%> (ø)`
sigal/signals.py	`100.00% <100.00%> (ø)`
sigal/writer.py	`83.87% <100.00%> (+0.82%)`	⬆️
sigal/image.py	`89.81% <0.00%> (-2.46%)`	⬇️
... and 1 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 972f0f1...a3b0876. Read the comment docs.

dbw9580 · 2020-04-06T16:36:18Z

I rewrote most of the frontend code with Service Worker. The decryption process is now (almost) transparent to the gallery app and the user. Image srcs are no longer replaced with blob: urls, which means many browser extensions that rely on these links to work (e.g. image bulk downloaders, thumbnail previewers) can now function properly. Also original images can be opened in a separate tab without showing an error (which was impossible with the previous approach that based on hijacking image load events).

Speaking of original images, I find that they are handled differently than the resized ones and the thumbnails, in terms of how they end up where they belong in the output directory.

They are copied here:

sigal/sigal/gallery.py

Line 120 in 0d8a3c8

copy(self.src_path, big_path, symlink=s['orig_link'],

while the resized and thumbnails are directly saved from PIL here:

sigal/sigal/image.py

Line 144 in 0d8a3c8

save_image(img, outname, outformat, options=options, autoconvert=True)

and here:

sigal/sigal/image.py

Line 163 in 0d8a3c8

save_image(img, outname, outformat, options=options, autoconvert=True)

This means if I were to use signals to handle the encryption, I would need one signal for the original files, and another for the resized and thumbnails. Which seems a little overkill to me.

Your thoughts?

dbw9580 · 2020-04-10T17:26:30Z

@saimn I've added the test. I think this PR is ready for review.

… setup until window load

saimn

Looks very good, thanks for the hard work in this !
I will try to review more carefully in the coming days.

sigal/writer.py

tests/test_encrypt.py

saimn · 2020-04-14T02:48:45Z

sigal/plugins/encrypt/encrypt.py

+  ``kdf_iters`` defaults to 10000. Do not specify them in the config file unless 
+  you have good reasons to do so.
+- ``encrypt_symlinked_originals``: Force encrypting original images even if they 
+  are symlinked. If you don't know what it means, leave it to ``False``.


I think this should be True by default, and we could maybe even remove this setting (and send a warning that orig_link should not be used in this case). When using this plugin clearly the goal is to protect privacy by encrypting all images including the original ones.

Yes, but we need to also make sure people don't get their original pictures overwritten if they accidentally have both keep_orig and orig_link set to True, and have not made a backup. I think it's better we abort the encryption and send a warning, to make sure the user understands the situation, then it's up to them to decide. It's trivial to restart an aborted gallery build, while to recover the originals by decrypting them is not.

Agreed! So I think you can remove this setting ?

Remove it from the doc or the code?

Remove from the code, as you said above we should abort if keep_orig and orig_link are True, so I don't think we should keep this setting to force encryption. People should just choose between orig_link and using this plugin.

Good point. Changed in a3b0876.

saimn

More comments, mostly minor ones but I think the error handling in the Python code could be improved. The Abort exception used everywhere makes it a bit difficult to follow the error cases. You should use standard exceptions when possible (e.g. missing or invalid parameter), and reserve use of an exception specific to the plugin for problems with the encryption. And in particular I think the plugin should always fail the build if something could prevent encyption of the files.

sigal/plugins/encrypt/encrypt.py

sigal/plugins/encrypt/endec.py

dbw9580 · 2020-04-14T08:56:44Z

It occurs to me that the zip_gallery option will be meaningless if this plugin is enabled. The zip archive will contain encrypted images, and the viewers will not be able to decrypt them if they download the archive. We should add a note to the documentation about this.

dbw9580 · 2020-04-18T03:07:22Z

Remove from the doc? 2020年4月18日 10:23, 10:23，在 Simon Conseil <notifications@github.com> 已写: @saimn commented on this pull request.

…

________________________________ In sigal/plugins/encrypt/encrypt.py<http://encrypt.py>: + 'kdf_iters': 10000, + 'encrypt_symlinked_originals': False + } + +- ``password``: The password used to encrypt the images on gallery build, + and decrypt them when viewers access the gallery. No default value. You must + specify a password. +- ``ask_password``: Whether or not viewers are asked for the password to view + the gallery. If set to ``False``, the password will be present in the HTML files + so the images are decrypted automatically. Defaults to ``False``. +- ``gcm_tag``, ``kdf_salt``, ``kdf_iters``: Cryptographic parameters used when + encrypting the files. ``gcm_tag``, ``kdf_salt`` are meant to be randomly generated, + ``kdf_iters`` defaults to 10000. Do not specify them in the config file unless + you have good reasons to do so. +- ``encrypt_symlinked_originals``: Force encrypting original images even if they + are symlinked. If you don't know what it means, leave it to ``False``. Agreed! So I think you can remove this setting ? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

saimn

Looks very good, just a small question.

saimn · 2020-04-20T02:29:46Z

sigal/plugins/encrypt/encrypt.py

+    theme_path = os.path.join(settings["destination"], 'static')
+    copy(os.path.join(ASSETS_PATH, "decrypt.js"), theme_path, symlink=False, rellink=False)
+    copy(os.path.join(ASSETS_PATH, "keycheck.txt"), theme_path, symlink=False, rellink=False)
+    copy(os.path.join(ASSETS_PATH, "sw.js"), settings["destination"], symlink=False, rellink=False)


Why sw.js is not in the static directory with the other static files ?

As per Service Worker specification, the worker script has no access to resources outside its directory tree. So putting sw.js in static will break it.

Ok, thanks for the info!

saimn · 2020-04-20T13:31:07Z

Great job, thanks @dbw9580 !

Plugin encrypt

dbw9580 added 2 commits Mar 30, 2020

Add plugin encrypt

87f2970

Add name in AUTHORS

654cedb

Add plugin example settings in config template

2b8451b

saimn reviewed Apr 1, 2020

View changes

sigal/plugins/encrypt/encrypt.py Show resolved Hide resolved

sigal/plugins/encrypt/encrypt.py Outdated Show resolved Hide resolved

tests/sample/sigal.conf.py Outdated Show resolved Hide resolved

dbw9580 added 3 commits Apr 1, 2020

Add signal to inject JS files via Jinja templates

015a48a

Hardcode all JS files to copy

181d0bb

Prevent empty passwords

9ebab43

dbw9580 reviewed Apr 1, 2020

View changes

setup.cfg Outdated Show resolved Hide resolved

Rewrite with service worker

a7332fc

dbw9580 added 4 commits Apr 7, 2020

Handle service worker hibernation and re-initialization

50938c3

Small fixes

0680fb9

Add test

28f187f

Fix keycheck path

052ad58

Fix password prompt disappearing instantly by delaying service worker…

228a36b

… setup until window load

saimn reviewed Apr 13, 2020

View changes

sigal/writer.py Outdated Show resolved Hide resolved

tests/test_encrypt.py Outdated Show resolved Hide resolved

dbw9580 added 2 commits Apr 13, 2020

Fix writer.py and test_encrypt.py

2c4fdd6

Fix line endings

f80dff1

saimn reviewed Apr 14, 2020

View changes

dbw9580 added 2 commits Apr 14, 2020

Make kdf_gen_key accept str; minor fixes

6c242b3

Improve error handling

25f054b

Update doc

8185354

dbw9580 force-pushed the plugin_encrypt branch from 29b370f to 8185354 Compare Apr 14, 2020

Avoid encrypting twice in case 'use_orig' is used

d0db5c1

Remove encrypt_symlinked_originals option

a3b0876

saimn reviewed Apr 20, 2020

View changes

saimn merged commit 0ba9b10 into saimn:master Apr 20, 2020
0 of 2 checks passed

saimn added this to the 2.1 milestone Apr 20, 2020

kontza pushed a commit to kontza/sigal that referenced this issue Aug 28, 2020

Merge pull request saimn#385 from dbw9580/plugin_encrypt

41fde2e

Plugin encrypt

Plugin encrypt #385

Plugin encrypt #385

dbw9580 commented Mar 30, 2020

dbw9580 commented Mar 30, 2020 •

edited

saimn commented Apr 1, 2020

saimn commented Apr 1, 2020

dbw9580 commented Apr 1, 2020

saimn commented Apr 2, 2020

codecov bot commented Apr 4, 2020 •

edited

dbw9580 commented Apr 6, 2020

dbw9580 commented Apr 10, 2020

saimn left a comment

saimn Apr 14, 2020

dbw9580 Apr 14, 2020

saimn Apr 18, 2020

dbw9580 Apr 18, 2020

saimn Apr 19, 2020

dbw9580 Apr 19, 2020

saimn left a comment

dbw9580 commented Apr 14, 2020

dbw9580 commented Apr 18, 2020

saimn left a comment

saimn Apr 20, 2020

dbw9580 Apr 20, 2020

saimn Apr 20, 2020

saimn commented Apr 20, 2020

Plugin encrypt #385

Plugin encrypt #385

Conversation

dbw9580 commented Mar 30, 2020

dbw9580 commented Mar 30, 2020 • edited

saimn commented Apr 1, 2020

saimn commented Apr 1, 2020

dbw9580 commented Apr 1, 2020

saimn commented Apr 2, 2020

codecov bot commented Apr 4, 2020 • edited

Codecov Report

dbw9580 commented Apr 6, 2020

dbw9580 commented Apr 10, 2020

saimn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

saimn left a comment

dbw9580 commented Apr 14, 2020

dbw9580 commented Apr 18, 2020

saimn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

saimn commented Apr 20, 2020

dbw9580 commented Mar 30, 2020 •

edited

codecov bot commented Apr 4, 2020 •

edited