New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: Blob encryption #1

Merged
merged 3 commits into from Apr 22, 2016

Conversation

Projects
None yet
3 participants
@pcdummy
Contributor

pcdummy commented Feb 1, 2016

We implemented Bob encryption here.
Its a bit of a hack as keas.kmi and ZODB doen't support file streams/fd passing.

So we have to save the encrypted file somewhere on the filesystem and return the filename of
that temporary encrypted file.

Is there a better way to solve this?

pcdummy added some commits Nov 11, 2015

Add .gitignore.
Signed-off-by: Rene Jochum <rene@jochums.at>
PEP8.
Signed-off-by: Rene Jochum <rene@jochums.at>

@pcdummy pcdummy changed the title from Blob encryption to WIP: Blob encryption Feb 1, 2016

@frisi

This comment has been minimized.

Show comment
Hide comment
@frisi

frisi Feb 2, 2016

Member

@agroszer, @mgedmin and @strichter as you are the people that contributed to this package and are listed as owners on pypi i'm mentioning your names here.

we are trying to add support for encrypting files in the blobstorage and would like to get your feedback on the work done by now.

encryption is working well - i think. decryption is done by creating a temporary decrypted file in $INSTANCE_HOME/var/tmp and return its filename instead of the filename of the original blobfile.

@pcdummy already started to implement "junk encoding" the files to save ram and gain performance. we'd need to use https://github.com/webmeisterei/keas.kmi/tree/enc_dec_file to get this working.

one of the open issues is the "garbage collection" of encrypted files in $INSTANCE_HOME/var/tmp.
a simple idea would be to delete files older than 2 minutes that are not yet opened using cron.
ideally loadBlob and storeBlob should work with filedescriptions instead of passing filenames and data. this way we could return enc- and decrypted streams and do not worry about cleaning up temporary files.

we'd love to get your feedback. what needs to be done to get this merged?

we are happy to add tests for our blob related additions as soon as we got your feedback and and can be sure this gets merged. maybe you could help us to fix the tests?

Member

frisi commented Feb 2, 2016

@agroszer, @mgedmin and @strichter as you are the people that contributed to this package and are listed as owners on pypi i'm mentioning your names here.

we are trying to add support for encrypting files in the blobstorage and would like to get your feedback on the work done by now.

encryption is working well - i think. decryption is done by creating a temporary decrypted file in $INSTANCE_HOME/var/tmp and return its filename instead of the filename of the original blobfile.

@pcdummy already started to implement "junk encoding" the files to save ram and gain performance. we'd need to use https://github.com/webmeisterei/keas.kmi/tree/enc_dec_file to get this working.

one of the open issues is the "garbage collection" of encrypted files in $INSTANCE_HOME/var/tmp.
a simple idea would be to delete files older than 2 minutes that are not yet opened using cron.
ideally loadBlob and storeBlob should work with filedescriptions instead of passing filenames and data. this way we could return enc- and decrypted streams and do not worry about cleaning up temporary files.

we'd love to get your feedback. what needs to be done to get this merged?

we are happy to add tests for our blob related additions as soon as we got your feedback and and can be sure this gets merged. maybe you could help us to fix the tests?

@mgedmin

This comment has been minimized.

Show comment
Hide comment
@mgedmin

mgedmin Feb 2, 2016

Member

I'm not using cipher.encryptingstorage, and I'm not interested in maintaining it. Feel free to take over, if nobody else objects.

One obvious thing that would be good to fix is to make the tests not fail on Travis ;)

(I haven't checked but this is probably the Python 3.2 thing, where the best way forward is probably drop Python 3.2 support from travis.yml, tox.ini and setup.py.)

Member

mgedmin commented Feb 2, 2016

I'm not using cipher.encryptingstorage, and I'm not interested in maintaining it. Feel free to take over, if nobody else objects.

One obvious thing that would be good to fix is to make the tests not fail on Travis ;)

(I haven't checked but this is probably the Python 3.2 thing, where the best way forward is probably drop Python 3.2 support from travis.yml, tox.ini and setup.py.)

:returns: The path to the temporary file.
TODO: Currently theres no code that handles the deletion of the file.

This comment has been minimized.

@mgedmin

mgedmin Feb 2, 2016

Member

This seems like kind of a big thing... ?

@mgedmin

mgedmin Feb 2, 2016

Member

This seems like kind of a big thing... ?

This comment has been minimized.

@pcdummy

pcdummy Feb 2, 2016

Contributor

Yes, thats a big thing. But currently theres filename passing instead of file descriptor passing in the Blob code, which means we currently don't know when the file gets closed.

I'm sure i'll get the time to implement monkey patches of ZODB for that and maybe upstream patches.

@pcdummy

pcdummy Feb 2, 2016

Contributor

Yes, thats a big thing. But currently theres filename passing instead of file descriptor passing in the Blob code, which means we currently don't know when the file gets closed.

I'm sure i'll get the time to implement monkey patches of ZODB for that and maybe upstream patches.

This comment has been minimized.

@pcdummy

pcdummy Feb 3, 2016

Contributor

Do you have a better solution? Any help?

@pcdummy

pcdummy Feb 3, 2016

Contributor

Do you have a better solution? Any help?

This comment has been minimized.

@mgedmin

mgedmin Feb 3, 2016

Member

I don't -- but unencrypted copies of sensitive data hitting the disk for indeterminate amounts of time worry me.

I suppose you could set up a tmpfs or something for the unencrypted blob shadow directory, but ick, another thing you can easily and silently get wrong.

@mgedmin

mgedmin Feb 3, 2016

Member

I don't -- but unencrypted copies of sensitive data hitting the disk for indeterminate amounts of time worry me.

I suppose you could set up a tmpfs or something for the unencrypted blob shadow directory, but ick, another thing you can easily and silently get wrong.

@pcdummy

This comment has been minimized.

Show comment
Hide comment
@pcdummy

pcdummy Apr 11, 2016

Contributor

@mgedmin can you add me and @frisi as owner to pypi?

Mine is pcdummy and the one of frisi is also frisi.

Contributor

pcdummy commented Apr 11, 2016

@mgedmin can you add me and @frisi as owner to pypi?

Mine is pcdummy and the one of frisi is also frisi.

@mgedmin

This comment has been minimized.

Show comment
Hide comment
@mgedmin

mgedmin Apr 11, 2016

Member

@frisi, @pcdummy: you're now cipher.encryptingstorage maintainers on PyPI. Use your new powers for good!

Member

mgedmin commented Apr 11, 2016

@frisi, @pcdummy: you're now cipher.encryptingstorage maintainers on PyPI. Use your new powers for good!

@pcdummy

This comment has been minimized.

Show comment
Hide comment
@pcdummy

pcdummy Apr 13, 2016

Contributor

FYI we use a simple cronjob to clean unencrypted files:

/usr/bin/find ${buildout:directory}/var/tmp -type f -amin +5 -exec sh -c "fuser -s {} || rm -f {}" ;

Contributor

pcdummy commented Apr 13, 2016

FYI we use a simple cronjob to clean unencrypted files:

/usr/bin/find ${buildout:directory}/var/tmp -type f -amin +5 -exec sh -c "fuser -s {} || rm -f {}" ;

@frisi

This comment has been minimized.

Show comment
Hide comment
@frisi

frisi Apr 13, 2016

Member

@frisi, @pcdummy: you're now cipher.encryptingstorage maintainers on PyPI. Use your new powers for good!

thanks @mgedmin - we will do so ;-)

Member

frisi commented Apr 13, 2016

@frisi, @pcdummy: you're now cipher.encryptingstorage maintainers on PyPI. Use your new powers for good!

thanks @mgedmin - we will do so ;-)

@frisi

This comment has been minimized.

Show comment
Hide comment
@frisi

frisi Apr 22, 2016

Member

i'd squash the last 2 commits. then this is ready to merge @pcdummy

Member

frisi commented Apr 22, 2016

i'd squash the last 2 commits. then this is ready to merge @pcdummy

Implement encryption and decryption of blobs.
Signed-off-by: Rene Jochum <rene@jochums.at>

@frisi frisi merged commit 39d0713 into zopefoundation:master Apr 22, 2016

1 check was pending

continuous-integration/travis-ci/pr The Travis CI build is in progress
Details

@pcdummy pcdummy deleted the webmeisterei:blob_encryption branch Apr 22, 2016

@frisi

This comment has been minimized.

Show comment
Hide comment
@frisi

frisi Apr 22, 2016

Member

1.1 released - thanks @pcdummy and @mgedmin

Member

frisi commented Apr 22, 2016

1.1 released - thanks @pcdummy and @mgedmin

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment