XML compression #27022

satyaakam · 2020-04-01T10:18:39Z

https://dimagi-dev.atlassian.net/browse/IIO-742

SUMMARY

Changes in BlobDB API were made as per the suggestions on #26813

FEATURE FLAG

ICDS-CAS

PRODUCT DESCRIPTION

None

corehq/blobs/mixin.py

corehq/blobs/s3db.py

corehq/blobs/util.py

snopoke

I did a quick pass and left some comments. Overall looks like a good start.

I don't think you meant to commit these:

corehq/blobs/s3db.py

corehq/blobs/interface.py

corehq/blobs/util.py

corehq/blobs/s3db.py

corehq/form_processor/models.py

corehq/blobs/s3db.py

corehq/blobs/util.py

corehq/blobs/s3db.py

…ompress-xml

corehq/blobs/migrations/0011_blobmeta_compressed.py

corehq/blobs/mixin.py

corehq/blobs/s3db.py

corehq/blobs/util.py

corehq/blobs/tests/test_util.py

corehq/blobs/exceptions.py

corehq/blobs/tests/test_util.py

corehq/blobs/exceptions.py

corehq/blobs/tests/test_util.py

corehq/blobs/export.py

Co-Authored-By: Daniel Miller <millerdev@gmail.com>

snopoke · 2020-04-17T13:04:47Z

corehq/blobs/metadata.py

@@ -31,6 +32,8 @@ def new(self, **blob_meta_args):
                    "keyword arguments are incompatible with `meta` argument")
            return blob_meta_args["meta"]
        timeout = blob_meta_args.pop("timeout", None)
+        if blob_meta_args.get('type_code') == CODES.form_xml:
+            blob_meta_args['compressed_length'] = -1


I don't really like this. It does got overwritten during later but still feels off

corehq/blobs/interface.py

corehq/blobs/metadata.py

corehq/blobs/s3db.py

Co-Authored-By: Daniel Miller <millerdev@gmail.com>

Make the error more generic so it can be used in other places if need be. Currently this error is never expected to occur unless there is a bug.

Both GzipStream and GzipStream.Buffer had a `_buf` attribute. The class composition made it easy to confuse the two.

millerdev

I fixed some bugs and created a new PR against the original branch in @satyaakam's fork: satyaakam#1

@snopoke I'm not sure how you were able to push directly to this PR since it's originating from a fork. I'd be interested in learning how you did that.

Gzip stream fixes for XML compression

snopoke · 2020-04-21T09:04:27Z

I'm not sure how you were able to push directly to this PR since it's originating from a fork. I'd be interested in learning how you did that.

I got given write access to the forked repo

snopoke · 2020-04-21T13:01:07Z

@millerdev build is passing but will let you do a final review before merging

shubham1g5 · 2020-05-22T05:53:58Z

Flagging that this PR incorrectly updated the commcare-translations submodule reverting it back to an older 2.47 version. @orangejenny has made the required correction on master pushing it back to the latest commit.

XML compression

adbdc3a

satyaakam requested review from calellowitz, millerdev, Rohit25negi and snopoke as code owners April 1, 2020 10:18

stickler-ci reviewed Apr 1, 2020

View reviewed changes

corehq/blobs/mixin.py Outdated Show resolved Hide resolved

corehq/blobs/s3db.py Show resolved Hide resolved

corehq/blobs/s3db.py Outdated Show resolved Hide resolved

corehq/blobs/util.py Outdated Show resolved Hide resolved

snopoke reviewed Apr 1, 2020

View reviewed changes

millerdev reviewed Apr 1, 2020

View reviewed changes

satyaakam added 2 commits April 3, 2020 12:56

Changes made as per the suggestions in PR 27022

1851c6d

Resolving conflits

6ff7e25

stickler-ci reviewed Apr 3, 2020

View reviewed changes

corehq/blobs/s3db.py Outdated Show resolved Hide resolved

satyaakam added 4 commits April 3, 2020 23:43

merged master into compress-xml

55fd72e

Merge branch 'master' of https://github.com/dimagi/commcare-hq into c…

787604b

…ompress-xml

Changes to the compress xml as review

c843658

Merge branch 'master' of https://github.com/dimagi/commcare-hq into c…

ed006c5

…ompress-xml

snopoke reviewed Apr 10, 2020

View reviewed changes

corehq/blobs/migrations/0011_blobmeta_compressed.py Show resolved Hide resolved

Fixing fucntion call

071b011

millerdev reviewed Apr 10, 2020

View reviewed changes

corehq/blobs/mixin.py Outdated Show resolved Hide resolved

corehq/blobs/s3db.py Outdated Show resolved Hide resolved

corehq/blobs/util.py Outdated Show resolved Hide resolved

corehq/blobs/tests/test_util.py Show resolved Hide resolved

satyaakam added 4 commits April 14, 2020 13:30

Missing steps to make Bower work as normal user

5511232

Merge branch 'master' of https://github.com/dimagi/commcare-hq

233f255

Updated Tests and custom exceptions added

8f63335

Merge branch 'master' into compress-xml

1ea2d13

stickler-ci reviewed Apr 15, 2020

View reviewed changes

corehq/blobs/exceptions.py Outdated Show resolved Hide resolved

corehq/blobs/exceptions.py Outdated Show resolved Hide resolved

corehq/blobs/tests/test_util.py Show resolved Hide resolved

corehq/blobs/tests/test_util.py Outdated Show resolved Hide resolved

Fixed Lint issues

7a43156

stickler-ci reviewed Apr 15, 2020

View reviewed changes

corehq/blobs/exceptions.py Outdated Show resolved Hide resolved

corehq/blobs/tests/test_util.py Outdated Show resolved Hide resolved

snopoke mentioned this pull request Apr 16, 2020

compress form XML in blobdb #27181

Closed

snopoke added 3 commits April 16, 2020 10:17

simplify validate_args

b8a190b

nits

acb621f

add missing type_code kwargs

7756fad

stickler-ci reviewed Apr 16, 2020

View reviewed changes

corehq/blobs/export.py Outdated Show resolved Hide resolved

snopoke and others added 5 commits April 17, 2020 13:49

simplify blobmeta.open()

b13480f

Co-Authored-By: Daniel Miller <millerdev@gmail.com>

reusue util to get sizes

5f6e847

minor simplication

a4a90a8

prefer using meta for lookup if it's available

f4bc531

better test

77ed3e5

snopoke reviewed Apr 17, 2020

View reviewed changes

millerdev reviewed Apr 17, 2020

View reviewed changes

corehq/blobs/interface.py Outdated Show resolved Hide resolved

corehq/blobs/metadata.py Show resolved Hide resolved

corehq/blobs/s3db.py Show resolved Hide resolved

snopoke and others added 5 commits April 17, 2020 17:07

Update corehq/blobs/interface.py

820f81d

Co-Authored-By: Daniel Miller <millerdev@gmail.com>

merging upstream changes

70a4090

Merging upstream changes

48c9e0e

fix argument order

aa7348f

remove GzipCompressReadStream.Buffer.content_length

6d2f0c0

snopoke added the product/invisible Change has no end-user visible impact label Apr 20, 2020

snopoke and others added 5 commits April 20, 2020 16:58

Merge branch 'master' into compress-xml

6f6466f

GzipCompressReadStream -> GzipStream

468bdb1

GzipStreamAttrAccessBeforeRead -> GzipStreamError

d83f529

Make the error more generic so it can be used in other places if need be. Currently this error is never expected to occur unless there is a bug.

Move IO buffer out of GzipStream class

4e6677f

Both GzipStream and GzipStream.Buffer had a `_buf` attribute. The class composition made it easy to confuse the two.

Fix bugs in GzipStream

3b3c00f

millerdev reviewed Apr 20, 2020

View reviewed changes

Merge pull request #1 from dimagi/dm/compress-xml

470f2ca

Gzip stream fixes for XML compression

satyaakam requested a review from esoergel as a code owner April 21, 2020 09:24

snopoke added 2 commits April 21, 2020 12:31

save to file before extracting in test

7c5d4bf

fallback to partial read if seek not supported

05cadf1

snopoke force-pushed the compress-xml branch from 062475e to 05cadf1 Compare April 21, 2020 10:31

millerdev approved these changes Apr 21, 2020

View reviewed changes

snopoke merged commit 3465254 into dimagi:master Apr 21, 2020

snopoke deleted the compress-xml branch April 21, 2020 13:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

XML compression #27022

XML compression #27022

satyaakam commented Apr 1, 2020

snopoke left a comment

snopoke Apr 17, 2020

millerdev left a comment

snopoke commented Apr 21, 2020

snopoke commented Apr 21, 2020

shubham1g5 commented May 22, 2020

XML compression #27022

XML compression #27022

Conversation

satyaakam commented Apr 1, 2020

SUMMARY

FEATURE FLAG

PRODUCT DESCRIPTION

snopoke left a comment

Choose a reason for hiding this comment

snopoke Apr 17, 2020

Choose a reason for hiding this comment

millerdev left a comment

Choose a reason for hiding this comment

snopoke commented Apr 21, 2020

snopoke commented Apr 21, 2020

shubham1g5 commented May 22, 2020