New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow uploader.py to compress rlog without upload permission #471

Merged
merged 15 commits into from Feb 14, 2019

Conversation

Projects
None yet
4 participants
@eFiniLan
Copy link
Contributor

eFiniLan commented Dec 18, 2018

rlog does not compress to rlog.bzip2 until upload is allowed, which means users like me who only upload logs with WIFI will waste quite a bit of space.

rlog2

OP generates a folder each minute with at least 2 files in it:
rlog

after compression, rlog size is reduced to 7mb (vs 25.5mb)

1 hr drive will save roughly around 1G of space.

@eFiniLan eFiniLan changed the title Allow uploader to compress rlog without upload permission Allow uploader.py to compress rlog without upload permission Dec 18, 2018

@emmertex

This comment has been minimized.

Copy link
Contributor

emmertex commented Dec 19, 2018

Can I suggest you also add this.
That way it uploads in the correct order, starting with those compressed already.

127 # try to upload compressed log files first
128 for name, key, fn in self.gen_upload_files():
129 if name == "rlog.bz2":
130 return (key, fn, 0)
131
132 # then do those which are yet to be compressed
133 for name, key, fn in self.gen_upload_files():
134 if name == "rlog":
135 return (key, fn, 0)

emmertex pushed a commit to emmertex/openpilot that referenced this pull request Dec 19, 2018

emmertex pushed a commit to emmertex/openpilot that referenced this pull request Dec 19, 2018

eFiniLan added some commits Dec 19, 2018

let uploader to upload compressed log first (when allowed) then proce…
…ss uncompressed rlog (suggested by ku7)
Revert "let uploader to upload compressed log first (when allowed) th…
…en process uncompressed rlog (suggested by ku7)"

This reverts commit 98d7c45.
@eFiniLan

This comment has been minimized.

Copy link
Contributor Author

eFiniLan commented Dec 21, 2018

@emmertex I have changed it back, if we return rlog.bzip2 files first that means none of the rlog will be compressed until all the rlog.bzip2 are uploaded.

@emmertex

This comment has been minimized.

Copy link
Contributor

emmertex commented Dec 21, 2018

I seen that, after I tested a long drive.
Obvious now I look at it.
I'll attack it again later, but having said that, it only takes 5 seconds to compress 1 minn of data, maybe it's pointless anyway?

@rbiasini

This comment has been minimized.

Copy link
Contributor

rbiasini commented Dec 24, 2018

looks like a good change, thanks. Is it good to go? @emmertex or anybody else had a chance to test it?

@emmertex

This comment has been minimized.

Copy link
Contributor

emmertex commented Dec 24, 2018

@emmertex

This comment has been minimized.

Copy link
Contributor

emmertex commented Jan 15, 2019

So sorry I forgot about this..
All worked fine for me..

@emmertex

This comment has been minimized.

Copy link
Contributor

emmertex commented Jan 24, 2019

This (exactly as per this PR) has been integrated in my fork since start of the year

snippet of output of ls -R showing it working properly

`./2019-01-23--23-29-47--0:
bootlog

./2019-01-24--09-12-12--0:
fcamera.hevc
rlog.bz2

./2019-01-24--09-19-27--0:
fcamera.hevc
rlog.bz2

./2019-01-24--09-57-43--0:
fcamera.hevc
rlog.bz2

./2019-01-24--09-57-43--1:
fcamera.hevc
rlog.bz2

./2019-01-24--09-57-43--10:
fcamera.hevc
prcamera.mkv
rlog.bz2

./2019-01-24--09-57-43--2:
fcamera.hevc
rlog.bz2

./2019-01-24--09-57-43--3:
fcamera.hevc
rlog.bz2

./2019-01-24--09-57-43--4:
fcamera.hevc
prcamera.mkv
rlog.bz2

./2019-01-24--09-57-43--5:
fcamera.hevc
rlog.bz2
`

ErichMoraga added a commit to ErichMoraga/openpilot that referenced this pull request Jan 24, 2019

@eFiniLan PR 471
Allow uploader.py to compress rlog without upload permission commaai#471

eFiniLan added some commits Jan 25, 2019

@legonigel legonigel merged commit 88246af into commaai:devel Feb 14, 2019

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment