Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow uploader.py to compress rlog without upload permission #471

Merged
merged 15 commits into from Feb 14, 2019
Merged

Allow uploader.py to compress rlog without upload permission #471

merged 15 commits into from Feb 14, 2019

Conversation

eFiniLan
Copy link
Contributor

@eFiniLan eFiniLan commented Dec 18, 2018

rlog does not compress to rlog.bzip2 until upload is allowed, which means users like me who only upload logs with WIFI will waste quite a bit of space.

rlog2

OP generates a folder each minute with at least 2 files in it:
rlog

after compression, rlog size is reduced to 7mb (vs 25.5mb)

1 hr drive will save roughly around 1G of space.

@eFiniLan eFiniLan changed the title Allow uploader to compress rlog without upload permission Allow uploader.py to compress rlog without upload permission Dec 18, 2018
@emmertex
Copy link

Can I suggest you also add this.
That way it uploads in the correct order, starting with those compressed already.

127 # try to upload compressed log files first
128 for name, key, fn in self.gen_upload_files():
129 if name == "rlog.bz2":
130 return (key, fn, 0)
131
132 # then do those which are yet to be compressed
133 for name, key, fn in self.gen_upload_files():
134 if name == "rlog":
135 return (key, fn, 0)

emmertex pushed a commit to emmertex/openpilot that referenced this pull request Dec 19, 2018
emmertex pushed a commit to emmertex/openpilot that referenced this pull request Dec 19, 2018
@eFiniLan
Copy link
Contributor Author

@emmertex I have changed it back, if we return rlog.bzip2 files first that means none of the rlog will be compressed until all the rlog.bzip2 are uploaded.

@emmertex
Copy link

I seen that, after I tested a long drive.
Obvious now I look at it.
I'll attack it again later, but having said that, it only takes 5 seconds to compress 1 minn of data, maybe it's pointless anyway?

@rbiasini
Copy link
Contributor

looks like a good change, thanks. Is it good to go? @emmertex or anybody else had a chance to test it?

@emmertex
Copy link

emmertex commented Dec 24, 2018 via email

@emmertex
Copy link

So sorry I forgot about this..
All worked fine for me..

@emmertex
Copy link

emmertex commented Jan 24, 2019

This (exactly as per this PR) has been integrated in my fork since start of the year

snippet of output of ls -R showing it working properly

`./2019-01-23--23-29-47--0:
bootlog

./2019-01-24--09-12-12--0:
fcamera.hevc
rlog.bz2

./2019-01-24--09-19-27--0:
fcamera.hevc
rlog.bz2

./2019-01-24--09-57-43--0:
fcamera.hevc
rlog.bz2

./2019-01-24--09-57-43--1:
fcamera.hevc
rlog.bz2

./2019-01-24--09-57-43--10:
fcamera.hevc
prcamera.mkv
rlog.bz2

./2019-01-24--09-57-43--2:
fcamera.hevc
rlog.bz2

./2019-01-24--09-57-43--3:
fcamera.hevc
rlog.bz2

./2019-01-24--09-57-43--4:
fcamera.hevc
prcamera.mkv
rlog.bz2

./2019-01-24--09-57-43--5:
fcamera.hevc
rlog.bz2
`

ErichMoraga added a commit to ErichMoraga/openpilot that referenced this pull request Jan 24, 2019
Allow uploader.py to compress rlog without upload permission commaai#471
@legonigel legonigel merged commit 88246af into commaai:devel Feb 14, 2019
adhintz added a commit to adhintz/openpilot that referenced this pull request Feb 24, 2019
* Add "Accept-Encoding: gzip" header to mapd.py to reduce data consumption (commaai#525)

* Allow uploader.py to compress rlog without upload permission (commaai#471)

* Always Compress - Make sure uploader to bzip2 rlog once it's completed without any upload permission.

* Squashed 'panda/' changes from 9ee6285..39c1e39

39c1e39 Cadillac: added brake msg forwarding
aaa1b14 bump VERSION
e0dd558 Fixed undefined reference error when "make recover" in EON
c91f038 Toyota pedal support (commaai#159)
de1ba06 Build optimizer: back to Os now that 'make --recover' is fixed
fd23383 add -lgcc flag to fix "undefined reference to __aeabi_llsr" (commaai#156)
1cc74e2 Reverted code optimizer option to O2 after 'make recover' broke
1218d09 Cadillac ascm proxy (commaai#155)
064446f Panda: bumped version
93266a9 Hyundai: allow the same max steer torque as stock
d275fa8 Subaru: Forwarding (commaai#152)

git-subtree-dir: panda
git-subtree-split: 39c1e39

* Squashed 'opendbc/' changes from b632acf..45c3285

45c3285 Added support to Toyota pedal
cfb7070 Revert "Added pedal support for Toyota"
ca607f4 Added pedal support for Toyota
22efd56 Honda: for simplicity all cars now have BRAKE_HOLD signal
f969de0 Honda Nidec: VSA_STATUS msg is teh same for all
85b0df7 chrysler L gear (commaai#139)
b9618da Update Gear Packet with Eco and Sport button (commaai#135)
5259345 Update DBC for Hyundai Kona Support (commaai#138)
608caba Create subaru_outback_2015_eyesight.dbc (commaai#137)

git-subtree-dir: opendbc
git-subtree-split: 45c3285

* Squashed 'pyextra/' changes from 8cc1594..0d19c13

0d19c13 Revert adding reverse_geocoder
4a22fac Fix reverse_geocoder folder
35308c5 Add reverse_geocoder

git-subtree-dir: pyextra
git-subtree-split: 0d19c13e6e4e5fa82a470374146a86dc5d5afa35

* openpilot v0.5.9 release

* remove unneeded msg variable
minimize diff with upstream

* change error back to perm

* allow it to instantly ramp down to 0

* try new panda safety code

* minor panda cleanup
Chrispreilly pushed a commit to Chrispreilly/openpilot that referenced this pull request Nov 21, 2019
Add speed hump with bump attributes
dragonpilot pushed a commit to dragonpilot-community/dragonpilot that referenced this pull request Apr 7, 2020
…#471)

* Always Compress - Make sure uploader to bzip2 rlog once it's completed without any upload permission.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants