/apt-offline

Python 3 Porting #32

Closed
wants to merge 313 commits into
from

Conversation

Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
2 participants
@copyninja @rickysarraf
@copyninja
Collaborator

copyninja commented Sep 9, 2015

This PR provides currently set command implementation which works properly with both python3 and python2 and behavior is compatible with existing apt-offline.

copyninja added some commits Aug 25, 2015

@copyninja
Separate out build-dep tests 
Run the build-dep only if we are not on TRAVIS, only with latest apt
from experimental release this will work fine.
c932aae
@copyninja
Add compatibility tests for source operation 
This excludes build-dep due to the bug in apt which asks for super user
privilege for build-dep operation even with --print-uris option. (Fixed
only in apt 1.1~exp9 from experimental).
216b049
@copyninja
Add function to generate list of releases 
Releases of distribution depends on sources.list file. This function
uses python-apt to get enabled releases in one's system.
9dbec01
@copyninja
Introduce cmdline.py for CLI operations 
All CLI operations will be done by aptoffline.cmdline module. And the
entry_point is added to make aptoffline.cmdline:main as the apt-offline
command.
7d5999b
@copyninja
Test refactoring using testtools and fixtures
061b8ca
@copyninja
Provide universal_newlines support.
1cc28f8
@copyninja
Refactor test_aptget using testtools 
The test for build-dep still fails.
f7f18b9
@copyninja
Generate version_compare variable for apt 
We need this as there was a bug in apt < 1.1~exp9 which needed sudo
privilege for build-dep operation even when --print-uris specified.
f48e733
@copyninja
Use testtools skipTest feature.
47cf71c
@copyninja
Install python-apt and python3-apt in travis 
Now python-apt has become part of our tool.
68cd4f2
@copyninja
Installing python-apt using apt in travis has no effect.
417308b
@copyninja
Install python-apt via requirements.txt
c521462
@copyninja
Fix build-dep tests.
de9feef
@copyninja
Update readme about testing requirements and broken CI status.
17796d2
@copyninja
Drop python-apt installation from the requirements.txt
418a965
@copyninja
Let travis install python-apt specific to precise.
db95684
@copyninja
Changing python-apt to 0.8.9 
0.8.3 had issue with python3 branch and won't compile with python3.3 and
higher.
4c596ca
@copyninja
Move python-apt to 0.9.3 and hope for best. 
*sigh* print statement not valid python3
df2ff24
@copyninja
Modification to install-aptoffline script 
To run only if python version is < 3. Basically for circle ci integration.
ffb6d0e
@copyninja
Circle CI integration.
9e1426e
@copyninja
Circle.yml issue fixes
2725c6a
@copyninja
Fix the python-apt git path and versions
7af316a
@copyninja
Versions should be space separated.
ecc90a8
@copyninja
Hoping this works
4d05247
@copyninja
It looks like circle currently only works with 1 python?. 
Or I should use tox.
0a42791
@copyninja
Migrate to testtools 
Use AptOfflineTests as base class for TestCompatibility class and
refactor.
4d92af4
@copyninja
Call tearDown from TestCompatibility
b082b0e
@copyninja
Fix typo in baseclass 
Use outfile not self.outfile.
d9d4dc2
@copyninja
Implement _setter function 
This function handles apt-offline set command.
0515490
@copyninja
Organize source operation 
--src-build-dep should be clubed with --install-src-packages
  logically. i.e. this option doesn't enforce it only shows in help.
c659882
@copyninja
Allow testing of command parsing. 
main function now takes test_args which if set should be list of
arguments to be passed to ArgumentParser.parse_args function call
ce71ad0
@copyninja
Code for testing set command 
In initial stage not yet completely working.
eda3fcb
@copyninja
Add cleanup routine 
For executing tests go into workdir and on completion return back.
f064a81
@copyninja
Test --upgrade option.
916dfb5
@copyninja
pep8 confirmance
16bc541
@copyninja
Import only what is needed.
58be595
@copyninja
Exclude unreachable codes during testing from coverage.
0e78536
@copyninja
Drop if __name__ == '__main__' part
8d8a3a7
@copyninja
Save details dictionary of fixtures.FakeLogger 
This will be helpful to inspect the logging output during tests.
8fa930a
@copyninja
Convert releases to list for indexing purpose.
a5063de
@copyninja
Add more tests for set command options.
ec9767f
@copyninja
Add test for unprivileged --src-build-dep 
Skip test on latest apt i.e >= 1.1~exp9
30ebab9
@copyninja
Typo fix /criticial/critical
cba7261
@copyninja
More generic contains for apt-get source operation.
203d8ca
@copyninja
pep8 confirmance.
f4be97a
@copyninja
Avoid universal loading of python-apt 
python-apt is needed only in set command so load it in only related
functions and guard the creation of set and install command so it won't
be executing in Windows.
3d481d5
@copyninja
Use sys.platforms.startswith('linux') for platform 
This is because python2 its linux2 and python3.3 and later its changed
to linux. So the document suggests use of startswith for checking linux
platform.
8ee1d06
@copyninja
Use append mode for files 
Current apt-offline appends if multiple options are given
simultaneously. Also update tests to handle this properly.
1a56a7f
@copyninja
Move pyversion detection to aptoffline.tests.utils
66f8aef
@copyninja
os.truncate is available only from python 3.3 
Handle truncating of files for both python2 and python3 in truncate_file
function in base class.
1e65cbd
@copyninja
Truncate files after running each test in compatibility mode.
23aa892
@copyninja
Use sudo to truncate apt-offline generated file 
Since older apt-offline needed root privileges truncating file with
normal user permission was not possible. So use truncate command with
sudo privileges.
0577b85
@copyninja
All argument to subprocess.check_call should be string
6ca20d2
@copyninja
Add apt-offline and new versions set compatibility test.
ae8f724
@copyninja
Use apt instead of apt_pkg for releases
e17a58f
@copyninja
Owner

copyninja commented on e17a58f Sep 9, 2015

Looks like apt_pkg.Cache was much faster than the apt.Cache. Only benefit of apt.Cache is no message of apt is being put on screen. @rickysarraf can you see the difference?.

copyninja added some commits Sep 10, 2015

@copyninja
Update tox.ini to install python-apt from git 
For now this is for sid testing and expects python-all-dev is
installed.
332c5e6
@copyninja
Update circle ci settings.
43765b7
@copyninja
Override circle ci tests and use tox
b598b7a
@copyninja
Install python-tox
2fac2ce
@copyninja
Looks like precise doesn't have python-tox install from pip
61cace2
@copyninja
Document apt-get backend
3262e5c
@copyninja
Make AptGetSigParse sequence type containing AptGetSig 
AptGetSig represents signature file generated by apt-get using
--print-uris and AptGetSigParse will parse and hold all AptGetSig as
sequence of items.
763e51b
@copyninja
Test for AptGetSigParse
aa1520a
@copyninja
pep8 confirmance.
d25ecf3
@copyninja
For now skip the AptGetSigParse sequence emulating function from cove… 
…rage.
f9f9b8c
@copyninja
Rearrange options as per rrs's comments.
566817e
@copyninja
Add __len__ to AptGetSigParse
bbb2a16
@copyninja
Take only element 1 after splitting on MD5Sum:
6a7ac6f
@copyninja
Test correctness of parsing signature file.
e423f32
@copyninja
pep8 confirmance.
6302d83
@copyninja
Capture checksum_type also from the item line
f95ee00
@copyninja
Add options for get command.
3bff9ab
@copyninja
Half implemented _getter function
8a0c2db
@copyninja
Flake8 fixes 
2 issues still remain hence tests will fail.
54362f0
@copyninja
Add ZipArchiver class and tests 
ZipArchiver is used to create zip files, used by --bundle option of
apt-offline.
3b819c1
@copyninja
Merge remote-tracking branch 'origin/py3-port' into py3-port
0e21efb
@copyninja
Use --simulate for Simulation to avoid conflict 
-s is used by --cache-dir which conflicted with -s used as short hand
 for simulation.
8337f2a
@copyninja
Typo fixes
79197cd
@copyninja
aptwrapper as alternative to python-apt 
If python-apt is not installed on system, have an alternative for
whatever we are using from python-apt.
c991b29
@copyninja
Guard python-apt inclusion in aptoffline.util 
Use python-apt if its only available otherwise revert back to our
handwritten aptwrapper.
8118083
@copyninja
Update compare_version documentation
9bfa850
@copyninja
Comment out cloning of the python-apt in tox 
We have a back up so python-apt is not really needed.
f4bf4e5
@copyninja
Utility functions: cache search, checksum handling 
Functions added to search for cached file and check the file validity
using checksum.
76526ce
@copyninja
os.listdir only gives out filenames 
prepend /etc/apt/sources.list.d to output files from os.listdir
function.
9c26b69
@copyninja
Enable python-apt installation in tox 
One of the test for aptwrapper needs python-apt
983ff1f
@copyninja
Fix joining of /etc/apt/sources.list.d files 
I was trying to merge list with a string, use map and partial
appropriately to fix this.
c3a8d11
@copyninja
Typo fix s/haslib/hashlib/
25a4b09
@copyninja
Drop unused exception variable
d7867a2
@copyninja
Drop release calculation completely 
To calculate release properly we need to parse sources.list and files
under /etc/sources.list.d in case of missing python-apt. This doesn't
look feasible so avoid releases generation completely.

This change means --releases option is no longer providing choices
5dd6688
@copyninja
Drop unused imports
b6f0667
@copyninja
Drop python-apt completely we don't need it.
ef4a5bf
@copyninja
Drop unwanted print statement.
6d9e800
@copyninja
Enable sources in circle builder
8382d6d
@copyninja
Bring scripts into one folder 
Mostly used by CI will reside here.
23c2930
@copyninja
Update CI build configs.
ee9fdd2
@copyninja
Improve _getter function code 
Its far from complete but some part like using cached file is done but
not tested yet.
592b70a
@copyninja
Add embeeded code copy for apt-offline 
Inspired from requests module added in such a way packaging can simply
remove these packages and reference from the system.
380daca
@copyninja
Use modules from .packages 
While packaging embedded code copies should be dropped.
a79ff78
@copyninja
Ignore packages content from flake8 and coverage
76d11df
@copyninja
Add futures.concurrent package for python2 
This package is part of standard lib for python3
98ac8a0
@copyninja
Parse auth information if present in url 
If password protected repo are used then url will be in format

`http://user:passwd@repo/debian`

Parse that information into AptGetSig.user and AptGetSig.passwd fields.
965e83f
@copyninja
Rename download.py as progress.py 
This module contains progress bar hence rename.
c6e9b2e
@copyninja
Add wheel related option to setup.cfg
51eca1e
@copyninja
Provide only one option for proxy 
Previous apt-offline line had --proxy-host and --proxy-port but it
doesn't make sense to later merge both, as requests takes both
together and give additional option for password protected proxies.

New option takes http://host:port/ or http://user:passwd@host:port/ and
uses it directly.
6790962
@copyninja
Add test case file for password protected repo 
Forgot to add previously.
c8cc5f6
@copyninja
Add add_directory helper to ZipArchiver 
Its good to have helper function which can add contents of directory to
zip file. (Not perfect one)
1a4d937
@copyninja
Rough implementation of download logic 
Not perfect not tested just added.
6a0c496
@copyninja
Take over line overwriting variable from old code 
Take the screen clean up variables from old apt-offline. Not sure how
this works but still.
1d4a2bc
@copyninja
Add logger into download manager
cfbaf3e
@copyninja
Add logging to cache checking function.
bd6e97f
@copyninja
Add human_size function 
Give bytes we return human readable size string with KB/MB/GB suffix.
6aac948
@copyninja
Return false from _in_cache for consistency
38094f3
@copyninja
Add error handling and logging 
Added InvalidPackageName exception
6b475cc
@rickysarraf
Owner

rickysarraf commented Mar 7, 2017

Closing this for now. When python3 porting will be done, some of the porting lessons can be picked from this PR. This PR was a py3 port and a rewrite of many components, both at the same time, which is what makes it a difficult PR.

@rickysarraf rickysarraf closed this Mar 7, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment