Skip to content

Fix test_steps_source_svn on Python 3#2796

Merged
tardyp merged 2 commits intobuildbot:masterfrom
rodrigc:svn1
Feb 19, 2017
Merged

Fix test_steps_source_svn on Python 3#2796
tardyp merged 2 commits intobuildbot:masterfrom
rodrigc:svn1

Conversation

@rodrigc
Copy link
Copy Markdown
Contributor

@rodrigc rodrigc commented Feb 15, 2017

I tried various combinations to get this test to work on Python 2 and Python 3,
but due to some corner cases where "somestring" is unicode on Python 3,
and bytes on Python 2, and the bytes in this testcase can't be decoded as ASCII,
I couldn't get this to work any other way.

@mention-bot
Copy link
Copy Markdown

@rodrigc, thanks for your PR! By analyzing the history of the files in this pull request, we identified @djmitche, @in3xes and @srinupiits to be potential reviewers.

@codecov
Copy link
Copy Markdown

codecov bot commented Feb 15, 2017

Codecov Report

Merging #2796 into master will increase coverage by 0.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #2796      +/-   ##
==========================================
+ Coverage   87.31%   87.32%   +0.01%     
==========================================
  Files         308      308              
  Lines       32651    32693      +42     
==========================================
+ Hits        28508    28550      +42     
  Misses       4143     4143
Impacted Files Coverage Δ
master/buildbot/steps/transfer.py 93.96% <ø> (+0.21%)
master/buildbot/www/oauth2.py 91.41% <ø> (+0.44%)
master/buildbot/reporters/github.py 92.15% <ø> (+2.15%)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e99fb94...a0d6612. Read the comment docs.

Copy link
Copy Markdown
Member

@tardyp tardyp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

python2 and python3 definitively need to output the same thing.

"http://foo.com/\x10\xe6%", "http://foo.com/%10%E6%25")
"http://foo.com/\x10\xe6%",
("http://foo.com/%10%E6%25", # Python 2
"http://foo.com/%10%C3%A6%25" # Python 3
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wait a minute.. I think this just means the svnUriCanonicalize does not work as expected with python3.

if for svn, url canonyfication of http://foo.com/\x10\xe6% is supposed to return http://foo.com/%10%E6%25, we cannot make it return http://foo.com/%10%C3%A6%25.

I think what can be assumed here is that the input of svnUriCanonicalize is supposed to be bytes (and not native string). Wouldn't it simplify the problem you had?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using bytes doesn't work, and results in ascii decode errors on Python 3.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

normal, it is not ascii. We really need to fix svnUriCanonicalize to work

@rodrigc
Copy link
Copy Markdown
Contributor Author

rodrigc commented Feb 17, 2017

@noc0lour I need your help with your superior debugging skills. Can you help with this one?
If I set up a Python 3 environment with this: https://lists.buildbot.net/pipermail/devel/2017-February/012310.html

and then type:

trial buildbot.test.unit.test_steps_source_svn.TestSvnUriCanonicalize.test_quote_funny_chars

I get a test failure. I tracked it down to somewhere inside urllib unquote(), where
on Python 2, things get decoded with latin-1 codec, while on Python 3, things get decoded with
ascii, so things get decoded differently.

@noc0lour
Copy link
Copy Markdown
Contributor

With your branch I don't get a test failure on python3?

@rodrigc
Copy link
Copy Markdown
Contributor Author

rodrigc commented Feb 18, 2017

Re-run the test from master branch. @tardyp doesn't like the fix I did here. I need another pair of eyes and smart brain (yours!) to look at it from another angle.

@noc0lour
Copy link
Copy Markdown
Contributor

noc0lour commented Feb 18, 2017

@rodrigc noc0lour@6db1796
Not very pythonic though, urlparse behaves differently in Python2/Python3 and you can specify encoding in Python3 but not Python2 to fix it. So I just check for Python3 and set urlopen to the correct function. But it looks... dirty. Test passed now.

@rodrigc
Copy link
Copy Markdown
Contributor Author

rodrigc commented Feb 18, 2017

@noc0lour can you submit a PR for the patch you came up with? I don't think there is a clean solution for this, so let's just move forward with this and see where it goes.

@tardyp
Copy link
Copy Markdown
Member

tardyp commented Feb 18, 2017

I find @noc0lour solution pretty clean.

We may add it in from buildbot.compat import urlquote

@codecov-io
Copy link
Copy Markdown

codecov-io commented Feb 18, 2017

Codecov Report

Merging #2796 into master will increase coverage by <.01%.
The diff coverage is 93.33%.

@@            Coverage Diff             @@
##           master    #2796      +/-   ##
==========================================
+ Coverage   87.33%   87.33%   +<.01%     
==========================================
  Files         309      310       +1     
  Lines       32771    32784      +13     
==========================================
+ Hits        28620    28632      +12     
- Misses       4151     4152       +1
Impacted Files Coverage Δ
master/buildbot/steps/source/svn.py 94.7% <100%> (ø)
master/buildbot/compat.py 92.3% <92.3%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e59eb78...7c9d567. Read the comment docs.

@rodrigc
Copy link
Copy Markdown
Contributor Author

rodrigc commented Feb 18, 2017

@noc0lour based on commits done to my branch by @tardyp, I took your solution in a slightly different direction. Please take a look.

@noc0lour
Copy link
Copy Markdown
Contributor

I think your solution now is fine as well. i think I can close my PR now ;)

@rodrigc rodrigc closed this Feb 18, 2017
@rodrigc rodrigc reopened this Feb 18, 2017
@rodrigc
Copy link
Copy Markdown
Contributor Author

rodrigc commented Feb 19, 2017

@noc0lour thank you for investigating and leading to the solution!

@tardyp
Copy link
Copy Markdown
Member

tardyp commented Feb 19, 2017

great cooperation guys thank you!

@tardyp tardyp merged commit 7c7bb6a into buildbot:master Feb 19, 2017
@noc0lour
Copy link
Copy Markdown
Contributor

No problem, I'm glad I can help out :) Since my virtualenvs on my machines are organized using virtualevnwrapper it takes literally no time to test on py2/py3.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants