Port code to Python 3

[bdamele]

Currently there is no pressure on Python projects to switch to the new version of Python interpreter, as the process of switching, especially on larger projects can be cumbersome (due to the few backward incompatibilities). The switch will take place eventually, but currently it is a very low priority task.

[stamparm]

Good reference: http://python3porting.com/noconv.html

[stamparm]

As I've collected information regarding this over time, this will be maybe an impossible task to do because of one feature: "unicode strings". We currently extensively use unicode everywhere together with encoding/decoding for page responses. Problem is that Python 2 has a clear distinction between unicode (e.g. decoded page response using stated charset) and non-unicode data (e.g. raw page response), while Python 3 doesn't have.

Good reference: http://lucumr.pocoo.org/2010/1/7/pros-and-cons-about-python-3/

[badbob]

Is there any news for this issue? Does Python 3.4 have required functionality?

[stamparm]

@badbob I would say no, but that's my opinion. Only small projects are prone to conversion from 2 to 3. I have a feeling that this is a futile task for sqlmap.

[badbob]

Looks like the only way to reuse sqlmap code in Python 3 solutions is a REST API. Thanks for developers sqlmap has it. I can't see any other variant.

[Flimm]

Problem is that Python 2 has a clear distinction between unicode (e.g. decoded page response using stated charset) and non-unicode data (e.g. raw page response), while Python 3 doesn't have.

I would highly disagree with this statement. In Python 2, strings represented either textual data or raw data (often raw data of textual data in an assumed or unknown encoding), while unicode objects represented textual data. Strings in Python 2 were by nature ambiguous. In Python 3, bytes objects represent raw data, and string objects represent textual data. It's crystal clear in Python 3, and it's enforced with exceptions when mixing strings with bytes objects. This crystal clear distinction exposes bad design choices in certain systems (for instance, the ambiguous encodings of URLs, or of filenames), and it causes the programmer to have to consider these problems upfront. Python 2 just let you stuff everything into a string, and it gives you runtime exceptions sometime in the future when you try anything not in ASCII.

I'm not trying to argue that sqlmap should necessarily be ported to Python 3. I just felt someone needed to speak up and defend Python 3's explicitness when it comes to strings and bytes.

Python 2.7 will no longer be supported past 2020, as advertised by this website: https://pythonclock.org/ Many scientific libraries for Python have announced that they will also no longer support Python 2.x http://www.python3statement.org/

If the owners of this project would welcome efforts to port this project to Python 3, perhaps create an issue saying so, with the tag help-wanted.

[stamparm]

I'll just leave this here. It is not an argument for anything, just to get the clear picture of current situation in Python world.

Reference: https://hynek.me/articles/python3-2016/

[stamparm]

@Flimm sqlmap currently has around 64K LoC (excluding comments and blank lines). Running simple 2to3 is not an option here (if you think that 2to3 is enough then you are utterly wrong). As I can see from other projects, estimates for conversion goes from 3-6 months on projects of this size. As I've solely done the unicode support here, I believe that I have a right to say that switching of unicode related stuff from Python 2 to Python 3 is one of the biggest obstacles for this whole switch. This is not only my opinion. Just Google around.

Let us all together wait for the 2020. I have a feeling that it will pass as any other (Python) year. I bet that nothing will change as there were couple of similar (failed) attempts in past couple of years.

[Flimm]

So you wouldn't welcome efforts to port this to Python 3?

[stamparm]

@Flimm efforts of porting sqlmap in one go? Impossible. sqlmap is a dynamic project with steady commits. Making a parallel branch and keeping up the pace with HEAD Python 2 changes would be a Sisyphean task. Also, there are parts of sqlmap that maybe couple of non-sqlmap developers would understand. If a converter (person doing the conversion) would get stuck inside, he'll have a hell of a time.

My conclusion. Let's wait for the 2020. If Python 2 will die I promise that I'll do the whole conversion thing - as I believe that nobody else would be able to do it.

[Flimm]

OK. Fair enough. I trust your experience in this project.

[liwt31]

I'm also not arguing something, just to leave another picture, which I found in https://hynek.me/articles/python3-2016/:

[stamparm]

@liwt31 that article is excellent. Everybody should take 5 minutes and read it. IMO your picture is way out of context :). Article explains in a great way what's the status of old and newly started projects.

One important part from me. Python3 has been introduced in a bad way. It has split the community and introduced more problems than benefits. Now, whoever forces changes of big Python2 projects to Python3 is IMO not a programmer as programmers wouldn't do that kind of stuff to other fellow programmers. It's like forcing usage of objects and C++ to established C Linux/FOSS projects.

Already made a statement. If Python2 is going to be killed I'll convert/rewrite everything to Python3. Up until then new Python programmers can live in a lie that Python3 HAD to be hard forked from Python2

[armudgal]

Well, I am also currently porting a project from python2 to python2 and it has 64kLoC, and it's a real pita.
Although I started with making sure there is a test suite as extensive as possible so that the port becomes super comfortable.

[stamparm]

Done (https://twitter.com/sqlmap/status/1126092034274734080)