http.server - what is executable on Windows

[vpython]

BPO	10483
Nosy	@loewis, @orsenthil, @merwok, @ethanfurman, @MaxwellDupre
PRs	bpo-10483: http.server - what is executable on Windows #29624 gh-54692: fix definition of executable file for http.server on windows #30216

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2010-11-21.07:26:29.039>
labels = ['type-bug', 'library', '3.11']
title = 'http.server - what is executable on Windows'
updated_at = <Date 2021-12-20.16:42:03.912>
user = 'https://bugs.python.org/vpython'

bugs.python.org fields:

activity = <Date 2021-12-20.16:42:03.912>
actor = 'mikecmcleod'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2010-11-21.07:26:29.039>
creator = 'v+python'
dependencies = []
files = []
hgrepos = []
issue_num = 10483
keywords = ['patch']
message_count = 22.0
messages = ['121875', '121883', '122062', '122098', '122102', '122108', '122208', '405399', '405413', '405428', '405464', '405465', '405472', '406077', '406102', '406117', '406539', '406541', '406555', '406563', '406567', '407140']
nosy_count = 6.0
nosy_names = ['loewis', 'orsenthil', 'eric.araujo', 'v+python', 'ethan.furman', 'mikecmcleod']
pr_nums = ['29624', '30216']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue10483'
versions = ['Python 3.11']

[vpython]

The def executable for CGIHTTPRequestHandler is simply wrong on Windows. The Unix executable bits do not apply.

Yet it is not clear what to use instead. One could check the extension against PATHEXT, perhaps, but Windows doesn't limit itself to that except when not finding the exactly specified executable name. Or one could require and borrow the unix #! convention. As an experiment, since I'm mostly dealing the script files, I tried out a hack that implements two #! lines, the first for Unix and the second for Windows, and only consider something executable if the second line exists. This fails miserably for .exe files, of course.

Another possibility would be to see if there is an association for the extension, but that rule would permit a Word document to be "executable" because there is a way to open it using MS Word.

Another possibility would be to declare a list of extensions in the server source, like the list of directories from which CGIs are found.

Another possibility would be to simply assume that anything found in the CGI directory is executable.

Another possibility is to require the .cgi extension only to be executable, but then it is hard to know how to run it.

Another possibility is to require two "extensions"... the "real" one for Windows, and then .cgi just before it. So to make a program executable, it would be renamed from file.ext to file.cgi.ext

But the current technique is clearly insufficient.

[loewis]

I think it's fairly clear what should be considered executable in this context: anything that subprocess.Popen is able to launch (because if is_executable succeeds, this is what will happen to the file). This excludes word files, and also excludes files ending in .cgi; only files ending in .exe are executable.

[vpython]

Martin, that is an interesting viewpoint, and one I considered, but didn't state, because it seems much too restrictive. Most CGI programs are written in scripting languages, not compiled to .exe. So it seems the solution should allow for launching at least Perl and Python scripts, as well as .exe. Whether subprocess.Popen can directly execute it, or whether it needs help from the registry or a #! line to get the execution going is just a matter of tweaking the coding for what gets passed to subprocess.Popen. Declaring the definition based on what the existing code can already do is self-limiting.

Another possible interpretation of executable might be PATHEXT environment variable, but that is similar to declaring a list in the server source, which I did mention. One might augment the other.

[loewis]

Martin, that is an interesting viewpoint, and one I considered, but
didn't state, because it seems much too restrictive. Most CGI
programs are written in scripting languages, not compiled to .exe.
So it seems the solution should allow for launching at least Perl and
Python scripts, as well as .exe.

Notice that it does support launching Python scripts. I disagree that
Perl scripts need to be supported. The idea of CGI is really that
"programs" get run by the web server, with the notion of "programs"
clearly deviating from system to system. Window really doesn't support
"scripts" (in the hash-bang sense), and it isn't the function of
http.server to extend Windows here. At best, support for .bat files
might be negotiable, using cmd.exe to launch them (but I personally
would not).

Anybody who wants support for other kinds of scripts on Windows will
have to subclass CGIHTTPRequestHandler (and it might be useful to
simplify subclassing that class).

In any case, the bug as stated ("def executable is simply wrong")
has a clear resolution - make it match what the rest of the code
supports. Anything beyond that is a feature request.

[vpython]

The rest of the code has clearly never had its deficiencies exposed on Windows, simply because executable() has prevented that. So what the rest of the code "already supports" is basically nothing. Reasonable Windows support is appropriate to implement as part of the bugfix.

You state that it isn't the function of http.server to extend Windows, however, even MS IIS has extended Windows to provide reasonable web scripting functionality, albeit it its own way, thus convicting the Windows facilities of being insufficient. Attempting to use http.server to get a web testing environment running so that Python scripts can be tested locally requires some way of using an existing environment (except, of course, for "all new" web sites). I suppose you would claim that using http.server for a web testing environment is an inappropriate use of http.server, also.

Yet http.server on Unix appears to provide an adequate web testing environment: yes, some of that is because of Unix's #! feature. This would certainly not be the first case where more code is required on Windows than Unix to implement reasonable functionality.

My desire for support for Perl is not an attempt to convince Python developers to use Perl instead of Python, but simply a reflection of the practicality of life: There are a lot of Perl CGI scripts used for pieces of Web servers. Reinventing them in Python may be fun, but can be more time consuming than projects may have the luxury to do.

Your claim that it already supports Python CGI scripts must be tempered by the documentation claim that it provides "altered semantics". "altered semantics", as best as I can read in the code, is that the query string is passed to the Python script as a command line if it doesn't happen to contain an "=" sign. This is weird, unlikely to be found in a real web server, and hence somewhat useless for use as a test server also.

http.server has chosen to use subprocess which has chosen to use CreateProcess as its way of executing CGI. There are other Windows facilities for executing programs, such as ShellExecute, but of course it takes the opposite tack: it can "execute" nearly any file, via registry-based associations. Neither of these seem to be directly appropriate for use by http.server, the former being too restrictive without enhancements, the latter being too liberal in executing too many file types, although the requirement that CGI scripts live in specific directories may sufficiently rein in that liberality.

However, you have made me think through the process: it seems that an appropriate technique for Windows is to allow for a specific set of file extensions, and permit them to be executed using the registry-based association to do so. However, for .cgi files, which depend heavily on the Unix #!, emulation of #! seems appropriate (and Windows doesn't seem to have an association for .cgi files either).

Your suggestion of making CGIHTTPRequestHandler easier to subclass is certainly a good one, and is almost imperative to implement to fix this bug in a useful manner without implementing an insufficient set of Windows extensions (for someone's definition of wrong). There should be a way to sidestep the "altered semantics" for Python scripts (and Python scripts shouldn't have to be a special case, they should work with the general case), without replacing the whole run_cgi() function. There should be a hook to define the list of executable extensions, and how to run them, and/or a hook to alter the command line passed to subprocess.Popen to achieve same.

So is_executable and is_python both seem to currently be replacable. What is missing is a hook to implement cmdline creation before calling subprocess.Popen() (besides the other reported bugs, of course)

[loewis]

Your suggestion of making CGIHTTPRequestHandler easier to subclass is
certainly a good one, and is almost imperative to implement to fix
this bug in a useful manner without implementing an insufficient set
of Windows extensions (for someone's definition of wrong).

It's indeed the approach I would prefer over the alternatives you
suggested - I particularly dislike Python implementing a strategy
where #! files become considered on Windows (you then immediately
run into subsequent problems, such as /usr/bin/perl being no valid
filename on most Windows installations).

So I maintain that technically, in order to resolve the *reported*
issue (msg121875), it is sufficient to define that executables
on Windows are the files ending with .exe. To recall, the reported
issue is "is simply wrong ... is not clear what to use instead"
(to you as the reporter). My job as a maintainer is to resolve this,
and I will decide to resolve this as suggested. Even the refactoring
to allow substitution of process creation is an independent feature,
but I'm willing to accept patches.

[vpython]

Martin, you are splitting hairs about the "reported problem". The original message does have a paragraph about the executable bits being wrong. But the bulk of the message is commenting about the difficulty of figuring out what to replace it with.

So it looks like in spite of the hair splitting, we have iterated to a design of making run_cgi a bit friendlier in this regard.

I find it sufficient to define a method fully extracted from run_cgi as follows:

    def make_cmdline( self, scriptfile, query ):
        cmdline = [scriptfile]
        if self.is_python(scriptfile):
            interp = sys.executable
            if interp.lower().endswith("w.exe"):
                # On Windows, use python.exe, not pythonw.exe
                interp = interp[:-5] + interp[-4:]
            cmdline = [interp, '-u'] + cmdline
        if '=' not in query:
            cmdline.append(query)

This leaves run_cgi with:

            import subprocess
            cmdline = self.make_cmdline( scriptfile, query )
            self.log_message("command: %s", subprocess.list2cmdline(cmdline))

Apologies: I don't know what format of patch is acceptable, but this is a simple cut-n-paste change. I was sort of holding off until the hg conversion to figure out how to make code submissions, since otherwise I'd have to learn it twice in short order.

I have reimplemented my work-arounds in terms of the above fix, and they function correctly, so this fix would suffice for me, for this issue. (N.B. I'm sure you've noticed that I have entered a number of issues for http.server; I hope that was the right way to do it, to attempt to separate the issues.)

[MaxwellDupre]

Hi,
I would like to help on this issue.
Let me know what can be done?

[merwok]

I don’t know if CGI on windows servers is very relevant nowadays.

[MaxwellDupre]

Hi Éric,

Then would the easiest way of dealing with this issue, to close it without
any further work?
Possibly adding to the documents it doesn't work on windows..
I can move to the next issue.

Regards,
Mike

On Sun, 31 Oct 2021 at 18:03, Éric Araujo <report@bugs.python.org> wrote:

Éric Araujo <merwok@netwok.org> added the comment:

I don’t know if CGI on windows servers is very relevant nowadays.

----------
nosy: +eric.araujo
versions: -Python 3.2

---

Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue10483\>

---

[vpython]

As the original author of this report, getting significant pushback from the then maintainer of the code, I went ahead and implemented what I considered to be a useful set of features to make CGI work on Windows.

I never proposed them as a patch, because they were bundled with a bunch of other things that certainly would have been considered feature requests, and I never figured out the process for submitting patches, which has changed a couple times since the bug was submitted, and I didn't have time to do all of
(1) learn the churning patch mechanisms
(2) separate my code changes into independent chunks
(3) also get my projects completed

The only reason that CGI support on Windows may not be relevant today is that it is so outdated, with no support or encouragement from the maintainers to encourage patches to make it useful, discouraging its use on Windows. With my private, forked implementation, I have a quite useful Windows testbed for implementing web server code that can run on Linux Apache servers. If only there were such support built-in!

[ethanfurman]

Glen, which version of Python is your fork using?

[vpython]

Ethan, 3.7.2 at the moment. I doubt there'd be any issues moving it to 3.11, unless there have been significant changes to the main branch in that area, I just haven't bothered because it is working for me as is. I forget what version I started with, but I guess 3.6 was when I first reported it? I know I did have to tweak it a little when I upgraded Python once, and there had been changes to http.server in the meantime.