Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Windows support #524

Open
jezdez opened this issue May 1, 2013 · 108 comments
Open

Add Windows support #524

jezdez opened this issue May 1, 2013 · 108 comments
Assignees
Labels
- Mailing List - Feature/Core help wanted Open for everyone. You do not need permission to work on these. May need familiarity with codebase. Improvement Platform/windows

Comments

@jezdez
Copy link
Contributor

jezdez commented May 1, 2013

This is related to the things we discussed at PyCon

@ghost ghost assigned benoitc May 27, 2013
@benoitc
Copy link
Owner

benoitc commented May 27, 2013

wip. planned for R18.

@benoitc
Copy link
Owner

benoitc commented Aug 9, 2013

Some update on that:

  • SOCKET support is quite done, a branch will be posted online next week

TODO:

handle service start/stop/restart to use gunicorn as a full windows service. It is replacing somehow the use of signals under unix.

@benoitc
Copy link
Owner

benoitc commented Oct 31, 2013

the support will finally happen in R20. i will create the new PR to track the development of this feature next week.

Things to fix are:

  • adapt Worker notifications to windows
  • port the signaling to windows: reload and reexec
  • make sure sockets can be inherited among processes

@renierdbruyn
Copy link

What is the progress on adding windows support for gunicorn?. Or rather How far are you from R20? I am having the same problem as #587

@benoitc
Copy link
Owner

benoitc commented Jan 25, 2014

Slow progress. I want to release R19 with the threaded work now. I will push a version on monday. the windows support will arrive right after.

@helmus
Copy link

helmus commented Feb 7, 2014

Is this windows supported version available somewhere ?

@renierdbruyn
Copy link

Windows support will be in R20, so I guess it will be a while till guincorn will have windows support... I created a vm with Ubuntu, and this works for me.....

@robertlagrant
Copy link

Just curious as to any progress in this. As a first pass it doesn't even need to be fast; a big advantage will be just being able to develop with Gunicorn on Windows and push to a Linux server (e.g. Heroku) without changing config.

@benoitc
Copy link
Owner

benoitc commented Sep 22, 2014

@robertlagrant it's on my todo. I need to find a windows license first. R20 will be released in october,

@benoitc benoitc modified the milestones: R20, R20.0 Sep 22, 2014
@helmus
Copy link

helmus commented Sep 22, 2014

How about the VM images at modern.ie ? Those are free and legal
On Sep 22, 2014 3:30 PM, "Benoit Chesneau" notifications@github.com wrote:

@robertlagrant https://github.com/robertlagrant it's on my todo until
the end of the year. I need to find a windows license first.


Reply to this email directly or view it on GitHub
#524 (comment).

@tilgovi
Copy link
Collaborator

tilgovi commented Sep 22, 2014

I used modern.ie images to test a web app in IE8 recently. Easy to get going and worked great.

@cyberdelia
Copy link

@benoitc If needed, I believe the PSF can provide you with a windows license.

@berkerpeksag
Copy link
Collaborator

@benoitc is there a WIP branch for this? I'm not a Windows user, but I can take a look at this.

@benoitc
Copy link
Owner

benoitc commented May 12, 2015

@berkerpeksag no not yet :/ we should really start one asap.

@benoitc
Copy link
Owner

benoitc commented May 12, 2015

like @cyberdelia said we could ask a license of windows at the PSF that the gunicorn project could use for such development. Thoughts?

@berkerpeksag
Copy link
Collaborator

It's a good idea. https://www.python.org/psf/grants/ has all information about grant proposals.

@zet4
Copy link

zet4 commented Nov 14, 2015

Any updates on this?

@jamespedid
Copy link

I am also looking for updates to this.

@benoitc
Copy link
Owner

benoitc commented Nov 20, 2015

@berkerpeksag let's have a talk asap on IRC, just ping me sometimes before since i'm away from any instant thing these days :)

@benoitc benoitc removed this from the R20.0 milestone Dec 6, 2015
@johann-petrak
Copy link

Is there a rough idea how it could be done on Windows to support proper multiprocessing?
Do we know what the major obstacles are so far for this getting implemented?

@platinops
Copy link

i may have some cycle yo spend on it during march. I will need to find a way to execute windows somewhere. you can already run it in the terminal anyway.

Could the Win10 dev VM's be a solution for you to run Windows and check gunicorn compatibility?

@arihant2math
Copy link

arihant2math commented May 26, 2021

Perhaps we can use pywin32 for windows instead of fnctl on windows, it seems they have file locking capabilities .

@bjones1
Copy link

bjones1 commented Aug 28, 2021

I've written #2636 which fixes about half of the problems porting gunicorn to Windows. The other half requires finding a Windows-compatible alternative to using os.pipe. Could we simply use a multiprocessing listener and client instead of the existing os.pipe?

  • It doesn't require using any other part of multiprocessing, so it avoid that baggage/overhead.
  • It produces a Connection object, which allows sending raw bytes for efficiency.
  • It has a select-like function.
  • It's (of course) multi-platform.

@benoitc
Copy link
Owner

benoitc commented Aug 28, 2021 via email

@bjones1
Copy link

bjones1 commented Aug 28, 2021

@benoitc, thanks for your feedback. Do we need only inter-thread synchronization, or is this actually inter-process synchronization? I assume inter-process, since you mention sockets and inter-thread is well supported already by the Python library.

@benoitc
Copy link
Owner

benoitc commented Aug 28, 2021 via email

@bjones1
Copy link

bjones1 commented Aug 28, 2021

I would admit that windows is a really low priority for me. Gunicorn is
done first to be deployed on POSIX/UNIX platforms. That’s a desirable goal
anyway :)

Thanks for taking the time to look at this. I really appreciate it. I know your time is very limited.

But in any case we need a more native way to handle it close to windows
libraries/system rather than using anything from multiprocessing. So we
can understand how it works and optimise when needed without depending on a
third party. This should work until Python stops to expose low level
libraries.

I'm a bit confused here. multiprocessing is from the Python standard library, so it's not a third-party library.

You’re right I meant inter-process synchronisation (or system threads) .I
am pretty sure there is a signaling API that could do it.

That's what I thought, but I wanted to check...

@benoitc
Copy link
Owner

benoitc commented Aug 28, 2021 via email

@bjones1
Copy link

bjones1 commented Aug 28, 2021

multiprocessing is an abstraction over systems calls. I would really prefer
we handle directly native calls so we can change any time or use only the
part we need. If really needed we can also create a binding for the one not
exposed.

OK, sounds good. I'll look for signaling / socket IPC mechanisms and let you know what I find.

@tilgovi
Copy link
Collaborator

tilgovi commented Aug 31, 2021

I would love to use multiprocessing, if it's appropriate. These standard library abstractions are a great thing about the language, especially because the low level implementation details for CPython are often well specified in the documentation.

If multiprocessing, or any other standard module, implements the right windows platform features to do what we need, we should consider using it. If it doesn't, we won't.

@tilgovi
Copy link
Collaborator

tilgovi commented Aug 31, 2021

In the past, then I've read multiprocessing docs, I haven't been convinced! But I'd be open to being educated if it does seem appropriate.

I'm also open to measuring the impact if it turns out slower but vastly simpler.

@bjones1
Copy link

bjones1 commented Aug 31, 2021

@tilgovi, thanks for your feedback. Unless @benoitc disagrees, I'll pursue an initial implementation using multiprocessing. With a working, multi-platform solution, we can then stay with the standard library or customize this working code based on the feedback I receive. I'd appreciate any other thoughts / ideas on the architecture you have -- would you mind posting them on #2637? Thanks again.

@benoitc
Copy link
Owner

benoitc commented Aug 31, 2021

well, i hate to repeat myself. I would like to have a clean implementation not based on multiprocessing. It's not about optimisation but also in the process learning and make gunicorn a better citizen in windows.

Multiprocessing and gunicorn doesn't share the same philosophy in handling multiprocessing. It's OK however to use low level python bindings, some coming with the multiprocessing. But definitly not higher level. We have to be cautious. And definitly removing anything using shared memory.

@johann-petrak
Copy link

I do not see how something not based on a standard python module but some native library should be the "cleaner" solution?
Clearly, multiprocessing is a part of Python that will get maintained and receive bug fixes in the future while depending on some native library may easily break things in the future.
Sure it makes sense to restrict the implementation to a relevant part of the abstractions present in multiprocessing or deal with situations where something needed is missing from the module, but why reinvent the wheel?

@bjones1
Copy link

bjones1 commented Aug 31, 2021

I believe good code argues for itself. Would the gunicorn team be willing to look at an initial implementation based only on the inter-process communication components of multiprocessing (not the overall multiprocessing approach, which I also see as a poor fit for this project)? If this works, we can either manually edit the code (@benoitc's preference, I think), or rely on the ongoing maintenance and improvements provided by direct use of the Python standard library.

If not, I can look at a socket-based solution, or any of the alternatives listed in #2637.

@benoitc
Copy link
Owner

benoitc commented Aug 31, 2021

there are many reason to not use multiprocessing. One of them is to be able to work with other possible workers and provide a facility that work for all workers. It is also to ensure our maintenance window. Something really hard with multiprocessing. And I also like to not have depends on optinion of others about what is interprocess-communication.

My preference is the following:

  • Have al the code handling multiprocessing embed in Gunicorn. This allows easy checking, and isolation.
  • if C-code is needed then provide them directly in Gunicorn.
  • When the a stdlib implementation is safe (thread safe, not sharing memory between processes) then use it
  • Fork internally if needed for compatibility across version.

@bjones1 if you believe the code can be used for example to spawn workiers that include gevent and check their liveness we can go with it. Better to speak over the code anyway.

@bjones1
Copy link

bjones1 commented Aug 31, 2021

@benoitc, my only goal is to provide inter-process communication code. I don't want to change or even touch the way gunicorn spawns/manages processes. As you say, I'll write code and listen to your feedback. A quick note: I'm @bjones1; sadly, @bjones claimed that userid before I joined github :(.

@tilgovi
Copy link
Collaborator

tilgovi commented Aug 31, 2021

This is great. I see a lot of alignment. I'll give feedback on the issue you opened. Nice to see movement on this.

I agree with everyone so far. It is good to be in control, avoid high level abstractions that don't do exactly what we need, but also not reinvent the wheel. So we will evaluate deeper, together, as we look at real code possibilities.

@bjones1
Copy link

bjones1 commented Aug 31, 2021

I've started work on this in #2643. Comments welcome.

@Andrioden
Copy link

Any news here? : )

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
- Mailing List - Feature/Core help wanted Open for everyone. You do not need permission to work on these. May need familiarity with codebase. Improvement Platform/windows
Projects
None yet
Development

No branches or pull requests