Publish wheel on PyPI

[SidneyGuenther]

@fredrikaverpil I found a post about publishing azure artifacts to pythons package index.

PyPI Support for Azure Artifacts

Is this available for us yet?

[fredrikaverpil]

I believe so. But it could become difficult to maintain this unless we can build "manylinux" wheels: https://github.com/pypa/manylinux

[fredrikaverpil]

Another question is if we should really take on the responsibility to offer this package on PyPi... especially if we decide to call it oiio, in which case I think we should get the oiio devs involved on their thoughts on this too as it would make things "official".

I would like to first build wheels for e.g. Ubuntu/CentOS, macOS, Windows successfully and then decide on how big of a challenge and how time consuming it will be to maintain this.

We can always attach wheels to GitHub releases (on trigger such as successful build with tag), which will offer a way to pip install <URL-TO-WHL>. I would personally prefer if we do it this way for some time so that we can more evaluate whether a PyPi publish would be a smart thing to do.

[shdwdln]

Hey hey! I really appreciate your work here.
Do you happen to have a build for CentOS?

[fredrikaverpil]

Hi, thanks for the nice words!

I think the Ubuntu ones should work on CentOS too.

Unfortunately, I haven't worked much on this project after having switched work places and I don't have any plans on updating the wheels myself.

[Correct-Syntax]

@fredrikaverpil, It's sad that you're no longer updating this. :(

I've forked your repo and have been converting it to run with Github Actions rather than Azure Devops. I have also talked to one of the oiio devs about producing python wheels for oiio. Basically, it's a green light if we can get this all working.

@fredrikaverpil do you have any tips or anything you think I should know when doing this?

It seemed like you edited vcpkg ports somehow to make this work. Is there any "problems" creating manylinux wheels as you mentioned or would it be best to hold off on putting it on PyPI? I would also be thankful if you could basically share what problems you've had, what you've learned, etc so that I don't have to waste time making the same mistakes with this.

Thanks!

[fredrikaverpil]

@Correct-Syntax I am no longer updating this as I am no longer using oiio (no longer active as a vfx professional). But I'm glad you wish to pick this project up. 👍😊

@SidneyGuenther and I never got around attempting to produce "manylinux" wheels, and I have never done that before, so I'm actually not sure what that would entail for oiio.

When you have a successfully built installation of oiio, the python bindings can be used. So the idea of this project was to copy the built python bindings into a portable Python wheel - to make it easier to distribute oiio for use with Python only. The dream would be to simply pip-install the wheel and start coding without the need to build/install anything else.

@SidneyGuenther and I believed using vcpkg would help us take a few shortcuts so to more easily build oiio on multiple platforms, so that's why we opted for this approach. However, the oiio vcpkg port depends on other vcpg ports, which were either outdated or became updated in ways that would break the oiio vcpkg port. In summary, I don't think this project is easily maintained using vcpkg.

If I were you, I would talk directly with @lgritz and see if would be possible to extend their existing CI (travis, github) with building Python wheels based on the build they are doing now in their CI. It could be harder to supply Windows based wheels, as I don't think they're interested in Windows at all though.

[lgritz]

It could be harder to supply Windows based wheels, as I don't think they're interested in Windows at all though.

Not sure who you mean by "they." If you mean the OIIO project, we certainly are interested in supporting Windows better! We just lack the expertise to do it as well as we'd like because so many of us work in a Linux environment all day.

[Correct-Syntax]

Not sure who you mean by "they." If you mean the OIIO project, we certainly are interested in supporting Windows better!

This is similar to what I was going to mention. :) I know that the OIIO project has (some) support for windows and (obviously) wants to improve it.

I am personally not in need of windows wheels, but rather the linux ones as some windows wheels are at the "unofficial windows binaries": https://www.lfd.uci.edu/~gohlke/pythonlibs/#openimageio. Though, for "official" OIIO we obviously need both windows and linux wheels.

Perhaps, we could reach out to the person who builds these "unofficial wheels" to find out how to better support windows? Otherwise, @lgritz, you probably know what is involved for building from source on windows at least.

In summary, I don't think this project is easily maintained using vcpkg.

@fredrikaverpil, @SidneyGuenther and @lgritz , any alternatives you know of to do this?

[lgritz]

Otherwise, @lgritz, you probably know what is involved for building from source on windows at least.

No, no, this is what I try to keep explaining to people. I have never in my life programmed on Windows professionally (though DOS, yes!), and I have not personally owned a machine running Windows since the mid 1990's. From 2000-2001 at work I used a dual boot machine that stayed in Linux for all my real work, but for one hour a day I would boot Windows XP to play Counter-Strike with my work mates, and then immediately switch back to Linux for work again.

I can just barely keep from breaking things in our incredibly basic Windows CI, changing a line here or there, pushing a PR, and waiting 30 minutes for the CI system to show an error message for me to take my next guess, and sometimes I can guess right and the build completes. That is the extent of my Windows knowledge. I am more comfortable (and knowledgeable) on IRIX, HPUX, or even a VAX than I am in Windows. What version of Windows is current? 10? I've heard people talking about 10. Or is 11 the one everybody is using now?

I'm very much interested in good Windows support for this project, but I need somebody, or several somebodies, to really take ownership of the various Windows aspects to OIIO (and also OSL, if any of you are listening). Every minute I spend on it is not only going at 1% of the rate of progress that there would be if the task was done by somebody who actually knows Windows, but it's also preventing me from doing something else for which I probably really am the most qualified person. It's lose-lose.

[Correct-Syntax]

No, no, this is what I try to keep explaining to people. I have never in my life programmed on Windows professionally...

@lgritz I see. However, I am not much better at either windows or linux when it comes to building things from c++ source. I've done a bit of experimenting with C (graphics programming), but I don't have much experience with compilers, building c++, etc otherwise. I have more experience in web development than I do this.

I could probably do it with clear directions, but seeing how you're not able to give any help for windows I am not sure how to move forward on the windows-side. The only thing I can think of is to contact the author of the unofficial windows binaries page and see how he does the oiio wheels.

For Linux, between the two of us (if your willing to help me along) it could probably get done. Though, having just an official linux wheel wouldn't do very well. In that case, maybe I'd just try to build linux wheels for my own projects or something.

I can just barely keep from breaking things in our incredibly basic Windows CI, changing a line here or there, pushing a PR, and waiting 30 minutes for the CI system to show an error message for me to take my next guess, and sometimes I can guess right and the build completes.

You've almost perfectly described how I have been working on converting this oiio project from azure to github actions...

Every minute I spend on it is not only going at 1% of the rate of progress that there would be if the task was done by somebody who actually knows Windows, but it's also preventing me from doing something else for which I probably really am the most qualified person. It's lose-lose.

I hear you. However, I'm not sure I am the most qualified person for building oiio wheels in the first place.

I just really need/want them for my project and it would be great to open oiio to the Python community via python wheels (which, as of now -it's only open to those who want the pain of building from source -or windows users, ironically!).

I am willing to help the best I can. We just need to get the wheels rolling.... :)

Bright ideas anyone?

[fredrikaverpil]

It could be harder to supply Windows based wheels, as I don't think they're interested in Windows at all though.

Not sure who you mean by "they." If you mean the OIIO project, we certainly are interested in supporting Windows better! We just lack the expertise to do it as well as we'd like because so many of us work in a Linux environment all day.

@lgritz Oh, sorry 😅 I first looked at the travis.yml file and and it didn't have any Windows specifics in it. So when I wrote that I assumed you primarily tested for (and targeted) Linux. And I realize that statement came out much harsher than I intended.

But then I actually did find the github actions CI too. I edited my post to include a link to it but forgot to also edit that statement too.

[lgritz]

The travis is the old stuff. I should just get rid of that file. We haven't relied on it in a long time.

[lgritz]

The Windows CI for GitHub Actions was just recently made to kinda work. We are still failing several tests and are working our way through the fixes.

[fredrikaverpil]

I am personally not in need of windows wheels, but rather the linux ones

@Correct-Syntax why not begin there then?
You could start with extending the official CI with also copying the required files into a Python wheel.

Having a working Linux wheel is a great start.

[Correct-Syntax]

@fredrikaverpil, yes, that is probably the best route. :)

[Correct-Syntax]

@fredrikaverpil after some trial and error I finally have a working openimagio lib for python and somewhat understand what's going on. I used vcpkg and it seems to work fine. I know you mentioned having issues with vcpkg, so I am wondering if you could tell me a little bit more specifically what issues there were with using it? Maybe it is now better supported than when you were working with it?

[fredrikaverpil]

Nice!

From what I recall, outdated/incompatible dependencies were pulled in via vcpkg, which is why we created some ports.
But then also I found wheels built on e.g. MacOS to not function properly, where errors would be thrown when using very basic operations.

[Correct-Syntax]

From what I recall, outdated/incompatible dependencies were pulled in via vcpkg, which is why we created some ports.

Thank you. I will have to look into that.

But then also I found wheels built on e.g. MacOS to not function properly, where errors would be thrown when using very basic operations.

Okay. Well, I'm unable to test on Mac since I don't have one. But someone recently used brew to install openimageio and the python operations seemed to work (for them, at least).