New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Python files end up in /usr/share (or similar) #962
Comments
Why do you say the .py files aren't platform independent? Or do you mean the .pyc that get generated (at least for me) when they are loaded / run, but which aren't distributed with FreeOrion...? |
The python code in FreeOrion isn't a third party python module with public interface. It isn't intended to be used by third party programs. So I don't see a point in putting it into the "site-package" directory. However, I do see your point regarding the python cache. I will investigate how this is handled in the common distributions. |
What are the potential issues that could actually arise when leaving our Python scripts in FO's data folder? |
Because it's code. And python code is not guaranteed to be platform independent (the same for perl: there are in fact some perl scripts that do not work on all platforms).
No, the bytecompiled .pyc files are as platform independent as the .py files. But there is simply no guarantee, which is why...
... it is FHS incompatible, which refers to |
The cache is VM dependent, not platform dependent, unless the code is. As in: .pyc files do not impose additional platform incompatibility. |
As said, even |
So… there is nothing to do after all? http://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch04s11.html
https://bytes.com/topic/python/answers/25152-pyc-pyo-architecture-independent
Lets see what Debian defines as packaging policy: http://debian-python.readthedocs.io/en/latest/debian-policy.html#types-of-python-modules
http://debian-python.readthedocs.io/en/latest/debian-policy.html#program-shipping-private-modules
Fedora doesn't explicit state anything regarding the the install directory for private modules, but notes
|
You quoted FHS:
Python files are not data files and are not strictly architecture independent.
See my previous comment, I already cleared that up.
Debian is not the mother of all packaging policies. On distros like gentoo or exherbo this is a (minor) QA violation. |
Neither the python scripts nor the cache are cpu architecture dependent as several python developers confirmed in the linked thread. If you consider python scripts as data or not is up to your personal interpretation. But the point is: there is no architecture dependent code inside the FreeOrion python scripts, so it doesn't belong into /usr/lib.
Then please point out the gentoo policy which defines that python scripts must be located in /usr/lib. |
I don't think that's true. Python scripts are runnable code, that may or may not be architecture independent. I think "data could be everything" is what's a personal interpretation.
Sure, that might be true, which is why I don't consider this a big issue. However, you could say the same about perl, ruby, etc and put them all across the whole filesystem, because they might be architecture-dependent or not. So this is also about consistency... anything that technically can be architecture-dependent (doesn't matter if it actually is), has no place in
I've been a gentoo developer for over 4 years, so this is basically IME. But I've also talked to @mgorny about it, who is part of the Gentoo QA team and the Gentoo python team. Also, as an exherbo contributor, I can assure you they don't like it either. Debian also sees |
The FHS is clear here:
Python scripts are neither object files or libraries. They are not arch dependent so they don't belong there.
Other distributions exactly split packages in arch dependent and non depending on arch, where the arch dependent 'scripts' are just glue to a library also located in the same lib directory or just the the so file itself. Example: perl5, ruby. Also hyperboles about 'all over the filesystem' won't help your point.
Again, there IS no architecture dependent code in the FreeOrion python scripts.
Yeah, for python code that is partly implement as python c extension (like the previously mentioned 'glue' for other script languages), but not for python scripts in general. |
Please read the whole FHS, which even points out the common existence of perl files under
I don't even know what you mean by that. Do you want to debate filesystem layout (that's not the same as FHS) and multilib/multiarch approaches and all the relevent implications of that?
It was a consistency argument, which also has side-effects on distro-internal mechanisms. I'm not sure I want to bore you with that.
This is about consistency and guarantees. I don't know why we are arguing so strong about this while even debian sees
Yes, also for python scripts in general. You can have platform-dependent code. |
Also, I think you are fundamentally misunderstanding something. This is not about "oh, you did it wrong". This was rather a request to do it differently. Afais, all distros would be fine with |
Yes, binaries. Python scripts are not binaries. They may be executable but they are not binaries.
That's the whole point of the discussion. You're talking about 'arch-dependent' or 'platform-dependent' which is nothing else than having a binary containing some cpu-architecture dependent machine instructions written in some format like ELF. Python scripts are not cpu-architecture dependent (as I pointed out by quoting from the python mailing list) so they don't belong into /usr/lib (unless you're writing an extension, which again, contains some cpu-architecture dependent binary in ELF format or similar). You pointed out that perl and ruby place scripts into /usr/lib/ and I countered that this is only true for scripts, that directly interact with some cpu-architecture dependent binary (like an so file). Other python or perl scripts that don't interact with some cpu-architecture dependent binary don't need to be placed in /usr/lib and are placed in /usr/share by other distributions (like, as I pointed out, Debian or Fedora). The FreeOrion python code does not depend cpu-architecture, you can swap out the freeorion binaries with the arch-dependent version you want to use and the python scripts would still work. So there is no need to place it in /usr/lib/.
I didn't interpret it as 'you did it wrong', but the current packaging doesn't violate FHS. You're asking for some files to be placed into a directory they don't belong.
If your distribution doesn't allow the usage of '/usr/share' in general, regardless of what FHS states (e.g. |
No, that's not what all distros do. Some stuff perl/ruby scripts consistently into Sure, you can say you don't care. But there are reasons to do that consistently and it is FHS compatible. Also, I'm not interested in discussing distro decisions with you (that are FHS compatible) and I'm not going to explain them in detail to you either.
No, I am asking to put them into an alternative directory (or better: provide an option, so one can choose where they end up), which is FHS compatible with python scripts (see the debian policy you just linked yourself), which makes life easier for a broader spectrum of distros. If you said "well, if you can provide a PR, please do, but we are not going to work on it"... I could very well understand. Which is why I opened this issue. Instead, I get a massive amount of semi-hostile nitpicking, which pretty much killed my motivation to create such a PR now. |
@hasufell, let me chime in here, maybe I can add another aspect that needs to be considered here.
Aside from what @adrianbroher mentioned, I want to add: because moving our Python scripts to FreeOrion's design expects all user customizable (other games refer to that as "moddable") content (graphics, sound, scripts defining game content, texts/translations, etc.) in one central location, which is usually referred to as the resource directory (the default/vanilla contents of that resource directory can be found in the default folder). FOs Python scripts are part of that. Ripping those out and placing them in a different folder obviously makes things less convenient for people who want to customize the game content (called "modders" in other games, we refer to them as "content scripters"), but that's actually a very minor issue here. Another (also minor, but still) issue is, having this centralized resource directory is handled the same way across all platforms, which of course makes it very easy for content scripters to exchange customized resource directories. If we split off the Python scripts on Linux, that gets needlessly complicated. But the far more serious issue is that the centralized location approach would be broken. All parts of the game content (graphic and sound files, content scripts, Python scripts etc.) are dependent on each other. Currently a player can easily switch to a different resource directory (that can contain completely different game content) by just selecting it in the options dialog. As all the customizable stuff is located in one folder, nothing can get mixed up when switching to another resource directory. Now consider what happens if you take a part of the customizable content out of the location where all the rest resides, and put it into a different location. First of all, you need to have two independent configuration settings, for two different locations that contain game content stuff. Both need to be accessible/changeable in the options dialog. And all that needs to be handled by the code, which of course gets more complicated. Nothing that can't be done, but still additional effort and more complex code (which means more prone to bugs). Now, if players switch one of the resource directories, they also might need to switch the other. If a player fails to select two resource directories that are compatible with each other, the game won't work. To avoid unpredictable behaviour by the app, and to provide the player with some error messages that give them a clue what actually went wrong, we need to write more code to properly detect and handle those cases. I think at that point you get an idea where this is heading. And all this to accomodate what in my eyes is an overly strict policy of some Linux distros. Even you said that you don't consider this a big issue, especially since our Python scripts are in fact 100% platform independent, they have to be. I'm well aware that you can write Python code that is not platform independent, there is quite some stuff in the standard library that behaves differently on different platforms, or even won't work on all platforms. We specifically avoid using any of that, and if something slips in, it will be considered a bug and addressed accordingly. But please understand that an open source project like ours, which has only very limited resources has to draw the line somewhere when it comes to the question how far we can go to accomodate the myriad of different requirements/policies/recomendations on how to do various things on all the different platforms (and flavors thereof) we try to support. And IMO in this case accomodating the specific policy in question here (and I deliberatly don't want to argue the point if you or @adrianbroher is right as far as this policy is concerned, I just assume now it is how you presented it) isn't worth the effort required and the potential for bugs and issues that will be introduced. I appreciate your intention to bring to our attention to something we probably could do a bit better, but I really don't want to open that can of worm for a minor issue like this. |
Please try to see @adrianbroher's replies not as semi-hostile, he's just arguing (maybe a bit strongly) against something he obviously perceives as an overly strict interpretation of a policy he understands differently (if rightly so or not is a different question I don't want to try to answer, because honestly I don't know). I mean, you have to admit that requiring to put something that is not platform dependent in a location that is intended for platform dependent stuff just because it could potentially be platform dependent is a bit strict. And as I explained above, this isn't always as simple as just storing a few files in a different directory. In our case a PR would be far from sufficient to address all the issues such a change would introduce. See my explanaitions above. I hope we all can calm down a bit and return to a friendlier tone 😃 |
Sure, I can understand your points and they are all valid. At least you're not trying to lecture me on distro related topics. Maybe you should double-check how some of your contributors influence the environment, because I'm definitely not going through this kind of hostile discussion again. For any patch. |
In addition to what @Vezzra just said, speaking entirely as a scripter/modder, some of the Python files in FO specifically are data files, they're deliberately setup to be very easily changeable by a player without needing to understand even the basics of Python. In addition, there's a serious discussion within the project to move our existing data files from the bespoke FOCS language created for the project into Python, making a huge amount more of our Python content data, not script or even really code. As someone who switched between a Linux desktop and a Windows laptop, sharing specific amendments between them, but not a high end dev (I can barely read Python let alone write complex scripts), the idea that on different machines/platforms the shared content would be in different places is simply a REALLY bad idea. Specifically one of our most important AI scripters uses a Github repository for his Default folder, allowing people to point to a folder online or an unpacked zip, etc is a key aim of the project, that this can be done is a very useful feature. |
Well, I hope you can give that decision another thought, and not let one discussion that got a bit out of hand discourage you from contributing to a project you apparently have been interested in enough to start providing some input. As far as @adrianbroher is concerned, he is not just a contributor, but a core team member who's contributions and efforts have been very important and valuable so far. And although we have our moments where we get into more intense arguments, so far we all have managed to get along well enough. We usually manage to maintain a friendly climate, and I hope we can hold on to that... |
Come up with better arguments. Starting with "the code is arch-dependent", which is not true, and finishing with "that's how we do things in gentoo, but screw you I won't explain why" is just not conclusive. |
I've been an opensource contributor long enough to know when I meet toxic people (not you). Good luck with your project. |
To be fair, that's not what he said/meant, at least not how I understood what he was trying to say. His point is: Python scripts are code (which is basically true, I mean, Python is a programming language), and because Python scripts can be platform dependent, all Python scripts (regardless if a specific script actually is platform dependent) belong in And that's not just his personal view/preference/taste, but from the way he presented it I got the impression he knows what he is talking about, and apparently on the Linux distros he referred to that policy actually is that strict. That's why he perceived your answers as hostile, I hope you can try to see the entire discussion through his eyes a bit (I mean, if I'm a Gentoo dev and know how we do things in Gentoo, and then someone wants to tell me differently I'll probably get a bit irritated too). Although I also admit that "hostile" takes it too far, and the term "toxic people" in his last post was uncalled-for. Of course one can raise the question if such a strict policy is reasonable (it strikes me as overly strict too, but I know too less about all that to give a strong opinion here), but even if we don't agree, why not just leave it at that? He didn't make any demands, he just made a suggestion and even pointed out that it's not a big issue if we don't go with it. Sometimes you just have to let it go... 😉 (Please see my post just as an attempt to calm things down, I don't want to lecture, criticize or offend anyone, I just try to see both sides here... I hope you can understand that, even if you don't agree to all I said.) |
Python files (e.g. https://github.com/freeorion/freeorion/tree/27549f6bd4b121a2ed568e763828f343dde790d7/default/python/AI) end up in the data directory of freeorion.
Python files are not strictly platform independent, so distros usually assume they end up somewhere in e.g.
/usr/lib/python2.7
or/usr/lib/python2.7/site-packages/
.There seems to be no clean cmake-idiomatic way of getting the site-packages dir, so one must probably use hacks like https://stackoverflow.com/questions/1242904/finding-python-site-packages-directory-with-cmake
Even
/usr/lib/freeorion
or so could be seen as appropriate for python files.The text was updated successfully, but these errors were encountered: