Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python files end up in /usr/share (or similar) #962

Closed
hasufell opened this issue Sep 9, 2016 · 23 comments
Closed

Python files end up in /usr/share (or similar) #962

hasufell opened this issue Sep 9, 2016 · 23 comments
Assignees
Labels
category:bug The Issue/PR describes or solves a perceived malfunction within the game. component:build system The Issue/PR deals with CMake, Visual Studio or the general build process. component:internal The Issue/PR deals with any project component that has no explicit `component` label. status:invalid The Issue is not classified as a formal valid Issue report or applicable PR.

Comments

@hasufell
Copy link

hasufell commented Sep 9, 2016

Python files (e.g. https://github.com/freeorion/freeorion/tree/27549f6bd4b121a2ed568e763828f343dde790d7/default/python/AI) end up in the data directory of freeorion.

Python files are not strictly platform independent, so distros usually assume they end up somewhere in e.g. /usr/lib/python2.7 or /usr/lib/python2.7/site-packages/.

There seems to be no clean cmake-idiomatic way of getting the site-packages dir, so one must probably use hacks like https://stackoverflow.com/questions/1242904/finding-python-site-packages-directory-with-cmake

Even /usr/lib/freeorion or so could be seen as appropriate for python files.

@geoffthemedio
Copy link
Member

geoffthemedio commented Sep 9, 2016

Why do you say the .py files aren't platform independent? Or do you mean the .pyc that get generated (at least for me) when they are loaded / run, but which aren't distributed with FreeOrion...?

@Vezzra Vezzra added category:request component:internal The Issue/PR deals with any project component that has no explicit `component` label. labels Sep 9, 2016
@Vezzra Vezzra added this to the post 0.4.6 milestone Sep 9, 2016
@adrianbroher
Copy link
Contributor

adrianbroher commented Sep 9, 2016

The python code in FreeOrion isn't a third party python module with public interface. It isn't intended to be used by third party programs. So I don't see a point in putting it into the "site-package" directory.

However, I do see your point regarding the python cache. I will investigate how this is handled in the common distributions.

@adrianbroher adrianbroher self-assigned this Sep 9, 2016
@Vezzra
Copy link
Member

Vezzra commented Sep 9, 2016

What are the potential issues that could actually arise when leaving our Python scripts in FO's data folder?

@hasufell
Copy link
Author

hasufell commented Sep 9, 2016

Why do you say the .py files aren't platform independent?

Because it's code. And python code is not guaranteed to be platform independent (the same for perl: there are in fact some perl scripts that do not work on all platforms).

Or do you mean the .pyc that get generated

No, the bytecompiled .pyc files are as platform independent as the .py files. But there is simply no guarantee, which is why...

What are the potential issues that could actually arise when leaving our Python scripts in FO's data folder?

... it is FHS incompatible, which refers to /usr/share as Architecture-independent data. However, code is not strictly architecture-independent, nor is it data. So this causes minor (yeah, this isn't a big thing) nuisance for distro packaging, depending on the strictness of the QA policies.

@hasufell
Copy link
Author

hasufell commented Sep 9, 2016

However, I do see your point regarding the python cache.

The cache is VM dependent, not platform dependent, unless the code is. As in: .pyc files do not impose additional platform incompatibility.

@hasufell
Copy link
Author

hasufell commented Sep 9, 2016

So I don't see a point in putting it into the "site-package" directory.

As said, even /usr/lib/freeorion would be better if you don't want it in the python searchpath.

@adrianbroher
Copy link
Contributor

So… there is nothing to do after all?

http://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch04s11.html

The /usr/share hierarchy is for all read-only architecture independent data files.
This hierarchy is intended to be shareable among all architecture platforms of a given OS; thus, for example, a site with i386, Alpha, and PPC platforms might maintain a single /usr/share directory that is centrally-mounted. Note, however, that /usr/share is generally not intended to be shared by different OSes or by different releases of the same OS.

https://bytes.com/topic/python/answers/25152-pyc-pyo-architecture-independent

I know the bytecode can change between interpreter
versions and other interpreters like Jython, Stackless, and
PyPy (does that exist yet?) may not even choose to
make them. But given that the same interpreter is made
available, will they work on, say, an ARM processor,
a 68K, and a i386 sharing them on the same network?

Yes. .pycs are marshalled code objects (mostly) so the notes in
https://docs.python.org/2/library/marshal.html apply.

Lets see what Debian defines as packaging policy:

http://debian-python.readthedocs.io/en/latest/debian-policy.html#types-of-python-modules

There are two ways to distribute Python modules. Public modules are installed in a public directory as listed in Module path. They are accessible to any program. Private modules are installed in a private directory such as /usr/share/package-name or /usr/lib/package-name. They are generally only accessible to a specific program or suite of programs included in the same package.

http://debian-python.readthedocs.io/en/latest/debian-policy.html#program-shipping-private-modules

A program using /usr/bin/python as the interpreter can come up with private Python modules. These modules should be installed in /usr/share/, or ``/usr/lib/ if the modules are architecture-dependent (e.g. extensions).

Fedora doesn't explicit state anything regarding the the install directory for private modules, but
https://fedoraproject.org/wiki/Packaging:Python_Appendix#Manual_byte_compilation

notes

Doing this is useful when you have a python3 application that's installing a private module into its own directory. For instance, if the foobar application installs a module for use only by the command line application in %{_datadir}/foobar. Since these files are not in one of the python3 library paths (ie. /usr/lib/python3.1) you have to override %{__python} to tell brp-python-bytecompile to use the python3 interpreter for byte compiling.

@hasufell
Copy link
Author

hasufell commented Sep 9, 2016

You quoted FHS:

The /usr/share hierarchy is for all read-only architecture independent data files.

Python files are not data files and are not strictly architecture independent.

https://bytes.com/topic/python/answers/25152-pyc-pyo-architecture-independent

See my previous comment, I already cleared that up.

Lets see what Debian defines as packaging policy:

Debian is not the mother of all packaging policies. On distros like gentoo or exherbo this is a (minor) QA violation.

@adrianbroher
Copy link
Contributor

Python files are not data files and are not strictly architecture independent.

Neither the python scripts nor the cache are cpu architecture dependent as several python developers confirmed in the linked thread. If you consider python scripts as data or not is up to your personal interpretation. But the point is: there is no architecture dependent code inside the FreeOrion python scripts, so it doesn't belong into /usr/lib.

Debian is not the mother of all packaging policies. On distros like gentoo or exherbo this is a (minor) QA violation.

Then please point out the gentoo policy which defines that python scripts must be located in /usr/lib.

@hasufell
Copy link
Author

hasufell commented Sep 9, 2016

If you consider python scripts as data or not is up to your personal interpretation.

I don't think that's true. Python scripts are runnable code, that may or may not be architecture independent. I think "data could be everything" is what's a personal interpretation.

But the point is: there is no architecture dependent code inside the FreeOrion python scripts

Sure, that might be true, which is why I don't consider this a big issue. However, you could say the same about perl, ruby, etc and put them all across the whole filesystem, because they might be architecture-dependent or not. So this is also about consistency... anything that technically can be architecture-dependent (doesn't matter if it actually is), has no place in /usr/share.

Then please point out the gentoo policy which defines that python scripts must be located in /usr/lib.

I've been a gentoo developer for over 4 years, so this is basically IME.

But I've also talked to @mgorny about it, who is part of the Gentoo QA team and the Gentoo python team.

Also, as an exherbo contributor, I can assure you they don't like it either.

Debian also sees /usr/lib/<package-name> as an appropriate directory, so if you switch to that directory for the python files, then it should fit all distros needs.

@adrianbroher
Copy link
Contributor

adrianbroher commented Sep 9, 2016

I think "data could be everything" is what's a personal interpretation.

The FHS is clear here:

/usr/lib includes object files and libraries.

Python scripts are neither object files or libraries. They are not arch dependent so they don't belong there.

However, you could say the same about perl, ruby, etc and put them all across the whole filesystem, because they might be architecture-dependent or not.

Other distributions exactly split packages in arch dependent and non depending on arch, where the arch dependent 'scripts' are just glue to a library also located in the same lib directory or just the the so file itself. Example: perl5, ruby.

Also hyperboles about 'all over the filesystem' won't help your point.

So this is also about consistency... anything that technically can be architecture-dependent (doesn't matter if it actually is), has no place in /usr/share.

Again, there IS no architecture dependent code in the FreeOrion python scripts.

Debian also sees /usr/lib/ as an appropriate directory, so if you switch to that directory for the python files, then it should fit all distros needs.

Yeah, for python code that is partly implement as python c extension (like the previously mentioned 'glue' for other script languages), but not for python scripts in general.

@hasufell
Copy link
Author

hasufell commented Sep 9, 2016

Python scripts are neither object files or libraries. They are not arch dependent so they don't belong there.

Please read the whole FHS, which even points out the common existence of perl files under /usr/lib/perl5 and the sentence On some systems, it may also include internal binaries that are not intended to be executed directly by users or shell scripts..

Other distributions exactly split packages in arch dependent and non depending on arch, where the arch dependent 'scripts' are just glue to a library also located in the same lib directory or just the the so file itself. Example: perl5, ruby.

I don't even know what you mean by that. Do you want to debate filesystem layout (that's not the same as FHS) and multilib/multiarch approaches and all the relevent implications of that?

Also hyperboles about 'all over the filesystem' won't help your point.

It was a consistency argument, which also has side-effects on distro-internal mechanisms. I'm not sure I want to bore you with that.

Again, there IS no architecture dependent code in the FreeOrion python scripts.

This is about consistency and guarantees. I don't know why we are arguing so strong about this while even debian sees /usr/lib/<package-name> as an appropriate directory for private python modules.

but not for python scripts in general

Yes, also for python scripts in general. You can have platform-dependent code.

@hasufell
Copy link
Author

hasufell commented Sep 9, 2016

Also, I think you are fundamentally misunderstanding something. This is not about "oh, you did it wrong". This was rather a request to do it differently. Afais, all distros would be fine with /usr/lib/<package-name>, but not all with /usr/share/<package-name> regardless of FHS, which most distros don't adhere to 100% anyway.

@adrianbroher
Copy link
Contributor

adrianbroher commented Sep 9, 2016

Please read the whole FHS, which even points out the common existence of perl files under /usr/lib/perl5 and the sentence On some systems, it may also include internal binaries that are not intended to be executed directly by users or shell scripts..

Yes, binaries. Python scripts are not binaries. They may be executable but they are not binaries.

I don't even know what you mean by that. Do you want to debate filesystem layout (that's not the same as FHS) and multilib/multiarch approaches and all the relevent implications of that?

That's the whole point of the discussion. You're talking about 'arch-dependent' or 'platform-dependent' which is nothing else than having a binary containing some cpu-architecture dependent machine instructions written in some format like ELF. Python scripts are not cpu-architecture dependent (as I pointed out by quoting from the python mailing list) so they don't belong into /usr/lib (unless you're writing an extension, which again, contains some cpu-architecture dependent binary in ELF format or similar). You pointed out that perl and ruby place scripts into /usr/lib/ and I countered that this is only true for scripts, that directly interact with some cpu-architecture dependent binary (like an so file). Other python or perl scripts that don't interact with some cpu-architecture dependent binary don't need to be placed in /usr/lib and are placed in /usr/share by other distributions (like, as I pointed out, Debian or Fedora).

The FreeOrion python code does not depend cpu-architecture, you can swap out the freeorion binaries with the arch-dependent version you want to use and the python scripts would still work. So there is no need to place it in /usr/lib/.

Also, I think you are fundamentally misunderstanding something. This is not about "oh, you did it wrong"

I didn't interpret it as 'you did it wrong', but the current packaging doesn't violate FHS. You're asking for some files to be placed into a directory they don't belong.

Afais, all distros would be fine with /usr/lib/, but not all with /usr/share/ regardless of FHS, which most distros don't adhere to 100% anyway.

If your distribution doesn't allow the usage of '/usr/share' in general, regardless of what FHS states (e.g. Miscellaneous architecture-independent application-specific static files and subdirectories must be placed in /usr/share.), then it's an issue of that distribution.

@hasufell
Copy link
Author

hasufell commented Sep 9, 2016

You pointed out that perl and ruby place scripts into /usr/lib/ and I countered that this is only true for scripts, that directly interact with some cpu-architecture dependent binary (like an so file). Other python or perl scripts that don't interact with some cpu-architecture dependent binary don't need to be placed in /usr/lib and are placed in /usr/share by other distributions (like, as I pointed out, Debian or Fedora).

No, that's not what all distros do. Some stuff perl/ruby scripts consistently into /usr/lib, regardless of whether they actually interact with a shared C library or not.

Sure, you can say you don't care. But there are reasons to do that consistently and it is FHS compatible.

Also, I'm not interested in discussing distro decisions with you (that are FHS compatible) and I'm not going to explain them in detail to you either.

I didn't interpret it as 'you did it wrong', but the current packing doesn't violate FHS. You're asking for some files to be placed into a directory they don't belong.

No, I am asking to put them into an alternative directory (or better: provide an option, so one can choose where they end up), which is FHS compatible with python scripts (see the debian policy you just linked yourself), which makes life easier for a broader spectrum of distros.

If you said "well, if you can provide a PR, please do, but we are not going to work on it"... I could very well understand. Which is why I opened this issue.

Instead, I get a massive amount of semi-hostile nitpicking, which pretty much killed my motivation to create such a PR now.

@Vezzra
Copy link
Member

Vezzra commented Sep 9, 2016

@hasufell, let me chime in here, maybe I can add another aspect that needs to be considered here.

I don't know why we are arguing so strong about this

Aside from what @adrianbroher mentioned, I want to add: because moving our Python scripts to /user/lib/freeorion on Linux isn't as trivial as it might seem to you. Actually that would open a can of worms for us. Let me try to explain:

FreeOrion's design expects all user customizable (other games refer to that as "moddable") content (graphics, sound, scripts defining game content, texts/translations, etc.) in one central location, which is usually referred to as the resource directory (the default/vanilla contents of that resource directory can be found in the default folder).

FOs Python scripts are part of that. Ripping those out and placing them in a different folder obviously makes things less convenient for people who want to customize the game content (called "modders" in other games, we refer to them as "content scripters"), but that's actually a very minor issue here.

Another (also minor, but still) issue is, having this centralized resource directory is handled the same way across all platforms, which of course makes it very easy for content scripters to exchange customized resource directories. If we split off the Python scripts on Linux, that gets needlessly complicated.

But the far more serious issue is that the centralized location approach would be broken. All parts of the game content (graphic and sound files, content scripts, Python scripts etc.) are dependent on each other. Currently a player can easily switch to a different resource directory (that can contain completely different game content) by just selecting it in the options dialog. As all the customizable stuff is located in one folder, nothing can get mixed up when switching to another resource directory.

Now consider what happens if you take a part of the customizable content out of the location where all the rest resides, and put it into a different location. First of all, you need to have two independent configuration settings, for two different locations that contain game content stuff. Both need to be accessible/changeable in the options dialog. And all that needs to be handled by the code, which of course gets more complicated. Nothing that can't be done, but still additional effort and more complex code (which means more prone to bugs).

Now, if players switch one of the resource directories, they also might need to switch the other. If a player fails to select two resource directories that are compatible with each other, the game won't work. To avoid unpredictable behaviour by the app, and to provide the player with some error messages that give them a clue what actually went wrong, we need to write more code to properly detect and handle those cases.

I think at that point you get an idea where this is heading. And all this to accomodate what in my eyes is an overly strict policy of some Linux distros. Even you said that you don't consider this a big issue, especially since our Python scripts are in fact 100% platform independent, they have to be.

I'm well aware that you can write Python code that is not platform independent, there is quite some stuff in the standard library that behaves differently on different platforms, or even won't work on all platforms. We specifically avoid using any of that, and if something slips in, it will be considered a bug and addressed accordingly.

But please understand that an open source project like ours, which has only very limited resources has to draw the line somewhere when it comes to the question how far we can go to accomodate the myriad of different requirements/policies/recomendations on how to do various things on all the different platforms (and flavors thereof) we try to support.

And IMO in this case accomodating the specific policy in question here (and I deliberatly don't want to argue the point if you or @adrianbroher is right as far as this policy is concerned, I just assume now it is how you presented it) isn't worth the effort required and the potential for bugs and issues that will be introduced.

I appreciate your intention to bring to our attention to something we probably could do a bit better, but I really don't want to open that can of worm for a minor issue like this.

@Vezzra
Copy link
Member

Vezzra commented Sep 9, 2016

If you said "well, if you can provide a PR, please do, but we are not going to work on it"... I could very well understand. Which is why I opened this issue.

Instead, I get a massive amount of semi-hostile nitpicking, which pretty much killed my motivation to create such a PR now.

Please try to see @adrianbroher's replies not as semi-hostile, he's just arguing (maybe a bit strongly) against something he obviously perceives as an overly strict interpretation of a policy he understands differently (if rightly so or not is a different question I don't want to try to answer, because honestly I don't know).

I mean, you have to admit that requiring to put something that is not platform dependent in a location that is intended for platform dependent stuff just because it could potentially be platform dependent is a bit strict. And as I explained above, this isn't always as simple as just storing a few files in a different directory.

In our case a PR would be far from sufficient to address all the issues such a change would introduce. See my explanaitions above.

I hope we all can calm down a bit and return to a friendlier tone 😃

@hasufell
Copy link
Author

hasufell commented Sep 9, 2016

I appreciate your intention to bring our attention to something we probably could do a bit better, but I really don't want to open that can of worm for a minor issue like this.

Sure, I can understand your points and they are all valid. At least you're not trying to lecture me on distro related topics.

Maybe you should double-check how some of your contributors influence the environment, because I'm definitely not going through this kind of hostile discussion again. For any patch.

@hasufell hasufell closed this as completed Sep 9, 2016
@MatGB
Copy link
Member

MatGB commented Sep 9, 2016

In addition to what @Vezzra just said, speaking entirely as a scripter/modder, some of the Python files in FO specifically are data files, they're deliberately setup to be very easily changeable by a player without needing to understand even the basics of Python.

In addition, there's a serious discussion within the project to move our existing data files from the bespoke FOCS language created for the project into Python, making a huge amount more of our Python content data, not script or even really code.

As someone who switched between a Linux desktop and a Windows laptop, sharing specific amendments between them, but not a high end dev (I can barely read Python let alone write complex scripts), the idea that on different machines/platforms the shared content would be in different places is simply a REALLY bad idea. Specifically one of our most important AI scripters uses a Github repository for his Default folder, allowing people to point to a folder online or an unpacked zip, etc is a key aim of the project, that this can be done is a very useful feature.

@Vezzra
Copy link
Member

Vezzra commented Sep 9, 2016

Maybe you should double-check how some of your contributors influence the environment, because I'm definitely not going through this kind of hostile discussion again. For any patch.

Well, I hope you can give that decision another thought, and not let one discussion that got a bit out of hand discourage you from contributing to a project you apparently have been interested in enough to start providing some input.

As far as @adrianbroher is concerned, he is not just a contributor, but a core team member who's contributions and efforts have been very important and valuable so far. And although we have our moments where we get into more intense arguments, so far we all have managed to get along well enough.

We usually manage to maintain a friendly climate, and I hope we can hold on to that...

@adrianbroher
Copy link
Contributor

Come up with better arguments. Starting with "the code is arch-dependent", which is not true, and finishing with "that's how we do things in gentoo, but screw you I won't explain why" is just not conclusive.

@hasufell
Copy link
Author

hasufell commented Sep 9, 2016

Well, I hope you can give that decision another thought

I've been an opensource contributor long enough to know when I meet toxic people (not you). Good luck with your project.

@Vezzra
Copy link
Member

Vezzra commented Sep 9, 2016

Starting with "the code is arch-dependent", which is not true, and finishing with "that's how we do things in gentoo, but screw you I won't explain why" is just not conclusive.

To be fair, that's not what he said/meant, at least not how I understood what he was trying to say. His point is: Python scripts are code (which is basically true, I mean, Python is a programming language), and because Python scripts can be platform dependent, all Python scripts (regardless if a specific script actually is platform dependent) belong in /usr/lib/.

And that's not just his personal view/preference/taste, but from the way he presented it I got the impression he knows what he is talking about, and apparently on the Linux distros he referred to that policy actually is that strict.

That's why he perceived your answers as hostile, I hope you can try to see the entire discussion through his eyes a bit (I mean, if I'm a Gentoo dev and know how we do things in Gentoo, and then someone wants to tell me differently I'll probably get a bit irritated too). Although I also admit that "hostile" takes it too far, and the term "toxic people" in his last post was uncalled-for.

Of course one can raise the question if such a strict policy is reasonable (it strikes me as overly strict too, but I know too less about all that to give a strong opinion here), but even if we don't agree, why not just leave it at that? He didn't make any demands, he just made a suggestion and even pointed out that it's not a big issue if we don't go with it.

Sometimes you just have to let it go... 😉

(Please see my post just as an attempt to calm things down, I don't want to lecture, criticize or offend anyone, I just try to see both sides here... I hope you can understand that, even if you don't agree to all I said.)

@adrianbroher adrianbroher added status:won't fix The Issue won't be fixed as implementation or maintainance effort required would be execessive. category:feature The Issue/PR describes or implements a new game feature. labels Oct 6, 2016
@adrianbroher adrianbroher added component:build system The Issue/PR deals with CMake, Visual Studio or the general build process. category:bug The Issue/PR describes or solves a perceived malfunction within the game. status:invalid The Issue is not classified as a formal valid Issue report or applicable PR. and removed category:request category:feature The Issue/PR describes or implements a new game feature. status:won't fix The Issue won't be fixed as implementation or maintainance effort required would be execessive. labels Oct 6, 2016
@Vezzra Vezzra modified the milestones: Gateway to the Void, v0.4.7 Jul 19, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
category:bug The Issue/PR describes or solves a perceived malfunction within the game. component:build system The Issue/PR deals with CMake, Visual Studio or the general build process. component:internal The Issue/PR deals with any project component that has no explicit `component` label. status:invalid The Issue is not classified as a formal valid Issue report or applicable PR.
Projects
None yet
Development

No branches or pull requests

5 participants