No setup.py prevents pip install from VCS

[MarvBob]

I don't know if I am using pybuilder correctly but for me, as the setup.py file is generated by pybuilder, it should not be committed in a VCS.
The problem is that this file seems to be required by pip when installing from a VCS repository (as indicated in the pip documentation).
Is there an alternative solution to this problem?

[AlexeySanko]

@MarvBob, I think that if we talk about pip VCS installation, that we should commit to VCS files from base_dir/target/dist/package_name-A.B.C directory. Because only there are final version of package files. For example, we can filter __init__.py with ${version} or do any things and should commit to VCS files after that changes.
From my point of view:

1. build project with Pybuilder
2. copy files from target/dist to other directory (which cloned from VCS)
3. commit changes to this VSC

PS You can try to create simple plugin which will call specific new task like collect_for_vcs after publish and automatically copy files from target/dist to needed directory

[MarvBob]

Ok I understand this idea, but something bothers me a bit: if I want to pip install from VCS it means that it's not a final version of my package. The process you described means that we have to build the package before being able to pip install from VCS, and I'd like to be able to pip install from VCS to work on sources before building a final version of the package.
I don't know what's right or wrong, I just think that's an interesting point to keep in mind. Thanks for your replies btw

[AlexeySanko]

I think that it isn't for not-final-version work. Looks like that it's just workaround for teams which do not have their own PyPi server, but need to work with pip internally. In this case VCS is just replacement for PyPi. It doesn't remove necessity to 'build' or 'collect' package.

[MarvBob]

Well maybe. Actually you can use the find-links option if you don't have an internal PyPi server.

[AlexeySanko]

find-links could use ftp or shared directories? Is it possible to distribute packages with pip --find-links across all needed hosts?

[arcivanov]

@MarvBob @AlexeySanko
This shouldn't bother you. setup.py is generated and committed for one specific reason - there is no way around it when installing a PyB project from source.

When you install a regular setuptools project from source it checks out the project, uses committed setup.py and executes it. The setup.py in a regular setuptools project contains all the logic required.

When you install a PyB-based project from source, there is, ordinarily, no setup.py. The installation process requires you to install PyB and run an install target. Setup.py that is committed into the source control for PyB project is merely a shim that installs PyB and runs the several targets to generate the actual setup.py and then delegate to the generated setup.py.

You can see the code here:

pybuilder/src/main/python/pybuilder/scaffolding.py

Line 171 in feaf5ee

subprocess.check_call([sys.executable, "setup.py"] + setup_args, cwd=script_dir)

[arcivanov]

I actually re-read the original post and realize it's a question/documentation issue. PyB fully supports installing from VCS.
See https://github.com/pybuilder/pybuilder/pull/273/files#diff-2cd9bc4e1c643733314db34717761b94

[MarvBob]

@AlexeySanko Not sure about using find-links from ftp, I am using it with a Windows network drive.
@arcivanov My problem is that I didn't commit my setup.py file as it is generated in the /target folder when building a project and I think it is not a good practice to commit the built files, maybe I am wrong with that?

[arcivanov]

@MarvBob you're not supposed to commit anything inside target/. The setup.py is generated by pyb at the root of the project. Try pyb --update-project. See the link above.

[AlexeySanko]

@arcivanov looks like great feature, but, unfortunately, undocumented. Could You please add it to documentation?

[MarvBob]

@arcivanov works perfectly, thanks. As @arcivanov said, it just needs to be documented 👍

[margru]

@MarvBob you're not supposed to commit anything inside target/. The setup.py is generated by pyb at the root of the project. Try pyb --update-project. See the link above.

There are few problems with this approach:

• on Windows, the binary is called pyb_ and not pyb
• the generated setup.py expects directory structure different from the one that pyb package creates (that might have been changed in the build.py script) -> causing

Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "D:\ProjectsSrc\SVOX\lrqa\setup.py", line 60, in <module>
        dist_dir = glob.glob(os.path.join(script_dir, "target", "dist", "*"))[0]
    IndexError: list index out of range

[arcivanov]

What version are you on?

[margru]

What version are you on?

Python 2.7.14
PyBuilder 0.11.17
Windows 10 64bit