Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

No setup.py prevents pip install from VCS #508

Closed
MarvBob opened this issue Oct 12, 2017 · 18 comments
Closed

No setup.py prevents pip install from VCS #508

MarvBob opened this issue Oct 12, 2017 · 18 comments

Comments

@MarvBob
Copy link

MarvBob commented Oct 12, 2017

I don't know if I am using pybuilder correctly but for me, as the setup.py file is generated by pybuilder, it should not be committed in a VCS.
The problem is that this file seems to be required by pip when installing from a VCS repository (as indicated in the pip documentation).
Is there an alternative solution to this problem?

@AlexeySanko
Copy link
Contributor

AlexeySanko commented Oct 12, 2017

@MarvBob, I think that if we talk about pip VCS installation, that we should commit to VCS files from base_dir/target/dist/package_name-A.B.C directory. Because only there are final version of package files. For example, we can filter __init__.py with ${version} or do any things and should commit to VCS files after that changes.
From my point of view:

  1. build project with Pybuilder
  2. copy files from target/dist to other directory (which cloned from VCS)
  3. commit changes to this VSC

PS You can try to create simple plugin which will call specific new task like collect_for_vcs after publish and automatically copy files from target/dist to needed directory

@MarvBob
Copy link
Author

MarvBob commented Oct 12, 2017

Ok I understand this idea, but something bothers me a bit: if I want to pip install from VCS it means that it's not a final version of my package. The process you described means that we have to build the package before being able to pip install from VCS, and I'd like to be able to pip install from VCS to work on sources before building a final version of the package.
I don't know what's right or wrong, I just think that's an interesting point to keep in mind. Thanks for your replies btw

@AlexeySanko
Copy link
Contributor

I think that it isn't for not-final-version work. Looks like that it's just workaround for teams which do not have their own PyPi server, but need to work with pip internally. In this case VCS is just replacement for PyPi. It doesn't remove necessity to 'build' or 'collect' package.

@MarvBob
Copy link
Author

MarvBob commented Oct 12, 2017

Well maybe. Actually you can use the find-links option if you don't have an internal PyPi server.

@AlexeySanko
Copy link
Contributor

AlexeySanko commented Oct 12, 2017

find-links could use ftp or shared directories? Is it possible to distribute packages with pip --find-links across all needed hosts?

@arcivanov
Copy link
Member

@MarvBob @AlexeySanko
This shouldn't bother you. setup.py is generated and committed for one specific reason - there is no way around it when installing a PyB project from source.

When you install a regular setuptools project from source it checks out the project, uses committed setup.py and executes it. The setup.py in a regular setuptools project contains all the logic required.

When you install a PyB-based project from source, there is, ordinarily, no setup.py. The installation process requires you to install PyB and run an install target. Setup.py that is committed into the source control for PyB project is merely a shim that installs PyB and runs the several targets to generate the actual setup.py and then delegate to the generated setup.py.

You can see the code here:

subprocess.check_call([sys.executable, "setup.py"] + setup_args, cwd=script_dir)

@arcivanov
Copy link
Member

arcivanov commented Oct 13, 2017

I actually re-read the original post and realize it's a question/documentation issue. PyB fully supports installing from VCS.
See https://github.com/pybuilder/pybuilder/pull/273/files#diff-2cd9bc4e1c643733314db34717761b94

@MarvBob
Copy link
Author

MarvBob commented Oct 13, 2017

@AlexeySanko Not sure about using find-links from ftp, I am using it with a Windows network drive.
@arcivanov My problem is that I didn't commit my setup.py file as it is generated in the /target folder when building a project and I think it is not a good practice to commit the built files, maybe I am wrong with that?

@arcivanov
Copy link
Member

arcivanov commented Oct 13, 2017

@MarvBob you're not supposed to commit anything inside target/. The setup.py is generated by pyb at the root of the project. Try pyb --update-project. See the link above.

@AlexeySanko
Copy link
Contributor

AlexeySanko commented Oct 13, 2017

@arcivanov looks like great feature, but, unfortunately, undocumented. Could You please add it to documentation?

@MarvBob
Copy link
Author

MarvBob commented Oct 13, 2017

@arcivanov works perfectly, thanks. As @arcivanov said, it just needs to be documented 👍

@MarvBob MarvBob closed this as completed Oct 13, 2017
@margru
Copy link

margru commented Mar 3, 2020

@MarvBob you're not supposed to commit anything inside target/. The setup.py is generated by pyb at the root of the project. Try pyb --update-project. See the link above.

There are few problems with this approach:

  • on Windows, the binary is called pyb_ and not pyb
  • the generated setup.py expects directory structure different from the one that pyb package creates (that might have been changed in the build.py script) -> causing
Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "D:\ProjectsSrc\SVOX\lrqa\setup.py", line 60, in <module>
        dist_dir = glob.glob(os.path.join(script_dir, "target", "dist", "*"))[0]
    IndexError: list index out of range

@arcivanov
Copy link
Member

What version are you on?

@margru
Copy link

margru commented Mar 3, 2020

What version are you on?

Python 2.7.14
PyBuilder 0.11.17
Windows 10 64bit

@arcivanov
Copy link
Member

The first issue should go away in v0.12.0 once released in a few hours.
The second I'll have to check to see why this happens on Windows as it works on Linux.

@margru
Copy link

margru commented Mar 3, 2020

The first issue should go away in v0.12.0 once released in a few hours.

Great, good to know!

The second I'll have to check to see why this happens on Windows as it works on Linux.

Just mentioning again that I am modifying the dir_dist property in the build.py script if that makes any difference. Unfortunately, I cannot test it in Linux right now, maybe WSL if that helps.

@arcivanov
Copy link
Member

Ah... It does make a difference, I need to think how to get this done properly, i.e. maybe need to create a separate build target to get around this. I'll create a bug for this.

@margru
Copy link

margru commented Mar 3, 2020

Great, thanks for your work and support!

arcivanov added a commit to arcivanov/pybuilder that referenced this issue Apr 5, 2020
Distutils Plugin now:
1) Allows skip-existing version on Twine
2) Performs a Twine check for PyPi compatibility
pyproject.toml and generated setup.py properly handle overwritten `$dir_dist`

fixes pybuilder#657, pybuilder#508
arcivanov added a commit to arcivanov/pybuilder that referenced this issue Apr 5, 2020
Distutils Plugin now:
1) Allows skip-existing version on Twine
2) Performs a Twine check for PyPi compatibility
pyproject.toml and generated setup.py properly handle overwritten `$dir_dist`

fixes pybuilder#657, pybuilder#508
arcivanov added a commit to arcivanov/pybuilder that referenced this issue Apr 5, 2020
Distutils Plugin now:
1) Allows skip-existing version on Twine
2) Performs a Twine check for PyPi compatibility
pyproject.toml and generated setup.py properly handle overwritten `$dir_dist`

fixes pybuilder#657, pybuilder#508
arcivanov added a commit to arcivanov/pybuilder that referenced this issue Apr 5, 2020
Distutils Plugin now:
1) Allows skip-existing version on Twine
2) Performs a Twine check for PyPi compatibility
pyproject.toml and generated setup.py properly handle overwritten `$dir_dist`

fixes pybuilder#657, pybuilder#508
arcivanov added a commit to arcivanov/pybuilder that referenced this issue Apr 5, 2020
Distutils Plugin now:
1) Allows skip-existing version on Twine
2) Performs a Twine check for PyPi compatibility
pyproject.toml and generated setup.py properly handle overwritten `$dir_dist`

fixes pybuilder#657, pybuilder#508
arcivanov added a commit to arcivanov/pybuilder that referenced this issue Apr 5, 2020
Distutils Plugin now:
1) Allows skip-existing version on Twine
2) Performs a Twine check for PyPi compatibility
pyproject.toml and generated setup.py properly handle overwritten `$dir_dist`

fixes pybuilder#657, pybuilder#508
arcivanov added a commit to arcivanov/pybuilder that referenced this issue Apr 5, 2020
Distutils Plugin now:
1) Allows skip-existing version on Twine
2) Performs a Twine check for PyPi compatibility
pyproject.toml and generated setup.py properly handle overwritten `$dir_dist`

fixes pybuilder#657, pybuilder#508
arcivanov added a commit to arcivanov/pybuilder that referenced this issue Apr 6, 2020
Distutils Plugin now:
1) Allows skip-existing version on Twine
2) Performs a Twine check for PyPi compatibility
pyproject.toml and generated setup.py properly handle overwritten `$dir_dist`

fixes pybuilder#657, pybuilder#508
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants