Replacement of interlinking requirements.txt files

[sybrenstuvel]

Hi there,

I'd love to be able to replace our use of requirements.txt files with Pipenv. However, I don't quite see how yet, as our use of the requirements.txt files seems to be rather unique. Our setup consists of a top-level application project ('mainproject'), our own framework 'Pillar', and a couple of Pillar extensions. To manage the requirements, each of those have their own requirements.txt file that refers to the requirements.txt files of their dependencies. This allowed us to pin versions for each project and still have per-project files.

• pillar/requirements.txt contains all the framework's requirements, all with pinned versions.
• extension1/requirements.txt starts with -r ../pillar/requirements.txt to link in the framework requirements, then continues to list the extension-speciifc ones.
• mainproject/requirements.txt: starts with:

  -r ../extension1/requirements.txt
  -r ../extension2/requirements.txt
  -r ../pillar/requirements.txt
  

  and then lists its own requirements, if any.

This setup allows us to quickly enable/disable certain extensions, and avoid repeating requirements and pinned versions. We want to get rid of those files, though, as pinning each version is a nightmare to maintain (as you well know).

How would such a setup translate to Pipenv? We don't push Pillar to PyPi, nor do we push the extensions. In the end it's all collected into a docker container for deployment simply by checking out certain Git branches; there is no python3 setup.py bdist invocations to create distribution packages.

I've tried adding something like this to extension1/Pipfile:

[packages]
pillar = {source = '../pillar', editable = true, version = "*"}

This doesn't work, though, as pipenv install in the main project gives us:

Virtualenv location: /home/sybren/.virtualenvs/mainproject-O7T8y2wV
Pipfile.lock not found, creating...
Locking [dev-packages] dependencies...
Locking [packages] dependencies...

Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
Could not find a version that matches pillar (from -r /tmp/pipenv-jmw6x1mc-requirements/pipenv-x6d8in2v-constraints.txt (line 2))
No versions found
Was https://pypi.org/simple reachable?

For me it would be ideal if we could specify "include section X from pipfile Y", something along those lines in mainproject/Pipfile:

[packages]
__include__ = {pipfile = "../pillar/Pipfile"}
otherdependency = "*"

---

$ pipenv --support

Pipenv version: '2018.7.1'

Pipenv location: '/home/sybren/.local/lib/python3.6/site-packages/pipenv'

Python location: '/usr/bin/python3'

Other Python installations in PATH:

• 2.7: /usr/bin/python2.7

• 2.7: /usr/bin/python2.7

• 3.4: /usr/bin/python3.4m

• 3.4: /usr/bin/python3.4

• 3.5: /opt/python3.5/bin/python3.5m

• 3.5: /opt/python3.5/bin/python3.5

• 3.6: /usr/bin/python3.6m

• 3.6: /usr/bin/python3.6

• 2.7.15: /usr/bin/python

• 2.7.15: /usr/bin/python2

• 3.6.5: /usr/bin/python3

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.6.5',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '4.15.0-33-generic',
 'platform_system': 'Linux',
 'platform_version': '#36-Ubuntu SMP Wed Aug 15 16:00:05 UTC 2018',
 'python_full_version': '3.6.5',
 'python_version': '3.6',
 'sys_platform': 'linux'}

System environment variables:

• CLUTTER_IM_MODULE
• COLORFGBG
• COLORTERM
• DBUS_SESSION_BUS_ADDRESS
• DEFAULTS_PATH
• DESKTOP_SESSION
• DISPLAY
• GPG_AGENT_INFO
• GS_LIB
• GTK2_RC_FILES
• GTK_IM_MODULE
• GTK_RC_FILES
• HOME
• KDE_FULL_SESSION
• KDE_SESSION_UID
• KDE_SESSION_VERSION
• KONSOLE_DBUS_SERVICE
• KONSOLE_DBUS_SESSION
• KONSOLE_DBUS_WINDOW
• KONSOLE_PROFILE_NAME
• LANG
• LANGUAGE
• LC_ADDRESS
• LC_IDENTIFICATION
• LC_MEASUREMENT
• LC_MONETARY
• LC_NAME
• LC_NUMERIC
• LC_PAPER
• LC_TELEPHONE
• LC_TIME
• LOGNAME
• MANDATORY_PATH
• OLDPWD
• PATH
• PROFILEHOME
• PWD
• QT4_IM_MODULE
• QT_ACCESSIBILITY
• QT_AUTO_SCREEN_SCALE_FACTOR
• SESSION_MANAGER
• SHELL
• SHELL_SESSION_ID
• SHLVL
• SSH_AGENT_PID
• SSH_AUTH_SOCK
• TERM
• USER
• WINDOWID
• XAUTHORITY
• XCURSOR_SIZE
• XCURSOR_THEME
• XDG_CONFIG_DIRS
• XDG_CURRENT_DESKTOP
• XDG_DATA_DIRS
• XDG_RUNTIME_DIR
• XDG_SEAT
• XDG_SEAT_PATH
• XDG_SESSION_CLASS
• XDG_SESSION_DESKTOP
• XDG_SESSION_ID
• XDG_SESSION_PATH
• XDG_SESSION_TYPE
• XDG_VTNR
• XMODIFIERS
• _
• PS1
• GOPATH
• GOROOT
• EDITOR
• VISUAL
• ASAN_OPTIONS
• LS_COLORS
• WORKON_HOME
• VIRTUALENVWRAPPER_PROJECT_FILENAME
• VIRTUALENVWRAPPER_SCRIPT
• VIRTUALENVWRAPPER_HOOK_DIR
• PYTHONDONTWRITEBYTECODE
• PIP_PYTHON_PATH

Pipenv–specific environment variables:

Debug–specific environment variables:

• PATH: /usr/lib/ccache:/home/sybren/workspace/blender-git/arcanist/bin:/opt/go/bin:/home/sybren/workspace/go/bin:/home/sybren/.local/bin:/home/sybren/bin:/usr/lib/ccache:/home/sybren/workspace/blender-git/arcanist/bin:/home/sybren/bin:/opt/python3.5/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/opt/lib/alembic/bin:/opt/lib/abcview/bin:/home/sybren/.configfiles/fzf/bin
• SHELL: /usr/bin/zsh
• EDITOR: /usr/bin/vim
• LANG: en_GB.UTF-8
• PWD: /home/sybren/workspace/cloud/pipenvtest/mainproject

---

Contents of Pipfile ('/home/sybren/workspace/cloud/pipenvtest/mainproject/Pipfile'):

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
pillar = {source = '../pillar', editable = true, version = "*"}

[dev-packages]

[requires]
python_version = "3.6"

[uranusjr]

Does #2148 cover this usage? This is indeed something I am interested in providing (see discussion in the linked thread), but it would require modifications in the Pipfile specification.

[sybrenstuvel]

#2148 seems to be very similar. However, I'm a big fan of Pipenv and I have no desire to keep our current requirements.txt and requirements-dev.txt files around. If we can find a solution that only depends on Pipfile and Pipfile.lock and doesn't require generating other files, that would be perfect.

[uranusjr]

#2148 did indeed proposal a proposal (see #2148 (comment)) of a subproject syntax that links to a Pipfile.