python,python3: Add pypi makefile by jefferyto · Pull Request #10242 · openwrt/packages

jefferyto · 2019-10-14T21:30:03Z

Maintainer: me, @commodo
Compile tested: armvirt-64, 2019-10-04 snapshot sdk (tested with python-six, python-twisted)
Run tested: N/A

Description:
This adds pypi.mk, which can be included in Python packages that download their sources from PyPI, to auto-fill various PKG_* variables based on the value of PYPI_NAME.

This makefile should be included after $(TOPDIR)/rules.mk but before $(INCLUDE_DIR)/package.mk (and $(INCLUDE_DIR)/host-build.mk).

Signed-off-by: Jeffery To jeffery.to@gmail.com

jefferyto · 2019-10-14T21:42:58Z

This is how I imagine this makefile would be used (using python-six as an example):

include $(TOPDIR)/rules.mk

PKG_NAME:=python-six
PKG_VERSION:=1.12.0
PKG_RELEASE:=2

PYPI_NAME:=six
PKG_HASH:=d16a0141ec1a18405cd4ce8b4613101da75da0e9a7aec5bdd4fa804d0e0eba73

PKG_LICENSE:=MIT
PKG_LICENSE_FILES:=LICENSE
PKG_MAINTAINER:=Jeffery To <jeffery.to@gmail.com>, Alexandru Ardelean <ardeleanalex@gmail.com>

HOST_BUILD_DEPENDS:=python3/host

include ../pypi.mk
include $(INCLUDE_DIR)/host-build.mk
include $(INCLUDE_DIR)/package.mk
include ../python-package.mk
include ../python3-package.mk

This needs to be included before package.mk and host-build.mk because it sets PKG_BUILD_DIR and HOST_BUILD_DIR (which are used in package.mk and host-build.mk on include). (I couldn't add to python[3]-package.mk because those need to follow package.mk to override Build/Compile.)

PKG_BUILD_DIR is set in a way so that a custom PKG_UNPACK is no longer necessary. (This is based on a change I submitted to the main repo (openwrt/openwrt@e545fac), which I completely forgot about until a few days ago 🤦‍♂️)

Not having PKG_SOURCE and PKG_SOURCE_URL in the package makefile is a little weird, not sure how everyone feels about that.

I can't believe it took me to so long to think of refactoring these PKG_* variables into a reusable form 🤦‍♂️

neheb · 2019-10-15T02:15:06Z

+  PKG_SOURCE?=$(PYPI_NAME)-$(PKG_VERSION).$(PYPI_SOURCE_EXT)
+  PKG_SOURCE_URL?=https://files.pythonhosted.org/packages/source/$(PYPI_NAME_FIRST_LETTER)/$(PYPI_NAME)
+
+  PKG_BUILD_DIR:=$(BUILD_DIR)/pypi/$(if $(BUILD_VARIANT),$(PKG_NAME)-$(BUILD_VARIANT)/)$(PYPI_NAME)$(if $(PKG_VERSION),-$(PKG_VERSION))


curious, would PKG_SOURCE_SUBDIR be better?

I think PKG_SOURCE_SUBDIR is only used when the source is checked out of a repo (git, svn, etc.), not when it is extracted from a tarball?

commodo · 2019-10-15T06:12:21Z

Not having PKG_SOURCE and PKG_SOURCE_URL in the package makefile is a little weird, not sure how everyone feels about that.

I wonder if this interferes with uscan [the tool that emails us telling to upgrade our packages].
If it does, then I am not sure about this change.

jefferyto · 2019-10-15T15:05:51Z

Not having PKG_SOURCE and PKG_SOURCE_URL in the package makefile is a little weird, not sure how everyone feels about that.

I wonder if this interferes with uscan [the tool that emails us telling to upgrade our packages].
If it does, then I am not sure about this change.

I'm not sure... @sdwalker do we need to have PKG_SOURCE and PKG_SOURCE_URL in the actual package makefile?

sdwalker · 2019-10-16T23:39:05Z

Not having PKG_SOURCE and PKG_SOURCE_URL in the package makefile is a little weird, not sure how everyone feels about that.

I wonder if this interferes with uscan [the tool that emails us telling to upgrade our packages].
If it does, then I am not sure about this change.

I'm not sure... @sdwalker do we need to have PKG_SOURCE and PKG_SOURCE_URL in the actual package makefile?

Not for tarballs.
Yes for git packages since PKG_SOURCE_URL is passed to git ls-remote.
uscan scripts don't use PKG_SOURCE.

jefferyto · 2019-10-17T20:27:56Z

@sdwalker Thanks for clarifying - those email notifications are a life-saver 😂

commodo · 2019-10-18T06:21:44Z

I'll admit, I'm a bit fuzzy on how uscan does detection of new packages.
But, if it works with this change, then I don't need to know about it.

Thanks @jefferyto & @sdwalker :)

neheb · 2019-10-18T06:36:17Z

If placement in a Makefile is a concern, this could be useful to add:

https://github.com/openwrt/openwrt/blob/master/include/uclibc%2B%2B.mk#L1

jefferyto · 2019-10-18T12:16:37Z

@commodo I'm glad it occurred to one of us to ask the question 😂

jefferyto · 2019-10-18T12:26:17Z

@neheb I'm not sure I want to stop weird placements, it could be that the makefile author really knows what they are doing or just wants to experiment... Maybe a $(warning ...) or $(info ...) will be enough?

This adds pypi.mk, which can be included in Python packages that download their sources from PyPI, to auto-fill various PKG_* variables based on the value of PYPI_NAME. This makefile should be included after $(TOPDIR)/rules.mk but before $(INCLUDE_DIR)/package.mk (and $(INCLUDE_DIR)/host-build.mk). Signed-off-by: Jeffery To <jeffery.to@gmail.com>

jefferyto · 2019-10-18T16:17:36Z

Updated with a warning that pypi.mk should be included before package.mk.

neheb · 2019-10-18T20:26:35Z

Merging. if warning is the wrong approach, there is plenty of time to change before version 20.

jefferyto · 2019-10-19T06:06:55Z

@commodo @neheb I was thinking of converting most/all packages that download from PyPI to use this makefile... Do you think I should wait until 19.07 is frozen, or just do it now?

neheb · 2019-10-19T06:19:35Z

Well, 19.07 has already been branched. I wouldn't bother backporting a change like this. It's fairly cosmetic.

jefferyto · 2019-10-19T06:47:18Z

I wasn't considering backporting this, but using this in master may make cherry picking changes to the 19.07 branch more difficult.

neheb · 2019-10-19T18:49:21Z

I wouldn't worry too much about that.

BKPepe · 2019-11-02T16:19:11Z

I think we should cherry-pick it to openwrt-19.07 as many packages have been converted and newly added packages can not be included openwrt-19.07 without this.

neheb · 2019-11-02T17:22:49Z

Go right ahead. It's a cosmetic change anyway.

BKPepe · 2019-11-02T18:02:06Z

Here you go- 866b42c.

jefferyto · 2019-11-02T18:06:55Z

Thanks @BKPepe - can you cherry pick #10315 as well?

neheb reviewed Oct 15, 2019

View reviewed changes

commodo reviewed Oct 15, 2019

View reviewed changes

Comment thread lang/python/pypi.mk Outdated

commodo reviewed Oct 15, 2019

View reviewed changes

Comment thread lang/python/pypi.mk Outdated

jefferyto force-pushed the pypi-mk branch from ae7f657 to 0732497 Compare October 15, 2019 15:10

commodo approved these changes Oct 18, 2019

View reviewed changes

jefferyto force-pushed the pypi-mk branch from 0732497 to add4c42 Compare October 18, 2019 16:17

neheb merged commit 758865f into openwrt:master Oct 18, 2019

jefferyto deleted the pypi-mk branch October 19, 2019 06:07

peter-stadler mentioned this pull request Oct 29, 2019

sqlparse: add python3 package #10375

Merged

Conversation

jefferyto commented Oct 14, 2019

Uh oh!

jefferyto commented Oct 14, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

neheb Oct 15, 2019

Choose a reason for hiding this comment

Uh oh!

jefferyto Oct 15, 2019

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

commodo commented Oct 15, 2019

Uh oh!

jefferyto commented Oct 15, 2019

Uh oh!

sdwalker commented Oct 16, 2019

Uh oh!

jefferyto commented Oct 17, 2019

Uh oh!

commodo commented Oct 18, 2019

Uh oh!

neheb commented Oct 18, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jefferyto commented Oct 18, 2019

Uh oh!

jefferyto commented Oct 18, 2019

Uh oh!

jefferyto commented Oct 18, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

neheb commented Oct 18, 2019

Uh oh!

jefferyto commented Oct 19, 2019

Uh oh!

neheb commented Oct 19, 2019

Uh oh!

jefferyto commented Oct 19, 2019

Uh oh!

neheb commented Oct 19, 2019

Uh oh!

BKPepe commented Nov 2, 2019

Uh oh!

neheb commented Nov 2, 2019

Uh oh!

BKPepe commented Nov 2, 2019

Uh oh!

jefferyto commented Nov 2, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

jefferyto commented Oct 14, 2019 •

edited

Loading

neheb commented Oct 18, 2019 •

edited

Loading

jefferyto commented Oct 18, 2019 •

edited

Loading