Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fail2ban: fix 2to3 error #17341

Merged
merged 1 commit into from
Jan 10, 2022
Merged

fail2ban: fix 2to3 error #17341

merged 1 commit into from
Jan 10, 2022

Conversation

erdoukki
Copy link
Contributor

@erdoukki erdoukki commented Dec 14, 2021

Maintainer: me / @erdoukki
Run tested: (test done on MVEBU arch53 EspressoBin & EspressoBin-ULTRA, OpenWrt 21.02.x)
Compile tested: (arch=cortexa53, host=amd64)

Issue: 2to3 support has been removed in setuptools since version 58.0.0.

Fix: #17311

Requirements: 2to3/host #17429

To install/build for python3 from source, it is necessary to convert to py3
codebase before setup (invoke 2to3 or ./fail2ban-2to3 firstly).

./fail2ban-2to3
python3 setup.py build

Signed-off-by: Kerma Gérald gandalf@gk2.net

@erdoukki
Copy link
Contributor Author

@BKPepe
Can you check, please ?

erdoukki added a commit to erdoukki/packages that referenced this pull request Dec 25, 2021
2to3 is a Python program that reads Python 2.x source code and applies a
  series of fixers to transform it into valid Python 3.x code. The standard
  library contains a rich set of fixers that will handle almost all code. 2to3
  supporting library lib2to3 is, however, a flexible and generic library, so it
  is possible to write your own fixers for 2to3. lib2to3 could also be adapted
  to custom applications in which Python code needs to be edited automatically.

This tool is necessary for fail2ban package because of issue
  openwrt#17311
  openwrt#17341

Simple 2to3.py script from Debian, thanks to Matthias Klose <doko@ubuntu.com>

Signed-off-by: Kerma Gérald <gandalf@gk2.net>
erdoukki added a commit to erdoukki/packages that referenced this pull request Dec 25, 2021
2to3 is a Python program that reads Python 2.x source code and applies a
  series of fixers to transform it into valid Python 3.x code. The standard
  library contains a rich set of fixers that will handle almost all code. 2to3
  supporting library lib2to3 is, however, a flexible and generic library, so it
  is possible to write your own fixers for 2to3. lib2to3 could also be adapted
  to custom applications in which Python code needs to be edited automatically.

This tool is necessary for fail2ban package because of issue
  openwrt#17311
  openwrt#17341

Simple 2to3.py script from Debian, thanks to Matthias Klose <doko@ubuntu.com>
From: https://salsa.debian.org/cpython-team/python3-defaults

Co-authored-by: Jeffery To <jeffery.to@gmail.com>
Signed-off-by: Kerma Gérald <gandalf@gk2.net>
erdoukki added a commit to erdoukki/packages that referenced this pull request Dec 25, 2021
2to3 is a Python program that reads Python 2.x source code and applies a
  series of fixers to transform it into valid Python 3.x code. The standard
  library contains a rich set of fixers that will handle almost all code. 2to3
  supporting library lib2to3 is, however, a flexible and generic library, so it
  is possible to write your own fixers for 2to3. lib2to3 could also be adapted
  to custom applications in which Python code needs to be edited automatically.

This tool is necessary for fail2ban package because of issue
  openwrt#17311
  openwrt#17341

Simple 2to3.py script from Debian, thanks to Matthias Klose <doko@ubuntu.com>
From: https://salsa.debian.org/cpython-team/python3-defaults

Co-authored-by: Jeffery To <jeffery.to@gmail.com>
Signed-off-by: Kerma Gérald <gandalf@gk2.net>
erdoukki added a commit to erdoukki/packages that referenced this pull request Dec 25, 2021
2to3 is a Python program that reads Python 2.x source code and applies a
  series of fixers to transform it into valid Python 3.x code. The standard
  library contains a rich set of fixers that will handle almost all code. 2to3
  supporting library lib2to3 is, however, a flexible and generic library, so it
  is possible to write your own fixers for 2to3. lib2to3 could also be adapted
  to custom applications in which Python code needs to be edited automatically.

This tool is necessary for fail2ban package because of issue
  openwrt#17311
  openwrt#17341

Simple 2to3.py script from Debian, thanks to Matthias Klose <doko@ubuntu.com>
From: https://salsa.debian.org/cpython-team/python3-defaults

Co-authored-by: Jeffery To <jeffery.to@gmail.com>
Signed-off-by: Kerma Gérald <gandalf@gk2.net>
erdoukki added a commit to erdoukki/packages that referenced this pull request Dec 25, 2021
2to3 is a Python program that reads Python 2.x source code and applies a
  series of fixers to transform it into valid Python 3.x code. The standard
  library contains a rich set of fixers that will handle almost all code. 2to3
  supporting library lib2to3 is, however, a flexible and generic library, so it
  is possible to write your own fixers for 2to3. lib2to3 could also be adapted
  to custom applications in which Python code needs to be edited automatically.

This tool is necessary for fail2ban package because of issue
  openwrt#17311
  openwrt#17341

Simple 2to3.py script from Debian, thanks to Matthias Klose <doko@ubuntu.com>
From: https://salsa.debian.org/cpython-team/python3-defaults

Co-authored-by: Jeffery To <jeffery.to@gmail.com>
Signed-off-by: Kerma Gérald <gandalf@gk2.net>
erdoukki added a commit to erdoukki/packages that referenced this pull request Dec 25, 2021
2to3 is a Python program that reads Python 2.x source code and applies a
  series of fixers to transform it into valid Python 3.x code. The standard
  library contains a rich set of fixers that will handle almost all code. 2to3
  supporting library lib2to3 is, however, a flexible and generic library, so it
  is possible to write your own fixers for 2to3. lib2to3 could also be adapted
  to custom applications in which Python code needs to be edited automatically.

This tool is necessary for fail2ban package because of issue
  openwrt#17311
  openwrt#17341

Simple 2to3.py script from Debian, thanks to Matthias Klose <doko@ubuntu.com>
From: https://salsa.debian.org/cpython-team/python3-defaults

Co-authored-by: Jeffery To <jeffery.to@gmail.com>
Signed-off-by: Kerma Gérald <gandalf@gk2.net>
erdoukki added a commit to erdoukki/packages that referenced this pull request Dec 25, 2021
2to3 is a Python program that reads Python 2.x source code and applies a
  series of fixers to transform it into valid Python 3.x code. The standard
  library contains a rich set of fixers that will handle almost all code. 2to3
  supporting library lib2to3 is, however, a flexible and generic library, so it
  is possible to write your own fixers for 2to3. lib2to3 could also be adapted
  to custom applications in which Python code needs to be edited automatically.

This tool is necessary for fail2ban package because of issue
  openwrt#17311
  openwrt#17341

Simple 2to3.py script from Debian, thanks to Matthias Klose <doko@ubuntu.com>
From: https://salsa.debian.org/cpython-team/python3-defaults

Co-authored-by: Jeffery To <jeffery.to@gmail.com>
Signed-off-by: Kerma Gérald <gandalf@gk2.net>
erdoukki added a commit to erdoukki/packages that referenced this pull request Dec 25, 2021
2to3 is a Python program that reads Python 2.x source code and applies a
  series of fixers to transform it into valid Python 3.x code. The standard
  library contains a rich set of fixers that will handle almost all code. 2to3
  supporting library lib2to3 is, however, a flexible and generic library, so it
  is possible to write your own fixers for 2to3. lib2to3 could also be adapted
  to custom applications in which Python code needs to be edited automatically.

This tool is necessary for fail2ban package because of issue
  openwrt#17311
  openwrt#17341

Simple 2to3.py script from Debian, thanks to Matthias Klose <doko@ubuntu.com>
From: https://salsa.debian.org/cpython-team/python3-defaults

Co-authored-by: Jeffery To <jeffery.to@gmail.com>
Signed-off-by: Kerma Gérald <gandalf@gk2.net>
erdoukki added a commit to erdoukki/packages that referenced this pull request Dec 25, 2021
2to3 is a Python program that reads Python 2.x source code and applies a
  series of fixers to transform it into valid Python 3.x code. The standard
  library contains a rich set of fixers that will handle almost all code. 2to3
  supporting library lib2to3 is, however, a flexible and generic library, so it
  is possible to write your own fixers for 2to3. lib2to3 could also be adapted
  to custom applications in which Python code needs to be edited automatically.

This tool is necessary for fail2ban package because of issue
  openwrt#17311
  openwrt#17341

Simple 2to3.py script from Debian, thanks to Matthias Klose <doko@ubuntu.com>
From: https://salsa.debian.org/cpython-team/python3-defaults

Requirements: python-toolz (lang/python/host-pip-requirements/lib2to3.txt)

Co-authored-by: Jeffery To <jeffery.to@gmail.com>
Signed-off-by: Kerma Gérald <gandalf@gk2.net>
erdoukki added a commit to erdoukki/packages that referenced this pull request Dec 25, 2021
2to3 is a Python program that reads Python 2.x source code and applies a
  series of fixers to transform it into valid Python 3.x code. The standard
  library contains a rich set of fixers that will handle almost all code. 2to3
  supporting library lib2to3 is, however, a flexible and generic library, so it
  is possible to write your own fixers for 2to3. lib2to3 could also be adapted
  to custom applications in which Python code needs to be edited automatically.

This tool is necessary for fail2ban package because of issue
  openwrt#17311
  openwrt#17341

Simple 2to3.py script from Debian, thanks to Matthias Klose <doko@ubuntu.com>
From: https://salsa.debian.org/cpython-team/python3-defaults

Requirements: python-toolz (lang/python/host-pip-requirements/lib2to3.txt)

Co-authored-by: Jeffery To <jeffery.to@gmail.com>
Signed-off-by: Kerma Gérald <gandalf@gk2.net>
@erdoukki
Copy link
Contributor Author

erdoukki commented Dec 25, 2021

May not compile correctly without the required 2to3/host package from PR #17429

erdoukki added a commit to erdoukki/packages that referenced this pull request Jan 1, 2022
2to3 is a Python program that reads Python 2.x source code and applies a
  series of fixers to transform it into valid Python 3.x code. The standard
  library contains a rich set of fixers that will handle almost all code. 2to3
  supporting library lib2to3 is, however, a flexible and generic library, so it
  is possible to write your own fixers for 2to3. lib2to3 could also be adapted
  to custom applications in which Python code needs to be edited automatically.

This tool is necessary for fail2ban package because of issue
  openwrt#17311
  openwrt#17341

Simple 2to3.py script from Debian, thanks to Matthias Klose <doko@ubuntu.com>
From: https://salsa.debian.org/cpython-team/python3-defaults

Co-authored-by: Jeffery To <jeffery.to@gmail.com>
Signed-off-by: Kerma Gérald <gandalf@gk2.net>
erdoukki added a commit to erdoukki/packages that referenced this pull request Jan 1, 2022
2to3 is a Python program that reads Python 2.x source code and applies a
  series of fixers to transform it into valid Python 3.x code. The standard
  library contains a rich set of fixers that will handle almost all code. 2to3
  supporting library lib2to3 is, however, a flexible and generic library, so it
  is possible to write your own fixers for 2to3. lib2to3 could also be adapted
  to custom applications in which Python code needs to be edited automatically.

This tool is necessary for fail2ban package because of issue
  openwrt#17311
  openwrt#17341

Simple 2to3.py script from Debian, thanks to Matthias Klose <doko@ubuntu.com>
From: https://salsa.debian.org/cpython-team/python3-defaults

Co-authored-by: Jeffery To <jeffery.to@gmail.com>
Signed-off-by: Kerma Gérald <gandalf@gk2.net>
erdoukki added a commit to erdoukki/packages that referenced this pull request Jan 3, 2022
2to3 is a Python program that reads Python 2.x source code and applies a
  series of fixers to transform it into valid Python 3.x code. The standard
  library contains a rich set of fixers that will handle almost all code. 2to3
  supporting library lib2to3 is, however, a flexible and generic library, so it
  is possible to write your own fixers for 2to3. lib2to3 could also be adapted
  to custom applications in which Python code needs to be edited automatically.

This tool is necessary for fail2ban package because of issue
  openwrt#17311
  openwrt#17341

Simple 2to3.py script from Debian, thanks to Matthias Klose <doko@ubuntu.com>
From: https://salsa.debian.org/cpython-team/python3-defaults

Co-authored-by: Jeffery To <jeffery.to@gmail.com>
Signed-off-by: Kerma Gérald <gandalf@gk2.net>
erdoukki added a commit to erdoukki/packages that referenced this pull request Jan 5, 2022
2to3 is a Python program that reads Python 2.x source code and applies a
  series of fixers to transform it into valid Python 3.x code. The standard
  library contains a rich set of fixers that will handle almost all code. 2to3
  supporting library lib2to3 is, however, a flexible and generic library, so it
  is possible to write your own fixers for 2to3. lib2to3 could also be adapted
  to custom applications in which Python code needs to be edited automatically.

This tool is necessary for fail2ban package because of issue
  openwrt#17311
  openwrt#17341

Simple 2to3.py script from Debian, thanks to Matthias Klose <doko@ubuntu.com>
From: https://salsa.debian.org/cpython-team/python3-defaults

Co-authored-by: Jeffery To <jeffery.to@gmail.com>
Signed-off-by: Kerma Gérald <gandalf@gk2.net>
net/fail2ban/Makefile Outdated Show resolved Hide resolved
net/fail2ban/Makefile Outdated Show resolved Hide resolved
@jefferyto
Copy link
Member

Not sure why https://github.com/openwrt/packages/actions/runs/1662159088 passed even though the 2to3 package wasn't merged into master yet - ping @aparcar

@erdoukki please rebase on current master to pick up 2to3

@erdoukki
Copy link
Contributor Author

erdoukki commented Jan 7, 2022

It now works with the add of the fail2ban/fail2ban@196c55e commit from fail2ban upstream

@erdoukki
Copy link
Contributor Author

erdoukki commented Jan 8, 2022

TEST IN PROGRESS,
There is an issue on the OpenWrt target:

root@STARGATE:~# fail2ban-client --version
Traceback (most recent call last):
  File "/usr/bin/fail2ban-client", line 34, in <module>
    from fail2ban.client.fail2banclient import exec_command_line, sys
ModuleNotFoundError: No module named 'fail2ban'

I am looking at to solve it. Will update the PR with a fix.
EDIT:
BUILD and TEST compatibility issue here !
the master is against PYTHON v3.10.x and 21.02.x and 19.07.x still use only PYTHON v3.9.x...
So the package include bad version of python components and correct version are missing and get the tool broken !

@jefferyto @commodo
Any help welcome...

Not very "clean" but I am now trying to build with a simple tweak of python3-version.mk to build for the correct target version:

diff --git a/lang/python/python3-version.mk b/lang/python/python3-version.mk
index d85154de5..791dab6b9 100644
--- a/lang/python/python3-version.mk
+++ b/lang/python/python3-version.mk
@@ -7,8 +7,8 @@
 
 # Note: keep in sync with setuptools & pip
 PYTHON3_VERSION_MAJOR:=3
-PYTHON3_VERSION_MINOR:=10
-PYTHON3_VERSION_MICRO:=1
+PYTHON3_VERSION_MINOR:=9
+PYTHON3_VERSION_MICRO:=9
 
 PYTHON3_VERSION:=$(PYTHON3_VERSION_MAJOR).$(PYTHON3_VERSION_MINOR)

Will it be sufficient to get packages for 21.02.x and 19.07.x targets to be built and tested !?
EDIT:
below solution from @jefferyto of cloning openwrt-21.02 for testing in progress

@jefferyto
Copy link
Member

I suggest:

  1. Clone the packages repo into a new directory and check out the openwrt-21.02 (or openwrt-19.07) branch
  2. Copy the new package and dependencies (i.e. 2to3) into the new packages directory
  3. Set up the openwrt-21.02 sdk/buildroot to use the new directory as the packages feed

net/fail2ban/Makefile Outdated Show resolved Hide resolved
@erdoukki erdoukki force-pushed the fail2ban-2to3 branch 2 times, most recently from 94d9e28 to c4ed554 Compare January 8, 2022 18:44
Issue: 2to3 support has been removed in setuptools since version 58.0.0.

Fix: openwrt#17311

Requirements: 2to3/host openwrt#17429

Add upstream patch: fail2ban/fail2ban@196c55e

To install/build for python3 from source, it is necessary to convert to py3
codebase before setup (invoke 2to3 or ./fail2ban-2to3 firstly).
> ./fail2ban-2to3
> python3 setup.py build

Signed-off-by: Kerma Gérald <gandalf@gk2.net>
@erdoukki
Copy link
Contributor Author

erdoukki commented Jan 9, 2022

BUILD and TEST compatibility issue fixed with the correct version of Python.

root@ULTRA-5G:~# opkg install /root/fail2ban_0.11.2-4_aarch64_cortex-a53.ipk --force-reinstall
Removing package fail2ban from root...
Installing fail2ban (0.11.2-4) to root...
Configuring fail2ban.
uci: Entry not found

OpenWrt-21.02:

root@ULTRA-5G:~# fail2ban-client  --version
Fail2Ban v0.11.2

Tests done.
Upgrade fine.
Looks to work fine.

@erdoukki
Copy link
Contributor Author

erdoukki commented Jan 9, 2022

@jefferyto test successful on 21.02.

@erdoukki
Copy link
Contributor Author

@jefferyto
All tests works ok
Build : ok
Install : ok
Upgrade : ok

Copy link
Member

@jefferyto jefferyto left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@jefferyto jefferyto merged commit 15008be into openwrt:master Jan 10, 2022
@erdoukki erdoukki deleted the fail2ban-2to3 branch January 10, 2022 12:57
1715173329 pushed a commit to immortalwrt/packages that referenced this pull request Jan 12, 2022
2to3 is a Python program that reads Python 2.x source code and applies a
  series of fixers to transform it into valid Python 3.x code. The standard
  library contains a rich set of fixers that will handle almost all code. 2to3
  supporting library lib2to3 is, however, a flexible and generic library, so it
  is possible to write your own fixers for 2to3. lib2to3 could also be adapted
  to custom applications in which Python code needs to be edited automatically.

This tool is necessary for fail2ban package because of issue
  openwrt/packages#17311
  openwrt/packages#17341

Simple 2to3.py script from Debian, thanks to Matthias Klose <doko@ubuntu.com>
From: https://salsa.debian.org/cpython-team/python3-defaults

Co-authored-by: Jeffery To <jeffery.to@gmail.com>
Signed-off-by: Kerma Gérald <gandalf@gk2.net>
Signed-off-by: Tianling Shen <cnsztl@immortalwrt.org>
erdoukki added a commit to erdoukki/packages that referenced this pull request Jan 15, 2022
2to3 is a Python program that reads Python 2.x source code and applies a
  series of fixers to transform it into valid Python 3.x code. The standard
  library contains a rich set of fixers that will handle almost all code. 2to3
  supporting library lib2to3 is, however, a flexible and generic library, so it
  is possible to write your own fixers for 2to3. lib2to3 could also be adapted
  to custom applications in which Python code needs to be edited automatically.

This tool is necessary for fail2ban package because of issue
  openwrt#17311
  openwrt#17341

Simple 2to3.py script from Debian, thanks to Matthias Klose <doko@ubuntu.com>
From: https://salsa.debian.org/cpython-team/python3-defaults

(cherry picked from commit 2f91e1c)
Co-authored-by: Jeffery To <jeffery.to@gmail.com>
Signed-off-by: Kerma Gérald <gandalf@gk2.net>
utoni pushed a commit to utoni/openwrt-packages that referenced this pull request Jan 21, 2022
2to3 is a Python program that reads Python 2.x source code and applies a
  series of fixers to transform it into valid Python 3.x code. The standard
  library contains a rich set of fixers that will handle almost all code. 2to3
  supporting library lib2to3 is, however, a flexible and generic library, so it
  is possible to write your own fixers for 2to3. lib2to3 could also be adapted
  to custom applications in which Python code needs to be edited automatically.

This tool is necessary for fail2ban package because of issue
  openwrt#17311
  openwrt#17341

Simple 2to3.py script from Debian, thanks to Matthias Klose <doko@ubuntu.com>
From: https://salsa.debian.org/cpython-team/python3-defaults

Co-authored-by: Jeffery To <jeffery.to@gmail.com>
Signed-off-by: Kerma Gérald <gandalf@gk2.net>
Beginner-Go pushed a commit to coolsnowwolf/packages that referenced this pull request Feb 6, 2022
2to3 is a Python program that reads Python 2.x source code and applies a
  series of fixers to transform it into valid Python 3.x code. The standard
  library contains a rich set of fixers that will handle almost all code. 2to3
  supporting library lib2to3 is, however, a flexible and generic library, so it
  is possible to write your own fixers for 2to3. lib2to3 could also be adapted
  to custom applications in which Python code needs to be edited automatically.

This tool is necessary for fail2ban package because of issue
  openwrt/packages#17311
  openwrt/packages#17341

Simple 2to3.py script from Debian, thanks to Matthias Klose <doko@ubuntu.com>
From: https://salsa.debian.org/cpython-team/python3-defaults

Co-authored-by: Jeffery To <jeffery.to@gmail.com>
Signed-off-by: Kerma Gérald <gandalf@gk2.net>
graysky2 pushed a commit to graysky2/packages that referenced this pull request Feb 6, 2022
2to3 is a Python program that reads Python 2.x source code and applies a
  series of fixers to transform it into valid Python 3.x code. The standard
  library contains a rich set of fixers that will handle almost all code. 2to3
  supporting library lib2to3 is, however, a flexible and generic library, so it
  is possible to write your own fixers for 2to3. lib2to3 could also be adapted
  to custom applications in which Python code needs to be edited automatically.

This tool is necessary for fail2ban package because of issue
  openwrt#17311
  openwrt#17341

Simple 2to3.py script from Debian, thanks to Matthias Klose <doko@ubuntu.com>
From: https://salsa.debian.org/cpython-team/python3-defaults

Co-authored-by: Jeffery To <jeffery.to@gmail.com>
Signed-off-by: Kerma Gérald <gandalf@gk2.net>
theatischbein pushed a commit to theatischbein/packages that referenced this pull request Mar 24, 2022
2to3 is a Python program that reads Python 2.x source code and applies a
  series of fixers to transform it into valid Python 3.x code. The standard
  library contains a rich set of fixers that will handle almost all code. 2to3
  supporting library lib2to3 is, however, a flexible and generic library, so it
  is possible to write your own fixers for 2to3. lib2to3 could also be adapted
  to custom applications in which Python code needs to be edited automatically.

This tool is necessary for fail2ban package because of issue
  openwrt#17311
  openwrt#17341

Simple 2to3.py script from Debian, thanks to Matthias Klose <doko@ubuntu.com>
From: https://salsa.debian.org/cpython-team/python3-defaults

(cherry picked from commit 2f91e1c)
Co-authored-by: Jeffery To <jeffery.to@gmail.com>
Signed-off-by: Kerma Gérald <gandalf@gk2.net>
utoni pushed a commit to utoni/openwrt-packages that referenced this pull request May 30, 2022
2to3 is a Python program that reads Python 2.x source code and applies a
  series of fixers to transform it into valid Python 3.x code. The standard
  library contains a rich set of fixers that will handle almost all code. 2to3
  supporting library lib2to3 is, however, a flexible and generic library, so it
  is possible to write your own fixers for 2to3. lib2to3 could also be adapted
  to custom applications in which Python code needs to be edited automatically.

This tool is necessary for fail2ban package because of issue
  openwrt#17311
  openwrt#17341

Simple 2to3.py script from Debian, thanks to Matthias Klose <doko@ubuntu.com>
From: https://salsa.debian.org/cpython-team/python3-defaults

Co-authored-by: Jeffery To <jeffery.to@gmail.com>
Signed-off-by: Kerma Gérald <gandalf@gk2.net>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

fail2ban: does not compile (_2to3) is invalid
2 participants