Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adjustments for setup requirements #255

Closed
wants to merge 1 commit into from

Conversation

tiran
Copy link
Member

@tiran tiran commented Nov 18, 2016

Fix some typos, missing or surplus dependencies. ipatests is now
installable. Tests need further changes to be runable.

https://fedorahosted.org/freeipa/ticket/6468

Signed-off-by: Christian Heimes cheimes@redhat.com

'dnspython': 'dnspython >= 1.11.1',
'gssapi': 'gssapi > 1.1.2',
'gssapi': 'gssapi >= 1.2.0',
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why did you raised required version?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both gssapi and cryptography has lower version in our specfile. I don't want to have inconsistent versions between RPM and pip installs

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I want to be consistent between RPM and pip, therefore I require the same versions as are available in Fedora 24:

$ rpm -qa python-gssapi python2-cryptography
python2-cryptography-1.5.3-3.fc24.x86_64
python-gssapi-1.2.0-1.fc24.x86_64

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why should we care about F24 versions? We have minimal required versions specified in specfile and these should be used as minimal versions for pip too.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are treating PyPI packages like you are treating RPM packages. That is not how it works. Distribution packages have curated, well test combinations of package versions. Do you have enough resources to test the packages with every release of python-cryptography since 0.9?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with Martin. The spec file contains the actual minimal required versions of the packages that IPA is known to work with and you can track down in git history why exactly it is these particular versions. So NACK as well.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you verify that FreeIPA works with cryptography 0.9?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AFAIK we had a bug somewhere that 0.9 and older versions break Custodia actually.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, but in case you know that cryptography 0.9 doesn't work, you should open a ticket and this should be fixed on both levels, spacfile and setup.py, not just in the one half.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are still applying RPM packaging rules to PyPI packages.

'pyldap': 'pyldap >= 2.4.15',
'pytest': 'pytest < 3.0',
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why pytest < 3 ? in specfile we have >=2.6. If pytest > 3 does not work with IPA, a ticket should be opened and specfile fixed as well.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here

I couldn't run any test with pytest 3. With 2.x some tests were working out of the box.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please open a ticket

@MartinBasti
Copy link
Contributor

I found some changes in versions of dependencies I don't like, because there is no explanation why it is needed and it is out of sync between specfile and setup.py

@MartinBasti
Copy link
Contributor

Better now, but commit message missing explanation why bumping requires was needed.

@pvoborni
Copy link
Member

The commit message doesn't explain why python-gssapi version is raised. Is it required by something? It also doesn't explain if the minimal required version of python-cryptography should be 1.3. Review would be much smoother if this information was here since the beginning.

That said, answers to those questions are not important. Fedora 23+, RHEL 7.3, PyPi all have the same or newer versions and it is actually more work to install older versions. (Not sure about Debian) Or it might not even be possible. So there is no point to waste time with discussing why it needs to be bumped.

If it was a version which would not be on target platforms then it would be different story.

@mbasti-rh is there any other reason for having full explanation of version bumps which I don't see? If not can we move the review forward to unblock #263 ?

@MartinBasti
Copy link
Contributor

@tiran You can split patch to useful part and please send unneeded bumping of requires as separate pull request, we can continue with discussion there about bumping versions. It is unrelated part of patch and should be in separated commit anyway.

@tiran
Copy link
Member Author

tiran commented Nov 29, 2016

@mbasti-rh The bumped version numbers are required. gssapi needs to be bumped because 1.1.x has wrong dependency information for Python 3 (enum34). cryptography 0.9 does not build any more. gssapi 1.2 and cryptography 1.3 are the oldest releases that are actually been tested by QE. I did not bother to verify older releases because I consider it a waste of time and resources. In a couple of weeks we have to bump up cryptography to 1.7 anyway.

@tiran
Copy link
Member Author

tiran commented Nov 29, 2016

PS: There is no technical reason to bump the version of python-gssapi in freeipa.spec. The enum34 dependency issues is solely a Python packaging bug. It does not affect RPM packages. Since you insist on syncing PyPI versions with RPM versions, I had to bump both. Have it your way.

@MartinBasti
Copy link
Contributor

@mbasti-rh The bumped version numbers are required. gssapi needs to be bumped because 1.1.x has wrong dependency information for Python 3 (enum34).
So, this is broken fedora dependency on fedora side? If yes then this should be fixed by fedora downstream patch. I don't see reason why upstream version should have raised dependency just because fedora is broken.

cryptography 0.9 does not build any more. gssapi 1.2 and cryptography 1.3 are the oldest releases that are actually been tested by QE. I did not bother to verify older releases because I consider it a waste of time and resources. In a couple of weeks we have to bump up cryptography to 1.7 anyway.

I don't see reason why bumping requires just because we are unable to build on fedora. Fedora is not the only linux distro.

@MartinBasti
Copy link
Contributor

PS: There is no technical reason to bump the version of python-gssapi in freeipa.spec. The enum34 dependency issues is solely a Python packaging bug. It does not affect RPM packages. Since you insist on syncing PyPI versions with RPM versions, I had to bump both. Have it your way.

So finally we have reason to bump version, which should be docummented in git history as separate commit.

@tiran
Copy link
Member Author

tiran commented Nov 29, 2016

You said Fedora, I didn't. The build bug is not related to Fedora at all. Cryptography 0.9 does not build on any distribution or platform with a recent version of OpenSSL.

Touché, I said Fedora in the commit message.

@MartinBasti
Copy link
Contributor

So create a separate commits:

  • fixes ipasetup.py
  • bumps python-gssapi for pypi, with proper explanation in commit message and maybe comment in code may be helpful.
    And we will be happy because we have reason why it needs to be raised and this reason can be found in git history.

I'm still not persuaded with need for bumping cryptography.

@tiran
Copy link
Member Author

tiran commented Nov 29, 2016

Would you rather claim to be compatible with a broken, unsupported, and old version?

@tiran tiran force-pushed the setup_requirements branch 3 times, most recently from 7f0ea93 to d2936b3 Compare November 29, 2016 15:12
@MartinBasti MartinBasti removed their assignment Nov 29, 2016
@martbab
Copy link
Contributor

martbab commented Nov 29, 2016

Well from our (as upstream) POV 0.9 and later is required for Custodia to work correctly. This requirement was introduced by me in commit aa74995 when I was building 4.3 in Copr for CentOS 7. There was ye olde 0.8 something version and I found empirically that 0.9 or later is required for replica promotion to work (at that time 1.2.1 was the most up-to-date version built in Brew IIRC). Yes, this version is ancient and vast majority of distros does not support it anymore but then it is their job to provide newer version fullfilling our Required and I see no point in artificially bumping it in upstream unless some of our code depends on functionality of newer version.

I mentioned the CentOS story as an example that demonstrates that you never know on what distro your software is being ported.

That said, if you are afraid that it can break the PIP use-case then I am fine with bumping the version but as @mbasti-rh said, please split version bumps into a separate commit with clean explanation of the reasons (already provided in the commit message).

This makes it easier for our future selves to review the build/runtime requirements during spec file cleanups and similar work. I remember that @jcholast was very frustrated when he was cleaning up BuildRequires recently and was unable to find any reasonable explanation for many of them in git history.

@tiran
Copy link
Member Author

tiran commented Nov 29, 2016

@martbab Welcome to the party! This discussion has been running for a very long time and in multiple places. Let me bring you up to speed.

First of all the requirements in ipasetup.py are completely unrelated to distribution packaging (RPM, DEB, whatever). PyPI packaging follows slightly different rules. For example you don't get carefully curated packages, downstream patches for build issues or a known working set of packages. It's a bit more wild west and fast moving. I was against bumping the version in the spec file because the bump is not required for my work. The other insisted on it.

Next up a version information like "cryptography >= 0.9" means that any version equal or greater than 0.9 is known to work. If you follow upstream development of OpenSSL and Cryptography closely then you are aware that any version of cryptography < 1.3 does no longer compile against a recent version of OpenSSL 1.0.2. CFFI bindings are very sensitive to subtle changes in the ABI and C API. OpenSSL tend to break both every now and then.

Finally this discussion is pointless. I will bump the version requirements of cryptography to 1.7.0 in a matter of weeks. BZ for RHEL has been filed. The version 1.7.0 hasn't been released yet. it will contain two important fixes (lock and osrandom) and a new feature for @frasertweedale (multi RDN).

$ python3 -m venv /tmp/cryptovenv
$ . /tmp/cryptovenv/bin/activate
(cryptovenv) $ pip install 'cryptography==0.9' 
Collecting cryptography==0.9
  Downloading cryptography-0.9.tar.gz (302kB)
    100% |████████████████████████████████| 303kB 122kB/s 
Collecting idna (from cryptography==0.9)
  Using cached idna-2.1-py2.py3-none-any.whl
Collecting pyasn1 (from cryptography==0.9)
  Using cached pyasn1-0.1.9-py2.py3-none-any.whl
Collecting six>=1.4.1 (from cryptography==0.9)
  Using cached six-1.10.0-py2.py3-none-any.whl
Requirement already satisfied (use --upgrade to upgrade): setuptools in ./cryptovenv/lib/python3.5/site-packages (from cryptography==0.9)
Collecting cffi>=0.8 (from cryptography==0.9)
  Using cached cffi-1.9.1.tar.gz
Collecting pycparser (from cffi>=0.8->cryptography==0.9)
Installing collected packages: idna, pyasn1, six, pycparser, cffi, cryptography
  Running setup.py install for cffi ... done
  Running setup.py install for cryptography ... error
    Complete output from command /tmp/cryptovenv/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-_2z81799/cryptography/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-83qpivr4-record/install-record.txt --single-version-externally-managed --compile --install-headers /tmp/cryptovenv/include/site/python3.5/cryptography:
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.5
    creating build/lib.linux-x86_64-3.5/cryptography
...
    running build_ext
    building '_Cryptography_cffi_1251de2xc302a38b' extension
    creating build/temp.linux-x86_64-3.5
    creating build/temp.linux-x86_64-3.5/src
    creating build/temp.linux-x86_64-3.5/src/cryptography
    creating build/temp.linux-x86_64-3.5/src/cryptography/hazmat
    creating build/temp.linux-x86_64-3.5/src/cryptography/hazmat/bindings
    creating build/temp.linux-x86_64-3.5/src/cryptography/hazmat/bindings/__pycache__
    gcc -pthread -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/tmp/cryptovenv/include -I/usr/include/python3.5m -c src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_1251de2xc302a38b.c -o build/temp.linux-x86_64-3.5/src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_1251de2xc302a38b.o
    src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_1251de2xc302a38b.c:505:6: error: conflicting types for ‘BIO_new_mem_buf’
     BIO *BIO_new_mem_buf(void *, int);
          ^~~~~~~~~~~~~~~
    In file included from /usr/include/openssl/asn1.h:65:0,
                     from src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_1251de2xc302a38b.c:220:
    /usr/include/openssl/bio.h:692:6: note: previous declaration of ‘BIO_new_mem_buf’ was here
     BIO *BIO_new_mem_buf(const void *buf, int len);
          ^~~~~~~~~~~~~~~
    src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_1251de2xc302a38b.c:2019:15: error: ‘SSLv2_method’ redeclared as different kind of symbol
     SSL_METHOD* (*SSLv2_method)(void) = NULL;
                   ^~~~~~~~~~~~
    In file included from src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_1251de2xc302a38b.c:316:0:
    /usr/include/openssl/ssl.h:2287:19: note: previous declaration of ‘SSLv2_method’ was here
     const SSL_METHOD *SSLv2_method(void); /* SSLv2 */
                       ^~~~~~~~~~~~
    src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_1251de2xc302a38b.c:2020:15: error: ‘SSLv2_client_method’ redeclared as different kind of symbol
     SSL_METHOD* (*SSLv2_client_method)(void) = NULL;
                   ^~~~~~~~~~~~~~~~~~~
    In file included from src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_1251de2xc302a38b.c:316:0:
    /usr/include/openssl/ssl.h:2289:19: note: previous declaration of ‘SSLv2_client_method’ was here
     const SSL_METHOD *SSLv2_client_method(void); /* SSLv2 */
                       ^~~~~~~~~~~~~~~~~~~
    src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_1251de2xc302a38b.c:2021:15: error: ‘SSLv2_server_method’ redeclared as different kind of symbol
     SSL_METHOD* (*SSLv2_server_method)(void) = NULL;
                   ^~~~~~~~~~~~~~~~~~~
    In file included from src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_1251de2xc302a38b.c:316:0:
    /usr/include/openssl/ssl.h:2288:19: note: previous declaration of ‘SSLv2_server_method’ was here
     const SSL_METHOD *SSLv2_server_method(void); /* SSLv2 */
                       ^~~~~~~~~~~~~~~~~~~
    src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_1251de2xc302a38b.c: In function ‘_cffi_f_EC_GFp_nistp224_method’:
    src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_1251de2xc302a38b.c:24411:14: warning: implicit declaration of function ‘EC_GFp_nistp224_method’ [-Wimplicit-function-declaration]
       { result = EC_GFp_nistp224_method(); }
                  ^~~~~~~~~~~~~~~~~~~~~~
    src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_1251de2xc302a38b.c:24411:12: warning: assignment makes pointer from integer without a cast [-Wint-conversion]
       { result = EC_GFp_nistp224_method(); }
                ^
    error: command 'gcc' failed with exit status 1
    
    ----------------------------------------
Command "/tmp/cryptovenv/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-_2z81799/cryptography/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-83qpivr4-record/install-record.txt --single-version-externally-managed --compile --install-headers /tmp/cryptovenv/include/site/python3.5/cryptography" failed with error code 1 in /tmp/pip-build-_2z81799/cryptography

1.2

$ pip install 'cryptography==1.2'
...
    running build_ext
    generating cffi module 'build/temp.linux-x86_64-3.5/_padding.c'
    creating build/temp.linux-x86_64-3.5
    generating cffi module 'build/temp.linux-x86_64-3.5/_constant_time.c'
    generating cffi module 'build/temp.linux-x86_64-3.5/_openssl.c'
    building '_openssl' extension
    creating build/temp.linux-x86_64-3.5/build
    creating build/temp.linux-x86_64-3.5/build/temp.linux-x86_64-3.5
    gcc -pthread -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/tmp/cryptovenv/include -I/usr/include/python3.5m -c build/temp.linux-x86_64-3.5/_openssl.c -o build/temp.linux-x86_64-3.5/build/temp.linux-x86_64-3.5/_openssl.o
    build/temp.linux-x86_64-3.5/_openssl.c:737:6: error: conflicting types for ‘BIO_new_mem_buf’
     BIO *BIO_new_mem_buf(void *, int);
          ^~~~~~~~~~~~~~~
    In file included from /usr/include/openssl/asn1.h:65:0,
                     from build/temp.linux-x86_64-3.5/_openssl.c:445:
    /usr/include/openssl/bio.h:692:6: note: previous declaration of ‘BIO_new_mem_buf’ was here
     BIO *BIO_new_mem_buf(const void *buf, int len);
          ^~~~~~~~~~~~~~~
    error: command 'gcc' failed with exit status 1
    
    ----------------------------------------
Command "/tmp/cryptovenv/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-c4zo1h2l/cryptography/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-xlaxncs5-record/install-record.txt --single-version-externally-managed --compile --install-headers /tmp/cryptovenv/include/site/python3.5/cryptography" failed with error code 1 in /tmp/pip-build-c4zo1h2l/cryptography

@martbab
Copy link
Contributor

martbab commented Nov 30, 2016

As I said, if 0.9 break your PyPI work feel freee to bump it but please split the version bumps into a separate commit on top of ipasetup fixes.

* Fix some typos, missing or surplus dependencies.
* Remove setup requirement on wheel since it triggers download.

ipatests is now installable. Tests need further changes to be runable.

https://fedorahosted.org/freeipa/ticket/6468

Signed-off-by: Christian Heimes <cheimes@redhat.com>
@tiran
Copy link
Member Author

tiran commented Nov 30, 2016

I opened PR #289 and #290.

@martbab
Copy link
Contributor

martbab commented Nov 30, 2016

Thank you.

It seems that 'bdist_wheel' target is broken in your PR:

# make bdist_wheel 
mkdir -p ./dist/wheels
for dir in ipaclient ipalib ipaplatform ipapython; do \
    make  -C ${dir} bdist_wheel || exit 1; \
done
make[1]: Entering directory '/freeipa/ipaclient'
(cd .. && make  ipasetup.py)
make[2]: Entering directory '/freeipa'
sed                                             \
        -e 's|@VERSION[@]|4.4.90.dev201611301151+git785f924|g'                  \
        ipasetup.py.in > ipasetup.py
make[2]: Leaving directory '/freeipa'
rm -rf ../dist/wheels/ipaclient*.whl
/usr/bin/python "./setup.py" bdist_wheel --dist-dir=../dist/wheels
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: setup.py --help [cmd1 cmd2 ...]
   or: setup.py --help-commands
   or: setup.py cmd --help

error: invalid command 'bdist_wheel'
Makefile:586: recipe for target 'bdist_wheel' failed
make[1]: *** [bdist_wheel] Error 1
make[1]: Leaving directory '/freeipa/ipaclient'
Makefile:1172: recipe for target 'bdist_wheel' failed
make: *** [bdist_wheel] Error 1

Do i need some of your other pull-requests to build wheels or this is a genuine issue?

@tiran
Copy link
Member Author

tiran commented Nov 30, 2016

The bdist_wheel command requires the Python wheel package installed in the system. Since setup.py no longer contains setup_requires=["wheel"], the dependency is no longer resolved automatically by setuptools.

Does it makes sense to include the dependency in freeipa.spec as build requirement? Technically it's not a build requirement for RPMs.

@martbab
Copy link
Contributor

martbab commented Nov 30, 2016

Installing python-wheel worked, thanks. I have discovered some other missing dependencies in minimal Docker container. I will investigate them some more and open a ticket. I think there is no need to add python-wheel to BuildRequires now.

@martbab martbab added ack Pull Request approved, can be merged pushed Pull Request has already been pushed labels Nov 30, 2016
@martbab
Copy link
Contributor

martbab commented Nov 30, 2016

@martbab martbab closed this Nov 30, 2016
@tiran
Copy link
Member Author

tiran commented Nov 30, 2016

@martbab The wheel bundle and packages need some documentation. I have started some docs but they are not finished..

@tiran tiran deleted the setup_requirements branch November 30, 2016 13:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ack Pull Request approved, can be merged pushed Pull Request has already been pushed
Projects
None yet
5 participants