Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cacert.pem is missing #109

Closed
dkliban opened this issue Oct 19, 2015 · 11 comments
Closed

cacert.pem is missing #109

dkliban opened this issue Oct 19, 2015 · 11 comments
Assignees
Labels

Comments

@dkliban
Copy link

@dkliban dkliban commented Oct 19, 2015

After upgrading to 1.4.0 the following snippet

from redmine import Redmine
redmine = Redmine(REDMINE_URL, key=key)
redmine.issue.filter(query_id=24)

is producing this traceback:

Traceback (most recent call last):
  File "pulp_packaging/ci/redmine_bugzilla.py", line 198, in <module>
    main()
  File "pulp_packaging/ci/redmine_bugzilla.py", line 102, in main
    redmine_issues = [issue for issue in redmine.issue.filter(query_id=24)]
  File "/usr/lib/python2.7/site-packages/redmine/resultsets.py", line 102, in __iter__
    self._evaluate()
  File "/usr/lib/python2.7/site-packages/redmine/resultsets.py", line 81, in _evaluate
    offset=self.manager.params.get('offset', self.offset)
  File "/usr/lib/python2.7/site-packages/redmine/managers.py", line 62, in retrieve
    response = self.redmine.request('get', self.url, params=dict(self.params, limit=limit, offset=offset))
  File "/usr/lib/python2.7/site-packages/redmine/__init__.py", line 119, in request
    response = getattr(requests, method)(url, **kwargs)
  File "/usr/lib/python2.7/site-packages/redmine/packages/requests/api.py", line 69, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/lib/python2.7/site-packages/redmine/packages/requests/api.py", line 50, in request
    response = session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python2.7/site-packages/redmine/packages/requests/sessions.py", line 468, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/site-packages/redmine/packages/requests/sessions.py", line 576, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/site-packages/redmine/packages/requests/adapters.py", line 433, in send
    raise SSLError(e, request=request)
redmine.packages.requests.exceptions.SSLError: [Errno bad ca_certs: '/usr/lib/python2.7/site-packages/redmine/packages/requests/cacert.pem'] [('system library', 'fopen', 'No such file or directory'), ('BIO routines', 'BIO_new_file', 'no such file'), ('x509 certificate routines', 'X509_load_cert_crl_file', 'system lib')]
@dkliban
Copy link
Author

@dkliban dkliban commented Oct 19, 2015

The snippet I posted there is not quite what is in my script. The script actually attempts to do list comprehension on the result of redmine.issue.filter(query_id=24). Just wanted to explain why traceback has

redmine_issues = [issue for issue in redmine.issue.filter(query_id=24)]
@maxtepkeev maxtepkeev added the question label Oct 19, 2015
@maxtepkeev maxtepkeev self-assigned this Oct 19, 2015
@maxtepkeev
Copy link
Owner

@maxtepkeev maxtepkeev commented Oct 19, 2015

Starting from Python-Redmine v1.4.0 Requests is now bundled, currently used version is 2.8.1. Also starting from Requests v2.4.0 they extracted certificates from Requests into separate project called certifi. So, you need to adjust your connection options and either set verify to false or use certifi and provide it's certificate to Python-Redmine, see here and here more info about how to use it.

In the meantime I'll think about whether it's a good or bad idea to bundle this certificates with Python-Redmine and automatically set needed parameters for the bundled Requests library.

EDIT: Looks like you can just pip install certifi and everything will work without making any code modifications.

@bmbouter
Copy link

@bmbouter bmbouter commented Oct 19, 2015

I really like python-redmine. It was a big part of our choice to user Redmine overall. Thanks for your work so far on it! In the spirit of improvement, here are two related thoughts to this issue.

+1 to not bundling certificates into python-redmine. Recommend that people use certifi (as you did here). It's going to be more up to date and trust decisions should be an explicit user decision for security reasons.

+1 to not bundling python-requests with python-redmine at all. I can see why it would be helpful/easy given the difficulty with different versions users will run, but consider instead picking a minimum version of python-requests and letting easy_install or pip install the real python-requests. This may cause friction for some users, but they can choose to upgrade or not. As the developer, you'll focus on making your software integrate well with python-requests instead of repackaging python-requests inside of python-redmine. Also you'll get compatibility testing of newer versions of python-requests as users receive newer and newer versions and they can always downgrade to workaround the incompatibility until a newer python-redmine can be released that is compatible with the latest python-requests. This will make it easier for python-redmine to switch newer versions of python-requests which eventually it will need to do.

Here's a security example relevant also motivating the two above points.

(1) Consider that a CA was deemed insecure by Mozilla and/or certifi and revoked. If it was carried inside of python-redmine you'll be distributing insecure trust info until you update python-redmine and have your users upgrade to the new python-redmine.

(2) What if a security vulnerability is discovered in python-requests 2.8.1? A new version will be released with the fix, but instead of upgrading just certifi and feeling safe, we also would have to upgrade python-redmine assuming a "new" version has been released.

In both cases additional, unnecessary work had to be done and it's still vulnerable for longer. Thanks again for the very excellent python-redmine!

@dkliban
Copy link
Author

@dkliban dkliban commented Oct 20, 2015

Pip installing certifi fixed the problem. Perhaps you can add certifi as dependency for python-redmine to make installing simpler?

@maxtepkeev
Copy link
Owner

@maxtepkeev maxtepkeev commented Oct 20, 2015

@dkliban Good to hear. Thanks.
@bmbouter Thanks for your opinion. I appreciate every point of view. Of course everything you say is correct and all the points are valid, but let me put it another way. I assume that you came here because you got this cacert.pem is missing error after upgrading to python-redmine v1.4.0. You got it because you didn't upgrade your version of python-requests for some time, I believe you should have had a version below 2.4.0 because that was the version where they moved certificates into separate package.

So, what if python-redmine didn't bundle the python-requests ? Everything would continue to work of course, but when would you upgrade your python-requests, how long would you continue to use this outdated but working version installed in your system ? How many security issues have been already discovered and which of them affects your outdated python-requests version ?

What I'm trying to say is that you, as a user of python-redmine, are only interested in this concrete software and you only check if there are new versions of python-redmine, you don't check for new versions of python-requests or certifi or whatever dependency a software might have (well maybe you do, but believe me, most people don't) and this is a really big problem, it creates a lot more security issues than you described.

A lot of people only upgrade python-redmine and don't care about whether there are new versions of python-requests, but with this new model, when python-requests is bundled within python-redmine users don't need to check for new versions of the dependencies anymore, I will do this for them, it is my responsibility, as the author of the library, to provide a secure software. You can say that I don't need to bundle python-requests for that, I just need to change the required version number for the most recent secure version, but that would still require to release a new version of python-redmine plus if there is any other software that uses python-requests it might just stop working and a user would have more problems (creating separate virtualenvs etc.).

Finally, I believe that bundling python-requests is a right decision, I also believe that certifi should also be bundled BUT I have a proposal to make power users happy. In the new version (to be released) python-redmine will check if there is a python-requests or certifi installed in the system and their version(s) are greater than the ones used by python-redmine internally, python-redmine will use them, if not it will fall back to the bundled ones. This way users can instantly upgrade python-requests and certifi if needed.

Waiting for your opinions about this proposal.

@bmbouter
Copy link

@bmbouter bmbouter commented Nov 3, 2015

@maxtepkeev Thanks for the detailed response. Sorry it has taken me soooo long to get back to you. I've read through your answer and I agree if end users only upgrade python-redmine and never upgrade python-requests then they will be insecure and likely end up with broken software. This would especially happen if changes in python-redmine require a specific version of python-requests and the user upgrades one without the other. Given that I can see how carrying it a resolution to that problem, but I think there is another way.

Setuptools supports dependency declaration using the install_requires mechanism. This way when users pip install python-redmine or python setup.py install from a checked out version of python-redmine, their dependencies will be upgraded with it. The current setup.py on master doesn't use install_requires. If that were added, I think the issue of upgrading python-redmine to version X and having any dependency version X requires such as python-requests being at a minimum version Y would be a non-issue.

What do you think about using that to manage dependencies instead of carrying internal versions of them?

@maxtepkeev this response is in the spirit of collaboration and improvement. I appreciate how much you've put into thinking about this issue and writing out your thoughts.

@maxtepkeev
Copy link
Owner

@maxtepkeev maxtepkeev commented Nov 4, 2015

@bmbouter Thanks for your response. I already described the cons of the setuptools install_requires mechanism in my previous response, but it seems you missed that, no problem, I'll repeat my point of view about that in more details.

Current version of setup.py in master branch doesn't have an install_requires directive because now python-redmine bundles (vendors) all the dependencies inside itself. Before version 1.4.0 there was an install_requires directive which specified the minimum required python-requests version for python-redmine to work.

python-requests is a very very popular library which is used by A LOT of projects, so, let's imagine that a new version of python-requests was released and I'll change the install_requires directive to point to that version, to be sure that when users of python-redmine will get a new version they will also get a new version of python-requests, so far so good. But as I said python-requests is a very popular library, so imagine that you also have another library installed that will stop working because of this new version of python-requests, I think you won't be happy because of that. Of course there is a solution to this problem, you'll have to create two separate virtualenvs and put python-redmine in one of them and your second library in another, but that will require you to do that and to waste your time on that. I don't want to waste the time of users of python-redmine no matter how good my intentions are.

So, bundling (vendoring) the software (with the automatic fallback to using the newest version of python-requests if it's available in the system of course, like I described in my proposal in my previous response) is the only way to make the software secure, up-to-date and with a happy user base.

@bmbouter
Copy link

@bmbouter bmbouter commented Nov 5, 2015

@maxtepkeev Thanks for your response. Bundling it for compatibility sounds ok as long as you're sure that if the system has a newer version of python-requests that it will be used. Where in the code does it do that?

Given that, if the bundled version is insecure and the user hasn't installed python-requests explicitly (which would give them the secure version) then they will be vulnerable until you release a rebundled version. This is the classic tradeoff between usability and security. Systems that are perfectly secure are generally unusable. There are upsides and downsides to both. You've clearly thought it through so that is good. Since you're choosing usability, I assume you'll be following python-requests community closely for security vulnerabilities discovered since you'll be responsible for quickly repackaging them.

Thanks for the very excellent python-redmine which we use everyday. Thanks also for the thoughtful responses.

@maxtepkeev
Copy link
Owner

@maxtepkeev maxtepkeev commented Nov 5, 2015

@bmbouter Thanks for the response.

Where in the code does it do that?

This code isn't written yet. My free time currently is very limited, but I'll try to do that ASAP.

Given that, if the bundled version is insecure and the user hasn't installed python-requests explicitly (which would give them the secure version) then they will be vulnerable until you release a rebundled version.

You're absolutely right, but the same situation would be if I stopped vendoring python-requests. User would get a new python-requests implicitly only after upgrading the python-redmine. The big difference is that with vendoring we won't brake any other packages that don't work with the new version of python-redmine. That is why I don't see any cons in the chosen approach.

I'm glad that you liked python-redmine and I appreciate the time you spent writing these responses. Thanks.

@languitar
Copy link

@languitar languitar commented Nov 17, 2015

I am also against using bundled versions per default. This completely works around the established package management standards in the python ecosystem and prevents the user or potential distribution package maintainers from easily addressing security issues in upstream libraries.

maxtepkeev added a commit that referenced this issue Nov 26, 2015
@maxtepkeev maxtepkeev added bug improvement and removed question labels Nov 26, 2015
@maxtepkeev
Copy link
Owner

@maxtepkeev maxtepkeev commented Nov 26, 2015

Python-Redmine v1.5.0 introduces "smart imports" which fixes all the above issues, see docs for details.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants
You can’t perform that action at this time.