New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade all packages using `pur` tool #2916

Merged
merged 10 commits into from Mar 15, 2018

Conversation

Projects
None yet
6 participants
@humitos
Member

humitos commented May 28, 2017

These ones were not upgraded since they are incompatible:

  • django
  • docker-py
  • celery
  • elasticsearch
  • pyelasticsearch

This is just a way to run all the test with this environment and also to discuss with other people if this is something we really want.

Closes #1893

@humitos

This comment has been minimized.

Member

humitos commented Nov 16, 2017

@agjohnson is this something that we want or do we prefer to upgrade packages one by one? Let me know, so I can update the PR or just close it.

@humitos

This comment has been minimized.

Member

humitos commented Dec 21, 2017

Updated today to the latest versions compatible with our codebase.

@ericholscher @agjohnson please, let me know if you are interested on these kind of upgrade or not.

  • I didn't touch docker-py here because it's done at #3243
  • django needs extra work, which is being done at #3372
  • elasticsearch didn't touch since Eric is working on that #3373
@ericholscher

This comment has been minimized.

Member

ericholscher commented Dec 21, 2017

I think it's a good practice. I will let this sit until Friday then merge it, so we can do some local testing etc on it before deploy.

@humitos

This comment has been minimized.

Member

humitos commented Dec 23, 2017

It's Friday night here, nobody is around and Xmas is comming this weekend. So, maybe it's a better idea to merge this next week :)

@humitos humitos referenced this pull request Dec 29, 2017

Closed

Minor virtualenv upgrade #3463

@humitos humitos self-assigned this Jan 15, 2018

mkdocs==0.14.0
django==1.9.12
six==1.10.0
mkdocs==0.17.2

This comment has been minimized.

@agjohnson

agjohnson Jan 15, 2018

Contributor

We don't support mkdocs==0.17, not sure if we want this here.

This comment has been minimized.

@humitos

humitos Jan 15, 2018

Member

Why not? Do we you have an example where it fails?

I could write a test to make it fail so we don't hit this issue again and downgrade to 0.14.0 as it was before this PR.

This comment has been minimized.

@agjohnson

agjohnson Mar 7, 2018

Contributor

mkdocs 0.17 doesn't work at all with RTD. They removed a feature we required. 0.16.x was the last working version

This comment has been minimized.

@humitos

humitos Mar 8, 2018

Member

This is confusing for me.

First, why do we need mkdocs for RTD itself? In fact, can we remove it from here? This is the requirements for the RTD project not the mkdocs installed by default in the user builds.

I did a quick grep and I found that we have 0.14.0 in the RTD reqs and 0.15.0 in the reqs for user builds:

$ rg mkdocs=
requirements/pip.txt
9:mkdocs==0.14.0

readthedocs/doc_builder/python_environments.py
215:            requirements.append('mkdocs==0.15.0')

This comment has been minimized.

@humitos

humitos Mar 9, 2018

Member

I pinned to 0.15.0 --the same that we install by default in the venvs

This comment has been minimized.

@davidfischer

davidfischer Apr 24, 2018

Contributor

Do you know which feature was removed in 0.17? There's at least one ticket (#3991) where the recommended fix from an mkdocs contributor was to upgrade.

@humitos

This comment has been minimized.

Member

humitos commented Jan 24, 2018

@agjohnson I updated this PR with the latest packages

@humitos

This comment has been minimized.

Member

humitos commented Jan 27, 2018

I didn't touch the deploy.txt requeriments, but just in case we want to upgrade it, here is the output:

$ pur --skip django-tastypie,django,docker-py,elasticsearch,pyelasticsearch,commonmark,stripe,djangorestframework -r requirements/deploy.txt 
New version for readthedocs-build found (2.0.9), but current spec prohibits updating: readthedocs-build<2.1
Updated gunicorn: 19.1.0 -> 19.7.1
Updated pysolr: 2.0.13 -> 3.7.0
Updated django-redis-cache: 1.6.3 -> 1.7.1
Updated django-mailgun: 0.2.2 -> 0.9.1
Updated hiredis: 0.1.2 -> 0.2.0
Updated cssselect: 0.7.1 -> 1.0.3
All requirements up-to-date.

Should I commit this changes? Is it safe?

This was referenced Feb 21, 2018

@xrmx

This comment has been minimized.

Contributor

xrmx commented Feb 22, 2018

You may want to upgrade to linaro_django_pagination==2.0.3 as seen on #3305 too

@humitos

This comment has been minimized.

Member

humitos commented Feb 22, 2018

@xrmx I didn't upgrade it because it pinned to a very specific commit from a fork of the original repo (I tried upgrading it and test failed) --and as I don't know the main reason I prefered to leave it as it was.

In fact, in the issue you linked here, they are talking about some problems with that package and they don't know how to fix it.

So, I wouldn't touch it without knowing the implicances. Maybe, as they are saying is good to remove that dependency after a research on how it would affect.

@humitos

This comment has been minimized.

Member

humitos commented Feb 23, 2018

Tow more related PRs that I wanted to include in this branch:

  • upgrade django pagination #3666
  • upgrade django taggit #3667
@humitos

This comment has been minimized.

Member

humitos commented Feb 23, 2018

@xrmx maybe you are interested in my last two PRs from my last comment :)

@xrmx

This comment has been minimized.

Contributor

xrmx commented Feb 27, 2018

@humitos nice, thanks!

@xrmx

This comment has been minimized.

Contributor

xrmx commented Feb 27, 2018

Wondering if pyelasticsearch can be dropped, git grep pyelasticsearch returns nothing and (current) haystack uses elasticsearch-py.

@humitos

This comment has been minimized.

Member

humitos commented Feb 28, 2018

@xrmx maybe. How elasticsearch works currently in production is a mistery for me, so I didn't want to get involved on that. As I don't have experience with that I didn't touch it.

@xrmx

This comment has been minimized.

Contributor

xrmx commented Feb 28, 2018

@humitos if nobody uses it, what can it break? Anyway i've removed it from my local installation and nothing changed.

@humitos

This comment has been minimized.

Member

humitos commented Feb 28, 2018

@xrmx I'm not saying that nobody uses it. I'm saying that I don't know how it's used, if it is and that's why I didn't wanted to remove it.

The search engine is complex (in fact, you opened another issue saying that it doesn't work your local instance). So, you don't know if it doesn't break things.

@agjohnson

This comment has been minimized.

Contributor

agjohnson commented Mar 7, 2018

elasticsearch is indeed used, it shouldn't be removed.

This has a high breakage potential. We should plan on merging on a week where we have plenty of eyes on the deploy. I don't have any strong feedback here otherwise, so effectively +1. Perhaps next week we can merge early, QA, then deploy midweek?

@humitos

This comment has been minimized.

Member

humitos commented Mar 8, 2018

@agjohnson agree with you. Let me know what to do with the mkdocs package, and we will be ready to go.

@xrmx

This comment has been minimized.

Contributor

xrmx commented Mar 8, 2018

@agjohnson we were talking about pyelasticsearch not elasticsearch :)

@ericholscher

This comment has been minimized.

Member

ericholscher commented Mar 8, 2018

@humitos we have mkdocs there so that people running it locally can build mkdocs docs. This is less important now that we have docker support. There is likely a lot that could be removed that is a build dependency, but not required for basic dev.

@humitos

This comment has been minimized.

Member

humitos commented Mar 9, 2018

@ericholscher thanks for the explanation about this. Considering what you have said, I think it's a good idea to split our requirements/pip.txt to have all the RTD specific requirements and another one for local builds (using LocalBuildEnvironment class). So, if you are running Docker you don't need to install them.

Besides, it easier to know what dependency is for. What do you think?

I will downgrade mkdocs to keep it compatible as @agjohnson said and if you like my proposal, I will achieve that in another PR.

@ericholscher

This comment has been minimized.

Member

ericholscher commented Mar 9, 2018

Big +1 on simplifying the requirements file for local dev. It should def. be in another PR, but I think it is a big reason that people have a hard time getting started, so reducing the "time to running local instance" is a good metric to increase contribution!

humitos added some commits May 28, 2017

Upgrade all packages using `pur` tool
These ones were not upgraded since they are incompatible:

* docker-py
* celery
* elasticsearch
* pyelasticsearch
Upgrade all packages with `pur`
    pur --skip django,docker-py,elasticsearch,pyelasticsearch

and some packages pinned manually to avoid conflicts with our tests.
Downgrade commonmark to 0.5.5
In 6.x changelog

> DocParser has been renamed to Parser.

https://github.com/rtfd/CommonMark-py/releases/tag/0.6.0

and that it's not compatible with the latest version of
recommonmark (0.4.0)
`assert_not_called` replaced by `call_count`
It doesn't receive arguments and can't be used at that point because
it was called previously by the other POST on the API:

https://docs.python.org/3/library/unittest.mock.html#unittest.mock.Mock.assert_not_called
Upgrade packages with `pur`
$ pur --skip django-tastypie,django,docker-py,elasticsearch,pyelasticsearch,commonmark,stripe,djangorestframework
Pin django-allauth to 0.32.0
django-allauth 0.33.0 doesn't support Django 1.9.x which is the
version we are using at the moment.
Pin mkdocs to 0.15.0 for compatibility reasons
This version is the one installed by default on the venv to build the
user's documentation.
@agjohnson

This comment has been minimized.

Contributor

agjohnson commented Mar 15, 2018

Rebased this, should hopefully be able to be merged after tests.

@stsewd

Mocks and random methods names...

Proper call to the superproject symlink on Project.save
`Project.superprojects.all()` returns `ProjectRelationship` instances
where we need to use the `parent` or `child` attribute to access to
the project itself.

(we were sending the `ProjectRelationship.pk`)
@humitos

This comment has been minimized.

Member

humitos commented Mar 15, 2018

I think we are all good now.

Some context:

  • mock==1.0 probably wasn't checking that the assert_ method exist in the mock object and was considering it as another mocked method (arbitrary) that you can call --that's why it didn't fail in the test.
  • now, mock==2.0 is checking this and the test start failing because the name with the final s was a mistake
  • finally, there was a bug on how the broadcast method was being called inside the Project.save since it was using the ProjectRelationship.pk instead of the proper pk from the superproject :)
@agjohnson

This comment has been minimized.

Contributor

agjohnson commented Mar 15, 2018

Nice! I wonder if this actually resolves some of the problems with symlinking that we noticed in the deploys.

I'll go ahead and merge this and the docker upgrade as well.

@agjohnson agjohnson merged commit 50e7d4b into master Mar 15, 2018

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

@agjohnson agjohnson deleted the humitos/pip/upgrade-packages branch Mar 15, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment