New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade to django-1.4 #758

Merged
merged 13 commits into from Jan 11, 2016

Conversation

Projects
None yet
2 participants
@zeehio
Copy link
Contributor

zeehio commented Jul 2, 2015

This pull request closes issue #751.

After some weeks using this patched version, today I had the time to rebase the commits into some meaningful history. It took me a while, because I had never had the need to order so many commits and changes to files.

I have tried to make the commit description as meaningful as possible, although some commits are so big that it is impossible for me to describe all the details in the commit message. I could try to split them if you want me to, but I wanted to make the commits on the master branch able to pass the testsuite as much as possible.

The first commit writes a requirements.txt file that replaces the original metashare 3.0.2 lib/ directory that contained all the bundled dependencies. The only change is an upgrade from django 1.3.3 to django 1.3.7 with no side effects. Additionally, the install-dependencies.sh script is updated to use virtualenv and pip, so no further PATH modifications are required anymore.

The second commit deals with the test environment and its automation using travis, sauce labs and coveralls, tools free to use for free and open source software. Local tests are of course also supported, but by using travis we can test every commit on every push to github automatically. It is set up for github.com/zeehio/metashare, but you can create an account for the metashare/* organization on the three services and customize the .travis.yml values to use it too. If you want me to help you doing that, feel free to ask 馃憤

The third commit updates some of the metashare dependencies to more recent versions, still compatible with django-1.3. The fourth commit changes all the static files from MEDIA_URL to STATIC_URL. See https://docs.djangoproject.com/en/1.4/releases/1.3/#extended-static-files-handling to learn why this was done (to separate user contributed files from static files from the site).

The fifth commit upgrades django to 1.4 and fixes several issues related to the upgrade. The sixth commit updates the documentation and installation instructions accordingly.

Finally, some deprecation warnings are corrected to ease the transition to future django versions, the django-selectable dependency is upgraded to use a standard version instead of a customized one and another final update of the metashare requirements is done to make sure we use versions as recent as possible (given django-1.4).

As an important comment, the requirements.txt file ends up describing the required dependencies. It is important to mention that we are depending on a slightly modified version of:

  • django-1.4. It includes a fix that will be applied in django-1.7. At that point we will be able to drop the modified dependency. This is the modification: zeehio/django@233117e
  • django-haystack: We use a commit between v1.2.0 and v2.0.0. I did not have the time to analyse what needs to be changed in metashare to adapt the code to the 2.0 upgrade. This is the commit we depend on: https://github.com/zeehio/django-haystack/commits/for_metashare_3.0.2.9000
  • django-kronos: This is the modified change to the official version: zeehio/django-kronos@f22f638 I did not have time to debug and fix this issue, but it does not seem very hard to address.
  • pysolr==2.1.0-beta: This is an official version, but it is an outdated beta and should be updated too. I have not analysed anything else.

Overall, I would make a release as soon as issue #748 is fixed. I can't provide a timeline for that though, not even a commitment to be able to fix it, although any efforts on reviewing this pull request will for sure motivate me to want to dig into it.

It is a pity that there is none of the original developers maintaining the code. I will do my best to fix the issues on my spare time, but if someone is able to get the money to hire somebody to fix them, that would be great.

EDIT: Add information on django-haystack version dependency

zeehio added some commits Nov 15, 2014

Improve testsuite and dependency tracking
 - Update requirements.txt to reflect real metashare-3.0.1 dependencies.
 - Update django from 1.3.5 to 1.3.7 (still has a patch added to it)
 - New install-dependencies.sh script:
    - Use a virtualenv and pip to download and install all python
      dependencies. Virtualenv is the recommended way for working
      with Python projects.
    - Remove the lib/ directory.
    - Remove the Python bundled file (more recent 2.7.9 is downloaded
      if required).
    - System PATH does not need to be modified anymore
Continuous Integration: Travis + SauceLabs + Coveralls
Travis builds and tests the code, with the help of SauceLabs to
run the Selenium tests. Coveralls evaluates the code coverage
of the whole testsuite. Travis also takes care to regenerate and
upload the documentation automatically.

The following jobs are run:
 - Basic testsuite (As suggested in the installation manual)
 - Selenium tests
 - Node syncing tests
 - Documentation update

To do that, selenium had to be updated and some tests fixed:
 - Update selenium and django-selenium to be compatible with recent browser
   versions
 - Use selenium implicitwait instead of time.sleep

Tests can also run locally, without Travis nor Sauce Labs.
Tests will now print DeprecationWarnings, to help make metashare code easier
to upgrade.
Update dependencies to recent versions compatible with django-1.3.x
 - six
 - Unidecode
 - django-kronos
 - pycountry
 - requests

Some of the modified dependencies depend on github.com/zeehio/*.
I won't delete the repositories, but I believe it would make sense
for metashare to host their own copy under github.com/metashare
just in case.

Additionally, in the long term the code should be ported to depend on
standard versions of the dependencies.
Use STATIC_URL instead of MEDIA_URL.
This is recommended as part of the port to django-1.4.
[DOC] Update installation instructions
Reflect latest virtualenv, pip and django changes
Fix django-1.4 warnings:
 - getchildren is deprecated
 - switch_to_window deprecated
 - Add SECRET_KEY for salting passwords
 - Add ALLOWED_HOSTS (hosts this django site can serve)
 - django.conf.urls.defaults -> django.conf.urls
 - Switch function-based generated views to class-based views
Drop customized django-selectable.
Use django-selectable 0.5.2
Update requirements
  - requests to 2.6.0
  - Update httplib2, psycopg2, pycountry, Unidecode
  - Update several dependencies.
  - Updating pygeoip, python-dateutil, requests, selenium and six
  - Update django-analytical
@jsteffen

This comment has been minimized.

Copy link
Member

jsteffen commented Jul 3, 2015

Thanks for all the work! I'll see that I find some time to get a rough idea of what you've done (with my restricted knowledge of Django and Python) . But at first glance, it looks very thorough and promising.

FIX: Don't delete distributionInfo in the resource editor.
There is probably a more clean solution than this one.
Closes issue #748

@zeehio zeehio force-pushed the zeehio:master branch from 58f8b13 to 4e133ed Sep 11, 2015

@zeehio zeehio force-pushed the zeehio:master branch from 4e133ed to ff9ca00 Sep 12, 2015

Fix #759, UnboundLocalError
Set `geovisits = []` case to make sure it is defined even if a random GET view= argument is used.
Closes #759. Thanks to mmatthiesencsc for reporting the issue.
Close #759. Define visitstitle.
Fixed following tests by mmatthiesencsc

jsteffen added a commit that referenced this pull request Jan 11, 2016

@jsteffen jsteffen merged commit 493cb39 into metashare:master Jan 11, 2016

@jsteffen

This comment has been minimized.

Copy link
Member

jsteffen commented Jan 11, 2016

At last I found time to look at your pull request. Better late than never I guess...
Anyway, thanks a lot again for your effort! The META-SHARE setup is indeed much better now with a clear separation of third party libraries, Django and META-SHARE itself. I set up META-SHARE from scratch on Ubuntu and everything basically works fine. I had trouble with the local Selenium tests, though. For unknown reasons, Selenium could just not connect to Firefox, so I deactivated Selenium tests for local tests for now.
I also did some adaptations in #761 so that the tests (excluding Selenium test) run properly on our old-style Jenkins CI server.
And I finally tried the Travis CI integration, but here the build fails:
In the multi tests, some tests fail from time to time. This also happens on our Jenkins. I guess this is a timeout issue. All other tests in the "BASIC" test suite succeed. And then some of the Selenium tests in the second test suite fail, too. This is strange, as they seem to work in your Travis integration and I didn't change any of them. Could you please run them again for your zeehio/META-SHARE and check if they still succeed? I remember that sometimes we had to adapt the Selenium tests for new Firefox versions.

@zeehio

This comment has been minimized.

Copy link
Contributor Author

zeehio commented Jan 11, 2016

Great to see that it got merged! :-)

I remember having many issues with Selenium tests to make them work. I will try to run them again as soon as possible, although probably the Selenium dependency now fails due to Firefox updates (see SeleniumHQ/selenium#1385).

My advice would be to upgrade the Selenium dependency to the latest released version 2.48.0 and run the tests again.

Some of the multitest tests were failing for me too from time to time, I have not looked into it as I haven't found the time to check how they work in detail.

Unfortunately, Django-1.4 is now unsupported (see https://www.djangoproject.com/download) so there is still work needed to port metashare to 1.5, 1.6, 1.7 up to Django 1.8 if we want to avoid being hit by security issues (see https://docs.djangoproject.com/en/1.9/releases/security/, there already is one after django-1.4 got EOL). Too much work and not enough time.

I'll come back to this thread once I have been able to test if I can fix the Selenium tests. Happy new year!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment