Skip to content
This repository

Why do we have django_trunk in the repo? #1

Closed
mbrochh opened this Issue · 31 comments

5 participants

Martin Brochhaus Calvin Cheng Daniel, Dao Quang Minh Victor Neo Atamert Ölçgen
Martin Brochhaus
Collaborator

On the server I will do pip install django (and pip install everything else needed). The stuff will then be installed into a virtual environment that is private to our user into /home/pythonsg/lib/python2.7/. That path is where django will reside and if we need to symlink to it, the links should go there. I don't think that we will have a lot of fun keeping the trunk in sync with upstream. Also I would not really want to use django trunk. We should stick to the stable releases.

My vote: Delete this folder altogether. The point of this repo should not be to do git clone and have a fully functional project. We should include a README.rst that describes how to install this project locally (that will be just a few steps anyways) so there is no need to host stuff here that is already hosted on other repos (DRY!).

Any thoughts on this?

Calvin Cheng
Owner

Ah... the idea of having django-trunk inside each project I work on is so that I don't need to use virtualenv! :-))

It also ensures that as a distributed development team, there's no need to waste time asking each other "hey, which version of django are you using for project Y?" Everyone uses the same trunk version for a specific project. When django gets updated, one of us just need to do an "svn update" on the django-trunk and fix any breakages on that specific project as a result of new implementations on django-trunk. Everyone else will simply do a git pull and they are set. No additional configuration needed.

I know this means that django-trunk is "duplicated" on every project but different projects progress at different rates and sometimes an older project may not want the latest and the greatest from django trunk and they want to stay at, say, 1.2.###. This project layout strapping in django-trunk caters specifically to such a problem.

Martin Brochhaus
Collaborator

I really don't feel good about this. Why not add a requirements.txt? That file will clearly say what django version is needed. If anything magically breaks, a dev can just do "pip install --update -r requirements.txt" and will get all the latest needed versions of all repos.

Also when having django==1.2 in requirements.txt we can stay in django 1.2 as long as we want.

Is there anything wrong with using virtualenv??

Calvin Cheng
Owner

No issue with using virtualenv if you really want that. I just feel it's a little unnecessary. I have never bothered with virtualenv so far because using this solves the problem of having to coordinate django trunk version with everyone.

If we are using virtualenv, just be sure that each person still has a django trunk on his local machine, the virtualenv file points to that django trunk and we will have to talk to each other that the virtual env pointing at each developer's django trunk (outside of the repo) is exactly the same revision. Extra communication overhead, in my opinion. But yes, it can be done.

If the django trunk is moved outside of the repo, we will need to make more changes to the custom manage.py - because the manage.py I use is customized to load django trunk into the system path.

Btw, using such a directory schema isn't strange. One can also "strap" in external libraries using git submodule feature. In this case, I am strapping the django library as an external library without using git submodule because django trunk is managed by subversion.

Daniel, Dao Quang Minh
dqminh commented

personally, i feel that not using virtualenv is just wrong. It is just easy enough to use virtualenv with virtualenvwrapper.

If the team is distributed, a combination of virtualenv and vagrant will certainly work most of the time.

Martin Brochhaus
Collaborator

The thing is... when we use django-cms we need at least 4 or 5 other dependencies already and following your philosophie we would need to add them all to this repo as well. Nobody does that. In fact django-cms did that with mptt some time ago and they ran into all kinds of horrible dependency hell and needed to take huge efforts to get rid of that internal (always outdated) copy of mptt again.

So we will need to run virtualenvs no matter what. I would also want to create all apps that we use as reusable apps that should reside in their own repositories - this repo will probably always stay very slim and consist mainly of some templates and the settings.py.

Calvin Cheng
Owner

Btw, what happens if I upgraded django trunk on my local django trunk and other developers haven't?

Won't that break your local instance of python.sg?

If you have 10 projects and each project use a different revision of django trunk, virtualenv or no virtualenv, you will still need 10 copies of django trunk on your local machine, don't you? (Because different projects might want 'updates' at different pace based on the needs of the project owner.)

Calvin Cheng
Owner

I agree that dependencies can be handled well by virtualenv.

django trunk though is a special case because of the reason I explained above.

Martin Brochhaus
Collaborator

Of course! I have a virtualenv for each project and each virtualenv has its own django repo.

One more thing: You keep referring to django-trunk. Would you insist on using the trunk? I would feel more relaxed if we used the latest stable release...

Calvin Cheng
Owner

Hey Martin, don't worry too much about the issue with django trunk... we have been using django trunk for all our projects and they are extremely stable. It's one of the most stable trunk open source software out there. Following trunk so that we have the latest updates is one of the best ways to stay cutting edge and learn more about what the core developers are doing.

Hopefully, some day, we can contribute actively to django core too. At least, I hope I can eventually be skilled enough to contribute actively to django trunk some day. And it all begins by staying close to the trunk and being able to report bugs to the django core team if we discover one first.

It's also the same reason why I think it is better to empower individual developers to perform direct code pushes into production server via git deployment hooks for our own live projects.

Assume trunks are stable - in the case of python.sg and in the case of django (and, also, in the case of Gentoo linux distro). :-)

It's not as crazy as it sounds. :-))

We have been doing this for many many projects. Even a large scale social network for a customer of ours - with an uptime of more than 1 year at that!

Martin Brochhaus
Collaborator

Agreed on the django-trunk thing. Still don't think we need it in our repo, though.

Calvin Cheng
Owner

We can remove it of course but this would mean that

  • each developer will need to have his own manage.py file to point at his own local copy of django-trunk since everyone would keep their local copy at their own directory of their preference.
  • when one of us decide to grab the latest version from django trunk by running an svn update, he will need to manually 'svn info' and tell everyone that he is now using django r16286 for example.

What's the point of these (albeit small) extra steps/communication work? Won't it better for him to have upgraded django, fix any breakages as a result of the django trunk upgrade and push it all up into the repo. Zero communication overhead (git has done the communication for us). Everyone involved in this project just do a pull and that's it, they get the updates for free and can focus on doing what they need to do next.

Martin Brochhaus
Collaborator

Huh? Everyone has a virtualenv and can just use manage.py. The virtualenv will know where django is.

I have to read all comments in a calm hour and think about it. It just sounds totally wrong to me. But I also don't get the big picture, yet. I think you don't assume that we will create reusable apps and host them in their own repos. You assume that this repo will grow and grow and get many folders like 'homepage', am I right? I'm not sure if that is the right approach. But I can see your concerns about communication overhead. I will try to think of a good solution tomorrow.

Calvin Cheng
Owner

Virtualenv requires each developer to point to their own copy of django trunk (which may be of different revision at any point in time depending on when that developer in question run an "svn update").

What happens if my local django trunk is r16121 and your local django trunk is r16214 and there are some backward incompatible features between that two revisions? On my end, it would appear that "everything works"; but on your end, you will see some strange behavior and start fixing it. When you push up your fixes and I git pull your fixes, I may run into trouble because my local django is not the same as yours.

Virtualenv does not solve this problem.

Martin Brochhaus
Collaborator

I think I am beginning to understand your point :)

Martin Brochhaus
Collaborator

But we would get into that trouble with EVERY app that we add to our stack? django-cms has a develop trunk as well. Should we use that?

What if we forward our django-trunk to the latest version and find out that it breaks django-cms? Will we then put an older version of django-trunk into our repo and freeze that until django-cms has released a new version?

Daniel, Dao Quang Minh
dqminh commented

So the issue here is what tools will we use to manage version, is it ? Indeed, virtualenv doesnt solve this problem but pip does. A pip freeze diff should point out the version difference. If you freeze the requirements into requirements.txt, you can use -e git://github.com/django/django.git@<commit>#egg=django-trunk to specify the commit

Martin Brochhaus
Collaborator

Good point @dqminh

Calvin Cheng
Owner

@dqminh it's not about version control per say. It's more about using the appropriate tool for the appropriate development scenario.

Virtualenv use cases

In my opinion, virtualenv is useful for managing the version of specific Python dependencies relating to a project. For example, when I run a django project on a specific version of pypy (and not cpython), virtualenv is my weapon of choice because I am not going to have the time to track the trunk of pypy on a regular basis. It would certainly be silly, in this context, to load the entire pypy library into the git repository!

Django trunk in git

Django is very different compared to the pypy example I have given above. I am very keen on tracking every update which the django core team makes which is why I configure it to be placed outside the control of virtualenv and let git do the job of controlling it and making sure that when any team member updates the django trunk, all team members working on this very same project always gets the same django trunk version by simply doing a git pull.

pip

When building features using django and a team member updates the django trunk and fixes the problems associated with the updates, I can just as easily view the difference with git log (and read about what he fixed and why he did what he did) and move on with productive work by a simple git pull.

Why would I use pip to figure out the version difference when git already does precisely that?

I don't want to know the difference and then later figure out why my local instance is broken. Because I know that if someone did a django trunk update, it is his/her responsibility to have run unit tests on his machine before pushing up the fresh django trunk together with the fixes for anything broken from the fresh django trunk.

When would I want to track trunk of a 'library'?

We do not necessarily want to track every 3rd party django app or external library by trunk.

As an example, if a django app is a very generic and reusable, git submodule comes into play perfectly fine. Generic/reusable apps include things like "django-pagination", for example. "Libraries" or django apps such as these are certainly worth tracking by trunk (via a git submodule for example).

I won't classify django-cms as an app that we need to track by trunk because chances are, we need to micromanage our own templates, customize specific urls and perhaps write custom view functions or templatetags in it. Meaning, the initial django-cms we load in will be heavily customized for our site's objectives.

So whether we want to track the trunk for a specific app or library depends on what that app or library is; and how much interest we have in following it closely.

Martin Brochhaus
Collaborator

No way! We will not make any changes to django-cms. That's the whole point of the cms! You can extend everything by just writing normal reusable apps and add cms_apps.py or cms_plugins.py to them.

Martin Brochhaus
Collaborator

Ok I tried to get the project working locally and create an easy to install setup. I still feel that this django trunk thing needs some discussion.

First of all I don't plan to use python.sg as a way to dive into the django trunk. We will not develop any fancy things here. We will chose existing reusable apps and tie them nicely together. Minimal effort. Maximal profit. Eventually we will make some changes to the apps that we use (for example django-cms, filer, cmsplugin_blog etc) and contribute them back but I can't see where we will ever need to change anything in django_trunk. I think this is not the goal of python.sg.

Secondly the need to change the manage.py file looks like black magic to me. It is not transparent and even though I knew that this change is in place I was cussing and swearing for half an hour and wondering why I get an exception that tells me that I am using django 1.4?!? I have never seen this practice in any other django project and I am sure it will confuse any new contributor who wants to jump on board in the future.

Thirdly having a modified manage.py and django_trunk in our repo will collide with the usefulness of pip, I believe. I think it is an industry standard to provide a requirements.txt and let people create a virtualenv. So I thought I'd just leave out django since we have that in our repo already but pip doesn't know about this. Therefore it would try to install django again because it is a dependency of django-cms. I mean - let pip install it, thanks to our manage.py that django version in my env will never be used but hell, all this is a horrible and confusing mess. I have not tested this but I guess a workaround would be to put a symlink to the django_trunk into the virtualenv so that pip believes that django is already there.

Lastly, I really really really think that we will never ever need any bleeding edge feature of django. I am running more than 20 django sites and I love every new django release. I read the release notes at the seconds they are published and I start migrating my projects to the latest django version almost immediately. But that's it. I never thought something like "BOY! If only I were running my sites on trunk and had this feature 3 months ago already." I strongly vote for putting django==1.3.5 into requirements.txt and thats it. Ease of use should be #1 priority (for every open source project, methinks). If we were a startup and we were developing hot new stuff and we would know that our work would need to vastly improve django, I would totally accept that we would want to work on django trunk because it would be obvious that we will fork it and add patches sooner than later. I can't see that usecase for python.sg.

We have 5 maintainers here... I would love to hear what everyone thinks.

Martin Brochhaus mbrochh reopened this
Calvin Cheng
Owner
Martin Brochhaus
Collaborator

Okay. I fully understand your vision and motivation but I disagree on all points. To me python.sg will only be a slim and leightweight platform to publish content. Raising awareness of our local usergroup is the first thing that we need to do. For that we need a beautiful website with a nice CMS in the background. That's it. Later we add some user profiles so that we can build a local Python developer directory or so but that would be solved by reusable apps as well. Later we can add a calendar integration of some sorts - reusable app or cmsplugin again.

Once we have a brilliant idea on how to monetize as a Python user group that project might be hosted under the python.sg domain as well (but probably would have its own domain ) but it should go into its own repository.

Anyhow I don't want to be the person blocking everything here. Let's have a skype chat soon so that I can learn from you how we can create an easy to setup environment based on git submodules. I don't see how this will work right now.

Martin Brochhaus
Collaborator

Just had a discussion with some other experienced django devs and the idea of having trunk in our repo shocked them all.

I also played a bit with pip. How about this approach:

  • we add a requirements.txt
  • first line will be this: -e git+git://github.com/django/django.git@18472a8#egg=django
  • that would be the latest commit of django
  • any dev can update that to the latest commit at any time
  • whenever a dev does a git pull of our repo he should to a pip install -r requirements.txt as well
  • we can even streamline that process with fabric and add a fab pull command that will do both tasks automatically
  • if someone changed the commit id to any later (or earlier) version, pip will checkout the correct django trunk in a matter of seconds

I think that way we would be able to work on trunk with minimal communication overhead (that is what you want) and we would not need to have trunk in our own repo and be able to use virtualenv and pip (that is what I want). Everyone would be happy?

Calvin Cheng
Owner
Calvin Cheng
Owner
Calvin Cheng
Owner

I was thinking of writing a detailed step-by-step blog post about the use of git submodule, but google did the trick for me. Here's the git/git submodule approach towards managing dependencies external to a project - http://twoguysarguing.wordpress.com/2010/11/14/tie-git-submodules-to-a-particular-commit-or-branch/

Works just as well for other dependencies that are on git (django trunk still runs on svn, so not relevant of course) and where we want to micromanage those specific dependencies.

Virtualenv/PIP is certainly not the only way to achieve dependencies management. Although, it is true that it is a popular approach in the Python community.

Many ways to get something done.

Victor Neo

Allow me to add my 5 cents worth on this matter.

python.sg redesign is done by 4 (I think) developers right now and this project was created by Calvin as a starting point for it. As of now I only see the two of your interaction so I am unaware of the other 2 developers' opinion.

Now since this is done as a open-source project, is there a leader for this project? If not, was there a meeting to finalize details like using virtualenv or project structure?

Both of you are experienced devs so you have your own working style for your own projects - but now you should all start with a discussion to get all the details kinked out first. Establishing a common ground for this project is vital, even more so that the devs for this redesign are working separately.

Martin Brochhaus
Collaborator

@victorneo thanks for your comment. Finally the community is waking up :)
We had a kickoff meeting that only Sean and me attended. We both preferred a virtualenv/pip style setup and were confused about trunk in the repo.

You are proposing to have a discussion. Well... we are having it here right now :)
Why don't you tell us your opinion about the issue?

Victor Neo

@mbrochh Honestly, any form of setup required is okay. Since both Sean and you are more familiar with the virtualenv/pip style, I see nothing wrong with implementing that. Afterall, it's a very common style that most developers will know and can easily setup (or even assume that it is the norm).

I'm not saying that Calvin's method is not common or anything, just that if it isn't what the rest of you devs know then there should be documentation that makes it easy for the rest to setup as well. I believe Calvin has mentioned something about this so we should wait for his opinion.

Another point to take point is that despite this being an open-source project, I highly doubt many people will fork this project and redeploy it for their own needs. The most common fork-ers (pardon the language) would be users helping with bug fixes or accessibility fixes. Hence any development setup for this project is fine as long as there is documentation for them to setup and help out.

(ps. I am not familiar with Calvin's style, I've only used virtualenv / pip for myself so I rather not get into the Dev War :) )

Atamert Ölçgen
Collaborator
muhuk commented

I didn't read the whole tread to be honest. I started with @victorneo's 1st message. Our organization, or the lack of it, is a little confusing. But now Calvin is back I am pretty sure it will be all right soon.

Also confusing is; we are talking about forking and developing etc, but we are using a CMS. In that case most of the development occurs in the database, am I wrong? Are we going to be developing apps as well? On a related note this kind of development was why I signed up in the 1st place and I would still like to do my best to contribute.

Yesterday's meeting was short and to the point. It cleared a lot of things up for me. But there is just one thing I couldn't figure out; are we trying to reach social goals exclusively by development? I don't think Django-trunking or virtualenvving will bring us to our goal of increasing awareness one bit. Don't get me wrong, this doesn't mean we shouldn't upgrade the architecture of the website. But it would really help to understand if we had a clear understanding of where we are trying to go in terms of software and design. At least for me.

Having written this, I realized this obscure place is not exactly the right place to post. :D

Cheers,

Martin Brochhaus
Collaborator

@muhuk we will surely create apps. Let's assume we want to have a event calendar at the site. There is no way to create one with pure django-cms, so we would need to create a (small, but still its an app!) cmsplugin.

Let's assume we want to have a way for user group members to create profiles on our page - another app.
Let's assume we want a way for local startups / companies to post to some kind of job board - another app.

Of course we could do all this with the current wordpress instance. But well... we are a python user group. It's also about the fun, isn't it? For some it might be the first time to contribute to an open source project. It's just so much easier to get involved if you know the maintainers personally and have the chance of them guiding you with setting up local dev environments and giving you an introduction to the current codebase etc.

Increasing awareness will surely be done by creating content on the new site. You are right. We could do it with the current wordpress site right away but we'd love to "waste" some time here in order to create something pythonic first. Just for the fun of it.

BTW the issue should be closed as I have come to an agreement with Calvin on how to proceed :) I will leave it open for a while in case you want to keep this slightly off-topic discussion going.

Martin Brochhaus mbrochh closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.