New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Include a Django project with ICEkit. #3

Merged
merged 140 commits into from Aug 22, 2016

Conversation

Projects
None yet
3 participants
@mrmachine
Member

mrmachine commented Aug 15, 2016

With this change, we can make ICEkit more like an off the shelf CMS that can be installed and used immediately in its default configuration.

Currently, it is more like a library that must be installed into a new or existing Django project, and we have a separate project template that we use to create new ICEkit projects.

Important things to note:

  • We can run ICEkit via Docker or directly, with or without a Django project. This should make it easier to develop ICEkit itself. Just clone the repository, make changes and run it to test your changes.
  • Settings and other boilerplate code from ixc-project-template (e.g. celery.py, context_processors.py, jinja2.py, urls.py, wsgi.py, etc.) have been added to icekit.project. This is the Django project ICEkit will use by default.
  • There is no need to juggle a project template that is not immediately useful (without being rendered as a new project) and an code library that must be installed into a new/temporary project to test.
  • ICEkit now includes its own project template (and a startproject.sh script to render it) that is much simpler than ixc-project-template and includes much less boilerplate code (most of which is now part of ICEkit).
    • Hooks for Bower components, Node modules, Python requirements, settings, static files, and templates, to override and extend ICEkit defaults.
    • Files that configure Docker to build and run an image and run the project. The image is built on top of an official ICEkit image, which makes build time much faster because the ICEkit image already contains everything except project overrides via the hooks above.
    • A go.sh entrypoint script that makes running directly work similarly to when running via Docker. Database name is derived from the project directory name and git branch. Bower components, Node modules and Python requirements are updated automatically. Migrations are applied automatically.
  • When running with Docker, we now run celery (worker, beat, flower) in separate containers for independent scaling. Since these all use the same image, which tries to apply outdated migrations at startup, we now use Redis locks to ensure that only one container executes such code at the same time.
  • ICEkit can still be used as a library, just as it is today. Existing projects don't need to change anything. But they could be updated to inherit from ICEkit's base settings instead of duplicating ~700 hundred lines of settings and other boilerplate code from the project template, to reduce future divergence.
  • We now have a Deploy to Docker Cloud button in the README. You won't be able to customise your project when deploying the official ICEkit Docker image this way, but you can easily get a site wit the default ICEkit configuration and styling.
  • Tests still pass for django17-py27, but this and other Django 1.7 environments are no longer tested automatically.
  • We were only using django-generic for one thing, to add SITE_NAME to the context for RequestContext objects. Since django-generic isn't on PyPI and I wasn't sure that we should put it on there (it's a fork of DrMeers repo), I removed it instead.
  • I have added a very basic intro page (fallback when DEBUG=True and when no Fluent pages exist).

Follow the instructions in README.md and docs/contributing.md to get started.

mrmachine added some commits Aug 9, 2016

WIP: Make it possible to install and use ICEkit more like a product.
* Make `icekit` a Django project, not just an app (settings modules,
  root URLconf, etc.) so we run ICEkit as-is (via Docker Compose) or
  install (via Pip), without needing to customise or maintain any
  boilerplate project code.

* An ICEkit project directory is now just a container for docker compose
  and stack files, static file and template overrides (front-end
  customisation) and runtime files (media, logs, etc.) Anticipate
  feature customisations will be facilitated via `django-model-settings`
  or possibly a restricted JSON config placed in the project directory
  that is parsed into allowed ICEkit/Django settings at startup. E.g.
  thumbnail aliases.

* Get started with a bash one-liner that creates a new project directory
  that you can immediately run via Docker Compose or deploy to Docker
  Cloud. No build step required.

* When run with Docker Compose, services are decoupled (no more
  `manage.py supervisor`) so they can be scaled separately.

* Database migrations are applied and local Node modules and Python
  requirements are installed automatically as required during container
  startup. A Redis lock is acquired for each command to prevent multiple
  containers from trying to update a shared resource (e.g. database,
  `node_modules` directory, etc.) at the same time.

* Alternatively, `pip install icekit` and then use `manage.py icekit ...`
  to create a project directory and `manage.py supervisor` to run it.
  You must install Elasticsearch, Redis, PostgreSQL, etc., manually.

* Alternatively, continue to install and use `django-icekit` as a
  library in a full blown custom project.

* From a git working copy, just run `docker-compose build` and then
  `docker-compose up`. This should be much easier to develop ICEkit in
  isolation, without needing to develop as part of a client project
  (which probably has many custom templates and features) or run the
  `startproject.sh` script from a working copy to create a functional
  version of ICEkit to test changes against.

* Replace Bower with NPM to simplify front-end asset management.

* Use `pip` instead of `pip-accel` because it wasn't playing nice with
  the alternate installation user scheme. Anticipating fewer builds
  being required, the speed of Pip should be less of an issue.

* Decouple PostgreSQL environment and database setup.

* Just use `manage.py runserver` for WSGI process for now. Anticipate
  adding Nginx and Gunicorn.

* Get master password from environment, making it easy to specify at
  runtime without having to create a compose override file and without
  having to commit a default weak password.

* Because of the way Docker extends services (you cannot extend a
  service with links, sequences are combined/extended and not replaced),
  we need to split the service definition into two files. A compose file
  (missing links and exposed ports) and an override file. Since we don't
  need to provide AWS credentials (for pip-accel) or a master password
  in the override file, this should not be a problem. We can also resort
  to using `env_file` to provide secrets.

* Get rid of `SITE_VERBOSE_NAME` and add `SITE_SLUG`. `SITE_NAME` can be
  verbose.

* Say "This site" (instead of "ICEkit") "is built with ICEkit".

* Default ICEkit default/home/sidebar templates are of questionable use,
  but are kept for now to reduce the amount of required boilerplate just
  to get something that works. We should be improving our default
  templates, not removing them.

* Use `SLUTE_SLUG` in default `SITE_DOMAIN`, now that we are not
  rendering a project template, we need sensible defaults for all
  settings.

* Add `_root` suffix to `media` and `static` directories, to avoid
  conflict level `static` directory and to match up with settings.

* Is it safe to use the celery broker URL (Redis DB 0) for cronlock, or
  should we use a separate database? TBC

* Use collected static_root directory as included path for node-sass.

* Keep fluent-contents `markup` plugin disabled for now. Seems our
  forked version is not compatible with Django 1.8.

* Combine the old project template root URLconf with the ICEkit app
  configurable URLs (which it seems we were never using).

* Make the hard coded home page URL a fallback. Anticipate adding
  welcome and getting started content here, instead of adding to the
  database via data migration.

* Add S3 storage classes.

* Allow package to be installed without a Git working copy (e.g. inside
  a Docker container).

* Add `manage.py` as a script, so we can run it from anywhere.
WIP: icekit as a product.
* Use bower components for front-end assets that will be collected. The
  `node_modules` directory is about 17 times larger (not ideal for
  including in images or packages), and takes longer to install.

* Move node modules and bower components into the `icekit` package, for
  manual installation into a virtualenv.

* Don't print that we are still waiting every second. Count the number
  of seconds we waited and print that when we are done.

* Simplify printed docstrings for shell scripts to reduce noise.

* Give waitlock.sh sensible default configuration when running outside
  of Docker, with no configuration in the environment.

* Use database result backend for Celery.

* Don't use New Relic by default. It was raising an exception, and might
  be going away soon anyway (pricing).

* Rename `/opt/icekit` to `/opt/django-icekit` to match package name.

* Install shell scripts from `setup.py`.

* Tweak icekit extras installed into Docker image.

* Remove goss, since we are not using it yet.

* Move files for bundled project to `icekit.project` package, for
  clarity.

* Load local override settings from project, instead of from `local`
  module in ICEkit settings package.

* Move `manage.py` into `icekit` package to trick django-supervisor into
  being able to locate `supervisord.conf`.

* Install `manage.py` as a console entrypoint, so we can run it from
  anywhere, especially when installing manually from PyPI, so we don't
  need a `manage.py` script in the project.

* Remove default weak master password from development base settings.

* Create required project templates directory at runtime.

* Use ICEkit base template as project base template by default.

* Update setup.py requirements. Remove some Django 1.7 era version
  restrictions. Add project requirements in an extra. Remove publishing
  extra - we always want publishing. Add slideshow extra.

* Fallback to import `BASE_SETTINGS_MODULE` as a fully qualified dotted
  path, instead of an ICEkit settings module.
Move console scripts to `manage.py`, where we can access settings.
* Only print `BASE_SETTINGS_MODULE` when executing management commands.
Don't assume project directory is current directory. This might accid…
…entally create a bunch of subdirectories whenever `manage.py` is run from the wrong location.

mrmachine and others added some commits Aug 14, 2016

We do actually execute some management commands without the `entrypoi…
…nt.sh`, so let's hide warnings and use a random hash seed again.
Remove django-model-utils requirement & fix manager method pass-through
- replace usage of obsolete `PassThroughManagerMixin` from
  django-model-utils with Django's preferred manager-generating methods
- fix "pass-through" of manager method calls so custom queryset methods
  get invoked when the same method names are called on the corresponding
  manager (this behaviour got lost due to changes in upstream reqs)
- cleanup unused imports and commented-out print statements
- add extra req to "test" requirements.
Revert to defining `VIRTUAL_HOST: *` for Django service. Trying to ma…
…ke it the default by leaving this out didn't seem to work.
Fix settings to properly handle publishable items
- add publishing middleware to display draft/published items correctly
  depending on the request context
- set `FLUENT_PAGES_PARENT_ADMIN_MIXIN` to properly display only draft
  items in admin
- show publishing status filters in page parent (listing) admin.
Add monkey-patch workaround to fix infinite recursion of Redirect
Work around infinite recursion issue when using Fluent Pages' Redirects
with django-polymorphic>=0.8 by renaming an attribute of the admin
class.

See django-fluent/django-fluent-pages#110
@coveralls

This comment has been minimized.

coveralls commented Aug 16, 2016

Coverage Status

Changes Unknown when pulling 66b4dd0 on feature/project into * on develop*.

mrmachine added some commits Aug 16, 2016

Re-jig console scripts and `django-icekit` install method.
* Don't install our shell scripts with Setuptools. Doing so means we
  have to:

  * Copy them into the image before we do `pip install -e .` in
    `Dockerfile`, so setuptools can install them.

  * Reinstall them every time they change during development, to update
    the installed version.

* Instead, move our bin directory into the `icekit` package, so it is
  included as package data, and is added to `PATH` in `Dockerfile` and
  `go.sh`.

* Don't install `manage.py` as a console script. Instead, move it to
  `icekit/bin`, and leave an empty file to trick django-supervisor.
@coveralls

This comment has been minimized.

coveralls commented Aug 16, 2016

Coverage Status

Changes Unknown when pulling 8822ca9 on feature/project into * on develop*.

@coveralls

This comment has been minimized.

coveralls commented Aug 16, 2016

Coverage Status

Changes Unknown when pulling e7c3190 on feature/project into * on develop*.

1 similar comment
@coveralls

This comment has been minimized.

coveralls commented Aug 16, 2016

Coverage Status

Changes Unknown when pulling e7c3190 on feature/project into * on develop*.

@coveralls

This comment has been minimized.

coveralls commented Aug 18, 2016

Coverage Status

Changes Unknown when pulling 8d01048 on feature/project into * on develop*.

@mrmachine mrmachine self-assigned this Aug 22, 2016

@mrmachine mrmachine merged commit 8d01048 into develop Aug 22, 2016

3 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details
coverage/coveralls Coverage remained the same at 0.0%
Details

@mrmachine mrmachine deleted the feature/project branch Aug 23, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment