Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.
Sign upGNIP: GeoNode Backup and Restore #2401
Comments
This comment has been minimized.
This comment has been minimized.
Nice idea, thanks. Regarding this: A part from getting the workflow going from the Django admin, why not to make available a django management command that is doing everything from the command line? |
This comment has been minimized.
This comment has been minimized.
Yes, the management command should be also part of this proposal. Anyone has experience and/or pointers with the Django serialization? In GeoServer Professional Services from the experts! Visit http://goo.gl/NWWaa2 for more information.Ing. Alessio Fabiani GeoSolutions S.A.S. http://www.geo-solutions.it AVVERTENZE AI SENSI DEL D.Lgs. 196/2003 Le informazioni contenute in questo messaggio di posta elettronica e/o The information in this message and/or attachments, is intended solely for On Fri, Jan 15, 2016 at 5:00 PM, Paolo Corti notifications@github.com
|
This comment has been minimized.
This comment has been minimized.
I'm going to report here the latest blocking issues I have found while working on this GNIP, just to keep track of the progresses here too. "Finally I found the issue (but not yet a beautiful solution). The problem is the following one:
The issue: When we dump the data everything works just fine. The data can be successfully dumped to JSON and, paying attention to the order, can be almost all deserialized again on a fresh database. There is no problem in restoring everything except the Layers. In fact, it seems like DJango tries to create the Layer but somehow loses the information on the resourcebase id when trying to restore the many to many relationships. In other words, it tries to create again the contactroles and the links associated to the Layer/ResourceBase, but this time imposing the resourcebase foreignkey as NULL. Relaxing the model and allowing contactroles and links to accept also NULL resources (which means basically have orphan records into the DB in the end), everything works fine, except that we have duplicated contactroles and links, half with the correct resourcebase id reference and half with a null reference. The DJango mailing list seems not providing any solution for this, or at least I was not able to find any at this time. Most probably this issue has been fixed from DJango 1.7a+ since there are similar issues around related to duplicate keys. The trivial workaround:
The non-trivial possible solutions:
Any feedback/help/idea welcomed. Regards, |
This comment has been minimized.
This comment has been minimized.
Just curious if this will also support geogig repositories? |
This comment has been minimized.
This comment has been minimized.
Hello Daniel, In theory should not be a big issue since the geogic repositories are We can extend the proposal to geogig as soon as they become an official Best Regards, GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information.Ing. Alessio Fabiani GeoSolutions S.A.S. http://www.geo-solutions.it AVVERTENZE AI SENSI DEL D.Lgs. 196/2003 Le informazioni contenute in questo messaggio di posta elettronica e/o The information in this message and/or attachments, is intended solely for On Mon, Feb 8, 2016 at 9:05 PM, Daniel Berry notifications@github.com
|
This comment has been minimized.
This comment has been minimized.
@afabiani this work is complete now correct? Do we have docs? (apologies if I missed them) |
This comment has been minimized.
This comment has been minimized.
Hi Jeffrey, We have documentation for this work available here which should be ported into the official GeoNode documentation and enriched |
* GeoNode Backup and Restore Management Commands - GeoServer Vectorial/Raster Data Full Backup&Restore - Backup & Restore Admin Page Bakup&Restore Admin Gui migrations - Document headers - Backup&Restore PR logging and minor fixes - fix migration issue - Backup & Restore Documentation - Minot update Backup & Restore docs * Update backup.py
This comment has been minimized.
This comment has been minimized.
@afabiani can we migrate this documentation into the main GeoNode Docs now? |
This comment has been minimized.
This comment has been minimized.
Documentation is already there but it is not deployed by the continuous |
This comment has been minimized.
This comment has been minimized.
Something was erroring on the docs job and I disabled it recently. @afabiani do you have time to take a look at see if you can re-enable it? |
This comment has been minimized.
This comment has been minimized.
The problem is the only jenkins job I can see with my user is empty. It is Maybe Ariel or Simone could send some pointers? Best Regards, GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information.Ing. Alessio Fabiani GeoSolutions S.A.S. http://www.geo-solutions.it AVVERTENZE AI SENSI DEL D.Lgs. 196/2003 Le informazioni contenute in questo messaggio di posta elettronica e/o The information in this message and/or attachments, is intended solely for On Tue, Oct 18, 2016 at 8:22 PM, Jeffrey Johnson notifications@github.com
|
This comment has been minimized.
This comment has been minimized.
http://readthedocs.org/projects/geonode/builds/4531366/ this is what I see. OSError: Could not find library geos_c or load any of its variants 2016-10-19 14:29 GMT+02:00 Alessio Fabiani notifications@github.com:
Simone |
This comment has been minimized.
This comment has been minimized.
Mmm, it looks Shapely version issue. Shapely (if I recall correctly) has a There was a similar issue some time ago with the shapely version declared Best Regards, GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information.Ing. Alessio Fabiani GeoSolutions S.A.S. http://www.geo-solutions.it AVVERTENZE AI SENSI DEL D.Lgs. 196/2003 Le informazioni contenute in questo messaggio di posta elettronica e/o The information in this message and/or attachments, is intended solely for On Wed, Oct 19, 2016 at 4:09 PM, Simone Dalmasso notifications@github.com
|
This comment has been minimized.
This comment has been minimized.
That jenkins job never worked. We had it setup with a post-commit hook in On Wed, Oct 19, 2016 at 5:29 AM, Alessio Fabiani notifications@github.com
|
This comment has been minimized.
This comment has been minimized.
Yep, I'll try to fix it. Best Regards, GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information.Ing. Alessio Fabiani GeoSolutions S.A.S. http://www.geo-solutions.it AVVERTENZE AI SENSI DEL D.Lgs. 196/2003 Le informazioni contenute in questo messaggio di posta elettronica e/o The information in this message and/or attachments, is intended solely for On Wed, Oct 19, 2016 at 4:58 PM, Jeffrey Johnson notifications@github.com
|
This comment has been minimized.
This comment has been minimized.
Jeffrey, readthedocs fails due to "requirements.txt" file including requirements.txt should be used only by CIs (as far as I know) since pip This is the commit reverting it the only way to make readthedocs working is to workaround the Once solved this issue, the next one seems to be related to the generation !pdfTeX error: pdflatex (file ./shp_table.png): libpng: internal error I guess here only an administrator of the project can change the options in For the moment I can't do much more than this. Let me know how to proceed. Best Regards, GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information.Ing. Alessio Fabiani GeoSolutions S.A.S. http://www.geo-solutions.it AVVERTENZE AI SENSI DEL D.Lgs. 196/2003 Le informazioni contenute in questo messaggio di posta elettronica e/o The information in this message and/or attachments, is intended solely for On Wed, Oct 19, 2016 at 5:04 PM, Alessio Fabiani <
|
* - Restoring missing parts of the docs * - update ext pakages versions * Lock in version of gsconfig Lock in the version until issue GeoNode#2544 is resolved * gsconfig now utilizes an attribution dictionary and bumped to newest version * - update geonode exts version in order to fix pavement sync db * README: fix syntax error Heading "Docker Usage" was not rendered correctly. * ol3-preview * keep geoext as default * Missing quotation mark * Add support for EPSG:4326 * bump user-messages * don't assume 'geoserver' in style url management * tiles_url should be a XYZ source * set_metadata returns four variables, which was causing an error of too many values to unpack * 'name' no longer a ContentType field in Django 1.8+ * Added handling of postgres backed geogig stores * fix layer style dropdown values * Default to public schema if not defined * added exception handler for when multiple legends are returned * just update the tiles link url if needed, this will respect any changes to the url made by other apps like contrib.mp * register models for actstream 0.5+ / Django 1.8+ * re enable publish as layer group * adjusted text and html metadata links in the templates for documents and layers * Changes in manual installation docs (GeoNode#2561) minor changes in docs * fix layer group endpoint * fix for port changes: can't concatenate str & int * check if layerfile_set exists * Refactor to align with updated PG Geogig plugin: calls the plugin only to create the geogig repo, and calls gsconfig to create the datastore * fix client/server handling for multi-file upload/import by storing the upload id in the session * use .get explicitly instead of .all().filter(...)[0] * use elif with mutually exclusive comparisons * Invalidate layer in GeoWebCache after style_update PUT/POST * send content-type:text/xml header * use str.join() properly * fix status code check * update url for cache refresh * prevent auto redirect when additional information is required. * bump gsconfig to 1.0.6 * - Paver setup_geoserver using the new 2.9.x * - remove layer_is FK constraints from layer_styles migrations * Added documentation for configuring an AWS S3 Bucket with GeoNode * - test travis script with Java 8 (GeoNode#2584) - test travis script with Java 8 - test travis script with Java 8 * Adding instructions to add windows dependencies * Adding gdal environment variables * Update README * Update README * Update README * Revert " - remove layer_is FK constraints from layer_styles migrations" This reverts commit ea2b292. * fix broken og:image link in meta tag in head * fix exception during renders of admin forms using autocomplete_light * change LayerStyles related name to snake_case * Create new CONTRIBUTING file Using CLAHub and OSGeo CLA * restructure get_context_resourcetype for clarity * add profiles & groups autocomplete config * Groups have titles, not names * Docker for development (GeoNode#2593) * Refactor to align with updated PG Geogig plugin: calls the plugin only to create the geogig repo, and calls gsconfig to create the datastore * prevent auto redirect when additional information is required. * bump gsconfig to 1.0.6 * - Paver setup_geoserver using the new 2.9.x * - remove layer_is FK constraints from layer_styles migrations * Added documentation for configuring an AWS S3 Bucket with GeoNode * fix broken og:image link in meta tag in head * Revert " - remove layer_is FK constraints from layer_styles migrations" This reverts commit ea2b292. * - test travis script with Java 8 (GeoNode#2584) - test travis script with Java 8 - test travis script with Java 8 * No specific versions in setup.py, only minimum * Bump version to start 2.5 cycle * Use published image for development * Added docker-compose and docker-compose.override * Pinned requirements * Pull image in deployment, build image in development * Makefile for tasks * Added pull target * Avoid autocompletelight 3 * Switch to 'latest' geonode/docker * Requirements from travis * Downgrade requirements * Use requirements file * Don't use project_name * Use specific docker images * s/genode/geonode * Use geoserver latest * Use latest tag for nginx * Fix migration problems * Use run instead of exec * Fix requirements.txt * Revert "Fix migration problems" This reverts commit aa33ce6. * Fix httplib version * Fix tests * Call docker-django from latest * Use geonode/django in celery too (GeoNode#2594) * make test works in docker (GeoNode#2595) * Add context-specific search to /groups page Templates that were originally built for ResourceBase-type content are being overridden here for group objects (e.g., _group_search_content.html) * groups do not have date or popular_count fields * let custom search_content template work for both profiles & groups * Custom sort filters depending on data type * Add context-specific search (filtering by username) within /people * if searching from /people context, use username instead of title * url fix in README * add name to AUTHORS * geonode-user-accounts 1.0.13 (GeoNode#2603) * Fix for issue 2599. The issue was that apache was allowing directory browsing. Added the options directive to remove it from the current list of options enforced for that directory. * Use environment variables in settings.py (GeoNode#2596) * Missing migrations (GeoNode#2604) * geonode-user-accounts 1.0.13 * Missing migrations * pep8 for migrations * pep8 * Updated readme for docker (GeoNode#2607) * Using docker images from hub * - (issue GeoNode#2070) Layer name cannot be a style name already on Geoserver * - (issue GeoNode#2075) Add lxml back into paver win_install_deps * adding missing trans for GeoNode#1769 * sycing tsf * use layer instance to revoke layer permissions, fixes GeoNode#2531 (GeoNode#2614) use layer instance instead of resourcebase to revoke permission, fixes GeoNode#2531 * include data files in downstream installations should fix GeoNode#2452 (GeoNode#2615) include data files in downstream installations should fix GeoNode#2452 * - Modify some settings and code to make geonode able to use QGIS Server as backend. - Add some views, templates and resources to make geonode able to use QGIS Server as backend. - Add sample setting to use QGIS Server backend. * should fix GeoNode#2464 (GeoNode#2616) bump haystack versions to support django 1.8 * Fixed migrations, settings.py, and pavement for development * pavement: moved migrate_apps to yaml * fix GeoNode#1718 * make debug default to true * sycing for GeoNode#1769, after GeoNode#2612 merge * fix upload issue, check if the js property exists first * hierarchical tags support * fix indentation * add freeboard to requirements.txt * bump taggit * Don't pin versions in setup.py use >= or <= Pinning should be done in requirements.txt Reasoning is explained in https://packaging.python.org/requirements/ * Pin versions in requirements.txt * Set to 2.5 final version for release * update keywords integration test * fix filename * we dont use submodules anymore * Updated instruction for configure ssl on geonode: with version 2.4 some path are changed (GeoNode#2571) and is not necessary the step regarding python and java configuration because ssl is managed by apache mod_wsgi * 2.5.0 is out, move master branch to 2.5.1-dev * Delete fig.yml * Do not flake8 migrations * python-properties is now usually pre-installed in Ubuntu And GeoNode works well with Ubuntu 14.04 * Fix typos in installation dev docs * Fix GeoGig stats/logs URL for Layer info page. (GeoNode#2633) The repoURL was using the GeoServer DataStore connection_parameters value to contstruct the GeoGig repository URL, which looks like: http://<server>/geoserver/geogig/geoserver://<repo name> The statistics and logs URL for GeoGig needs to use a repo URL like this: http://<server>/geoserver/geogig/repos/<repo name> * Do not pin versions on setup.py * Upgrade django mptt * Use a pattern for adding package data, fixes missing fixtures on packages * Set development version for 2.5.3 * Add volume data container to geoserver * Exclude remote services layers from map download feature. * Make download links open in new tab (GeoNode#2642) * Fix race condition during Raster upload/import - Removing a duplicate block of code for PENDING raster imports - Don't re-send the Import request if it's already been sent * Add variables for building base_url * local_settings.py should be called only once * Provide defaults for optional ogc_server_settings (GeoNode#2647) * Provide defaults for optional ogc_server_settings MAPFISH_PRINT_ENABLED, PRINT_NG_ENABLED, GEONODE_SECURITY_ENABLED, GEOGIG_ENABLED * fixup * Move master to 2.5.4 * Minor text fix for profile update * fix QGIS path log when using qgis server backend (GeoNode#2648) * Bugfix for updating style through UI * Clear comment modal after adding comment * Issue GeoNode#2401: GNIP: GeoNode Backup and Restore (GeoNode#2651) * GeoNode Backup and Restore Management Commands - GeoServer Vectorial/Raster Data Full Backup&Restore - Backup & Restore Admin Page Bakup&Restore Admin Gui migrations - Document headers - Backup&Restore PR logging and minor fixes - fix migration issue - Backup & Restore Documentation - Minot update Backup & Restore docs * Update backup.py * Respond correctly when user removes the last related file for a layer upload (GeoNode#2660) * bump guardian and manage the anonymous pk internally * use names instead of slugs for keywords * ini file was not an allowed file type for MANIFEST.in. This was causing the settings.ini file to not be included in the package. * Fix small bug where {{ repoUrl }} was being used as geogig email due to incorrect input type in HTML * Fixing missing and broken links to actions on the user's activity page * Updating pycsw to the latest released version 2.0.2 * bump pycsw * Update conf.py * Update setup.py * Update conf.py * Update conf.py * Update requirements.txt * Update conf.py * - Fix the docs * Updating Choose Files button to allow user to reupload the same file again without having to use Clear (GeoNode#2681) * Use new UserActivity view (GeoNode#2680) * truncate upload names over length to allow upload (GeoNode#2671) One approach to the problem described in GeoNode#2646 This patch uses the Django field definition to get the length, so this can be controlled from that one place and won't get out of sync. * remove mutable default values for kwargs (GeoNode#2668) This subtle misunderstanding of Python tends to cause confusing bugs. Suppose you have a definition statement like 'def function(keys=[]): ...' This def statement is evaluated once, at import time, to create a function object. It is at this time that '[]' is evaluated, and it is that ONE list object which is retained as the default value for 'keys'. Python doesn't store the expression and re-evaluate it every time the default is needed, it evaluates it once at import time and the resulting value is passed for every call where that arg is not given. To explain this with code, the effect is something like this: default_keys = [] def function(keys=default_keys): ... When function() is called with a value for keys, no problem will be seen. Even when it is called with no value for keys, and we default to that list, you might not see a problem. The problem that tends to happen unpredictably is that the mutable default value is mutated. Those changes are persistent and globally visible to any call of the function which uses the default. Then the default value is non-empty. Or information leaks across calls that shouldn't be leaking. Or the same dict keeps growing forever. To avoid this, we can use a sentinel value such as None, and then check for that value in the body of the function. It is a cheap safeguard against situations that can be very annoying. * first of many docs fixes (GeoNode#2684) * Fix the Resources section on profile detail page to work when no data exists in the category yet * Incubation documentation fixes (GeoNode#2687) * Fixed dead link [ci skip] * Fixed some doc links [ci skip] * Fixed License header in some po files * Fixed packaging files copyright * Added OSGeo copyright and CC-SA in documentation * Fixed bug breaking Group Activity view * Updated changelog for version 2.5.4 * Updating release info and some improvements to pavement for Xenial releases * Fixing travis build * previous adjustments to the setup.py and MANIFEST.ini were not allowing certain filetypes to be packaged, loading.gif was the most noticeable. * Updated changelog for version 2.5.5 * Update release version * Make resourcebase info panel template extendable * Fixing syntax error: celery_app (GeoNode#2697) * Sync auto generate avatar sizes with avatar sizes used in templates (GeoNode#2693) * 2696 Part 1 - GNIP - Better management of View and Download permissions. 2696 Part 1 - GNIP - Better management of View and Download permissions. - Fix flake8 formatting issues * Updated AUTHORS list * Update views.py - Fix minor typo with brackets * Update AUTHORS * Enable update layers management command to accept the permissions dictionary. (GeoNode#2691) * Insert missing sudo to chmod commands This fixes "[Errno 13] Permission denied: '/home/geonode/geonode/geonode/uploaded/layers/" encountered by user when trying to upload layer files. * Clean up extra ':'s (from ::: to ::) * Language cleanup, deleted duplicate section * Minor cleanup * Process inbox messages using AJAX * - removed import on INSTALLED_SCHEMES since its usage was removed on this commit (aff43c0). - limiting the version of celery to less then the 4.0 release, due to django-celery not having a release in over a year which would include the updates in master to support the new version of celery. If you currently install Geonode it will install celery 4.0, but when attempting to use any django commands an ImportError occurs due to no module named timeutils. * Split geoserver.helpers.set_attributes() (GeoNode#2699) * Add conditional to geosites so signals are connected only if it is in INSTALLED_APPS. * Added test for geonode.utils.set_attributes(). * Split helpers.set_attributes() into 2 functions. * Renamed set_attributes() to set_attributes_from_geoserver(). * Moved really_set_attributes(), to geonode.utils and renamed to set_attributes(). * Removed unnecessary variable 'Attribute' from utils. * Added check for layer/field pairs not in attribute_stats. * flake8 formatting fixes. * require celery version to be at least 3.1.18 When using `pip install -r requirements.txt` I was finding that my Celery worker was not booting properly. ``` ... File "/env/lib/python2.7/site-packages/django/core/management/base.py", line 534, in check self.stderr.write(msg, lambda x: x) TypeError: function takes exactly 1 argument (2 given) ``` This is an incompatibility between Celery 3.1.17 and Django 1.8. celery/celery#2536 Fix: upgrade to Celery 3.1.18, which supports Django 1.8. http://docs.celeryproject.org/en/latest/history/changelog-3.1.html#version-3-1-18 * The standalone TEMPLATE_* settings were deprecated in Django 1.8 (GeoNode#2711) - The standalone TEMPLATE_* settings were deprecated in Django 1.8 and the TEMPLATES dictionary takes precedence. You must put the values of the following settings into your default TEMPLATES dict: TEMPLATE_DIRS, TEMPLATE_CONTEXT_PROCESSORS, TEMPLATE_DEBUG. https://docs.djangoproject.com/en/1.8/topics/templates/#support-for-template-engines - historic pep8 fixes * Set SITEURL to server's host name
* Adds support for s3 storage and forces layer uploads to use the FileSystemStorage * Fix issue GeoNode#2526 * Added LDAP install and configure steps in Admin workshop * Add a name to AUTHORS * Fill empty value for DATASTORE variable in docs * Added an autocomplete template and additions to enable search by catalogue * Created template and edited files to enable .txt and .html formats export for metadata * Fixes admin interface for layers AttributeError: 'NoneType' object has no attribute 'name' django-polymorphic/django-polymorphic#201 * dj_database_url formats the port as an integer, whereas gsconfig is wanting a string for creating the feature store. * addition to previous patch: format port as str * bump some packages version * - Restoring missing parts of the docs * - update ext pakages versions * Lock in version of gsconfig Lock in the version until issue GeoNode#2544 is resolved * gsconfig now utilizes an attribution dictionary and bumped to newest version * - update geonode exts version in order to fix pavement sync db * README: fix syntax error Heading "Docker Usage" was not rendered correctly. * ol3-preview * keep geoext as default * Missing quotation mark * Add support for EPSG:4326 * bump user-messages * don't assume 'geoserver' in style url management * tiles_url should be a XYZ source * set_metadata returns four variables, which was causing an error of too many values to unpack * 'name' no longer a ContentType field in Django 1.8+ * Added handling of postgres backed geogig stores * fix layer style dropdown values * Default to public schema if not defined * added exception handler for when multiple legends are returned * just update the tiles link url if needed, this will respect any changes to the url made by other apps like contrib.mp * register models for actstream 0.5+ / Django 1.8+ * re enable publish as layer group * adjusted text and html metadata links in the templates for documents and layers * Changes in manual installation docs (GeoNode#2561) minor changes in docs * fix layer group endpoint * fix for port changes: can't concatenate str & int * check if layerfile_set exists * Refactor to align with updated PG Geogig plugin: calls the plugin only to create the geogig repo, and calls gsconfig to create the datastore * fix client/server handling for multi-file upload/import by storing the upload id in the session * use .get explicitly instead of .all().filter(...)[0] * use elif with mutually exclusive comparisons * Invalidate layer in GeoWebCache after style_update PUT/POST * send content-type:text/xml header * use str.join() properly * fix status code check * update url for cache refresh * prevent auto redirect when additional information is required. * bump gsconfig to 1.0.6 * - Paver setup_geoserver using the new 2.9.x * - remove layer_is FK constraints from layer_styles migrations * Added documentation for configuring an AWS S3 Bucket with GeoNode * - test travis script with Java 8 (GeoNode#2584) - test travis script with Java 8 - test travis script with Java 8 * Adding instructions to add windows dependencies * Adding gdal environment variables * Update README * Update README * Update README * Revert " - remove layer_is FK constraints from layer_styles migrations" This reverts commit ea2b292. * fix broken og:image link in meta tag in head * fix exception during renders of admin forms using autocomplete_light * change LayerStyles related name to snake_case * Create new CONTRIBUTING file Using CLAHub and OSGeo CLA * restructure get_context_resourcetype for clarity * add profiles & groups autocomplete config * Groups have titles, not names * Docker for development (GeoNode#2593) * Refactor to align with updated PG Geogig plugin: calls the plugin only to create the geogig repo, and calls gsconfig to create the datastore * prevent auto redirect when additional information is required. * bump gsconfig to 1.0.6 * - Paver setup_geoserver using the new 2.9.x * - remove layer_is FK constraints from layer_styles migrations * Added documentation for configuring an AWS S3 Bucket with GeoNode * fix broken og:image link in meta tag in head * Revert " - remove layer_is FK constraints from layer_styles migrations" This reverts commit ea2b292. * - test travis script with Java 8 (GeoNode#2584) - test travis script with Java 8 - test travis script with Java 8 * No specific versions in setup.py, only minimum * Bump version to start 2.5 cycle * Use published image for development * Added docker-compose and docker-compose.override * Pinned requirements * Pull image in deployment, build image in development * Makefile for tasks * Added pull target * Avoid autocompletelight 3 * Switch to 'latest' geonode/docker * Requirements from travis * Downgrade requirements * Use requirements file * Don't use project_name * Use specific docker images * s/genode/geonode * Use geoserver latest * Use latest tag for nginx * Fix migration problems * Use run instead of exec * Fix requirements.txt * Revert "Fix migration problems" This reverts commit aa33ce6. * Fix httplib version * Fix tests * Call docker-django from latest * Use geonode/django in celery too (GeoNode#2594) * make test works in docker (GeoNode#2595) * Add context-specific search to /groups page Templates that were originally built for ResourceBase-type content are being overridden here for group objects (e.g., _group_search_content.html) * groups do not have date or popular_count fields * let custom search_content template work for both profiles & groups * Custom sort filters depending on data type * Add context-specific search (filtering by username) within /people * if searching from /people context, use username instead of title * url fix in README * add name to AUTHORS * geonode-user-accounts 1.0.13 (GeoNode#2603) * Fix for issue 2599. The issue was that apache was allowing directory browsing. Added the options directive to remove it from the current list of options enforced for that directory. * Use environment variables in settings.py (GeoNode#2596) * Missing migrations (GeoNode#2604) * geonode-user-accounts 1.0.13 * Missing migrations * pep8 for migrations * pep8 * Updated readme for docker (GeoNode#2607) * Using docker images from hub * - (issue GeoNode#2070) Layer name cannot be a style name already on Geoserver * - (issue GeoNode#2075) Add lxml back into paver win_install_deps * adding missing trans for GeoNode#1769 * sycing tsf * use layer instance to revoke layer permissions, fixes GeoNode#2531 (GeoNode#2614) use layer instance instead of resourcebase to revoke permission, fixes GeoNode#2531 * include data files in downstream installations should fix GeoNode#2452 (GeoNode#2615) include data files in downstream installations should fix GeoNode#2452 * - Modify some settings and code to make geonode able to use QGIS Server as backend. - Add some views, templates and resources to make geonode able to use QGIS Server as backend. - Add sample setting to use QGIS Server backend. * should fix GeoNode#2464 (GeoNode#2616) bump haystack versions to support django 1.8 * Fixed migrations, settings.py, and pavement for development * pavement: moved migrate_apps to yaml * fix GeoNode#1718 * make debug default to true * sycing for GeoNode#1769, after GeoNode#2612 merge * fix upload issue, check if the js property exists first * hierarchical tags support * fix indentation * add freeboard to requirements.txt * bump taggit * Don't pin versions in setup.py use >= or <= Pinning should be done in requirements.txt Reasoning is explained in https://packaging.python.org/requirements/ * Pin versions in requirements.txt * Set to 2.5 final version for release * update keywords integration test * fix filename * we dont use submodules anymore * Updated instruction for configure ssl on geonode: with version 2.4 some path are changed (GeoNode#2571) and is not necessary the step regarding python and java configuration because ssl is managed by apache mod_wsgi * 2.5.0 is out, move master branch to 2.5.1-dev * Delete fig.yml * Do not flake8 migrations * python-properties is now usually pre-installed in Ubuntu And GeoNode works well with Ubuntu 14.04 * Fix typos in installation dev docs * Fix GeoGig stats/logs URL for Layer info page. (GeoNode#2633) The repoURL was using the GeoServer DataStore connection_parameters value to contstruct the GeoGig repository URL, which looks like: http://<server>/geoserver/geogig/geoserver://<repo name> The statistics and logs URL for GeoGig needs to use a repo URL like this: http://<server>/geoserver/geogig/repos/<repo name> * Do not pin versions on setup.py * Upgrade django mptt * Use a pattern for adding package data, fixes missing fixtures on packages * Set development version for 2.5.3 * Add volume data container to geoserver * Exclude remote services layers from map download feature. * Make download links open in new tab (GeoNode#2642) * Fix race condition during Raster upload/import - Removing a duplicate block of code for PENDING raster imports - Don't re-send the Import request if it's already been sent * Add variables for building base_url * local_settings.py should be called only once * Provide defaults for optional ogc_server_settings (GeoNode#2647) * Provide defaults for optional ogc_server_settings MAPFISH_PRINT_ENABLED, PRINT_NG_ENABLED, GEONODE_SECURITY_ENABLED, GEOGIG_ENABLED * fixup * Move master to 2.5.4 * Minor text fix for profile update * fix QGIS path log when using qgis server backend (GeoNode#2648) * Bugfix for updating style through UI * Clear comment modal after adding comment * Issue GeoNode#2401: GNIP: GeoNode Backup and Restore (GeoNode#2651) * GeoNode Backup and Restore Management Commands - GeoServer Vectorial/Raster Data Full Backup&Restore - Backup & Restore Admin Page Bakup&Restore Admin Gui migrations - Document headers - Backup&Restore PR logging and minor fixes - fix migration issue - Backup & Restore Documentation - Minot update Backup & Restore docs * Update backup.py * Respond correctly when user removes the last related file for a layer upload (GeoNode#2660) * bump guardian and manage the anonymous pk internally * use names instead of slugs for keywords * ini file was not an allowed file type for MANIFEST.in. This was causing the settings.ini file to not be included in the package. * Fix small bug where {{ repoUrl }} was being used as geogig email due to incorrect input type in HTML * Fixing missing and broken links to actions on the user's activity page * Updating pycsw to the latest released version 2.0.2 * bump pycsw * Update conf.py * Update setup.py * Update conf.py * Update conf.py * Update requirements.txt * Update conf.py * - Fix the docs * Updating Choose Files button to allow user to reupload the same file again without having to use Clear (GeoNode#2681) * Use new UserActivity view (GeoNode#2680) * truncate upload names over length to allow upload (GeoNode#2671) One approach to the problem described in GeoNode#2646 This patch uses the Django field definition to get the length, so this can be controlled from that one place and won't get out of sync. * remove mutable default values for kwargs (GeoNode#2668) This subtle misunderstanding of Python tends to cause confusing bugs. Suppose you have a definition statement like 'def function(keys=[]): ...' This def statement is evaluated once, at import time, to create a function object. It is at this time that '[]' is evaluated, and it is that ONE list object which is retained as the default value for 'keys'. Python doesn't store the expression and re-evaluate it every time the default is needed, it evaluates it once at import time and the resulting value is passed for every call where that arg is not given. To explain this with code, the effect is something like this: default_keys = [] def function(keys=default_keys): ... When function() is called with a value for keys, no problem will be seen. Even when it is called with no value for keys, and we default to that list, you might not see a problem. The problem that tends to happen unpredictably is that the mutable default value is mutated. Those changes are persistent and globally visible to any call of the function which uses the default. Then the default value is non-empty. Or information leaks across calls that shouldn't be leaking. Or the same dict keeps growing forever. To avoid this, we can use a sentinel value such as None, and then check for that value in the body of the function. It is a cheap safeguard against situations that can be very annoying. * first of many docs fixes (GeoNode#2684) * Fix the Resources section on profile detail page to work when no data exists in the category yet * Incubation documentation fixes (GeoNode#2687) * Fixed dead link [ci skip] * Fixed some doc links [ci skip] * Fixed License header in some po files * Fixed packaging files copyright * Added OSGeo copyright and CC-SA in documentation * Fixed bug breaking Group Activity view * Updated changelog for version 2.5.4 * Updating release info and some improvements to pavement for Xenial releases * Fixing travis build * previous adjustments to the setup.py and MANIFEST.ini were not allowing certain filetypes to be packaged, loading.gif was the most noticeable. * Updated changelog for version 2.5.5 * Update release version * Make resourcebase info panel template extendable * Fixing syntax error: celery_app (GeoNode#2697) * Sync auto generate avatar sizes with avatar sizes used in templates (GeoNode#2693) * 2696 Part 1 - GNIP - Better management of View and Download permissions. 2696 Part 1 - GNIP - Better management of View and Download permissions. - Fix flake8 formatting issues * Updated AUTHORS list * Update views.py - Fix minor typo with brackets * Add ol3-react based viewer as default map viewer Add a new template for editing/showing new maps and change just the viewer to use the new ol3-react based viewer add all kinds of assets/deps for the ol3-react based viewer ol3-cesium is a little tedious but is needed for 3d globe
Overview
The scope of this proposal is to point out a comprehensive list of functionalities, requirements and constraints we envisage a strong, reliable and automatic GeoNode/GeoServer backup and restore procedure should provide, and finally depict a technical design along with technological choices for the implementation of such tool.
Proposed by
Alessio Fabiani (GeoSolutions)
Assigned to release
None yet.
Motivation
Currently does not exist any simple and automatic procedure to execute a backup and restore of a GeoNode instance nor a way to migrate from a version to another.
In the GeoNode documentation it is described a semi-automatic multi-step procedure allowing a tech-savvy user to perform a backup and restore of GeoNode and GeoServer, but the process is prone to errors that could result in data losses.
Proposal
Purpose and Requirements and Constraints:
The Admin command to backup and restore GeoNode data on a given version and to migrate data from one version to another, from our perspective can be splitted in three big conceptual modules:
Extract
The tool should be able to extract consistently the GeoNode and GeoServer data models in a serializable meta-format which can be interpreted later by the restore procedure in order to exactly rebuild the whole structure, accordingly to the current instance version (which may be different from the starting one).
In particular the tool must guarantee that the following resources are correctly extracted and serialized on the storage and deserialized on the target GeoNode/GeoServer instance:
a. Layers (both raster and vectors)
b. Security model along with auth filters configuration, users and credentials
c. Workspaces
d. Stores (both DataStores and CoverageStores)
e. Layers
f. Styles
a. OWS Services configuration and limits
b. Security model along with auth filters configuration, users and credentials
c. Workspaces
d. Stores (both DataStores and CoverageStores)
e. Layers
f. Styles
Backup and Restore
The tool must be able to save and restore GeoNode/GeoServer binary data. In particular the data which must be backed up are:
a. Uploaded Document data (pdf, txt, doc, xls, zip) and images (jpg , png, tif) are stored locally in GeoNodes www upload folders: geonode/uploaded/documents/
a. Uploaded geo-spatial data
- GeoNode uploaded Vector data (shp files, etc); stored in PostGIS DB, managed via GeoServer
- GeoNode uploaded Raster data (GeoTiff); stored locally in GeoServer as its own "Datastore"
b. User vectorial/raster data configured in GeoServer
c. GeoServer Styles
d. GeoServer plugin folders and workspaces/datastores external images and Freemarker templates
Features and Constraints
The tool must provide the following functionalities and respect several constraints:
a. A Sequence ID with the Backup Date
b. The status of the backup (valid or not valid)
c. The version of the GeoNode/GeoServer backed up
d. The possibility of delete old backups
Technical Proposal
Backup and Restore - Functional Perspective
The diagram depicted on Fig. 1 represents a functional view of the backup/restore tool we are proposing.
Fig. 1
Diagram in Fig.1 shows how the responsibilities have been functionally separated into two different components:
Each component is developed separately but the whole transaction is orchestrated by the backup/restore tool, also developed as a DJango extension and integrated with GeoNode, which drives the entire procedure making sure that the workflow is executed entirely and properly.
The advantage of such approach is that GeoNode is not aware about the GeoServer configuration and/or version.
From a functional perspective each component performs its tasks by following the conceptual model described in the introductory section:
Backup and Restore - Technological Perspective
In this section we are going to describe the technological choices we are proposing in order to implement the conceptual solutions designed in the previous section.
Extract
GeoNode Extract
GeoNode is developed as a DJango project and takes advantage of the framework capabilities and plugin.
In DJango there are several ways to stream the data model into a serialized format. As an instance the plugin “django-smuggler”(1) allows to easily dump/load fixtures via the automatically-generated administration interface. A fixture is a file containing serialized model data (e.g. JSON or XML format may be used) that Django knows how to import into the database. Smuggler is especially useful for transporting database data between production and development environments, but can also be used as a backup tool.
The Smuggler plugin is not the only one possible choice and the final technical proposal will be discussed with the GeoNode developer community and with the GFDRR’s Innovation Lab. In any case the specific technological choice won’t impact much the approach we propose here.
GeoServer Extract
We envisage to build the whole GeoServer backup and restore functionality as a plugin interacting directly with the GeoServer catalog and data model.
Since the plugin will be integrated in GeoServer, it will be able to access to the GeoServer core api. Moreover the GeoServer catalog is currently already serialized as XML fixtures.
Under those assumptions, the GeoServer backup and restore plugin will be able to store the data model by accessing to the GeoServer core apis directly.
Backup
GeoNode Backup
We envisage the GeoNode backup component process (not the whole backup process but only the one provided by the specific GeoNode extension, responsible only for the backup of the GeoNode instance data model and static/binary data) as composed by four different phases:
Uploaded documents folders
Images
Web content folders
HTTPD web content
GeoServer Backup
The GeoServer backup plugin will perform the backup process in three different steps:
a. Data folder containing raw binary data
b. Styles SLDs
c. Workspaces and DataStores Freemarker Templates
d. Other extension and plugin folders and files
It is worth to point out that this procedure will filter out stuff that should not be saved, like:
a. GeoServer logs and temporary folder contents
b. GeoWebcache tiles and temporary content
Restore
GeoNode Restore
We envisage the GeoNode restore component process as composed by four different phases:
GeoServer Restore
The GeoServer backup plugin will perform the restore process in three different steps:
Useful links