Skip to content

Commit

Permalink
Split codebase into vocabulary definition and python application code…
Browse files Browse the repository at this point in the history 
… (in submodule - sdopythonapp) (#2365)

* First pass at removing sdopythonapp files and introduction of required new files

* More appless setup

* More basic setup for appless

* Adjustments for code being moved into sdopythonapp

* Moved version information into versions.json

* Report PythonApp version in output

* Added path inserts for sdopythonapp

* tweaks

* Added redirect in siteconfig.json to enable scripted switching between local & remote data

* Changed config variable SCHEMAORGLOC to a more generic VOCABDEFLOC

* Set default to './site'  - gets overriden by buildsite scripts

* test

* tests

* Tweaks

* Added sdopythonapp module (from RichardWallis/sdopythonapp)

* Update to siteconfig

* Tweaks

* Remove force of CONFIGFILE

* Update submodule

* Mods for requesting deployment to specific sites

* Final tweaks

* Updated software description

* Only copy handlers.yaml

* updated for split

* Moved runpythonpreprepare.sh into scripts directory

* Ling to submodule https://github.com/schemaorg/sdopythonapp.git

* emoved reference to test submodule

* Added sdopythonapp submodule

* Bring in sync with sdopythonapp

* Coppied in utils from sdopythonapp

* Tweak to description

* Sync with sdopythonapp

* Added git submodule update to run scripts
  • Loading branch information
@RichardWallis @danbri
RichardWallis authored and danbri committed Oct 17, 2019
1 parent 8fa9aa3 commit 72a38a4
Show file tree
Hide file tree
Showing 306 changed files with 467 additions and 176,780 deletions.
3 changes: 3 additions & 0 deletions .gitmodules
View file
@@ -0,0 +1,3 @@
[submodule "sdopythonapp"]
path = sdopythonapp
url = https://github.com/schemaorg/sdopythonapp.git
2 changes: 2 additions & 0 deletions README.md
View file
Expand Up @@ -4,6 +4,8 @@ Welcome to Schema.org

This is the Schema.org project repository. It contains all the schemas, examples and software used to publish schema.org. For the site itself, please see [Schema.org](https://schema.org/) instead.

Note: Much of the supporting software is imported from a sub module: 'sdopythonapp'

Issues and proposals are managed here by participants of the [W3C Schema.org Community Group](https://www.w3.org/community/schemaorg/). If you are interested to participate please
join the group at the [W3C](https://www.w3.org/community/schemaorg/), introduce yourself and find or file issues here that engage your interest. If you are new to Git and GitHub, there's a useful [introduction to Github](https://www.w3.org/2006/tools/wiki/Github) in the W3C Wiki.

Expand Down
123 changes: 40 additions & 83 deletions RELEASING.TXT
View file
Expand Up @@ -13,27 +13,28 @@ chosen in /docs/releases.html (check markup assigns an HTML ID too).

The following steps assume a general healthy freeze (tested / QA as below).

in sdoapp.py:

SCHEMA_VERSION=2.2 # or 2.x, as above.

in sdoapp.py:

releaselog = { "2.0": "2015-05-13", "2.1": "2015-08-06" }

in versions.json:

"schemaversion": "3.9",
"releaseLog": {
.
.
"3.8": "2019-07-01",
"3.9": "2019-08-01"
}
becomes

releaselog = { "2.0": "2015-05-13", "2.1": "2015-08-06", "2.2": "2015-11-05" }

"schemaversion": "4.0",
"releaseLog": {
.
.
"3.8": "2019-07-01",
"3.9": "2019-08-01",
"4.0": "2019-10-01"
}

... this should have a release date for the current release and all
previous releases that are archived under data/releases/{version}/*

TODO: Add a unit test to ensure this.

TODO: Re-order this document to avoid forward references.

Under data/releases/{version}/ we need a versioned-snapshot structure:

e.g.
data/releases/2.1
Expand All @@ -54,18 +55,6 @@ Example usage:
scripts/buildreleasefiles.sh 3.9


* Inspect snapshot, add to git and push to repo.
* ls -l data/releases/2.2/
* git add data/releases/2.2/*
* git push
* TODO: more QA and automation would be highly appropriate here.



TODO: list optional tasks post-launch (blog, email, refresh dydra sparql db etc.).




1) General preconditions / process and QA for release.

Expand Down Expand Up @@ -116,71 +105,39 @@ e.g. see http://webschemas.org/docs/qa.html
There is often some final tweaking, bugs, etc. When things settle down, a
final round of sanity-check tests is useful:

* Run again './scripts/buildsitemap.py' make sure sitemap is up to date
* Run again: PYTHONPATH=/usr/local/google_appengine ./scripts/run_tests.py
* Run again 'git status', confirm no uncommited files or changes.
* Run again: './scripts/updateAppEngine.sh' for development/upstream site sync.
* Run again './scripts/buildreleasefiles.py'

3.) Previously: "Merge release branch into master"; now we develop in "master."

4.) Publish from a clean checkout of 'master' branch to schema.org.
Using gcloud deployment script
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Using gcloud app deploy
~~~~~~~~~~~~~~~~~~~~~~~
Use python deploment script:

From 3.4.1 onwards use app deploy script (does a bit of housekeeping):
scripts/appdeploy.sh --project schemaorgae --no-promote --version=3-5-1 schemaorg.yaml

./scripts/deployschema.org.sh - to deploy to main Schema.org site
./scripts/deploywebschemas.org.sh - to deploy to main webschemas.org site

* Choose Local option to deploy local configuration files
* Choose Remote option to pull files from a URL (eg. the github repository)

* Provide an appropriate version string - different to currently running version
* Optionally pass -e to bypass site exercise option
* Optionally pass -m to bypass merge of new version


Recommended Method For Making Release Live
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

1) Use app deploy script as above providing a version identifier not in use on schemaorgae appengine.

2) Access that version directly via the URL of following format:
https://3-5-1-dot-schemaorgae.appspot.com

Note: The application will take a few minutes before responding to its first request.
If you get a timeout message, try again in 30 seconds or so.

3) Try a few samples requests to the direct URL

4) In directory from which the deploy took place run the following script, with the approriate site (-s) value:
scripts/exercisesite.py -s https://3-5-1-dot-schemaorgae.appspot.com

The script requests a page for each Type and property, causing them to be cached in the application's storage
area. It averages a few seconds per page, the whole process takes 60-90 minutes.

5) When exercisesite is complete, use the Traffic Spiting feature of the App Engine Console to progressively increase
the percentage of traffic being allocated to the new version (5%, 10%, 25%, 50%, 75%, 100%) with at least a few minutes
between each.

During all the above steps, monitor the log for the new version, filtering for errors being the min requirement. Note: receiving occasional errors from requests such as _start, _stop, can be expected.


##### original deploy: gcloud app deploy --no-promote --project schemaorgae --version=3.4 schemaorg.yaml

Note: the --no-promote flage deploys the version but does not transfer traffic to it.
This can then be done when required via the appengine console, versions screen:
https://console.cloud.google.com/appengine/versions?project=schemaorgae&serviceId=default&versionssize=50

===========OLD============
Either by editing app.yaml to make sure 'application: schemaorgae' is active,
or by passing the name of the live appengine ID on the commandline.
===========OLD============

Currently @rvguha (as Steering Group chair) and @danbri (as Community Group chair and
webmaster / dogsbody) have write access to the corresponding appengine account.

===========OLD============
* e.g. update schemaorg.yaml then: appcfg.py update schemaorg.yaml -A schemaorgae
* Count to thirty. Log into appengine and you should see a new version.
* Send 5% of traffic to this version to catch any errors, then move to 100%.
===========OLD============
For Webschema.org
Use ./scripts/deploywebschemas.org.sh -e
No real benefit in exercising low traffic site

For Schema.org.

Note that appengine versions are visible using URLs with this pattern:
Either run ./scripts/deployschema.org.sh
This will put the new site live when site exercise is complete (takes some time)
or run ./scripts/deployschema.org.sh -m
At some time after site exercise is complete login to console and manually migrate traffic

http://3-3.schemaorgae.appspot.com/docs/releases.html

5.) Tag the code at the current relese state
* Use a local copy of the relesed version of the 'master' branch:
Expand Down
79 changes: 22 additions & 57 deletions SOFTWARE_README.md
View file
Expand Up @@ -5,79 +5,44 @@ Schema.org Software
This document describes the software that underpins schema.org. Most collaborators will only need to be able to run
it. At this time we do not solicit active collaboration on the codebase itself from the general public.

* see https://github.com/schemaorg/schemaorg/blob/sdo-phobos/LICENSE for opensource license info (Apache2)
* see https://github.com/schemaorg/schemaorg/blob/master/LICENSE for opensource license info (Apache2)

Software
========

The site codebase is a simple Python application. It uses Google App Engine, and is designed to allow schema.org contributors to explore new or improved schemas. The code has a bias towards simplicity and minimal dependencies,
rather than elegance and re-usability. We expect collaboration will focus more on schemas and examples than
on our supporting software.
The site codebase is a simple Python application. It uses Google App Engine, and is designed to allow schema.org contributors to explore new or improved schemas. The code has a bias towards simplicity and minimal dependencies.

The app reads its schemas and examples from the data/ directory when it starts up. These
are expressed in simple text formats. Proposals to schema.org can be provided as diffs
or github pull requests.
This repository only contains the vovcabulary definition, and examples files, supporting documetation, and Schema,org specifi tests and build scripts.

Internals
=========

Internally, the app uses a simple RDF-like graph data model, and has a parser for
the RDFa Lite subset that we use to represent schemas. Potential contributors are
cautioned that this code is not designed to become a general purpose framework, and
that we're comfortable with it being hardcoded in various ways around the needs and
approaches of schema.org. If that's not too discouraging, do let us know if you find
interesting uses for it or have ideas for improvements.
The core software is included via a sub module 'sdopythonapp'.

See also wiki: https://github.com/schemaorg/schemaorg/wiki/Contributing

External Software
=================
Working with a local version of Schema.org
==========================================

In addition to AppEngine and Python itself, this repository
contains copies of the following opensource libraries in the
lib/ directory tree.
To work on the vocabulary and run locally firstly clone the repository on a local system:
`git clone git clone --recurse-submodules https://github.com/schemaorg/schemaorg.git`

(If you forget the `--recurse-submodules` option, run the command `git submodule update --init --recursive`)

1.) html5lib/
To locally run the application run `./runpythonapp.sh`

https://pypi.python.org/pypi/html5lib (MIT License)
To deploy to a Google appengine run `./deploypythonapp.sh`

2.) isodate/
There are preconfigured scripts `./scripts/deployschema.org.sh` & `./scripts/deploywebschemas.org.sh` to deploy to the main schema.org sites (with relevant permissions).

https://pypi.python.org/pypi/isodate (BSD License)
Note: If you are informed of an update to the sdopythonapp submodule, use the command `git submodule update --remote` to synchronise with the local version

3.) markdown2/

https://github.com/trentm/python-markdown2 (MIT License)

4.) pyparsing.py
Internals
=========

From lib/pyparsing.py
Internally, the app uses a simple RDF-like graph data model, and has a parser for
the RDFa Lite subset that we use to represent schemas.

# Copyright (c) 2003-2015 Paul T. McGuire
#
# Permission is hereby granted, free of charge, to any person obtaining
# a copy of this software and associated documentation files (the
# "Software"), to deal in the Software without restriction, including
# without limitation the rights to use, copy, modify, merge, publish,
# distribute, sublicense, and/or sell copies of the Software, and to
# permit persons to whom the Software is furnished to do so, subject to
# the following conditions:
#
# The above copyright notice and this permission notice shall be
# included in all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
# IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
# CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
# TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
# SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
See also wiki: https://github.com/schemaorg/schemaorg/wiki/Contributing

5.) rdflib/ and rdflib_jsonld/
External Software
=================

https://github.com/RDFLib
https://github.com/RDFLib/rdflib/blob/master/LICENSE
Checkout ./sdopythonapp/SOFTWARE_README.md

https://github.com/RDFLib/rdflib-jsonld
https://github.com/RDFLib/rdflib-jsonld/blob/master/LICENSE.md

0 comments on commit 72a38a4

Please sign in to comment.