After you have installed Weaver, you can customize its behaviour using multiple configuration settings.
All settings are configured using a weaver.ini
configuration file. A `weaver.ini.example`_ file is provided
with default values to help in the configuration process. Explanations of respective settings are also available in
this example file.
The configuration file tell the application runner (e.g. `Gunicorn`_, pserve
or similar WSGI HTTP Server), how to
execute Weaver as well as all settings to provide in order to personalize the application. All settings specific to
Weaver employ the format weaver.<setting>
.
Most configuration parameters for the manager portion of Weaver (i.e.: WSGI HTTP server for API endpoints) are
defined in the [app:main]
section of `weaver.ini.example`_, while parameters specific to the worker (task queue
handler) are within [celery]
section. Note that multiple settings are shared between the two applications, such as
the mongodb.[...]
configuration or weaver.configuration
options. When parameters are shared, they are usually
expected to be placed in [app:main]
section.
Following is a partial list of most predominant settings specific to Weaver. Many parameters provide alternative or extended functionality when employed in conjunction with other settings. Others are sometimes not necessarily required to be defined if default behaviour is desired. Refer to the relevant details that will describe in which condition they are optional and which default value or operation is applied in each situation.
Note
Refer to `weaver.ini.example`_ for the extended list of applicable settings. Some advanced configuration settings are also described in other sections of this page.
weaver.configuration = ADES|EMS|HYBRID|DEFAULT
(default:DEFAULT
)Tells the application in which mode to run.EnablingADES
for instance will disable someEMS
-specific operations such as dispatching :ref:`Workflow` process steps to known remoteADES
servers.ADES
should be used to only run processes locally (as the working unit).EMS
will always dispatch execution of jobs to otherADES
except for :ref:`Workflow` processes that chains them.WhenHYBRID
is specified, Weaver will assume bothADES
andEMS
roles simultaneously, meaning it will be able to execute local processes by itself and monitor dispatched execution of registered remote providers.Finally,DEFAULT
configuration will provide very minimalistic operations as all other modes will be unavailable.weaver.url = <url>
(default:http://localhost:4001
)Defines the full URL (including HTTP protocol/scheme, hostname and optionally additional path suffix) that will be used as base URL for all other URL settings of Weaver.
Note
This is the URL that you want displayed in responses (e.g.: processDescriptionURL
or job location
).
For the effective URL employed by the WSGI HTTP server, refer to [server:main]
section of `weaver.ini.example`_.
weaver.schema_url = <url>
(default:${weaver.url}/json#/definitions
)Defines the base URL of schemas to be reported in responses.When not provided, the running Web Application instance OpenAPI JSON path will be employed to refer to the schemadefinitions
section. The configuration setting is available to override this endpoint by another static URL location where the corresponding schemas can be found if desired.
.. versionadded:: 4.0.0
weaver.wps = true|false
(default:true
)Enables the WPS-1/2 endpoint.
Warning
At the moment, this setting must be true
to allow job execution as the worker monitors this endpoint.
This could change with future developments (see issue #21).
weaver.wps_path = <url-path>
weaver.wps_url = <full-url>
(default: path/ows/wps
)Defines the URL to employ as WPS-1/2 endpoint.It can either be the explicit full URL to use or the path relative toweaver.url
.Settingweaver.wps_path
is ignored if its URL equivalent is defined.The path variant SHOULD start with/
for appropriate concatenation withweaver.url
, although this is not strictly enforced.weaver.wps_output_s3_bucket = <s3-bucket>
(default:None
)AWS S3 bucket where to store WPS outputs. Used in conjunction withweaver.wps_output_s3_region
.When this parameter is defined, any job result generated by a process execution will be stored (uploaded) to that location. If no bucket is specified, the outputs fall back to using the location specified byweaver.wps_output_dir
.
.. versionadded:: 1.13.0
.. seealso:: `Configuration of AWS S3 Buckets`_
weaver.wps_output_s3_region = <s3-region>
(default:None
)AWS S3 region to employ for storing WPS outputs. Used in conjunction withweaver.wps_output_s3_bucket
.When this parameter is defined as well asweaver.wps_output_s3_bucket
, it is employed to define which S3 to write output files to. If not defined butweaver.wps_output_s3_bucket
is specified, Weaver attempt to retrieve the region from the profile defined in AWS configuration files or environment variables.
.. versionadded:: 1.13.0
.. seealso:: `Configuration of AWS S3 Buckets`_
weaver.wps_output_dir = <directory-path>
(default: path/tmp
)Location where WPS outputs (results from :term:`Job`) will be stored for stage-out.Whenweaver.wps_output_s3_bucket
is specified, only :term:`WPS` :term:`XML` status and log files are stored under this path. Otherwise, :term:`Job` results are also located under this directory with a sub-directory named with the :term:`Job` ID.This directory should be mapped to Weaver's :term:`WPS` output URL to serve them externally as needed.
.. versionchanged:: 4.3.0 The output directory could be nested under a *contextual directory* if requested during :term:`Job` submission. See :ref:`exec_output_location` and below ``weaver.wps_output_context`` parameter for more details.
weaver.wps_output_context = <sub-directory-path>
(default:None
)Default sub-directory hierarchy location to nest :term:`WPS` outputs (:term:`Job` results) under.If defined, this parameter is used as substitute context whenX-WPS-Output-Context
header is omitted. When not defined,X-WPS-Output-Context
header can still take effect, but omitting it will store results directly underweaver.wps_output_dir
instead of default context location.
.. versionadded:: 4.3.0
.. seealso:: See :ref:`exec_output_location` for more details about this feature and implications of this setting.
weaver.wps_output_path = <url-path>
weaver.wps_output_url = <full-url>
(default: path/wpsoutputs
)Endpoint that will be employed as prefix to refer to :term:`WPS` outputs (:term:`Job` results).It can either be the explicit full URL to use or the path relative toweaver.url
.Settingweaver.wps_output_path
is ignored if its URL equivalent is defined.The path variant SHOULD start with/
for appropriate concatenation withweaver.url
, although this is not strictly enforced.
Note
The resulting weaver.wps_output_url
endpoint, whether directly provided or indirectly
resolved by weaver.url
and weaver.wps_output_path
will not be served by Weaver itself.
This location is returned for reference in API responses, but it is up to the infrastructure that
hosts Weaver service to make this location available online as deemed necessary.
weaver.wps_workdir = <directory-path>
(default: uses automatically generated temporary directory if none specified)Prefix where process job worker should execute the process from.weaver.wps_restapi = true|false
(default:true
)Enable the WPS-REST endpoint.
Warning
Weaver looses most, if not all, of its useful features without this, and there won't be much point in using it without REST endpoint, but it should technically be possible to run it as WPS-1/2 only if desired.
weaver.wps_restapi_path = <url-path>
weaver.wps_restapi_url = <full-url>
(default: path/
)Endpoint that will be employed as prefix to refer to WPS-REST requests(including but not limited to |ogc-proc-api|_ schemas).It can either be the explicit full URL to use or the path relative toweaver.url
.Settingweaver.wps_restapi_path
is ignored if its URL equivalent is defined.The path variant SHOULD start with/
for appropriate concatenation withweaver.url
, although this is not strictly enforced.weaver.wps_metadata_[...]
(multiple settings)Metadata fields that will be rendered by either or both the WPS-1/2 and WPS-REST endpoints (:ref:`GetCapabilities`).weaver.wps_email_[...]
(multiple settings)Defines configuration of email notification functionality on job completion.Encryption settings as well as custom email templates are available. Default email template defined in `email-template`_ is employed if none is provided. Email notifications are sent only on job completion if an email was provided in the :ref:`Execute` request body (see also: :ref:`Email Notification`).
Note
Since Weaver employs `Celery`_ as task queue manager and `MongoDB`_ as backend, relevant settings for the configuration of Celery and the configuration of MongoDB Backend should be referred to. Processing of task jobs and results reporting is accomplished according to the specific implementation of these services. Therefore, all applicable settings and extensions should be available for custom server configuration and scaling as needed.
Any AWS S3 bucket accessed by Weaver needs to be accessible by the application, whether it is to fetch input files or to store output results. This can require from the server administrator to specify credentials by one of reference |aws-cred-support|_ to provide necessary role and/or permissions. See also reference |aws-config|_ which list various options that will be considered when working with S3 buckets.
Note that Weaver expects the |aws-config|_ to define a default profile from which the AWS
client can infer which region it needs to connect to. The S3 bucket to store files should be defined by
weaver.wps_output_s3_bucket
setting as presented in the previous section.
The S3 file references for input and output in Weaver are expected to be formatted as:
s3://<bucket>/<file-key>
This implicitly tells Weaver to employ the S3 bucket it was configured with as well as the automatically retrieved region from the AWS server configuration.
Alternatively, the reference can be provided with the more explicit AWS S3 link such as:
https://s3.[region-name.]amazonaws.com/<bucket>/<file-key>
In this situation, Weaver will parse it as equivalent to the prior shorthand reference format, as long as the AWS server configuration matches with all associated details from the HTTP URL variant. If this is not the case, Weaver will still attempt to fetch the file as standard HTTP reference, but read access should be granted accordingly to the corresponding bucket and file such that Weaver can access it.
Finally, in the above references, file-key
is used as anything after the bucket name. In other words, this value
can contain any amount of /
separators and details. For example, Weaver will store process output results to S3
using file-key
as a combination of <WPS-UUID>/<output-id>.<ext>
, therefore forming the full job result file
references as:
https://s3.<region-name>.amazonaws.com/<bucket>/<WPS-UUID>/<output-id>.<ext>
Note
Value of WPS-UUID
can be retrieved from Weaver internal job storage from :meth:`weaver.datatypes.Job.wps_id`.
It refers to the process execution identifier that accomplished the WPS request to run the Application Package.
A typical :term:`Data Source` file is presented below. This sample is also provided in `data_sources.yml.example`_.
.. literalinclude:: ../../config/data_sources.yml.example :language: yaml
Both JSON
and YAML
are equivalent and supported. The data_sources.yml
file is generated by default in the
configuration folder based on the default example (if missing).
Custom configurations can be placed in the expected location or can also be provide with an alternative path
using the Weaver.data_sources
configuration setting.
Note
As presented in the above example, the :term:`Data Source` file can also refer to :ref:`opensearch_data_source` which imply additional pre-processing steps.
.. seealso:: More details about the implication of :term:`Data Source` are provided in :ref:`data-source`.
Weaver allows the configuration of services or processes auto-deployment using definitions from a file formatted
as `wps_processes.yml.example`_. On application startup, provided references in processes
list will be employed
to attempt deployment of corresponding processes locally. Given that the resources can be correctly resolved, they
will immediately be available from Weaver's API without further request needed.
For convenience, every reference URL in the configuration file can either refer to explicit process definition (i.e.: endpoint and query parameters that resolve to :ref:`DescribeProcess` response), or a group of processes under a common WPS server to iteratively register, using a :ref:`GetCapabilities` WPS endpoint. Please refer to `wps_processes.yml.example`_ for explicit format, keywords supported, and their resulting behaviour.
Note
Processes defined under processes
section registered into Weaver will correspond to a local snapshot of
the remote resource at that point in time, and will not update if the reference changes. On the other hand, their
listing and description offering will not require the remote service to be available at all time until execution.
.. versionadded:: 1.14.0 When references are specified using ``providers`` section instead of ``processes``, the registration only saves the remote WPS provider endpoint to dynamically populate WPS processes on demand. Using this registration method, the processes will always reflect the latest modification from the remote WPS provider.
To specify a custom YAML file, you can define the setting named weaver.wps_processes_file
with the appropriate path
within the employed weaver.ini
file that starts your application. By default, this setting will look for the
provided path as absolute location, then will attempt to resolve relative path (corresponding to where the application
is started from), and will also look within the |weaver-config|_ directory. If none of the files can be found, the
operation is skipped.
To ensure that this feature is disabled and to avoid any unexpected auto-deployment provided by this functionality,
simply set setting weaver.wps_processes_file
as undefined (i.e.: nothing after =
in weaver.ini
).
.. seealso:: - `weaver.ini.example`_ - `wps_processes.yml.example`_
.. todo:: complete docs
weaver.ssl_verify
.. versionadded:: 1.8.0
`request_options.yml.example`_
.. todo:: complete docs
make start
(or similar command)
- need to start
gunicorn/pserve
(example `Dockerfile-manager`_) - need to start
celery
worker (example `Dockerfile-worker`_)