Repository navigation
Admin page: export form definitions and data with attachments #779
Description
Activity
Also exporting data with dependencies should also be an option.
Also requested: ability to export selected subset of data only. See customer thread.
The requirement also calls for:
- doing this manually via a Form Runner UI
- doing this via an API
+1 from customer as an option to migrate between providers (although another, explicit way of migrating between providers might be better).
First step: write "archive export API". Parameters:
-
date range
- all
last-modified-time-gt/last-modified-time-gelast-modified-time-lt/last-modified-time-le(optional)
-
data vs. form vs. both (vs. attachments?)
content=form-definition|form-data|attachments?
-
app/form/form-version
- app: part of the path, optional
- form: part of the path, optional
form-version: URL parameter or header?- Search API uses
Orbeon-Form-Definition-Versionheader for latest/specific/all- but that might not have been the best choice vs. a URL parameter
- Search API uses
document-id: part of the path, optional (requires app/form)- scenario: export a single instance of form data with its attachments
-
P2: "dry run" to just return the count(s) of operations
New entry point:
/fr/service/persistence/export/$app/$form/$document-id?form-version=$form-version$app/$form/$document-id/form-versionare all optional, but in constrained ways:- no parameter: export everything
- we can imagine allowing
form-version=latest|all, butspecificdoesn't make much sense here
- we can imagine allowing
$apponly: export everything under the app- we can imagine allowing
form-version=latest|all, butspecificdoesn't make much sense here
- we can imagine allowing
$appand$formform-version=latest|specific|all
$appand$formanddocument-idform-version: can be omitted or specific, but if so must match
-
We are implementing this mostly at the level of the persistence proxy, not directly in the providers.
- a request that needs to search for form definitions and their versions, use the Form metadata API
- a request that needs to search for data uses the Search API
GETting data and attachments use the CRUD API
32 remaining items
Permissions: I think we had decided that this was a admin function, which would export all data regardless of permissions.
- check whether this is indeed the current behavior
- document that aspect
To check the permissions behavior:
- as user
clerk, enter data that only user can read, with no permissions for useradmin - as user
admin, export form and data
- as user
Conclusion of the above:
- as
adminI am getting a zip with the form definition, but no data - as
clerk(which also can access the Admin page), I am not getting any download (and no error), but there is a 403 in the logs
- as
The 403 comes when:
- calling
/fr/service/mysql/history/test/export-permissions/2cadd6415e082486f6939a65c4aeaa4ea527783f?page-size=100&page-number=1 findAuthorizedOperationsOrThrow()doesn't find any operations
However, this is wrong, because this takes the same code path as if for a
GETof individual data. But this calls the/historyAPI, and ends up with the "PUTand no data, so this is a 1" branch, andauthorizedOperationsForNoData(), which returns no permissions.So even if we wanted to honor permissions, this would be wrong.
But does this point to a bug with the
/historyAPI?- check and enter
/historyAPI bug if needed
However, we do not want permissions to be honored, as per the decision above.
- calling
If we want the current permissions not to be honored, we need some new sort of setting to override the default behavior. This is not something we have right now. This would be the same as passing
Permissions.Undefined, which would produceAnyOperation. That, or passingPermissions.Defined()grantingReadtoAnyonewould work.However, of course, we'd need to be very careful and to enable this by passing an encrypted parameter for export, which would only be passed in this case.
An external service call API would at this point not support this, and would honor permissions (but see possible
/historybut above).In addition, the Search API also needs to support all operations so that all the data is returned.
So the idea is:
- Admin page calls export/purge API with the encrypted admin param
- export/purge API reads and then passes/forwards that parameter to the relevant persistence APIs
- Search
- CRUD
- History (which currently ignores it?)
Instead of an encrypted param, now using a token with expiration.
Closing as all major items are done or new RFEs have been entered.
The idea of using a token is that we have some trusted source, in this case the Admin page, calling a service, in this case
/fr/service/persistence/export. We'd like that source to tell the service that it has some special permissions, in this case "I can list and read everything" (and, in the case of/fr/service/persistence/purge, "I can list and read and delete everything"). This information has to be forwarded to the Search, CRUD, and History APIs down the line.My first idea was to say that we just add an internal URL parameter that says "I have these rights". But I didn't want that parameter to be usable by everyone, at least not until we figure out all the security implications. It is true that, for service calls, we currently allow any authorized service caller to then impersonate any user. However, after that, permissions still kick in. A regular, non-enrypted parameter could be ok, or not. Also, services can also be called internally, for example by a form service. Someone could, by design or mistakenly, design a form that calls the CRUD API with that parameter, and gain access to other users' data, bypassing permissions.
So then I figured I'd just use an encrypted parameter, as we do for mode changes. However, in case of an error, the internal URL could be output to the logs, and someone obtaining the log could then know what encrypted parameter value to pass to gain more access, forever (or until the encryption password is changed).
So instead, I used a token like we use for the access token, which has the benefit of having an expiration time. Even if someone has access to the token through logs or otherwise, the token will expire. In this case, it expires in a matter of seconds, enough for an internal request to go through, but hopefully not enough to give time to someone to gain access if by any chance a token value was obtained. At last, this covers the case of logs being shared.
I cannot use the exact logic used by the access token, because that requires forms that enable token access. So currently, the token says "I am an admin and I can do anything with the data". We could change this to be more like the access token and list individual operations. This would allow
/exportto only requestreadandlistpermissions, for example.One last question is the creation of the token. If someone could have access to the function that creates the token, that could be a problem. This function is internal, but we should harden it. One idea would be, for example, to whitelist pages and services where the function can be successful. Specifically:
/fr/admin/fr/service/persistence/export/fr/service/persistence/purge
Tasks:
- whitelist allowed token creators
Metadata
Metadata
Projects
- Status
Feature
The form definition, its dependencies (images, attachments), and/or form data and dependencies, are exported into a zip archive, which can be imported back into a Form Builder/Form Runner installation.
Use cases
Migrating from one persistence implementation to another (e.g. eXist to relational)Location of functionality
Steps
Service
It might be good to implement most of the functionality as a service first. This would also enable a scripted mode which would not require the Orbeon Forms user interface.
NOTE: This however would still require a running Form Builder instance.