New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Option to store data in database using newer format #3110

Closed
ebruchez opened this Issue Feb 8, 2017 · 7 comments

Comments

Projects
1 participant
@ebruchez
Collaborator

ebruchez commented Feb 8, 2017

Following #635 in 4.8, we had picked "Option C": for backward compatibility with existing data, we keep the 4.0 data format in the database.

We now have a request to support storing the 4.8 format to the database. If we are to do this, what is needed?

  • an option (property?) to determine the default database data format
  • store the data format alongside the data, e.g. <form fr:data-format-version="4.8">
  • migration must depend on property for writing, and data version for reading
  • search
    • relational: we now index data upon writing
      • what about reindexing? need to check if changes needed
    • eXist: check whether expressions need adjusting (probably)
  • later: ability to migrate data in database

+1 from customer

@ebruchez ebruchez added the Form Runner label Feb 8, 2017

@ebruchez ebruchez self-assigned this Feb 8, 2017

@ebruchez

This comment has been minimized.

Show comment
Hide comment
@ebruchez

ebruchez Feb 8, 2017

Collaborator

It would be good to do #1947, or we will have 3 database formats once we do it. But maybe the cat is out of the bag as the 4.8 format is visible with the send action.

Collaborator

ebruchez commented Feb 8, 2017

It would be good to do #1947, or we will have 3 database formats once we do it. But maybe the cat is out of the bag as the 4.8 format is visible with the send action.

@ebruchez

This comment has been minimized.

Show comment
Hide comment
@ebruchez

ebruchez Mar 9, 2017

Collaborator

Currently:

  • FormRunnerActions.trySaveAttachmentsAndData() has access to the form definitions's metadata, and converts the data to 4.0.0 format if needed (that is, if there is any migration information).
  • FormRunnerPersistenceProxy receives and transmits 4.0.0 only.

Now if we set the database format to 4.8.0, and trySaveAttachmentsAndData() converts down, then FormRunnerPersistenceProxy has to convert back up, which is not optimal.

We could do this:

  • FormRunnerActions.trySaveAttachmentsAndData()
    • reads the property for the given provider, and if it matches the current data format, doesn't convert it.
    • passes data-format-version="4.8.0|4.0.0" as URL parameter (specific, not edge)
  • FormRunnerPersistenceProxy
    • reads data-format-version (default will be 4.0.0 for backward compatibility)
    • if it matches the provider's data format, doesn't convert
    • if it doesn't match, currently returns an error, but in the future should convert
      • RelationalUtils.readFormMetadata can retrieve the form definition's metadata, although right now there is no caching of the underlying form definition
Collaborator

ebruchez commented Mar 9, 2017

Currently:

  • FormRunnerActions.trySaveAttachmentsAndData() has access to the form definitions's metadata, and converts the data to 4.0.0 format if needed (that is, if there is any migration information).
  • FormRunnerPersistenceProxy receives and transmits 4.0.0 only.

Now if we set the database format to 4.8.0, and trySaveAttachmentsAndData() converts down, then FormRunnerPersistenceProxy has to convert back up, which is not optimal.

We could do this:

  • FormRunnerActions.trySaveAttachmentsAndData()
    • reads the property for the given provider, and if it matches the current data format, doesn't convert it.
    • passes data-format-version="4.8.0|4.0.0" as URL parameter (specific, not edge)
  • FormRunnerPersistenceProxy
    • reads data-format-version (default will be 4.0.0 for backward compatibility)
    • if it matches the provider's data format, doesn't convert
    • if it doesn't match, currently returns an error, but in the future should convert
      • RelationalUtils.readFormMetadata can retrieve the form definition's metadata, although right now there is no caching of the underlying form definition
@ebruchez

This comment has been minimized.

Show comment
Hide comment
@ebruchez

ebruchez Mar 9, 2017

Collaborator

If, in the immediate future, FormRunnerPersistenceProxy doesn't do any conversions:

  • Form Runner reads data passing data-format-version=4.0.0|4.8.0
    • converts based on format needed and the provider format
  • Form Runner writes data passing data-format-version=4.8.0|4.0.0 (one or the other)
    • converts based on provider's desired format

Also:

  • check what to do for attachments
  • Currently data conversion when writing is done in FormRunnerActions.trySaveAttachmentsAndData()
  • add version number attribute on data when writing
    • Q: Who does this? It would be good if the proxy could check/add the version, but then it would need to parse the data. A: For now, not the proxy, but the migration code.
  • GET: check version when reading
    • Q: Do we need to parse the XML to check the version attribute? A: Not for now.
  • search
    • currently, Index.reindex() calls findIndexedControls to get paths
    • paths can depend on whether migration information is found
    • here too, some adjustment is needed depending on which format is in the database
  • eXist
    • Q: What about search? A: produce paths consistent with provider data format version.
  • FormRunnerSummary.duplicate()
    • should use raw both ways?
    • or maybe read as raw, convert to database format if possible, then write as desired format
    • since we do not support more than one format per provider, read like persistence-model.xml by passing the provider's data format version, and write using the same version
  • /attachments service
    • read like persistence-model.xml as well
Collaborator

ebruchez commented Mar 9, 2017

If, in the immediate future, FormRunnerPersistenceProxy doesn't do any conversions:

  • Form Runner reads data passing data-format-version=4.0.0|4.8.0
    • converts based on format needed and the provider format
  • Form Runner writes data passing data-format-version=4.8.0|4.0.0 (one or the other)
    • converts based on provider's desired format

Also:

  • check what to do for attachments
  • Currently data conversion when writing is done in FormRunnerActions.trySaveAttachmentsAndData()
  • add version number attribute on data when writing
    • Q: Who does this? It would be good if the proxy could check/add the version, but then it would need to parse the data. A: For now, not the proxy, but the migration code.
  • GET: check version when reading
    • Q: Do we need to parse the XML to check the version attribute? A: Not for now.
  • search
    • currently, Index.reindex() calls findIndexedControls to get paths
    • paths can depend on whether migration information is found
    • here too, some adjustment is needed depending on which format is in the database
  • eXist
    • Q: What about search? A: produce paths consistent with provider data format version.
  • FormRunnerSummary.duplicate()
    • should use raw both ways?
    • or maybe read as raw, convert to database format if possible, then write as desired format
    • since we do not support more than one format per provider, read like persistence-model.xml by passing the provider's data format version, and write using the same version
  • /attachments service
    • read like persistence-model.xml as well
@ebruchez

This comment has been minimized.

Show comment
Hide comment
@ebruchez

ebruchez Mar 10, 2017

Collaborator

The assumption that the database could contain more than one format causes problems:

  • extracting the version upon GET
  • implementing search, which won't work with two different formats unless we standardize on canonical paths in the index no matter what the data format is for relational
  • eXist doesn't have a separate index, so this won't work there

So I suggest that for now, we put as a requirement that for a given provider, all data must be in a single format.

Collaborator

ebruchez commented Mar 10, 2017

The assumption that the database could contain more than one format causes problems:

  • extracting the version upon GET
  • implementing search, which won't work with two different formats unless we standardize on canonical paths in the index no matter what the data format is for relational
  • eXist doesn't have a separate index, so this won't work there

So I suggest that for now, we put as a requirement that for a given provider, all data must be in a single format.

@ebruchez

This comment has been minimized.

Show comment
Hide comment
@ebruchez

ebruchez Mar 10, 2017

Collaborator
  • test
    • manual test with relational
  • doc
    • <property as="xs:string" name="oxf.fr.persistence.*.data-format-version" value="4.0.0|4.8.0"/>
      • cannot mix and match data formats in same provider
    • fr-get-instance-from-service: currently still alway assumes as external format 4.0.0
    • POST instance to page: currently migrates unless edge
Collaborator

ebruchez commented Mar 10, 2017

  • test
    • manual test with relational
  • doc
    • <property as="xs:string" name="oxf.fr.persistence.*.data-format-version" value="4.0.0|4.8.0"/>
      • cannot mix and match data formats in same provider
    • fr-get-instance-from-service: currently still alway assumes as external format 4.0.0
    • POST instance to page: currently migrates unless edge
@ebruchez

This comment has been minimized.

Show comment
Hide comment
@ebruchez

ebruchez Mar 10, 2017

Collaborator

For attachments: if data-format-version is provided, proxy checks validity, otherwise ignores. No conversion needed.

Collaborator

ebruchez commented Mar 10, 2017

For attachments: if data-format-version is provided, proxy checks validity, otherwise ignores. No conversion needed.

ebruchez added a commit that referenced this issue Mar 13, 2017

Implement #3110 "Option to store data in database using newer format"
- assumption
  - a given provider contains data only in one format
- GET data
  - pass `data-format-version` param
  - migrate response data depending on database format
- PUT data
  - pass `data-format-version` param
- persistence proxy
  - checks `data-format-version` param is compatible with provider
- duplicate, attachments services
  - pass `data-format-version` param
- search
  - generate paths depending on database format

@ebruchez ebruchez added this to the 2017.1 milestone Mar 14, 2017

@ebruchez ebruchez added this to Needs Doc in Orbeon Forms 2017.1 Apr 27, 2017

@ebruchez ebruchez closed this May 12, 2017

@ebruchez ebruchez moved this from Missing Doc to Done in Orbeon Forms 2017.1 May 12, 2017

@ebruchez ebruchez removed the Missing Doc label May 25, 2017

@ebruchez

This comment has been minimized.

Show comment
Hide comment
Collaborator

ebruchez commented May 25, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment