COMPONENT METADATA: translation

[bkueng]

@hamishwillee this adds the translation support as discussed in #1656.

• Uses a summary json file with urls and modification timestamps to individual translated (and compressed) .ts files.
  I setup an example repo under https://github.com/bkueng/px4_metadata_translations. It explains the work-flow, CI integration is not done yet. @hamishwillee do you want to look into that? In particular for the CrowdIn integration.
• It's generic so new metadata types do not require GCS changes.
• Versioning: there's different approaches, probably the simplest is to use branches in the metadata repo.
• QGC implementation: Metadata translation support qgroundcontrol#10522

TODO (outside of mavlink):

• CI integrations & crowdin sync
• Documentation

[hamishwillee]

Does this imply we need to iterate the "version" property number?

[bkueng]

We could, I don't remember when it should be increased.

[hamishwillee]

It should be increased for sure if there is a breaking change - i.e. a removal. I'm not sure otherwise.

Probably doesn't need it unless we have a feature where we want users to be able to know whether something is not present vs not supported. In this case I don't think it matters or is needed but we might want to get into habit of just updating on any change - because it is hard to retrofit the ability to spot a change after this is out in the wild?

[bkueng]

Yes that's fine with me.

[hamishwillee]

Where does this URL come from? I ask because it is a 404

[bkueng]

It matches the file name. We don't make use of it though. In general it can be used to point to the file (e.g. online url).

[hamishwillee]

@bkueng This is my last day for a bit, so if there is urgency on this we need some more people involved. The stuff here all looks good - but I'm not a JSON or schema expert, and I've almost forgotten what was agreed.

So to kick this off, let me summarise what I remember, and also what am seeing. Then you can tell me if my understanding is correct. So using the example of parameters.metadata.json

1. The general.metadata.json file on vehicle references the parameter metadata json and has a CRC for that file (it is invariant). Parameter.metadata.json might be on the vehicle (mavftp) or on a server on the internet. On PX4 it is usually on the vehicle.
2. Parameter.metadata.json has a URL to translation schema summary file.
   • There is no associated CRC for translation summary because it (and other translations) can change
   • Translation file is on a server accessible by HTTPS (I think we said no to MAVFTP for the translations right?)
   • GCS uses link to get summary file - when does it check it? I think we said that it should check on every session/connection to a vehicle right?
3. Summary translation file lists the translations available, their URLs and their last modified date. These are .ts files that may or may not be .xz compressed.
4. GCS gets this, then updates its translations if any values have changed.

That right?

Re the questions for me:

CI integration is not done yet. @hamishwillee do you want to look into that? In particular for the CrowdIn integration.
It's generic so new metadata types do not require GCS changes.

What repo will all this live in?

Setting up a crowdin github integration within a single repo is pretty easy - essentially you specify the source folder and a name translation format for the output files. I SHOULD be able to apply this exactly to your current repo structure.
Crowdin then normally polls every hour or so for changes to the source. It pushes changes as PRs - I think we can probably make it auto accept them too, but would need to check that.

Then you could have a git action to trigger rebuilding the summary based on the translation files changing or a PR being submitted.

Versioning: there's different approaches, probably the simplest is to use branches in the metadata repo.

You can certainly do that; I guess it doesn't matter to the "system" as long as the summary contains the right information for finding any changed files for the current vehicle.

You can maintain multiple translation branches in crowdin and continue to update them separately. I think that we should just maintain the main branch in crowding. So we'd update the main branch progressively and branch our repo for new releases. If we want then we can choose to manually take translations into older branches via git. Make sense?

[bkueng]

Thanks for the review! Didn't want to add more work to your plate before holidays.

@bkueng This is my last day for a bit, so if there is urgency on this we need some more people involved.

Not at all, this has time. I was just wrapping up some longer-time outstanding things.

Parameter.json has a URL to translation schema summary file.

The URL is in component general. Otherwise you're right.

when does it check it? I think we said that it should check on every session/connection to a vehicle right?

[EDIT, forgot to answer this one]: I implemented a 3 day caching policy in the GCS to avoid unnecessary frequent requests.

What repo will all this live in?

The repo I shared above, but moved under PX4/ when ready.

Then you could have a git action to trigger rebuilding the summary based on the translation files changing or a PR being submitted.

Yes that's the idea

Make sense?

Yes. We probably need to see how complete translations will be.

[hamishwillee]

Thanks!

What repo will all this live in?

The repo I shared above, but moved under PX4/ when ready.

There is no point doing the crowdin integration until this is in the right location (or it will all have to be done again). Happy to do this when you move it.

Otherwise, I hope to have mav call tonight (then away two weeks) and see if there are any further suggestions.

[hamishwillee]

@bkueng There was no particular feedback on this - it appears to match what was discussed/agreed so this can go in (20230104-Dev-Meeting)

Separately there was a discussion on actuator metadata definitions and whether they are truly autopilot agnostic - at least for ArduPilot and PX4 (clearly if actuator definitions are not mapped using parameters then this won't work). My recollection is that we thought that they probably were based on the fact that both systems "just write parameters", though it is hard to be sure without an implementation. Upshot, there might need to be further iteration required on that file.