Triton: run IOC on Triton control PC: implementation

[Tom-Willemsen]

As a member of the cryogenics team, I would like the Triton IOC to run on the Triton control PC rather than the instrument PC so that I can run diagnostics easily, without depending on which beamline the fridge is on.

Acceptance criteria

• As a user, I can control the Triton system when the IOC is running on the Triton control PC
• As a user, I can see any alarms that my IOC is generating in the alarms view
• As a user, I can create blocks pointing at the Triton IOC PVs
• As a member of the cryogenics team, I can look at archived data for Triton PVs (regardless of which beamline the fridge was on at the time)
• As a member of the cryogenics team, I can view and control PVs from the triton system
• As a member of the cryogenics team, I can configure which beamline the fridge is currently on
• As a developer, I can see IOC logs from the Triton system
• As a developer, I can start and stop the Triton IOC.

Notes

1. The purpose of this ticket is to implement the solution described in the comments of Triton: run the IOC on the triton PCs rather than the instruments #3091 for running the Triton IOC on the Triton's control PC

[DominicOram]

Rough picture of the proposed architecture: (should be covered in linked ticket but add a better drawn picture to the wiki as part of ticket)

Criteria of each group

[FreddieAkeroyd]

I guess configuring the IOC will be via globals.txt ?

[Tom-Willemsen]

Setting the instrument will be done via a PV, see comments of #3091

For the triton there are no other configuration options to worry about (it will always connect to the triton on localhost with a fixed port number)

[KathrynBaker]

This solution is also of interest to the Cryogenics group (as part of a small project) so that they can monitor the systems centrally (not part of the HLM work).

[Tom-Willemsen]

Re-requested by Cryogenics group, so re-proposing.

They say that better remote diagnostics would help them to prevent somewhere around ~1 problem/cycle (where a "problem" has a lost beamtime impact of multiple hours and sometimes days).

Things to do in this ticket:

• Implement service that serves some PVs that we can use to control the "remote ioc"
• Be able to start and stop the IOC via this service we've decided this doesn't really make sense to do in general - the IOC will restart when the configuration changes as normal though.
• Implement gateway which aliases the fixed names to a particular instrument
• Provide some mechanism(s) for configuring the IOC (macros etc)
• Implement access security on the gateway(s) so that the given instrument + cryogenics + us can write but no-one else
• Log values into a local/central archive
• Log values into the instrument archive (maybe not needed?)
  • We have decided not to do this (in this ticket).
• Push alarms to instrument db
• Push ioc status to instrument db
• Push interesting pvs to instrument db

[Tom-Willemsen]

After discussion with @John-Holt-Tessella regarding MySQL (instrument) archives specifically:

• We will run the central archive with a static logging config that always points at all of the remote IOCs
  • This will be manually updated when we add a new type of "remote" ioc. This is because we can't just use the standard archive config tool so when we want to do this automatically it will need a new tool writing. For this ticket (which just relates to Tritons) a static logging config + documenting how to add extra iocs in future will be sufficient
  • The PVs will be archived as ME:TRITON1:..., not IN:LET:...
• We won't run an instrument archive on the remote PCs
  • This might turn out to be required in future (there may be a need to access archived values from when the fridge was not on the network) but does not conflict with the archive above so could be added in a later ticket.
  • We don't yet have an adequate management strategy for how we would manage resources on a computer we can't see (if it's not on the network). This is a much wider discussion point, and should get it's own ticket if we decide/need to do it.
• We will not push a logging config from the remote IOC to the instrument PC.
  • If scientists want it logged, the answer has always been that they should put a block on it
  • If we decide to do this in future, we would need to write an extra tool that merges the logging config of the remote IOC and the rest of the instrument. The tool would also need to know how to translate ME:TRITON:... into the IN:LET:... aliases in it's logging config.

[FreddieAkeroyd]

How inportant is archiving the PVS? The central archive server is not critical for instrument operation and is also not on a system that is under our direct control.

[ChrisM-S]

If the central archive server is not related to the ISIS data archive (where all cite-able data from ISIS is archived) and is not something under our direct control in IBEX where/what on earth is it ?

[DominicOram]

@Tom-Willemsen do we need to update the _base component in the master branch too?

[Tom-Willemsen]

I think last time this was discussed we said that the _base component stays on the older version (with an appropriate config_version.txt) and then the upgrade script takes care of any changes from that version? If I remember rightly this was to try to reduce merge conflicts between master and instruments.

I'm happy to change it if I've misremembered though, let me know. Either way, we should document the decision on the wiki...

[John-Holt-Tessella]

I don't remember, either we should have written it down my bad. Do we run the upgrade script over a new config set by default? If not we should add it to base because other wise new configs won't get that change.

[DominicOram]

Is master regularly merged into instrument configs? It seems rather counter intuitive to me that master is actually incorrect and you need to run an additional script to correct it.

[Tom-Willemsen]

Yes, I think the deploy script automatically merges master into the instrument configs every time we deploy (see https://github.com/ISISComputingGroup/ibex_utils/blob/master/installation_and_upgrade/ibex_install_utils/install_tasks.py#L519 ). That might not be the best solution though.