Today OTP reads all its input from the local file system and writes the graph to the same local disk. This is ok in a static deployment, but in a cloud deployment it creates some overhead of moving files from permanent storage into a cluster node and back when the process (OTP) is done. When the OTP process fails or freeze it is difficult to detect, and make the system less robust.
In a continuous automated devops ecosystem we would like OTP to directly integrate with the rest of the system, not having to wrap OTP and copy files around. We want to change the in/out-put files to read and write directly to the cloud storage and to track the progress. This is most relevant when building a graph, but a solution should, if possible, not be limited to that.
At Entur we need to change the current way we do this, so we will implement support for pluggable file access, so the default (current) way this work can be switched to accessing Google Cloud storage.
We post this issue here to let people know we do this in our private fork, and if there is an interest for this, we can make a PR to integrate this into OTP2.
We are not going to introduce dependencies to any Google Cloud specific libraries, just provide a pluggable extension point in OTP to swap in an alternative implementation. We will provide links to our GCS implementation in the Entur GitHub repo, if someone want to copy/use it.
Today OTP reads all its input from the local file system and writes the graph to the same local disk. This is ok in a static deployment, but in a cloud deployment it creates some overhead of moving files from permanent storage into a cluster node and back when the process (OTP) is done. When the OTP process fails or freeze it is difficult to detect, and make the system less robust.
In a continuous automated devops ecosystem we would like OTP to directly integrate with the rest of the system, not having to wrap OTP and copy files around. We want to change the in/out-put files to read and write directly to the cloud storage and to track the progress. This is most relevant when building a graph, but a solution should, if possible, not be limited to that.
At Entur we need to change the current way we do this, so we will implement support for pluggable file access, so the default (current) way this work can be switched to accessing Google Cloud storage.
We post this issue here to let people know we do this in our private fork, and if there is an interest for this, we can make a PR to integrate this into OTP2.
We are not going to introduce dependencies to any Google Cloud specific libraries, just provide a pluggable extension point in OTP to swap in an alternative implementation. We will provide links to our GCS implementation in the Entur GitHub repo, if someone want to copy/use it.