BVE5 Routefile Parser

[leezer3]

Tracking issue, split from discussion at the end of #81

WIP branch is here: https://github.com/leezer3/OpenBVE/tree/BVE5

What I've got at the minute is very much based upon Michelle's original parser, just adapted to the BVE5 syntax, although I'm also using it as a testbed for the cleanup I'm intending to do in the original parser.
In essence, each routefile command is split into it's own function, which are contained in the file openBVE/OpenBve/Parsers/Bve5ScenarioParser.Commands.cs

For testing, I've primarily been using a copy of Aln Valley (Simple single-track RouteBuilder route).
This is what's currently working:

• Basic route parser
• Legacy curves
• Signalling
• Stations

Partially working:

• Repeaters (25m block length only)
• Structure placements

Not implemented, planned:

• Beacons
• Secondary rails
• Gradient
• Train formats (Shouldn't be too hard, they appear to be BVE4 trains split up into multiple files)

[marcriera]

I will take a look at this. I have been reading the BVE5 file format documentation from Mackoy's site and know how many of the commands work. At first sight, some of them look like 1:1 equivalents to BVE4 commands, and I think I could manage to add support for them to the parser. Others (specially the track-related ones) have a completely different behaviour, and will need more work.

As soon as I get stable results, I will make sure to pull request changes to this new branch.

[marcriera]

Some progress I have made:

• BVE5 scenario detection in route file browser (.txt files which are not BVE5 routes are not shown)
• Route picture support in main menu.
• Improved file handling ('//' comments were not detected and blank lines resulted in an exception)
• Less strict file version check (1.00 scenario files exist too, for example)
• More reliable encoding checker

I cannot pull request at the moment (I only have access to the GitHub website and I have not cloned the BVE5 branch to my fork yet because of the rail cycle pull request), but I wanted to tell you about this so we do not end up wasting effort working on the same. I will keep working on the main menu for the moment.

[leezer3]

OK, so I've just added basic handling of secondary tracks, and this brings up a (minor) issue.

BVE5 accepts commands in any order within a block declaration. For just about anything this doesn't matter, but unfortunately it allows things to be attached to non-existant tracks.
Two ways I see of handling this:

1. Twiddle things so that the block is sorted before any of the commands are processed.
2. Add an 'exists' bool to each rail, and just expand our arrays appropriately. In the postprocessing stage, just discard anything that isn't attached to an existing rail.

Solution 2 appears a little easier on paper, and I suspect that it'll handle multiple files better.
What I'm a little concerned about though is whether there are any actual 'rules' defining command order. I can't see any from a brief skim of the documentation in English or Japanese, but this is the sort of thing that could easily trip stuff up later down the line.

[marcriera]

After some testing in BVE5, I think I have a better idea of secondary rails and how they work. Basically, as there is no command to start (or end) a rail, the first command making a reference to a secondary rail serves as the start point. Weird things such as placing a Repeater on a non-existing rail are possible (in that case the position matches that of the main rail, as everything is zero). It would be good if the documentation explained this, but it is mentioned nowhere.

So yes, I guess solution 2 is the best choice we have. When the parser finds a reference to a secondary rail, it should check a list with all the rail keys, and if it does not exist, simply start it with zero coordinates. Sorting blocks should then become unnecessary.

[leezer3]

Things are working much better now (Famous last words!)

This shot shows Barnt Green from Birmingham X-City South converted to BVE5 format.
There's still some sort of bug going on when ending a repeater which I'm looking into, but generally speaking converted routes look reasonably OK.

[leezer3]

And fixed....

I'd forgotten to increase the track position offset when cloning a repeater into a non-declared block, hence the object was actually appearing in the previous block.

A converted BVE4 route will now actually work and look 99% as intended.
There are still some kinks to be worked out in terms of exact object orientations, but there we go.

[leezer3]

Getting a lot closer still!

(This is the first station on the default Kesei route supplied with BVE5. Getting this working and looking correct is the primary target)

Basic handling of repeating objects with differing block lengths is now in, just need to sort out the rotations properly.
(I'm hoping that using a 1m or 50cm fixed block length may well work without trying to do anything silly fancy)

Still missing:

• Backgrounds
• Interpolation of height/ curves (Related to the block issues above)
• Route sounds
• Beacons
• Pretrains
• 'New' gradient format
• Massive code cleanup.....

[marcriera]

Amazing progress! The Keisei route is a very good start, I think it uses all (or nearly all) the commands possible in BVE5. If it works, the other routes will mostly work too.

I was looking at backgrounds yesterday and it will require quite a lot of changes outside of the BVE5 parser. Backgrounds are handled by a different class, and currently it creates a 3D model on runtime and attaches the background texture to it. This was made with legacy routes in mind and works well. However, BVE5 backgrounds are .x objects loaded from the route folder, so the code must be rewritten to support those too.

The idea of 1m or 0.5m blocks has been in my mind too for some days, but I am not sure if it really is the best solution, as short blocks lead to longer loading times and probably memory issues too. The best alternative I could come up with is event arrays. This would mean ditching the block system. Basically, we would have different arrays for each type of route feature that would store all the information from the command and the track position. After parsing the commands, we would loop through the arrays and place the events and objects on the track. We would know the span of curves and transition curves in advance, for example, so interpolation would probably become easier.

Repeaters should not give problems either (other than rotation, which is tricky) if we place each 'state' of the cycle as a FreeObj. Again, a loop would place the objects in the right place until the distance specified at the Repeater.End() command, or until the end of the route. This is essentially the same to what the 'CreateMissingBlocks' function does now.

This may be 'silly fancy' at first sight (it requires an important code rework), but the results are worth the effort.

[leezer3]

Backgrounds shouldn't be too difficult-
All the renderer code is already written and ready to go.
All it really requires is the adding of a new object array for the backgrounds, and rendering them absolutely first.

Supporting the use of an object rather than just a texture for the background is actually a really good idea, and I think this is probably an improvement that the original CSV route format would benefit from.
Very little needing to be changed either, just add a hook object parsing code if the selected background is an object not a texture.

Your freeobjects idea for repeaters is essentially how they're being handled at the minute.
The primary issue at present, is that an object requires two points of reference in order to create a rotation matrix.
Take a BVE2/4 route:
For any secondary track, the rotation matrix is built around the block in which the object is placed, and the points are calculated as the object position, and the end of the block. (Where the object position is in a straight line between these two points)
For the primary track, it's a little more complex as they're transformed by the curve itself, rather than just being placed upon the straight.

A quick demonstration-
Load the Keisei route with my current source and find yourself a curve, and take a look at the track.

The ideal solution to this is obviously to discard the block itself as a point of reference for object rotations, but this is considerably more complicated than it looks, as in order for this to work, we require the curve radius of the track, the starting and ending points of the track, and any interpolation, rotations, etc. we're applying-
If this structure looks familiar, it's because it's a block, and hence the question is a little circular!

Looking into this a little deeper. the solution at first would appear to be variable block lengths, but this is again somewhat of a fallacy.
Whilst this solves the problem of positioning and rotation, it's still got the fundamental problem that tracks will always be of differing lengths due to the basic lengths of curves properties that I mentioned above, and therefore, an object may easily end up spanning multiple blocks.

It's going to need a function writing to calculate the points of reference based upon the blocks and the curve radii.....
(sorry, this ended up being rather complex technobabble......)