Project files should be human readable

[mansellan]

Is your feature request related to a problem?
Modern source control systems expect to deal with non-binary files. Their diff tools are set up to optimise for this convention.

Describe the solution you'd like
Either JSON, XML, or (even) INI project files.

Additional context
Currently, twinBASIC uses:

1. workspace files (json)
2. twinproj files (binary, with source code embedded (as UTF-8?))

.Net (with the .Core reset) has moved to minimal, convention-based project files, expressed in XML form. This decision made it much easier to diff commits and critique any changes made (as there was much less boilerplate).

VB6, for all its faults with indeterministic serialisation, tended to keep with renderable file formats, except for binary resource files. Everything except .**x files could be diffed for source control.

Using a binary project file seems like a step backwards.

[WaynePhillipsEA]

You have to consider the twinproj file as a virtual file system. It's like a zip file.

There are good reasons that the twinproj file format is not text based, in the same way that there are good reasons ZIP files, VDI files etc are not text based.

The decision mostly comes down to efficiency, and serialization simplicity. There is also meta-data that does not lend itself well to text-based storage, such as indices. Soon, you'll also be able to include binary files such as graphics inside the twinproj file once the one-file limit has been removed.

When thinking of SCC, you wouldn't diff a ZIP file, and so you wouldn't diff a twinproj file. You'd diff the extracted content, and once we support SCC from inside VS code, that's exactly how it'll work with tB projects.

If the twinproj file wasn't a self-contained virtual file system, then a text-based human readable file format would likely have been chosen.

[RokeJulianLockhart]

SCC?

[mansellan]

Source Code Control.

[mika76]

I must say I agree with the original poster, dealing with binary files in source control is a pain in the bum. There's nothing more frustrating that .frx files changing and you have no idea why.

I also think it's pretty standard these days for projects files, and many many other files developers have to work with to be humanly readable, but even more importantly to be comparable and easily differenced.

What about when you have to do merges between people adding new files to a project, or as you say images but with the same file names?

People are even choosing to work with svg files more because of these features.

[mika76]

I played with it a bit now, and I do understand what you mean, but I'm struggling to figure out why it's like this?

What is the purpose of it being in a container, and not just a normal folder/folders with files?

The decision mostly comes down to efficiency, and serialization simplicity. There is also meta-data that does not lend itself well to text-based storage, such as indices. Soon, you'll also be able to include binary files such as graphics inside the twinproj file once the one-file limit has been removed.

I see you wrote the above, but it just seems to be adding complexity for very little gain:

• There's nothing simpler than serializing/de-serializing single files (which you'd still have to do in a virtual file system anyway), and meta data (including indices) can be kept in xml/json/base64 whatever is required.
• What if the file is corrupt, even with a single byte? Does that mean you've lost ALL your source code?
• Currently git/tfs/svn only sees the .twinproj file, so I assume you'd have to hack vscode to "see" the files in the virtual system so that you can use the tools to check in and out? What if someone wants to use a source control which is not standard, like mercurial or bazaar for example? Or even a different editor (like as mentioned in Other IDEs #82)

I personally feel that the industry has moved away from using binary files many years ago (for source assets) and probably for a good reason.

[WaynePhillipsEA]

What is the purpose of it being in a container, and not just a normal folder/folders with files?

The virtual file system used by twinBASIC is designed for versatility. As one example, the future plan is to be able to support in-memory VBA streams (where the underlying storage is based on IStorage), and the virtual file system design is PERFECT for this. The design of our virtual file system also pretty much matches the VS code API virtual file system.

I see you wrote the above, but it just seems to be adding complexity for very little gain...

You are not privy to the intricate details of the compiler, and our future plans, so I can't imagine how you could possibly come to such a conclusion.

One of the key features of the twinproj file, is that it allows for embedded files (as they are now), AND links to external files (not yet available in the preview). For example, VBP files will automatically be migrated to a twinproj with just links to the external files, so the source code in that case is not stored inside the twinproj file at all.

Currently git/tfs/svn only sees the .twinproj file, so I assume you'd have to hack vscode to "see" the files in the virtual system so that you can use the tools to check in and out

Nope, there's no 'hack' required at all. VS code fully supports virtual file systems, and indeed if you look at the recent VS code releases, a lot development work has been done in VS Code in support of the virtual file system approach.

What if someone wants to use a source control which is not standard, like mercurial or bazaar for example?

If VS Code doesn't support it, then neither will twinBASIC. You'll still be able to use the raw tools (coming soon) to extract from, and repopulate twinproj files, and then you can ultimately still make it work with any SCC you like.

The compiler has the virtual file system at its very core, and for very good reason. This isn't changing any time soon.

[bclothier]

Can I ask about how we will cope with corruption? Currently with a VBA project, if it's corrupted, Microsoft gives us 2 choices: 1) make a backup of the corrupted file or 2) throw it out. 😨 In this case, the option 1 isn't a great option because this requires spelunking around in binary contents to find and fix the corruption. It's usually "cheaper" (in terms of time and effort) to go back to the backup and redo the work.

Will it be able to recover as much as it can or at least make it possible for us to fix the corruption without having to revert to a prior backup?

One more thing that is not quite clear to me. Since the twinproj file won't be in the SCC, from where does VSC? tB compiler? has the information to rebuild the twinproj file from scratch? Imagine a scenario where I'm cloning a repository on another machine and I don't have the twinproj file but want to reconstitute the project from the SCC.

[WaynePhillipsEA]

Can I ask about how we will cope with corruption?

At the moment, the twinproj will be parsed as much as possible (i.e. 'recovery'), but obviously the success depends on many factors, and we don't want to have to rely on that. It is planned that we offer an auto-backup feature for the twinproj file.

from where does VSC? tB compiler? has the information to rebuild the twinproj file from scratch?

tB will be able to reconstruct the twinproj file fully from the repository. At the moment, things like breakpoints are stored as meta data rather than actual files in the virtual filesystem, but this will be changing soon so that literally everything will be able to be cloned to/from a repository.

FWIW, VS code just introduced a feature that creates a virtual file system for accessing github repositories 'live'. So by using a virtual file system we are very much on-board with where VS code is heading, IMO.

[dmrvb]

I also much prefer files in plain text (NOT XML) but if binary is essential then a mixed format file is usually preferable to separate files. Trouble with mixed format, especially if its "mostly text" is that sooner or later somebody (probably me) will try to edit it.

I have needed to create my own mixed format file definitions for a couple of my VB6 projects. I have two rules:
The first line or two of the file must be text and describe the file format, something like:

File Format = TwinBasic Project
File Version = 1.0

Any binary sections must be enclosed in a block and the btye size specified, something like:

Binary Bytes =100
Begin Block = Binary
////binary data here////
End Block = Binary

This makes it relatively easy to inspect the file (in Notepad) and to write code to extract data from the file whilst skipping over the binary blocks if they are not of interest.

[DaveInCaz]

It sounds like the twinproj file is almost like a local cache, rather than the 'master record'. (Except for project metadata which doesn't seem to have any text-only equivalent that would be able to be source controlled?) So why make the twinproj file user-visible at all... if its just needed for efficiency. This would be something like the .vs directory which Visual Studio creates when you work on a C# or other .NET solution, but ultimately can be totally discarded and rebuilt when needed, the user doesn't concern themselves with it.

[DaveInCaz]

I use diffing and source control as a fundamental work process. Files like FRX or MDBs which don't fit that model are much harder to work with. We've actually taken pains to eliminate binary files in as many places as possible so that work can be more productive and consistent. Nearly all popular programming languages historically have used a text-only approach, VB6 being the unfortunate exception with FRX's.

It would be ideal for us if tB would stick to the norm of easily diffable text-only files. Having to do a manual import/export procedure to work with tB would produce some disincentives for people to do things the "right" way (in terms of our normal process).

So indeed an approach where .twin files function more like .vbp files would be much appreciated! :) In that case I suggest there might be two distinct project "formats" with different names / file extensions. One analogous to VBP (or CSPROJ, VBPROJ, etc.) and a second where everything is all bundled together.

[WaynePhillipsEA]

As mentioned above, this is already planned.

For the time being, please accept the current situation and allow focus to continue on the UI side of things.

[DaveInCaz]

@WaynePhillipsEA sorry if my comment seemed pushy, that was not at all how I intended it!

I don't think this is a significant issue at all at the present time, I'm sure your current focus areas are spot on. I just wanted to register my personal opinion for the long haul.

[WaynePhillipsEA]

No worries @DaveInCaz, understood :) No apology necessary, it's all good.

[gh658804]

Hi @WaynePhillipsEA. Please clarify what is planned (whenever it is planned, it is understandable that it's not a priority at this particular moment). Is it this? ->

It sounds like the twinproj file is almost like a local cache, rather than the 'master record'. (Except for project metadata which doesn't seem to have any text-only equivalent that would be able to be source controlled?) So why make the twinproj file user-visible at all... if its just needed for efficiency. This would be something like the .vs directory which Visual Studio creates when you work on a C# or other .NET solution, but ultimately can be totally discarded and rebuilt when needed, the user doesn't concern themselves with it.

If the answer is Yes then great. Having plain text files (like i.e. .bas.twb or cls.twb) would indeed be great, and minimize the number of binary files as much as possible. IMO, VB's .frx files could have been text files from the beginning.

[rexxitall]

Fully agree ! Would make it way more easy to migrate things between vb6 vba and TB. As a first step it would be a help if the file extnesions would be same as in Vb6. To just name all .twin is unhandy and you can not just throw files around. Also The Module and Class statemenst should be hidden. Its unhandy by copy and paste a whole module. Even if the project file isnt VB6 compatible. its easy to create just a tb project export ist and create a new VB6 project with the exported files imported. In this case you has what you want (at least half the way) - one code base (tb) but you can see without a lot of work how the "original" would behave. Which might also a nice timesaver to track problems down ;)

[WaynePhillipsEA]

Fully agree ! Would make it way more easy to migrate things between vb6 vba and TB. As a first step it would be a help if the file extnesions would be same as in Vb6. To just name all .twin is unhandy and you can not just throw files around. Also The Module and Class statemenst should be hidden. Its unhandy by copy and paste a whole module. Even if the project file isnt VB6 compatible. its easy to create just a tb project export ist and create a new VB6 project with the exported files imported. In this case you has what you want (at least half the way) - one code base (tb) but you can see without a lot of work how the "original" would behave. Which might also a nice timesaver to track problems down ;)

Nope, none of that is happening, sorry, and it is non-negotiable.

We support legacy BAS/CLS files 100%. You can continue to use them without issue.

TWIN files have important differences. The main one is full Unicode support. BAS/CLS files are ANSI based (based on your system ANSI codepage). They cannot support Unicode, and any attempt to make them Unicode-friendly will break backwards compatibility with VBx that does not interpret them as Unicode. The second difference is that they were designed to allow for multiple components per file.

So as you can see, it is very important to offer the new format, and it is imperative be able to distinguish between legacy vs new format via the file extension. These decisions weren't made lightly or without significant thought.

[rexxitall]

Do as you want as usual. User needs are how to say ... just hindering :D

[mansellan]

Uh, wut? You had a comprehensive answer. It's not possible.

[Greedquest]

This discussion seems like an X/Y issue. Most of the desire to have human readable files seems to be around git and source control. I'm sure you've considered this @WaynePhillipsEA so can I ask what is your latest vision for source control support? I think if that is clear then most of this discussion is moot.

Re-reading the other posts in this discussion I see 3 options (not necessarily exclusive, multiple alternative options could be supported at once if there is no single "Right Way"):

1. Export on save dumps a copy of the live project in human readable format to the Disk. Sync with folder could bring changes from the human readable format back into the editor. This is the workflow RubberDuck allows us to do. It is annoying as if I make changes to the exported human readable files these are not automatically reflected in the live project open in the VBE (VBA/VB6 Editor).
   • RD could add a filesystem-watcher to spot changes in the exported files and automatically sync to the VBE, this is what Notepad++ does, but there is some risk of cache invalidation.
2. VBP style "symbolic links" to source code. This creates a live mirror between the project in the virtual filesystem in the IDE and the real filesystem. Make an edit in one place and it is immediately reflected in the other. No risk of cache invalidation if the Operating System manages it for you.
3. git integrated into the IDE, which means the files do not need to be exported since git would work within the virtual filesystem. This makes sense for tB as a VS Code extension as it has good support. However, I worry that unlike VSCode, there won't be resources available to polish user interface for the git integration to the same standard in tB's IDE. I know RD gave up on git integration for this reason, deemed beyond the scope and not bringing much value that wasn't already possible with code sync + external git tools. I like to use a 3rd party GUI GitKraken
   rather than command line git for source control, so I prefer editors that support method 2.

Additional Context:
If tB exposes some tokeniser/abstract syntax tree/reflection API, it will give the community a lot more power than was previously possible to create automatic code linting and formatting tools. Auto-formatters, indenters, code to re-order your Subs in alphabetical order, etc.

These could be worked in as IDE plugins if the IDE supports scripting. Or as Language Servers. Both those methods work with a virtual filesystem (LSP is FileSystemProvider agnostic, the LSP Server that provides the linting or indenting etc. has no idea where the file comes from, as this info is hidden behind abstraction and the responsibility of the LSP tool - i.e. the IDE). That said my favourite and simplest to author way is to create exes that run as git hooks/pre-commit hooks - e.g. run the smart-indenter every time I try to make a git commit. Issue: these git-hooks cannot work with a virtual filesystem.

[fafalone]

It could allow 3rd party source control tools that don't understand the .twinproj format to track the internal files by being able to access them as a part of the file system, rather than manually parsing the twinproj file itself.

[Greedquest]

@fafalone I thought a shell extension was more along the lines of how I can open a zip file in Windows explorer as if it is a regular folder. But if I try to commit the zip file to a repo with git, it still sees it as a big binary blob. That is not diffable. So I still don't quite follow how a shell extension can help existing source control tools without them having special treatment in the source control software? Or do the source control tools also need modification to make this work?

[fafalone]

It depends on how the shell extension is implemented.

The zip folder extension doesn't implement all features of a normal folder, and is in fact quite a bit different from a traditional shell extension; but you could.

For instance if you attempt to drag a file from zip folder onto GitHub, it won't allow you too. This is because it doesn't supply a CF_HDROP clipboard format; you could have the shell extension extract to a temporary physical location and supply a CF_HDROP format to it, then you'd be able to drop it on GitHub's website directly from the virtual folder; this would be transparent to the user.

[Greedquest]

@fafalone ok shell extensions are more powerful than I thought. Still, do you know of any flags that you could in theory support which would allow git to track changes to the contents of the twinproj file? Without dragging and dropping to GitHub to commit the files.

[fafalone]

I don't know how git works under the hood but if you skip the shell and just go low level APIs to copy files, no it won't be possible as you can only present it as a folder to higher level methods. But the point is it allows some options for browsing the files on disk and using them with other tools, as opposed to none when there's no way of browsing it as a folder.

[rexxitall]

I still think to implement something compatible to VBx would be the simplest and most flexible solution. Its also the most robust ones. If something weired happens you do not loose everything :) And a lot of code tools alredy written will have a chance to still work on :) There are reasons why all compilers i know relay on simple ascii files ;) Some kind of packing system might be a idea to archive things. But that would be on top.

[RokeJulianLockhart]

@rexxitall, you already got an answer to this at #131 (reply in thread)