Storage and retrieval of 25-line frames to and from the URL.

README.md

@@ -3,17 +3,19 @@
 This is a teletext editor implemented in JavaScript, so that people now 
 need no more than a JavaScript-enabled browser in order to create their 
 own teletext frames. It was written for the 
-[teletext40](http://www.teletext40.com/100/1) project and you can try the 
-editor out at http://editor.teletext40.com/ - however, it also works as 
-a general-purpose editor, and is intended to be completely independent of
-teletext40.
+[teletext40](http://www.teletext40.com/100/1) project and you can try 
+the editor out at http://editor.teletext40.com/ - however, it also works 
+as a general-purpose editor, and is intended to be completely 
+independent of teletext40. It may also be used for editing BBC Micro 
+mode 7 screens, or preparing viewdata frames.
 
 Most of the functionality of the editor is accessed through key 
 sequences beginning with the escape key. They are summarised in a table 
-to the right of the the editor. Pressing the escape key takes you into 
+to the left of the the editor. Pressing the escape key takes you into 
 command mode, in which the status bar is coloured yellow, and then 
 typing a (possibly shifted) letter will insert a control character or 
-perform some other function.
+perform some other function. The status bar can also be used to view 
+teletext metadata. The key sequence ESC-L toggles this.
 
 Everything is supplied in a single HTML file. There are no external 
 dependencies. You can email the HTML file to friends so that they can 
@@ -34,19 +36,39 @@ In the URL, the nybble before the colon describes the character set the
 page is encoded in (the least significant three bytes) and whether the 
 page is intended to be rendered with black foreground colours enabled 
 (the most significant bit enabled if it is). The part of the URL after 
-the colon contains a base-64-encoded sequence of bits, amounting to 1120 
+the colon contains a base-64-encoded sequence of bits, amounting to 1167 
 base 64 digits. The encoding is standard 'base64url' with URL and 
 Filename Safe Alphabet ([RFC 4648](https://tools.ietf.org/html/rfc4648) 
 §5 'Table 2: The "URL and Filename safe" Base 64 Alphabet'). After 
 decoding, the seven-bit character code for column _c_ and row _r_ 
 appears at bit positions _280r+7c_ to _280r+7c+6_ inclusive, the most 
-significant bit appearing first.
-
-A script, `url2raw.pl`, in the `tools/` directory, is provided to assist 
-with this conversion. Supply an editor URL on standard input and it will 
-output the raw frame, with lines delimited with newlines, on standard 
-output. Contributions of scripts to convert these URLs to formats 
-required by other teletext systems would be very welcome.
+significant bit appearing first. There are therefore two bits at the
+end of the encoding which are not used.
+
+The key combination ESC-E will pop up a box allowing you to export the 
+frame. It will appear as a data URI. There are three formats at present. 
+Two are raw and differ in how they deal with character codes in the 
+range 0x00 to 0x1f. One raw format leaves them untouched, the other sets 
+the high bit so they appear as characters in the range 0x80 to 0x9f when 
+exported. The other format is 8-bit TTI, designed for use with wxTED, by 
+@peterkvt80 - Peter Kwan. This includes metadata which can be viewed 
+with ESC-L but which unfortunately cannot yet be edited.
+
+Alternatively, a script, `url2raw.pl`, in the `tools/` directory, is 
+provided to assist with conversion on the command line. Supply an editor 
+URL on standard input and it will output the raw frame, with lines 
+delimited with newlines, on standard output.
+
+We are happy to implement export to other formats if an unambiguous 
+specification document can be found for them. Contributions of scripts 
+to convert these URLs to formats required by other teletext systems 
+would also be very welcome.
+
+Before September 2015, the editor frame was 24 lines and not 25, so 
+authors of conversion scripts and routines may wish to consider also 
+supporting encodings of length 1120, describing the first 24 lines. (In 
+this case, there are no left-over bits!). `url2raw.pl` supports both 
+frame lengths.
 
 ## The editor is licenced under GPLv3.
 
@@ -63,6 +85,26 @@ would like to contribute but cannot program, documentation and tutorials
 would also be very welcome. If you're interested in contributing in this 
 way, please comment on issue #3.
 
+## Hints, tips and caveats
+
+* The editor allows you to edit the whole frame, but if you are preparing
+  a frame for broadcast teletext, you should bear in mind that the last
+  line is used for Fastext links and the first one is used for the header
+  and will usually be overwritten by the broadcaster. Furthermore, the 
+  first eight characters of the first line are never transmitted, instead
+  being used to encode the page metadata. The television usually displays
+  the page number here instead. A reminder of these restrictions is given
+  when the grid is enabled (with ESC-X). Cells to avoid are not included
+  in the grid shown.
+* Another application of the editor is for designing BBC Micro frames.
+  There are subtle differences in the way each system displays the frame, 
+  notably in the handling of double height. The characters in the first
+  row of a double-height pair aren't automatically copied to the second
+  row. Instead, on a BBC Micro, the two rows need to have identical data.
+  This means that effects like the top and bottom of double-height
+  characters having different colours, or even different characters
+  entirely, may be achieved. This editor does not display such effects.
+
 ## Related links
 
 ### As seen at...
@@ -83,7 +125,8 @@ The editor has been used for various events and systems:
 
 * The Windows-based teletext editor [wxTED](http://teastop.co.uk/teletext/wxted/) by Peter Kwan - @peterkvt80 - has 
   implemented import from, and export to, the URLs used in this editor.
-  Peter has also produced a Raspberry Pi teletext inserter!
+  Peter has also produced a Raspberry Pi teletext editor and other cool 
+  teletext technologies.
 
 ### Test frames
 

index.html

@@ -822,14 +822,23 @@
 	}
 
 	// This block deals with the new base 64 format.
-	if ( hashstring.length == 1120 ) {
+
+	// We need to be able to handle two cases here, depending on the
+	// size of the frame. 24-line frames have 1120 characters, and
+	// 25-line frames, the new way we do things, have 1167 characters.
+	// 25-line frames have two bits at the end which are ignored and
+	// just exist for padding.
+
+	if ( hashstring.length == 1120 || hashstring.length == 1167 ) {
+		var numlines = 25;
+		if ( hashstring.length == 1120 ) { numlines = 24; } 
 
 		// As we scan across the hashstring, we keep track of the
 		// code for the current character cell we're writing into.
 		var currentcode = 0;
 
 		// p is the position in the string.
-		for ( var p = 0; p < 1120; p++ ) {
+		for ( var p = 0; p < hashstring.length; p++ ) {
 			var pc = hashstring.charAt(p);
 			var pc_dec = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_"
 				.indexOf(hashstring.charAt(p));
@@ -870,17 +879,28 @@
 				}
 			}
 		}
+
+		// If we only read in a 24-line file, we need to blank the final
+		// line.
+		if ( numlines == 24 ) { 
+			for ( var x = 0; x < 40; x++ ) { clear_char(x,24) }
+		} 
 	}
 }
 
 // Similarly, we want to save the page to the hash. This simply
 // converts the character set and page data into a hex string and
 // puts it there.
+
+// Now that the editor is 25 lines, this format is the one we 
+// save to. There are two whole left-over bits at the end of
+// the encoding. Yes, it's wasteful, but for now, we'll have to
+// let that go.
 function save_to_hash() { 
-    if (mouse_button == 1) {
-        // optimisation: don't update hash while drawing
-        return;
-    }
+	if (mouse_button == 1) {
+		// optimisation: don't update hash while drawing
+		return;
+	}
 
 	var encoding = "";
 
@@ -893,7 +913,7 @@
 	// Construct a base-64 array by iterating over each character
 	// in the frame.
 	var b64 = [];
-	for ( var r=0; r<24; r++ ) {
+	for ( var r=0; r<25; r++ ) {
 		for ( var c=0; c<40; c++ ) {
 			for ( var b=0; b<7; b++ ) {
 
@@ -915,7 +935,7 @@
 	}
 
 	// Encode bit-for-bit.
-	for ( var i = 0; i < 1120; i++ ) {
+	for ( var i = 0; i < 1167; i++ ) {
 		encoding += "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_".charAt(b64[i]); 
 	}
 	if (window.location.hash != encoding) { window.location.hash = encoding; }
@@ -926,7 +946,7 @@
 // converts the frame to raw format, and then edits the document to show a link
 // using the data URI scheme. The user can save this link to their own computer
 // without the server needing to store it.
-function export_raw() { 
+function export_frame() { 
 	// We can't substitute characters for the base64 in the address bar
 	// becase the output must constain newlines and the addressbar uses
 	// seven bits for each character. Therefore we must export in a
@@ -1425,7 +1445,7 @@
 		if ( code == 122 ) { matched = 1; redraw(); }
 
 		// E = export frame
-		if ( code == 69 || code == 101 ) { matched = 1; export_raw(); }
+		if ( code == 69 || code == 101 ) { matched = 1; export_frame(); }
 
 		// L = teletext metadata
 		if ( code == 76 || code == 108 ) {

tools/url2raw.pl

@@ -46,11 +46,13 @@
 # since the output is agnostic to it.
 $url =~ s/^[0-9a-fA-F]:(.*)$/$1/;
 
-if ( length($url) != 1120 ) { die "The encoded frame should be exactly 1120 characters in length"; } 
+if ( length($url) != 1120 && length($url) != 1167 ) {
+	die "The encoded frame should be exactly 1120 or 1167 characters in length";
+} 
 
 # Compute the characters in the output frame, bit-by-bit.
 my @cc = ();
-for ( my $i = 0; $i < 1120; $i++ ) { 
+for ( my $i = 0; $i < length($url); $i++ ) { 
 	my $val = index($alphabet, substr($url, $i, 1));
 	if ( $val == -1 ) { 
 		die "The encoded character at position $i should be one from the alphabet";
@@ -66,8 +68,11 @@
 	}
 }
 
+my $numlines = 25;
+if ( length($url) == 1120 ) { $numlines = 24; } 
+
 # Output the frame.
-for ( my $y = 0; $y < 24; $y++ ) { 
+for ( my $y = 0; $y < $numlines; $y++ ) { 
 	for ( my $x = 0; $x < 40; $x++ ) { 
 		# Set the high bit if needed.
 		if ( $sethighbit != 0 && $cc[$y*40+$x] < 32 ) {