NOTES

next for compiler:
resolve all var types

timer = f1() - doesn't know what var type timer is

need sure-fire way to tag/resolve function return types
analyse return usage at end of function block?


where to put the stack?
malloc routine/function ?
in temp var space ?
end of main scm space, reserve with padding ? ! ? (5,875 bytes free, carid2gxt is 3,475, leaving 2,400 free)
128 slots (512 bytes)
256 slots (1024 bytes)

reserved vars for:
all local boolean vars (32 bools)
stack index base (can access from 0)
stack index (can access from n)
stack index max (used for raising an error)

populate stack with:
0: 

dynamic or static stacks

stack size allocated in multiples of 16 ints


intermediate representation

[opcode_call [wait,[[int8 0]]]]
[function_call [my_func,[[int8 0]]]]
[gvar my_global_var]
[gvar my_global_var int]
[gvar 1234 int]
[ivar my_instance_var]
[ivar my_instance_var int]
[ivar 12 int]
[lvar my_local_var]
[lvar my_local_var int]
[function [ my_func , [[return_val int]] , [[arg1 int],[arg2 int]] ]]
[end_function]
[function_return [lvar1,lvar2]]
[assign [gvar my_global_var],[int8 0]]
[assign [gvar my_global_var],[lvar my_instance_var]]

self.stack_frames = [
  {
    returns:   { retval1: :int , retval2: :int },
    arguments: { arg1: :int , arg2: :int, arg3: :int },
  },
  {
    lvars:    { tmp1: :int , tmp2: :int }
  }
]

scan_for_lvasgn_in_block() - returns all lvar assign names in block



READ UP ON:
whitequark's parser
whitequark's transformer

assign vars on the stack in the following order:
* return values
* arguments
* temp vars

when function is called and returned, return values will be at the end of the temp vars
if assigned return values are new temp vars, adjust stack pointer so return vars are absorbed as temp vars

stack_counter_base, stack_counter = allocate_stack()
set_stack_vars(stack_counter_base, stack_counter)
release_stack(stack_counter_base)

can use only 1 stack if no scripts yield with a non-empty stack?

can repl use readline ?

reverse-rpc for logging?
some sort of logging system

surveying script

press buttons to record 3 coordinates in array
show xyz coords, heading/distance between each one and player coords

search all start_new_script commands:
ack "\(start_new_script \(\(label thread_(?\!(a_alap|a_cont|a_term|audiol|cleanau))"

r1 menu
hold down r1, use dpad to navigate
(X/O for accept/back? would interfere with normal controls)
record and lock radio station when in vehicle to prevent conflicts

support temp vars in comparisons and maths

taxi:
* accumulate fare while driving (percentage of start distance, only allow it to increment, no decrements)
* while driving: triangle = pull over to leave, square = trip skip, X = change driving style
* implement leaving taxi (square = leave illegally, triangle = leave legally with payment)

set up useful config menu
settings for:
* max number of spatial/car scripts
* blips for spatial/car scripts
* taxi costs (minimum + distance multiplier)

spatial scripts:
make collectables managers use them
weapon/pickup stashes
pack timers into int8s

garage menu
proper buildings/garage for it
store car variation, not dirt (fancy bitpacking needed for 2 values)

support constants in s-exps, ensure external loader indexes use them
use constants for start_new_streamed_script

rewrite watchdog script
just hook main loop
set `$scripts_spawned = true` when scripts are spawned
shut down when saving, kill all threads (allow graceful shutdown with global var) set `$scripts_spawned = false`, continue save





function pointer notes:

SCM base 1.0 = 10787168 (0xA49960)
SCM base 3.0 = 10664568
diff         =  -122600

structs = (4625296) 0x469390 
10933576

opcode function pointer = 0x466C50 - 122600

function pointer arrays: 0x465e74, 0x465fa0

main function? 4611300

doc'd steam addr: 0x46F75C

0x46F75C
0x465e74

diff between mine and steam version's addrs: 333128


Unused/debug code:
 55976 -  60030 :  4054       - unused after game boot, useful for temp variables? (within range, won't be saved into save)
 56124 -  56153 :    29       - jumped-over code
 56728 -  57945 :  1217       - jumped-over code during init
 59712 -  59754 :    42       - these launch_missions just hit terminate_this_script commands
 59818 -  59818 : first branch
 60030 -  61763 : main loop
 85611 -  86605 -   994       - collectables rewards, hook or term
 92228 -  94428 :  2194       - help thread, begin with terminate_this_script or hook it
127703 - 129492 :  1898 bytes - impound/export debug (impossible branch begins 127588) (patch 127573 to jump to 129471)
141175 - 141401 :   226       - impossible branch in gf init
154564 - 154860 :   296
152364 - 154839 :  2475       - unused gf debug

                :  7,217

intro mission launch site: 59976

reserve space for external global vars
whole external script shares the space
lvar31 = thread ID
lvar30 = (external address + external var offset) / 4
access vars with: $_0[lvar30],$_4[lvar30],$_8[lvar30]


reserve global vars for temp compiler vars
spatial index script
  - array of active points
  - run init/loop/kill callbacks 
  - how to do persistence/vars/execution? separate script instances?
  - global array of event ID => timers (timers are packed int8 minutes until event can trigger)
  - local array of active event IDs
  - when near event location:
    - if local array slot free
      - add ID to local array
      - set global timer to 255 (executing)
      - spawn script with event details as args
    - else
      - do nothing (no free slots)
  - each tick:
    - for event ID in local array
      - if global array [event ID] == 255
        - do nothing (script executing)
      - else
        - remove event ID from local array
  - each minute:
    - subtract 1 from each timer in global array
  - each script:
    - receives arguments: event ID, xyzh coords, trigger distance
    - expected to shut down when outside distance
    - set global array [event ID] to next timeout in minutes 
for loop syntax
compile standard lib into externals, ease limit on main code

global vars: 560 bytes @ 21700 = 140 int32s

Things needed for release:

Scroll all lists (threads, local vars, global vars, regions list, repl)
Trim binaries down to just the important useful ones (assember/disassembler/debugger, shove the rest in a `tools` dir)
No-BS fast way to set up code reloader and open editor with scaffold code
Bootstrapper (don't need custom SCM for RPC/breakpoints/repl)
Better first-boot intro (offer a few simple options: Decompile main.scm, compile source code, init code reloader)
Show routine names in stack traces
Data stack implementation
better syntax for routines/functions
sublime text integration (compile/build menu)

* just show menu when r1 is pressed ??
do array check for car features (8 max?) (needs to be global through savegame? or not if detect script is external too?)
reverse syscall
export symbols for array vars
show include sizes for externals

dope wars style system
8 drugs, 8 locations

- vegas
- desert
- bayside
- san fierro
- mountains
- countryside
- los santos east
- los santos west

supply[area][drug] = 0-15 (bitpack as 4 bits = 8 ints in int32)
demand[area][drug] = 0-15

demand has constant base, variable +/- effects
supply has base, can be changed by player deals (with decay to base), variable +/- effects

bootstrap script code required:

  $SCRIPTS_RUNNING - script ready to load vars (set bitmask once script is running)
  $SCRIPTS_TERMINATE - script ready to exit vars (set bitmask to tell script to self-terminate, clear when done)

  main script extension:
    * spawn_scripts()

  save script extension:
    * safely shut down all scripts

  rpc handler:

  breakpoint and repl handlers:

  spawn_scripts():
    * spawn script if $SCRIPTS_RUNNING[script_id] == 0

  shutdown_scripts():
    * set all $SCRIPTS_TERMINATE bits

  is_scripts_shutdown_complete():
    * return true if all $SCRIPTS_TERMINATE bits are unset






NEXT:

classes

SCM: holds raw bytes, lazy-loads or preloads instruction offsets/disassembled instructions
Assembler: takes raw tokens and assembles bytecode at defined positions, and symbols
Compiler: takes ruby code and emits raw tokens
Instruction: convenient reader/writer methods
Script: convenient reader/writer methods
Process: convenience methods + read SCM/scripts from memory, can load symbols
Analyser: loads bytecode, calculates jumps/blocks

GTAV memdump notes:
1130934 (main)
14921914 main

* debugger: run without bootstrap script (inject rpc/repl code, find+patch main loop to run them)

* filter opcodes to only ones used in-game (are there other supported ones that are unused?)
* debugger: add/remove gvars from console
* use `subl` app to show disassembly, new script template

* debugger: control debugger features from console (add gvars, read/write memory, etc.)
  - launch/kill game
  - inject any script
  - set script
  - use hex for input/output where appropriate
  - read/write local vars
  - read/write global vars
  - read/write scm memory
  - read/write game memory
  - disassemble instruction at offset
  - patch instruction at offset
  - insert breakpoint at offset
  - handle breakpoint in repl
* debugger: scrollable lists/tables everywhere
* debugger: regions list - maybe replaces code injector
  - detect MAIN + external script's memspaces + block allocator spaces
* debugger: disassemble bytecode in background thread

* compiler: dump all instruction offsets to symbols
* debugger repl: easy-to-use (don't require injection/attachment)
* debugger repl: use local vars instead of global?
* debugger: enable auto-reload, toggle settings, choose files

* debugger: flag on whether to alias script local vars to ruby local vars
* debugger: flag on whether to alias global local vars to ruby global vars
* debugger/compiler: generate symbols that can be used for injected code/strange base offsets
* debugger: DONE: hotkeys, convert all to ctrl+key, ensure no weird focus stuff
* debugger: allow copying text properly
* compiler: handle more errors


* RESOLVED: debugger+code: make breakpoint handler larger, dynamic? (need multiple code sites for multiple dynamic executions)
* RESOLVED: debugger: bulk instruction calls + return values
* RESOLVED: debugger: try to speed up (2nd-level proxy for executing in multiple repl scripts?)
* DONE: debugger: ensure breakpoints work with new repl
* DONE: debugger: cache process.threads calls on each update call

* DONE: generate call graph
  detect all opcodes that use labels (start_mission/switch/headers/etc.)

* stack for function call args?
  is it safe to use if each script leaves the stack empty before yielding?

* DONE: debugger: more mouse click events
* debugger: finalise script viewer
* debugger: persist panels settings
* DONE: debugger: hook up code reloader
* DONE: debugger: hook up repl


* docs: generate list of used/supported opcodes in SA



* code reloader:
  get malloc space
  insert header+code
  header: magic string, size of code, routine offsets for: init/loop/shutdown, base offsets (mem+scm relative)
  when new code is inserted, either:
    * trigger shutdown of old code, start new code
    * trigger old code to jump to new code, retaining local var state
    debug_string("binary data with header")
    init()
    jump(to_next_instruction)
    loop()
    jump(to_next_instruction)
    shutdown()
    jump(to_next_instruction)
    terminate_this_script()
    routines {
      init{}
      loop{}
      shutdown{}
    }

  to halt current code, look at loop_offset, read wait instruction, write return instruction immediately afterwards
  control whether it reloads or shutsdown/reinits by altering main body jumps
  entry/re-entry points: with init: body+0, without init: body+14

  UI columns:
  filename (16+)+3
  mode (6)+3
  injected at (9)+3
  status (the rest)

* game: garage manager - have drop-off areas for it, charge money?
* game: garage manager - deliver cars in traffic, don't just spawn?
* DONE: game: remake collectables finder in ruby
* game: remake collectables finder manager in ruby
  - need to handle respawn timer (and activity timer)
  - read/write these to global vars
* game: rewrite rpc thread in ruby (?)
* game: chill with homies
* game: get nearby cars, allows you to ride trams?
* game: inventory, bank snacks for health regen in levels (consume 1 snack every 10/8/6/4/2 seconds when below n% health)
* game: organise code into subdirs, update assembler to handle if needed

* DONE: compiler: gosubs as a condition (return_true/false opcodes)
* compiler: handle `return` with and without args (return $var == -1)
* compiler: perhaps make gosubs/jumps shorter by assembling as int16 where possible? (only useful for externals)
* ??? compiler: perhaps make gosubs/jumps shorter by making binary lookup table of offsets, then accessing with gosub(dmavar 50000 label_name) ?
* ??? compiler: perhaps make text lookups shorter by assembling binary table, then using global var array into it? (8 * 211+1 = 1696)
* compiler: better output/logging
* compiler: enumerator routines (all_locations().each do |loc_x,loc_y,loc_z|
* compiler: lvar array = use any local args by reference?
* compiler: lvar array = register remapping?
* compiler: trap/exception routines, goto to them to set global var + hit breakpoint in loop
* DONE: compiler: breakpoints: global var to set wait/no-wait on breakpoint loop
* compiler: tail-call optimisation (convert gosub+return into goto)
* compiler: treat routine names as arguments (emit as (label routine_name)). ie. gosub(routine) / goto(routine)
* compiler: function defs with global vars used for arguments/return values, assigned automatically around gosub to routine ( function(args: [:$bitpacker_value,:$bitpacker_bits], returns: [:$bitpacker_value]))

* disassembler: tag all jumps (start_mission/switch statements)
* disassembler: make list of label names, use in disassembly
* disassembler: make list of variable names, use in disasm
* disassembler: use dmavar, not var in disasm

* debugger: use electron for UI

* more RPCs:
  DONE: teleport player: x/y/z/heading/interior id
* DONE-FIXED: disable EmitNode feature, is dangerous (tries to put touchups into unassembled instructions)

* DONE: gxt compile pipeline (rebuild GXT files, insert new entries)
* DONE: generate HeaderExternal from files automatically
* DONE: build external scripts first, patch into script.img, generate header, compile main script
* DONE: main idle loop at 60030->61763
* DONE: patch idle loop to check if externals are running (/watchdog timer for threads)
* DONE: patch save game code to make save-game safe (kill threads/delete blips?)
* DONE: save thread: gosub to save opcode at 88020
* DONE: disassembly mission bytecode in-memory, patch ps2 keyboard checks (15 bytes total (4+4+7) -> goto (7), 4 no-ops (8))
* DONE: compiler header-less mission_label SCMs (with assembler directive?)
* DONE: insert them back into script.img
* DONE: load from main.scm with bootstrap code

* standard lib
* car-id-2-gxt if space permits? (~3600 bytes)
* DONE: bitpacker accumulator routine
* 2d/3d iterator helper
* in-game year/month/day/hour/minute time value routine (use bitpacking?)
* state machine helper?
* distance sorter helper?
* modulo helper
  a - (a/b)*b
  tmp1 = a
  tmp2 = a
  tmp2 /= b
  tmp2 *= b
  tmp1 -= tmp2
  tmp1
* get nearby cars/actors helper

useful for spawning cars? GET_PARKING_NODE_IN_AREA

* Ruby compiler
* DONE: singleton global variable assigner
* export symbols to subdir, name properly
* symbol export for routines, use routine names in stack view
* compiler-level include
* DONE: raw sexp output by assigning array to constant (ROUTINE_ADDR = [:label,:routine_label]; gosub(ROUTINE_ADDR))
* non-hacky string var support
* DONE: emit smallest size for immediate values possible
* DONE: routines {} block, avoids jumps around routines
* function calls using global vars for arguments/return values
* thread {} block, named + scoped local vars (+export symbols)
* DONE: conditional gosubs (routine call as condition, routine calls return_true/false + return)
* DONE: debugger/breakpoints (reuse rpc code?) (rpc_breakpoints_enabled,rpc_breakpoint_pc) (GXTs: CHEATON, FEM_PWT)
* DONE: fix parser bugs with global vars, multiple conditionals
* MOSTLY DONE: pipeline for code/img/gxt compiles/installs
* DONE: refactor menu input handling into routines for each menu with a variable for which button was pressed
* DONE: standard lib that can be compiled in
* DONE: function for bitpacking with counter + number of bits + packed value + unpacked value
* test mission jumps (probs not) (does base pc change? check mission threads)
* switch support (later)
* DONE: array handling
* DSL for patching original scm?

enable/disable keys (hold down r1, use face buttons to navigate, disabling original functions)
enter/ext vehicle: SET_PLAYER_ENTER_CAR_BUTTON
duck: SET_PLAYER_DUCK_BUTTON
shoot: SET_PLAYER_FIRE_BUTTON
jump: SET_PLAYER_JUMP_BUTTON
r1?: SET_PLAYER_DISPLAY_VITAL_STATS_BUTTON
r2/l2?: SET_PLAYER_CYCLE_WEAPON_BUTTON
get_current_population_zone_type
create_user_3d_marker



* Can access memory outside of Variables space
* Can kill all running threads with terminate_this_script
* Some .gxt entries require load_mission_text (OVALRIG/HOTR_05)
* Self-modifying code allowed
  * Limits? presumably only first 64kb due to int16 var size
* Integer overflow
  * Overflows from 2147483647 -> -2147483648
* Local variables
  * Per-thread
  * start_new_script takes (offset,*var_args), each nth arg gets assigned to lvar n-1
    ie. start_new_script(1000,50,333) sets (lvar 0) = 50, (lvar 1) = 333
* Out-of-bounds variables
  * var reads outside of memory appear to return 0
  * but writing anything out-of-bounds makes subsequent reads return -68
  * doesn't appear to be wrap-around or shadowing
  * same for lvars
  * terrifying

* Can crash if threads don't yield with wait (?)
* San Andreas can handle ~24,000 noops without crashing (confirm this - i think i just overwrote my main code)

Check:

* Can you use negative jump offsets in normal scripts? how do they behave?
* Out-of-bounds array access

Next
----

  * NodeSet class
    * Can refer to/return other nodes in the nodeset

  * Cache
    - Scm#nodes to json - what about binary data?

  * Rails frontend, resources for:
    - /scms
    - /scms/1

    - /nodes ? offset  = 1024
    - /nodes ? range[] = 1024 & range[] = 2048
    - /nodes ? offset  = 1024 & range   = 512

    - /opcodes/0002

    - /enums
    - /enums/objects

  * UI
    - Serialise scroll positions from the start
    - Combined code/graph viewer
    - Workspaces (windows/layout/target window for shortcuts)
    - Minimap for code view

  * Tests
    - Headers
    - Arg types