Creatures that prefer MELEE seem to not wanna cast damaging spells like Fireball , Missile, Lightning at close range.
Strangely enough creatures that prefer RANGED do it the right way spamming all their damaging spells including fist or sword.
Another issue is creature seem to not wanna use their defensive spells unless they are in melee mode AND under 50% hp. If there is a creature they can't reach or if they are fighting against a MELEE creature with a damaging ranged ability like say Demonspawn they will never Rebound the first ranged spell attack.
That creatures with a melee attack preference use their melee attack in melee range is by design.
As for the 'defensive spell problem' you describe, have you taken into consideration that those defensive spells have a much larger cooldown than the other spells and that the cooldown is active when you start the map?
If you want to make 2 creatures fight to see how they react, you can't do so by making a testmap with them locked in the same room from the beginning. You've got to wait a while before you let the fight start so that all the spells are active.
In my tests creatures seem to hold on to their defensive spells until they are blow 50% health or the enemy is very close to them. Strangely enough I noticed melee units use like Avatar use spells like invisibility strategically to hide and heal while ranged casters simply never use invisibility in combat.
I deduct that
While damaging spell ai behaviour may be based on design I propose that at least all DEFENSIVE spells should be cast THE MOMENT BATTLE BEGINS.