Having all minions attack player structures is a bit unrealistic and also not so fun. To make things a bit more interesting, have a minion dedicated to bringing down player structures.

Battering Ram

The battering ram is a minion whose only job is to seek out player-made gates and doors and smash them down. It can be a slow moving but heavily armoured ram on wheels.

Maybe it is weak to fire? So defending players can try to snipe it with fire arrows.

Siege Troll (melee)

A player-aligned troll whose job is big smashing. Walls? Smash. Gates? Smash. Everything smash.

Siege Troll (ranged)

A player-aligned troll whose only job is hurling rocks at player structures, specifically walls.