SAC-QMIX

Algorithm that applies SAC to QMIX for Multi-Agent Reinforcement Learning. Watch the demo here.

Requirements

SMAC

pytorch (GPU support recommanded while training)

tensorboard

StarCraft II

For the installation of SMAC and StarCraft II, refer to the repository of SMAC.

Train

Train a model with the following command:

python main.py

Configurations and parameters of the training are specified in config.json. Models will be saved at ./models

Test

Test a trained model with the following command:

python test_model.py

Configurations and parameters of the testing are specified in test_config.json. Match the run_name items in config.json and test_config.json.

Theory & Algorithm

Architecture

Computation Flow

Note that a_i is equivalent to \mu_i and s_i is equivalent to o_i in the architecture schema above.

Train Objective: policies that maximum

Q-values computed by networks:

Individual state-value functions:

Total state-values (alpha is the entropy temperature):

Q-values expressed with Bellman Function:

Critic networks update: minimum

Actor networks update: maximum

Entropy temperatures update: minimum

Result

Note that data of other algorithm are from SMAC paper. Therefore methods of evaluations are kept the same as SMAC paper did (StarCraftII version: SC2.4.6.2.69232).

Test Win Rate % of SAC-QMIX and other algorithms

(Mean of 5 independent runs)

Scenario	IQL	VDN	QMIX	SAC-QMIX
2s_vs_1sc	100	100	100	100
2s3z	75	97	99	100
3s5z	10	84	97	97
1c3s5z	21	91	97	100
10m_vs_11m	34	97	97	100
2c_vs_64zg	7	21	58	56
bane_vs_bane	99	94	85	100
5m_vs_6m	49	70	70	90
3s_vs_5z	45	91	87	100
3s5z_vs_3s6z	0	2	2	85
6h_vs_8z	0	0	3	82
27m_vs_30m	0	0	49	100
MMM2	0	1	69	95
corridor	0	0	1	0

Learning curves of SAC-QMIX and other algorithms

(Mean of 5 independent runs)