implement TD3+BC for offline RL

[nuance1979]

• I have marked all applicable categories:
  • exception-raising fix
  • algorithm implementation fix
  • documentation modification
  • new feature
• I have reformatted the code using make format (required)
• I have checked the code using make commit-checks (required)
• If applicable, I have mentioned the relevant/related issue(s)
• If applicable, I have listed every items in this Pull Request below

• implement TD3+BC for offline RL;
• fix a bug in trainer about test reward not logged because self.env_step is not set for offline setting;

[codecov-commenter]

Codecov Report

Merging #660 (ec95af3) into master (9ce0a55) will increase coverage by 0.03%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master     #660      +/-   ##
==========================================
+ Coverage   93.63%   93.66%   +0.03%     
==========================================
  Files          71       72       +1     
  Lines        4757     4786      +29     
==========================================
+ Hits         4454     4483      +29     
  Misses        303      303              

Flag	Coverage Δ
unittests	93.66% <100.00%> (+0.03%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
tianshou/policy/__init__.py	100.00% <100.00%> (ø)
tianshou/policy/imitation/td3_bc.py	100.00% <100.00%> (ø)
tianshou/trainer/base.py	96.89% <100.00%> (+0.03%)	⬆️

📣 Codecov can now indicate which changes are the most critical in Pull Requests. Learn more

[nuance1979]

I tried to turn on/off with --norm-obs 1/0 and found the differences were small but the improvement was consistent:

Task	w/ norm-obs	w/o norm-obs
halfcheeta-medium-v2	5741.13	5724.41
halfcheeta-expert-v2	11788.25	11665.77
walker2d-medium-v2	4051.76	3985.59
walker2d-expert-v2	5068.15	5027.75

Also note that the policy converges quickly so the default epoch of 200 can be reduced to 100.

[Trinkle23897]

I tried to turn on/off with --norm-obs 1/0 and found the differences were small but the improvement was consistent:

Task w/ norm-obs w/o norm-obs
halfcheeta-medium-v2 5741.13 5724.41
halfcheeta-expert-v2 11788.25 11665.77
walker2d-medium-v2 4051.76 3985.59
walker2d-expert-v2 5068.15 5027.75
Also note that the policy converges quickly so the default epoch of 200 can be reduced to 100.

Why not add the result in examples/offline/README.md?