Add evaluation environment specs to dataset metadata #155

rodrigodelazcano · 2023-10-22T04:40:02Z

Description

NOTE: this PR might have conflicts with this one: #137

Some datasets can be targeted for evaluation in an environment with different specifications than the environment used to collect the episodes of the dataset. This is the case of the Maze environments in D4RL (Ant/Point).

This PR adds an additional and optional metadata key to store separate specifications for the evaluation environment, eval_env_spec. Main changes are:

To add the environment specifications to a dataset: eval_env is an argument that can be passed during dataset creation (collector env and buffer) as a str dataset id, gym.Env or EnvSpec. eval_env will also be used to compute ref_min_score or ref_max_score when required.
Reduce redundant code: the metadata generation and the creation of the dataset path in both create_dataset_from_buffers() and create_dataset_from_collector_env() have been merged into two separate functions: _generate_dataset_path() and _generate_dataset_metadata().
Recover evaluation environment: an additional boolean argument has been added to the recover_environment() function in every MinariDataset, eval_env. If eval_env is set to True and the eval_env_spec metadata key is present in the dataset, a gym.Env instance of the evaluation environment will be returned. Otherwise, the environment used for collecting the dataset is returned.
Add pytest to check evaluatoin environment recovery and creation of datasets by passing eval_env with the different allowed types.

Dataset updates

In this PR I'm also including the AntMaze datasets and updating PointMaze to include eval_env (these datasets have been already uploaded to GCP). The scripts will be updated to add the evaluation environments in both sets of datasets in the following PR's:

AntMaze - rodrigodelazcano/d4rl-minari-dataset-generation#4
PointMaze - rodrigodelazcano/d4rl-minari-dataset-generation#3

Doc updates

Aside from adding the documentation for AntMaze datasets, this PR expands the documentation of each individual dataset by including in the table all of the specifications of a Gymnasium environment , another table for the evaluation environment specs, and notes to explain how to recover the environments. All of these docs are generated automatically. Example of the docs environment specs and note if no evaluation environment is found for the dataset:

Checklist:

I have run the pre-commit checks with pre-commit run --all-files (see CONTRIBUTING.md instructions to set it up)
I have run pytest -v and no errors are present.
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I solved any possible warnings that pytest -v has generated that are related to my code to the best of my knowledge.
I have added tests that prove my fix is effective or that my feature works
New and existing unit tests pass locally with my changes

younik

Looks good @rodrigodelazcano, thank you also for refactoring the utils file!
I just added few minors comments

younik · 2023-10-24T20:57:15Z

docs/_scripts/gen_dataset_md.py

+        env_spec = EnvSpec.from_json(dataset_spec["env_spec"])
+        env = gym.make(env_spec.id)


Can we support env_spec = None here? You can do a similar workflow you did for eval_env_spec

Yes we can. Will it be more helpful to wait for this PR #137 to be merged and then I can try to rebase the functionality in this one?

Wait, never mind, this is for the docs only. I'll start working on the env_spec=None case as well

younik · 2023-10-24T21:09:03Z

docs/_scripts/gen_dataset_md.py

-|ID| `{env_id}`|
-| Action Space | `{re.sub(' +', ' ', action_space_table)}` |
+|ID| `{env_spec.id}`|
 | Observation Space | `{re.sub(' +', ' ', observation_space_table)}` |
-
+| Action Space | `{re.sub(' +', ' ', action_space_table)}` |
+| entry_point | `{env_spec.entry_point}` |
+| max_episode_steps | `{env_spec.max_episode_steps}` |
+| reward_threshold | `{env_spec.reward_threshold}` |
+| nondeterministic | `{env_spec.nondeterministic}` |
+| order_enforce    | `{env_spec.order_enforce}`|
+| autoreset        | `{env_spec.autoreset}` |
+| disable_env_checker | `{env_spec.disable_env_checker}` |
+| kwargs | `{env_spec.kwargs}` |
+| additional_wrappers | `{env_spec.additional_wrappers}` |
+| vector_entry_point | `{env_spec.vector_entry_point}` |


we can probably factorize this to a function describe_env, to use for env_spec and eval_env_spec. This can also handle the None case

Great idea! I'll refactorize this part

minari/dataset/minari_dataset.py

younik · 2023-10-24T21:14:27Z

minari/dataset/minari_dataset.py

+        if eval_env and self._eval_env_spec is not None:
+            return gym.make(self._eval_env_spec)


I would add a message to the logger (INFO level?) in case eval_env and _eval_env_spec=None.

Yeap, agree. I will add this.

tests/utils/test_dataset_creation.py

minari/utils.py

rodrigodelazcano · 2023-10-27T16:13:54Z

I've removed flake8:E272 in gen_dataset_md.py in the pre-commit.

younik

LGTM; I think there are just two typos, then we can merge

docs/_scripts/gen_dataset_md.py

rodrigodelazcano requested a review from younik October 22, 2023 04:40

rodrigodelazcano mentioned this pull request Oct 22, 2023

[Proposal] Porting Antmazes to Minari. #152

Closed

younik reviewed Oct 24, 2023

View reviewed changes

rodrigodelazcano added 21 commits October 25, 2023 12:49

create dataset with eval_env

c42ac43

recover eval env

4ca95fd

fix doc

700f77e

add recover eval env tests

8548920

accept string id and env spec in create dataset from buffers

c5b47a0

test for different types of env and eval_env

69ae531

eval env docs

3fed0ef

update daset docs with eval env

af5bad3

remove datasets docs

782fd4b

href code permalink

3014578

initial main doc antmaze

63ed294

antmaze initial docs

7bacdef

eval-env create from collector env

dd64943

merge create from buffer and collector

9831215

update dataset docs

6cf47f6

restructure utils

4782bb1

update codelinks tests

b4eb64d

add logger info

d2f3b3a

pre-commit

372bea7

refactor env docs

e9d8082

revert dataset docs

069b635

rodrigodelazcano force-pushed the eval-env branch from 44ae214 to 069b635 Compare October 27, 2023 05:13

rodrigodelazcano added 3 commits October 27, 2023 11:29

possible pre-commit fix

dbd2b3c

indent text

b14c9a8

remove flake8:E272

514bc6b

younik reviewed Oct 27, 2023

View reviewed changes

docs/_scripts/gen_dataset_md.py Outdated Show resolved Hide resolved

docs/_scripts/gen_dataset_md.py Outdated Show resolved Hide resolved

through -> throw

063cf65

rodrigodelazcano force-pushed the eval-env branch from 9f0356b to 063cf65 Compare October 29, 2023 15:42

Merge branch 'main' into eval-env

656904b

younik approved these changes Oct 29, 2023

View reviewed changes

younik merged commit 98016ac into Farama-Foundation:main Oct 29, 2023
6 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add evaluation environment specs to dataset metadata #155

Add evaluation environment specs to dataset metadata #155

rodrigodelazcano commented Oct 22, 2023

younik left a comment

younik Oct 24, 2023

rodrigodelazcano Oct 25, 2023

rodrigodelazcano Oct 25, 2023

younik Oct 24, 2023

rodrigodelazcano Oct 25, 2023

younik Oct 24, 2023

rodrigodelazcano Oct 25, 2023

rodrigodelazcano commented Oct 27, 2023

younik left a comment

		env_spec = EnvSpec.from_json(dataset_spec["env_spec"])
		env = gym.make(env_spec.id)

		if eval_env and self._eval_env_spec is not None:
		return gym.make(self._eval_env_spec)

Add evaluation environment specs to dataset metadata #155

Add evaluation environment specs to dataset metadata #155

Conversation

rodrigodelazcano commented Oct 22, 2023

Description

Dataset updates

Doc updates

Checklist:

younik left a comment

Choose a reason for hiding this comment

younik Oct 24, 2023

Choose a reason for hiding this comment

rodrigodelazcano Oct 25, 2023

Choose a reason for hiding this comment

rodrigodelazcano Oct 25, 2023

Choose a reason for hiding this comment

younik Oct 24, 2023

Choose a reason for hiding this comment

rodrigodelazcano Oct 25, 2023

Choose a reason for hiding this comment

younik Oct 24, 2023

Choose a reason for hiding this comment

rodrigodelazcano Oct 25, 2023

Choose a reason for hiding this comment

rodrigodelazcano commented Oct 27, 2023

younik left a comment

Choose a reason for hiding this comment