Truncated Normal Distribution, `get_neighbours` and `to_uniform`

[eddiebergman]

PR #188 introduce a Normal Truncated Distribution type but there are perhaps some changes required to the get_neighbours and to_uniform functions.

Quoting the major response but please read the PR for more details.

• get_neighbors - Seems to n parameter values that are neighbors to the current value, by some definition of neighbor or close.
  • This function for NormalFloatHypereparameter as it stands has an odd definition of neighbors, it just creates a normal distribution centered on whatever value is passed and then draws n neighbors from that distribution.
    neighbours = [ rs.normal(value, self.sigma) for _ in range(n) ].
  • I'm not sure of my own definition but to me it would make more sense to have this sample from some reduced sigma, i.e.
    neighbours = [ rs.normal(value, self.sigma / 3) for _ in range(n) ], where 3 is arbitrary and a more reasonable value should be used.
  • What would your definition of a neighbor be?
  • The bounding method you use creates a non-normal distribution. For example, consider the case where lower and upper are close to value (mean) but where self.sigma is quite wide. This bounding would result in a large amount of neighbors with the same value as lower or upper. I don't really know how to tackle this to be honest. Would users still expect neighbors to be normally distributed within these bounds or be aware of the fact that with tight bounds, most of the neighbors will be on the boundary.

              new_value = rs.normal(value, self.sigma)
              if self.lower is not None and self.upper is not None:
                  new_value = min(max(new_value, self.lower), self.upper)
  

• to_uniform - Well, converts a normally distributed parameter to a uniform one.
  • You're right, NormalFloatHyperparameter.get_neighbors could return a value outside of the UniformFloatHypereparameter that is created by to_uniform. While for our own use cases, I don't think this is an issue (maybe?) I could see how this might cause unexpected behaviors.
  • I'm not sure there's a fully sound solution to this but please correct me and suggest if I am wrong here.
    • A normal distribution is technically unbounded in the values it could return. (without it being specified by params as you did)
    • A uniform distribution is bounded (or at least in how we have it implemeneted)
    • There is no fully safe way to specify a boundary on the normal distribution and so how can it be set for the uniform distribution.
    • Without a formally correct way then in practice we have to pick some value. I believe mu +- 3*sigma is reasonable but if you have other suggestions, do let us know.

[eddiebergman]

Closed by #346