Consider raising default memory limit for the operator

[david-kow]

ECK memory usage scales not only with the number of resources it manages, but also with the size of resources in the K8s cluster in general. This is because we are not able to filter watched resources by labels - client caches include all Pods, Secrets and any other resources of the same type that ECK watches. In #2981 we can see that caused the operator pod to be OOMKilled.

As the memory limit is fairly low right now (150Mi) and seems insignificant in comparison with memory requirements of most ES clusters, we should consider raising it to allow larger clusters to work out of the box.

[sebgl]

It's a bit arbitrary, but based on users feedback 512Mi seems to be a fair memory limit.

[barkbay]

It's a bit arbitrary, but based on users feedback 512Mi seems to be a fair memory limit.

👍 Also in order to be consistent maybe increase the request ? Half of the limit looks reasonable I guess.

[sebgl]

👍 Also in order to be consistent maybe increase the request ? Half of the limit looks reasonable I guess.

We could do 150Mi for the memory requests?