Move from skimage to opencv

[Seon82]

Optimizing the transform function as much as possible could be quite interesting for future image sequence conversions.

A bit of profiling seems to reveal that calls to skimage functions are the major bottleneck (most notably equalize_adapthist, resize, median, and rgb<->hsv conversions, which account for ~80% of the time spent in transform when boost=True).
These functions all have heavily optimized cv2 equivalents, maybe we could speed up the code by using them instead?

Timer unit: 1e-06 s

Total time: 0.374083 s
File: <ipython-input-6-3ea2832d4ec7>
Function: transform at line 315

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   315                                               def transform(self, X, y=None):
   316                                                   """Transform image to pyxelated version"""
   317         1         26.0     26.0      0.0          assert self.is_fitted, "Call 'fit(image_as_numpy)' first before calling 'transform(image_as_numpy)'!"
   318         1          4.0      4.0      0.0          h, w, d = X.shape
   319         1          3.0      3.0      0.0          if self.find_palette:
   320         1          3.0      3.0      0.0              assert h * w > self.palette, "Too many colors for such a small image! Use a larger image or a smaller palette."
   321                                                   else:
   322                                                       assert h * w > len(self.palette), "Too many colors for such a small image! Use a larger image or a smaller palette."
   323                                                   
   324         1          8.0      8.0      0.0          new_h, new_w = self._get_size(h, w)  # get desired size depending on settings
   325         1          3.0      3.0      0.0          if d > 3:
   326                                                       # image has alpha channel
   327                                                       X_ = self._dilate(X)
   328                                                       alpha_mask = resize(X_[:, :, 3], (new_h, new_w), anti_aliasing=True)
   329                                                   else:
   330                                                       # image has no alpha channel
   331         1          2.0      2.0      0.0              X_ = X
   332         1          2.0      2.0      0.0              alpha_mask = None
   333         1          3.0      3.0      0.0          if self.depth:
   334                                                       # change size depending on the number of iterations
   335         1          5.0      5.0      0.0              new_h, new_w = new_h * (self.sobel ** self.depth), new_w * (self.sobel ** self.depth)
   336         1      49371.0  49371.0     13.2          X_ = resize(X_[:, :, :3], (new_h, new_w), anti_aliasing=True)  # colors are now 0. - 1.        
   337                                                   
   338         1          5.0      5.0      0.0          if self.boost:
   339                                                       # adjust contrast
   340         1     113935.0 113935.0     30.5              X_ = rgb2hsv(equalize_adapthist(X_))
   341         1       1638.0   1638.0      0.4              X_[:, :, 1:] *= self.HIST_BRIGHTNESS
   342         1      45119.0  45119.0     12.1              X_ = hsv2rgb(np.clip(X_, 0., 1.))
   343                                                   
   344                                                   # pyxelate iteratively
   345         2          8.0      4.0      0.0          for _ in range(self.depth):
   346         1          2.0      2.0      0.0              if self.boost and d == 3:
   347                                                           # remove noise
   348         1      78951.0  78951.0     21.1                  X_ = self._median(X_)
   349         1      16055.0  16055.0      4.3              X_ = self._pyxelate(X_)  # downsample in each iteration
   350                                                       
   351         1          3.0      3.0      0.0          final_h, final_w, _ = X_.shape
   352         1          2.0      2.0      0.0          if self.find_palette:
   353         1         63.0     63.0      0.0              X_ = ((X_ - .5) * self.SCALE_RGB) + .5  # values were already altered before in .fit()
   354         1          9.0      9.0      0.0          reshaped = np.reshape(X_, (final_h * final_w, 3))
   355                                                       
   356                                                   # add dithering
   357         1          2.0      2.0      0.0          if self.dither is None or self.dither == "none":
   358                                                       probs = self.model.predict(reshaped)
   359                                                       X_ = self.colors[probs]
   360         1          1.0      1.0      0.0          elif self.dither == "naive":
   361                                                       # pyxelate dithering based on BGM probability density
   362         1       4953.0   4953.0      1.3              probs = self.model.predict_proba(reshaped)
   363         1         92.0     92.0      0.0              p = np.argmax(probs, axis=1)
   364         1       1055.0   1055.0      0.3              X_ = self.colors[p]
   365         1         86.0     86.0      0.0              probs[np.arange(len(p)), p] = 0
   366         1        116.0    116.0      0.0              p2 = np.argmax(probs, axis=1)  # second best
   367         1        517.0    517.0      0.1              v1 = np.max(probs, axis=1) > (1.  / (len(self.colors) + 1))
   368         1        612.0    612.0      0.2              v2 = np.max(probs, axis=1) > (1.  / (len(self.colors) * self.DITHER_NAIVE_BOOST + 1))
   369         1          2.0      2.0      0.0              pad = not bool(final_w % 2)
   370      8763      10951.0      1.2      2.9              for i in range(0, len(X_), 2):
   371      8762      11332.0      1.3      3.0                  m = (i // final_w) % 2
   372      8762      10834.0      1.2      2.9                  if pad:
   373                                                               i += m
   374      8762      10942.0      1.2      2.9                  if m:
   375      4312       6475.0      1.5      1.7                      if v1[i]:
   376       862       2319.0      2.7      0.6                          X_[i] = self.colors[p2[i]]
   377      4450       5665.0      1.3      1.5                  elif v2[i]:
   378      1065       2790.0      2.6      0.7                      X_[i] = self.colors[p2[i]]
   379                                                   elif self.dither == "bayer":
   380                                                       # Bayer-like dithering
   381                                                       self._warn_on_dither_with_alpha(d)
   382                                                       probs = self.model.predict_proba(reshaped)
   383                                                       probs = [convolve(probs[:, i].reshape((final_h, final_w)), self.DITHER_BAYER_MATRIX, mode="reflect") for i in range(len(self.colors))]
   384                                                       probs = np.argmin(probs, axis=0)
   385                                                       X_ = self.colors[probs]
   386                                                   elif self.dither == "floyd":
   387                                                       # Floyd-Steinberg-like algorithm
   388                                                       self._warn_on_dither_with_alpha(d)
   389                                                       X_ = self._dither_floyd(reshaped, (final_h, final_w))
   390                                                   elif self.dither == "atkinson":
   391                                                       # Atkinson-like algorithm
   392                                                       self._warn_on_dither_with_alpha(d)
   393                                                       res = np.zeros((final_h + 2, final_w + 3), dtype=int)
   394                                                       X_ = np.pad(X_, ((0, 2), (1, 2), (0, 0)), "reflect")
   395                                                       for y in range(final_h):
   396                                                           for x in range(1, final_w+1):
   397                                                               pred = self.model.predict_proba(X_[y, x, :3].reshape(-1, 3))
   398                                                               res[y, x] = np.argmax(pred)
   399                                                               quant_error = (X_[y, x, :3] - self.model.means_[res[y, x]]) / 8.
   400                                                               X_[y, x+1, :3] += quant_error
   401                                                               X_[y, x+2, :3] += quant_error
   402                                                               X_[y+1, x-1, :3] += quant_error
   403                                                               X_[y+1, x, :3] += quant_error
   404                                                               X_[y+1, x+1, :3] += quant_error
   405                                                               X_[y+2, x, :3] += quant_error
   406                                                       # fix edges
   407                                                       res = res[:final_h, 1:final_w+1]
   408                                                       X_ = self.colors[res.reshape(final_h * final_w)]
   409                                                   
   410         1         14.0     14.0      0.0          X_ = np.reshape(X_, (final_h, final_w, 3))  # reshape to actual image dimensions
   411         1          1.0      1.0      0.0          if alpha_mask is not None:
   412                                                       # attach lost alpha layer
   413                                                       alpha_mask[alpha_mask >= self.alpha] = 255
   414                                                       alpha_mask[alpha_mask < self.alpha] = 0
   415                                                       X_ = np.dstack((X_[:, :, :3], alpha_mask.astype(int)))
   416                                                   
   417                                                   # return upscaled image
   418         1         88.0     88.0      0.0          X_ = np.repeat(np.repeat(X_, self.upscale[0], axis=0), self.upscale[1], axis=1)
   419         1         16.0     16.0      0.0          return X_.astype(np.uint8)

[sedthh]

Thanks for the suggestion. It's more of a personal choice: I personally prefer scikit-image over OpenCV for Python.
Scikit-image is much more flexible and has a similar thought process behind it to scikit-learn (it also plays nice with SciPy and other libs for DS).
Whereas the OpenCV wrapper is more of a black box with an un-pythonic API, with BGR color representation.

In my experience, even if some of the functions are faster in OpenCV, it lacks the flexibility and the regular updates for doing more complex image manipulation.

[sedthh]

But yeah, maybe it's worth switching a few of those bottleneck functions to OpenCV.