How to dynamically create movable Images

[alarm109]

Goal

I want to press add button, that will add a selected Image to the Canvas, that could be moved with GestureHandler. Element tracking works for my case, but I want to be able to add as many items on button press.

Problem

1. If I try to add GestureHandler to Canvas I get this error: Error: GestureDetector must be used as a descendant of GestureHandlerRootView. Otherwise the gestures will not be recognized.
2. If I create multiple MovableElement(see code below), I also create multiple canvas, which is not ideal, because I want to have one Canvas with all the Image

Code

import { Canvas, rect, Skia } from '@shopify/react-native-skia';
import { StyleSheet, View } from 'react-native';
import { useSharedValue } from 'react-native-reanimated';
import { HeartSticker } from '@assets/stickers';

import { GestureHandler } from './GestureHandler';
import { Image } from './Image';

const WIDTH = 300;
export const dimensions = rect(0, 0, WIDTH, WIDTH);

export const MovableElement = () => {
  const matrix = useSharedValue(Skia.Matrix());

  return (
    <View style={styles.container}>
      <Canvas style={styles.container} accessibilityIgnoresInvertColors>
        <Image height={370} image={HeartSticker} matrix={matrix} width={400} />
      </Canvas>

      <GestureHandler matrix={matrix} dimensions={dimensions} debug />
    </View>
  );
};

const styles = StyleSheet.create({
  container: {
    flex: 1,
  },
});

[laxman-stan]

It's possible to utilize multiple instances of MovableElement within a single canvas, as illustrated by @wcandillon in this example: link.

Alternatively, you can accomplish a similar effect using only one GestureDetector element. In this case, during the onStart event, you would check the ID of the selected image. This event provides the touch position within the GestureDetector, which you can then use to determine the specific image that needs to be moved.

[alarm109]

I think, I'm trying to achieve similar effect as you suggest in your alternative. But the problem is, that when I have an array of useSharedValue, the object doesn't move when it is updated in onChange.

When I set useSharedValue<SkMatrix>(Skia.Matrix()) it works fine. But I need to create a number of objects that is equal to the number of elements. For this reason I need to save the SkMatrix values for all the elements. And then move one dependant where the user drags. For debugging purpose right now I'm trying just to dynamically create new elements equal to the count of elements and then move the first one.

Do you have any suggestions how can I achieve this functionality?

const CanvasComponent = ({ elements }: Props) => {
  const matrixArray = useSharedValue<SkMatrix[]>([]);

  useEffect(() => {
    matrixArray.value = [...matrixArray.value, Skia.Matrix()];
  }, [elements, matrixArray]);

  const gesture = Gesture.Pan()
    .onStart(e => {
      console.log({ startX: e.x, startY: e.y });
    })
    .onChange(e => {
      const matrix = matrixArray.value[0];

      if (!matrix || matrix.translate === undefined) {
        return;
      }

      matrixArray.value[0] = matrix.translate(e.changeX, e.changeY);
    });

  return (
    <GestureDetector gesture={gesture}>
      <CanvasContainer>
        {matrixArray.value.length > 0 &&
          matrixArray.value.map((matrix, index) => (
            <MovableElement matrix={matrix} key={`key-${index}`} />
          ))}
      </CanvasContainer>
    </GestureDetector>
  );
};

const CanvasContainer = styled(Canvas)`
  flex: 1;
  background-color: cornflowerblue;
`;

[laxman-stan]

try this

const CanvasComponent = ({ elements }: Props) => {
  const matrixArray = useSharedValue<SkMatrix[]>([]);

  useEffect(() => {
    matrixArray.value = [...matrixArray.value, Skia.Matrix()];
  }, [elements, matrixArray]);

  const gesture = Gesture.Pan()
    .onStart(e => {
      console.log({ startX: e.x, startY: e.y });
    })
    .onChange(e => {
      const matrix = matrixArray.value[0];

      if (!matrix || matrix.translate === undefined) {
        return;
      }

      matrixArray.value[0] = matrix.translate(e.changeX, e.changeY);
      matrixArray.value = [...matrixArray.value];
    });

  return (
    <GestureDetector gesture={gesture}>
      <CanvasContainer>
        {matrixArray.value.length > 0 &&
          matrixArray.value.map((matrix, index) => {

            let mat = useDerivedValue(()=>{
              return matrixArray.value[index]
            }, [matrixArray])

            return <MovableElement matrix={mat} key={`key-${index}`} />
          })}
      </CanvasContainer>
    </GestureDetector>
  );
};

const CanvasContainer = styled(Canvas)`
  flex: 1;
  background-color: cornflowerblue;
`;

I've made two changes

1. updated matrixArray.value with new refrence like this matrixArray.value = [...matrixArray.value];
2. used a derived value in matrix prop

[tylerrrkd]

@wcandillon thank you soooo much, I am trying to use zustand to manage matrix value which stored by useSharedValue in the example. And it seems work now, but still need to be optimized.

[tylerrrkd]

@wcandillon Hiii friend, after using zustand to store matrix value, it works well with runOnJS mode, but if I wanna recognize the layer with toM4 transform, it will occur the problem like following.

And this is my code, copy from Stickers only add runOnJS chain with gesture in file GestureHelper, and remove worklet in file MatrixHelper

GestureHelper.ts

import type { SkMatrix, SkRect } from '@shopify/react-native-skia'
import { Skia } from '@shopify/react-native-skia'
import React from 'react'
import { Gesture, GestureDetector } from 'react-native-gesture-handler'
import Animated, { useAnimatedStyle, useSharedValue } from 'react-native-reanimated'

import { rotateZ, toM4, translate, scale } from './MatrixHelpers'

interface GestureHandlerProps {
  matrix: SkMatrix
  dimensions: SkRect
  debug?: boolean
  onMatrixChange?: (matrix: SkMatrix) => void
}

export const GestureHandler = ({
  matrix,
  dimensions,
  debug,
  onMatrixChange,
}: GestureHandlerProps) => {
  const { x, y, width, height } = dimensions
  const origin = useSharedValue(Skia.Point(0, 0))
  const offset = useSharedValue(Skia.Matrix())

  const pan = Gesture.Pan().runOnJS(true).onChange((e) => {
    onMatrixChange?.(translate(matrix, e.changeX, e.changeY))
  })

  const rotate = Gesture.Rotation().runOnJS(true)
    .onBegin((e) => {
      origin.value = Skia.Point(e.anchorX, e.anchorY)
      offset.value = matrix
    })
    .onChange((e) => {
      onMatrixChange?.(rotateZ(offset.value, e.rotation, origin.value))
    })

  const pinch = Gesture.Pinch().runOnJS(true)
    .onBegin((e) => {
      origin.value = Skia.Point(e.focalX, e.focalY)
      offset.value = matrix
    })
    .onChange((e) => {
      onMatrixChange?.(scale(offset.value, e.scale, origin.value))
    })

  const style = useAnimatedStyle(() => {
    return {
      position: 'absolute',
      left: x,
      top: y,
      width,
      height,
      backgroundColor: debug ? 'rgba(100, 200, 300, 0.4)' : 'transparent',
      transform: [
        { translateX: -width / 2 },
        { translateY: -height / 2 },
        {
          matrix: toM4(matrix),
        },
        { translateX: width / 2 },
        { translateY: height / 2 },
      ],
    }
  })
  const gesture = Gesture.Race(pinch, rotate, pan)
  return (
    <GestureDetector gesture={gesture}>
      <Animated.View style={style} />
    </GestureDetector>
  )
}

MatrixHelper.ts

import type { SkMatrix, Vector } from '@shopify/react-native-skia'
import { Skia, MatrixIndex } from '@shopify/react-native-skia'

export const scale = (matrix: SkMatrix, s: number, origin: Vector) => {
  const source = Skia.Matrix(matrix.get())
  source.translate(origin.x, origin.y)
  source.scale(s, s)
  source.translate(-origin.x, -origin.y)
  return source
}

export const rotateZ = (matrix: SkMatrix, theta: number, origin: Vector) => {
  const source = Skia.Matrix(matrix.get())
  source.translate(origin.x, origin.y)
  source.rotate(theta)
  source.translate(-origin.x, -origin.y)
  return source
}

export const translate = (matrix: SkMatrix, x: number, y: number) => {
  const m = Skia.Matrix()
  m.translate(x, y)
  m.concat(matrix)
  return m
}

export const toM4 = (m3: SkMatrix) => {
  const m = m3.get()
  const tx = m[MatrixIndex.TransX]!
  const ty = m[MatrixIndex.TransY]!
  const sx = m[MatrixIndex.ScaleX]!
  const sy = m[MatrixIndex.ScaleY]!
  const skewX = m[MatrixIndex.SkewX]!
  const skewY = m[MatrixIndex.SkewY]!
  const persp0 = m[MatrixIndex.Persp0]!
  const persp1 = m[MatrixIndex.Persp1]!
  const persp2 = m[MatrixIndex.Persp2]!
  return [sx, skewY, persp0, 0, skewX, sy, persp1, 0, 0, 0, 1, 0, tx, ty, persp2, 1]
}

[tylerrrkd]

@wcandillon I've solved the problem now. I was mixed up the UI & JS thread. Cause the value of GestureHandlerProps['matrix'] has been stored in zustand, I should transform the matrix in JS thread.

I changed the core transform code like below.

// ...
  const m4Matrix = useMemo(() => toM4(matrix), [matrix])

  const style = useAnimatedStyle(() => {
    return {
      position: 'absolute',
      left: x,
      top: y,
      width,
      height,
      backgroundColor: debug ? 'rgba(100, 200, 300, 0.2)' : 'transparent',
      transform: [
        { translateX: -width / 2 },
        { translateY: -height / 2 },
        {
          matrix: m4Matrix,
        },
        { translateX: width / 2 },
        { translateY: height / 2 },
      ],
    }
  }, [matrix])
// ...

[wcandillon]

this is how I updated the demo to run with dynamic images: https://gist.github.com/wcandillon/4ade39497f96a4e2745b57b0cb071d8c

useSharedValue is just makeMutable + memoization + clean up (cc @dmg46664).

[alarm109]

Great news! I will try this and then get back to you. 🚀

[gwendall]

How would you handle a dynamic stickers list? Can't get it to work with something like this:

https://gist.github.com/gwendall/219728ae019cc10a03b3a408363d955c

[tylerrrkd]

@alarm109 how about it now?

[anovius]

@alarm109 @wcandillon @tylerrrkd I have solved this issue you can add unlimited images dynamically I faced this problem earlier and opened a discussion but now I found a solution and it's working fine for me as I have a similar case where I have to add dynamic text on an image on button press

Find the code snippets here
#1752 (comment)

[tylerrrkd]

Hi friend, thanks for your reply. And congrats you make it. I used the other way to solve this problem and trying to adjust my code like @wcandillon did https://gist.github.com/wcandillon/4ade39497f96a4e2745b57b0cb071d8c, maybe performance improved.

[anovius]

Can you share your code and how you made it possible with this mentioned solution by @wcandillon with dynamic array (state)?

[VictorioMolina]

How to do this with Gesture.Simultaneous?