Saving inking data and restoring it

[pjcwilliamson]

Hi! I'm trying to use your InkView for a Surface Duo project, and I can't figure out how to retrieve inking data, and then redraw it (in order to make inking persist after closing and reopening the app). I can save as a bitmap, but am struggling to redraw the bitmap into the inkview. But it also feels like there should be higher quality inking data than bitmap. I'm still operating in java (sorry).

I can tell you what doesn't work for redrawing a bitmap, and it's this:

    Canvas canvas = new Canvas();
    canvas.setBitmap(bitmap);
    inkView.draw(canvas);

(But as I said, I'd prefer it be more specific data than a bitmap)

[parkerschroeder]

Hello!

This is a really great point, while saving to a bitmap is fine if you just want to save what the drawing looks like, it doesn’t help when loading and editing the data later. We encountered this same scenario when creating our note editor hero app - TwoNote which also uses the InkView.

TLDR - I’ll go over some of the complexities of saving ink at a higher quality level, but if you’re mainly interested in code, there’s some code at the bottom you can check out which allows you to edit the InkSDK to meet your needs.

To save and load notes as higher quality inking data, there are a couple of approaches. With all of these, we’re looking to convert the ink on our canvas into some other data type that can be persisted. There are a couple of common approaches to this issue.

1. Serializing the data into a String that can be easily saved (often formatted as JSON)
2. Creating a parser that converts the data into a differently structured format
   a. Converting to an open standard type like InkML is going to be more complex, but may have advantages like being supported by web pages or other apps

Regardless of the persistence type (e.g. saving in a file, saving in a Room database, etc.), we can often only save simple data types. Anything with references to a Context, Activity, or a class with specific references to the application instance are difficult to persist without causing memory leaks. Additionally, any interfaces or abstract classes with extensible properties require additional overhead to serialize.

For these reasons, the code examples I’m mentioning won’t include serializing DynamicPaintHandlers. It’s not impossible but complicates the explanation of saving and loading data.

Alright, the first step in saving ink data is deciding what we want to preserve when we save the ink. For this example, we’ll save the following for each stroke:

• Color
• Width
• Pen info (coordinates, tilt, etc) within each stroke

We can package this into a single class called a Brush (I’ll be doing the code in Kotlin to align with the project, but the same concepts apply to Java)

@kotlinx.serialization.Serializable
data class Brush (
    val color: Int,
    val strokeWidth: Float,
    val strokeWidthMax: Float,
    val stroke: InputManager.ExtendedStroke
)

Notice that by marking it as Serializable, we can easily convert it to JSON in Kotlin. Java has many tools that can be used for Serialization, one of which is Gson.

The returned value from the saveInk() function can be saved in whatever persistence medium you choose.

fun saveInk(): String {
    return Json.encodeToString(brushList)
}

To load the canvas with a serialized ink string, we simply need to deserialize the data, iterate through all the brush strokes, and draw them on the canvas!

fun loadInk(serializedBrushes: String) {
    drawCanvas.drawColor(Color.TRANSPARENT, PorterDuff.Mode.CLEAR)
    strokeList.clear()
    inputManager.currentStroke.reset()
    brushList.clear()

    val brushes: List<Brush> = Json.decodeFromString(serializedBrushes)

    for (brush in brushes) {
        brush.stroke.lastPointReferenced = 0
        strokeList.add(brush.stroke)
        brushList.add(brush)

        color = brush.color
        strokeWidth = brush.strokeWidth
        strokeWidthMax = brush.strokeWidthMax
        dynamicPaintHandler = brush.paintHandler
        inputManager.currentStroke = brush.stroke

        drawStroke()
    }
    redrawTexture()
}

There are a couple of small other configurations needed to get this working, but overall, that’s the main idea! I created a branch that you can check out for an example on how to load/save using the InkSDK sample. Also definitely check out our note taking hero app sample TwoNote. It also loads and saves serialized ink data using the InkSDK (although the implementation is a bit more complex).

Hope this helps!

[pjcwilliamson]

Thank you so much for your comprehensive reply. This'll help me a lot. I'll seek to implement this and see how it goes, but on my first look, it makes sense.

What this raises for me is a question of the right ways to interact with APIs such as yours. Forgive me if this is super basic stuff. I'm a hobbyist who's very much just figuring things out as I go along. I see that most of your code above is for the InkView.kt class, that is, it is editing the InkView itself (rather than interacting with an instance of an InkView). It seems to me that there is no way to save and load complex ink data just with the InkView class you provide. I originally tried to extend the InkView class to add some code similar (ish) to what you suggest above, but your InkView class cannot be extended (the default in Kotlin, right?).

So what would be the normal way for me to implement code such as yours to the InkView class directly? This may be more of a question of how to do open sourcing and using Github together. Like, should I just copy all Microsoft's code for InkView and just create an independent InkView class in my project which I can edit? Or should I do it a different way that remains dependent on your repository?

[parkerschroeder]

You're right, with the current version of the InkView class, there's no way to save and load complex ink by simply importing it.

Some modification is required on InkView to get this working, the easiest way to do this is to either copy both the InkView and InputManager classes into your project or paste in the entire :ink module like we do in TwoNote.

As you may have noticed, the downside to doing this is that you'll no longer be directly linked to our Github library.

Kotlin is unique in the fact that it declares classes as final by default, which explains why you are not able to extend the InkView class.

That being said, we now have a branch with most of this code working 🤔 We may merge the changes into main so others can access the things we've been talking about without having to edit everything.

[parkerschroeder]

We just merged in a change to address saving and loading ink data without users having to manually copy code and change it themselves. If you're interested, feel free to check it out by updating the version number in gradle!

    implementation "com.microsoft.device:ink:1.0.0-alpha5"

[pjcwilliamson]

Thanks Parker.

I am stuck on implementing InkView again, and I think I'm going to revert back to a custommade PaintView because I could get that working. But I wanted to reach out and give it one more chance.

Basically, I'm trying to implement an InkView inside a ViewPager. I know ViewPager is being deprecated in favor of ViewPager2, but I found ViewPager2 was much less reliable. So, you can ink on different InkViews in Fragments which are in a ViewPager which uses the FragmentStatePagerAdapter. In the Fragment, in order to load some saved ink onto the InkView, I have:

@Override
public View onCreateView(LayoutInflater inflater, ViewGroup container,
                         Bundle savedInstanceState) {
    // Inflate the layout for this fragment
    View v = inflater.inflate(R.layout.page, container, false);

    inkView = v.findViewById(R.id.inkView);
    txtPage = v.findViewById(R.id.txtPage);

    txtPage.setText(mName);

    PaintHandler paintHandler = new PaintHandler(getContext(), inkView);

    inkView.setDynamicPaintHandler(paintHandler);

    if (null != brushes)    {
        inkView.loadInk(brushes);
    }
    return v;
}

Where brushes is a List<InkViewPlus.Brush>.

But it crashes every time it gets to inkView.loadInk(brushes) and the error seems to be in
fun loadInk(brushes: List)

on the line:
drawCanvas.drawColor(Color.TRANSPARENT, PorterDuff.Mode.CLEAR)

Because it says that drawCanvas as not been initialized yet. I don't have this problem when not using a ViewPager. drawCanvas seems to be initialized in fun onSizeChanged(), but in the viewpager that seems not to have been called yet. So it just crashes.

I've tried a number of things, including editing InkView to call onSizeChanged at the start. Trying to loadInk with OnResume in the fragment. I'm pretty stuck.

Do you know why I would have so much trouble loading ink in an InkView which is in a fragment which is in a ViewPager?

[parkerschroeder]

Hi, thanks for reaching out again!
 
Unfortunately, we haven’t run into this issue before, it’s definitely an interesting use case that seems to be causing it.

At first glance it sounds like the loadInk() method is being called before the InkView has been fully rendered on the screen. It’s interesting that the drawCanvas still isn’t initialized by the time onResume() is triggered.

If we find a solution to the initialization issues you’re facing, we’ll push them up as a new library version. In the meantime, if you’ve hit a wall with debugging, continuing with a custom PaintView or other similar canvas is a good route to explore.
 
Best of luck and happy coding!