Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Capturing biometric data with the sign #213

Closed
kalihos opened this issue Jan 20, 2017 · 58 comments · Fixed by #566
Closed

Capturing biometric data with the sign #213

kalihos opened this issue Jan 20, 2017 · 58 comments · Fixed by #566

Comments

@kalihos
Copy link

kalihos commented Jan 20, 2017

Hello.

Is it possible to get biometric data associated to the sign? I say, writing speed, number of lines, etc. I need to be sure that the user doesn't reject the authenticity of his own sign for legal purposes. This component is the only one I've seen that writes a smooth sign, so I suppose that data is getting captured. The end goal is getting the end user sign + biometric information associated to that sign + legal third-party timestamp; then, encoding the whole pack, and send to the database. Is it possible anyway?

Thanks a lot.

@szimek
Copy link
Owner

szimek commented Jan 21, 2017

It depends on what data you exactly have in mind, but I think it would be possible. There's ISO/IEC 19794-7:2014 that "specifies data interchange formats for signature/sign behavioural data captured in the form of a multi-dimensional time series", so it would be awesome to use this standard instead of coming up with our own. Unfortunately, I don't have access to this document and don't know how the format actually looks like :/

@kalihos
Copy link
Author

kalihos commented Jan 21, 2017 via email

@szimek
Copy link
Owner

szimek commented Jan 22, 2017

@kalihos You can also try out the latest beta release (1.6.0-beta.1), which has #toData method that returns all points grouped in lines, but currently it returns only time, x and y coordinates for each point. It would be rather easy to add velocity and acceleration to it, but I don't think we can capture anything else.

You can try out this beta version on the demo page - just draw anything and then call signaturePad.toData() in dev tools console.

@kalihos
Copy link
Author

kalihos commented Jan 22, 2017 via email

@JD-Robbs
Copy link

I suppose acceleration and velocity could be inferred from the timestamps - but, for what it's worth, it would be really nice having an option to include those data points when calling toData, or to have a new method to impute them.

By the way - I'm loving the beta. Have been testing it out a bit and I look forward to being able to store the raw data in the DB in the future. Great stuff! 👍

@szimek
Copy link
Owner

szimek commented Jan 23, 2017

@JD-Robbs

By the way - I'm loving the beta.

Thanks! :)

Have been testing it out a bit and I look forward to being able to store the raw data in the DB in the future.

That's exactly why I'd prefer to use some existing standard for storing this stuff. Once people start storing it in their DB it will be hard to modify it in the next versions. Originally, it was meant to be private and used only to draw SVG and help with unit tests.

I'm just wondering what velocity and acceleration should be for the first point in each line - 0 or null/undefined?

@kalihos
Copy link
Author

kalihos commented Jan 23, 2017

I've tried your excellent beta version, and it works great! As you said, it would be fantastic to have in addition velocity and acceleration available. In the legal point, I think it's not relevant having 0 or null/undefined for the first point, because it would be only used to compare a sign with another previously saved one, so anyone can check if it's the same person who signed both times. But, technically speaking, it would maybe be better to set acceleration for first point to 0. I think it would be easier to store the full array, because a 0 is always an easier format to export to a file... otherwise we would store "null-5-3-7-2...", and it sounds a bit confusing in my opinion...

Respect to the standard, as I said, I'm trying to get detailed information about it, but the high legal differences in complexity between different countries doesn't help at all... For example, in Spain there is no legal certification about this kind of "digital-handwritten" sign, but it's accepted. But, in contrast with traditional handwritten sign or digital-certificated sign, here you are the one who need to demonstrate that your process is ok and you can ensure the authenticity of the sign, but based on a Spanish legal order (http://noticias.juridicas.com/base_datos/Admin/l59-2003.html - sorry, only Spanish).

@kalihos
Copy link
Author

kalihos commented Jan 24, 2017

Hi again!

I can't access anyway to the standard document, but I've been looking at the Spanish Law, and your currently captured data is more than ok. Only data such as pressure points and inclination are impossible to catch from a normal device, but they are not fully necessary. Anyway, velocity and acceleration would be amazing to ensure user's not rejecting his own sign. I'm starting my development integrating your component. If you add such additional data in the near future, I will try to adapt the data in my database. It's working really great...

Thanks a lot!

@szimek
Copy link
Owner

szimek commented Jan 24, 2017

I guess if a point already has velocity and acceleration, then it doesn't really need time anymore. I'll try to update it, though it might take a while. Maybe someone wants to create a PR? 😉

@JD-Robbs
Copy link

@szimek

That's exactly why I'd prefer to use some existing standard for storing this stuff. Once people start storing it in their DB it will be hard to modify it in the next versions. Originally, it was meant to be private and used only to draw SVG and help with unit tests.

That's a very good point - I haven't even thought that far with all my excitement (although I'll still be storing BLOBs, it's nice to have the raw biometric information as well). I suppose any future version will still be based on the basic information around coordinates and time - therefore, I guess, there could always be an upgrade/conversion path should things change (e.g. measures/scales). But, then again, I'm not really up-to-speed on how everything is implemented.

I'm just wondering what velocity and acceleration should be for the first point in each line - 0 or null/undefined?

I guess velocity would be 0, and acceleration the delta to the second point? Although, then, what would be the acceleration of the last point? Perhaps, both should be 0 - not sure.

I guess if a point already has velocity and acceleration, then it doesn't really need time anymore. I'll try to update it, though it might take a while. Maybe someone wants to create a PR? 😉

Alternatively, I think the toData and fromData behaviour could just remain as-is (just to keep the required data to the bare minimum).

Instead, there could, perhaps, be an additional method on the prototype which can turn the output of toData to {x: number, y: number, time: number, velocity: number, acceleration: number}[].

@kalihos
Copy link
Author

kalihos commented Jan 24, 2017 via email

@ovninosa
Copy link

ovninosa commented Feb 2, 2017

+1 on this.
This is a good info to accomplish the ISO 19794-7.

@szimek
Copy link
Owner

szimek commented Feb 2, 2017

@javenosa Do you have any info what data/format ISO 19794-7 actually specifies?

@ovninosa
Copy link

ovninosa commented Feb 2, 2017

X coord from pen (increments when moves to right)
Y coord from pen (increments when moves to top)
Z coord from pen (increment when moves the lift between the device and the pen, this doesn't apply here)
Horizontal velocity (difference between velocities)
Vertical velocity (difference between velocities)
Horizonal acceleration
Vertical acceleration
Time
Time difference
Pressure
Angle (this doesn't apply here)

With your plugin you can get the most items from this list.

https://image.slidesharecdn.com/onlinesignaturerecognition-130201011136-phpapp02/95/online-signature-recognition-16-638.jpg?cb=1359681345

@szimek
Copy link
Owner

szimek commented Feb 2, 2017

@javenosa Thanks! If there's anyone who wants to do PR to add horizontal and vertical velocities and acceleration and time difference, it would be awesome 😉 Like you said we can't really get Z coord, pressure or angle.

One difference is that in HTML canvas Y coord increases when moving to the bottom. I'm not sure if it makes sense to change it to comply with the spec, because it will be a bit annoying to use this data then to draw on a canvas and you'd probably need to know height of the canvas to know what to subtract from...

I'm also wondering if we should also have some other method (or an option for #toData) that returns pen color and max/min widths per line as well. It's not really necessary for signature biometric data, so maybe it can be added eventually later, if enough people complain about it.

@ovninosa
Copy link

ovninosa commented Feb 2, 2017

I think pressure can be get on the correct device (of course, with pressure support, iPhone 6s, iPhone 6s Plus, iPhone 7, iPhone 7 Plus, etc).

@szimek
Copy link
Owner

szimek commented Feb 2, 2017

I had no idea that it's possible to get these values in JS, that's pretty awesome.

Though looking at https://pressurejs.com source code it doesn't seem to be trivial to implement and I'm not sure it's worth the additional code, so maybe it can be postponed for now and eventually added in the next versions.

@ovninosa
Copy link

ovninosa commented Feb 2, 2017

Yep, pressurejs is one alternative (I just copy the iphone's version from their site hahah).
I understand your point, sound logic.

@kalihos
Copy link
Author

kalihos commented Feb 3, 2017

I tested pressurejs, and it seems to say always that device is not supported (I tried with normal Macbook trackpad, iPhone...). Anyway, looking at the spec information from @javenosa, I think we're are currently storing every necessary information into DB with toData method, except maybe pressure (difficult to get without a Wacom device or similar), and angle (similar to previous). I'm currently using your excellent plugin into my application, and it works fine. And, if I need later to analyze any signature, I can read biometric information and get additional data like velocity or acceleration. I think it's not necessary to store redundant information...

@szimek
Copy link
Owner

szimek commented Feb 3, 2017

@kalihos If we decide to skip the pressure for now, then you're right that all other data (i.e. velocities and accelerations) can be derived from time. Maybe for now it would be easier to just return time only and provide a script in the README file that can convert it to something that has this additional data.

I'm just wondering if we also need to return size of the canvas (or maybe better max/min X and Y coords), so that it's possible to reverse Y coords (0 at the bottom vs 0 at the top).

@kalihos
Copy link
Author

kalihos commented Feb 3, 2017

@szimek It's a good idea to provide max/min X and Y coords. But take in account that, when saving to SVG, such information could be confusing due to resizing possibility. Don't you think it would be better to store the signature only in PNG format if you want to store max X/Y?

Respect to the script in the README file, yes, it's another good idea. It would be only necessary to ensure a correct comparison for legal purposes (legal claim).

@szimek
Copy link
Owner

szimek commented Feb 3, 2017

@kalihos I was thinking about storing max/min X and Y only so that it's possible to reverse Y for biometric data. As @javenosa mentioned before, in the spec 0 is at the bottom, but in a canvas 0 is at the top. If we'd like to follow the spec, I'd need some way to "reverse" Y coords. Of course max/min X and Y can also be derived by iterating over all points returned by #toData method, so it doesn't necessarily need to be returned by #toData itself.

@kalihos
Copy link
Author

kalihos commented Feb 3, 2017

@szimek It's a good option to store max/min X and Y, so you can restore the full canvas, and you can accomplish the spec. Anyway, I don't think it's fully necessary to take the spec literally, if you can ensure the signature authenticity... but it's right to store it to be more "pure" ;)

@ovninosa
Copy link

Any update on this?
Acceleration and velocity would be an amazing feature to the "toData" method.

Thanks

@szimek
Copy link
Owner

szimek commented Feb 20, 2017

@javenosa I'm currently focusing on releasing 1.6 with SVG support, but don't have much free time, so it's progressing rather slowly.

if you want to see it in 1.6, PRs are welcome ;) Also, it should be quite easy to write a simple script that takes output from toData and adds velocities and acceleration - all you need to calculate those are time diffs, which are already returned.

@ovninosa
Copy link

I don't know how be the formulas to calculate the velocity and acceleration, if you can tell me that, I can do a PR.

@szimek
Copy link
Owner

szimek commented Feb 21, 2017

velocity = change in distance / change in time
acceleration = change in velocity / change in time

There's Point#velocityFrom(point) that will give you velocity between 2 points and once you got that you can use almost the same code to calculate acceleration:

// This one already exists
Point.prototype.velocityFrom = function (start) {
  return (this.time !== start.time) ? this.distanceTo(start) / (this.time - start.time) : 1;
};

// This one should be added
Point.prototype.accelerationFrom = function (start) {
  return (this.time !== start.time) ? this.velocityFrom(start) / (this.time - start.time) : 1;
};

Both methods should probably return 0 instead of 1 in case the times are the same, because the position should be the same as well...

@Lonzak
Copy link

Lonzak commented Jun 12, 2017

Interesting project! The ISO biometry standard is an ASN-1 encoded binary standard which we were about to implement. Luckily for us an XML version was released in 2015 which is much easier to implement.

Regarding the post from javenosa it is worth mentioning that most of the channels (they are called channels in the ISO standard) are optional. Only x,y and t are required and the rest is optional. So it may be nice to have pressure or Z-coords, angle etc. but it is not mandatory for a fully compliant biometric dataset. Additionally many other values can be calculated later from the existing x,y, and t values like (velocity, acceleration etc.). So I would start with a most basic dataset: x,y,t ...

@szimek
Copy link
Owner

szimek commented Jun 12, 2017

@Lonzak If anyone can provide info how exactly the data structure should look like compared to what it looks like now, it would be great. Because currently #toData method returns x, y coords and time, so I guess it's just a matter of transforming the current structure to whatever is expected by the ISO standard.

@mvayngrib
Copy link

@szimek awesome library, using it in a react native project and it's smooth as butter!
@kalihos do you know of a good signature recognition/matching sdk?

@kalihos
Copy link
Author

kalihos commented Sep 25, 2017 via email

@szimek
Copy link
Owner

szimek commented Dec 7, 2017

@mgierw Thanks! However, now that there's SignaturePad#toData, one could (probably) write a tiny script that takes output of this function and converts it to the format expected by this standard. It wouldn't even have to be a part of this library.

Something like:

const data = convertToISO(signaturePad.toData());

If someone comes up with such library, I'd add info about it to the README file.

@Filoz
Copy link

Filoz commented Mar 1, 2018

Hi,
this conversation is really interesting!
did someone tried to implement the ISO standard?
I would like to help

@jos1337
Copy link

jos1337 commented Jul 26, 2018

Any news on this?

@probert94
Copy link

As you were talking about speed and accelleration, what about force? There is a library called pressurejs, which allowes you to meassure the force of the touch on devices that support it. I am not in this topic, so I don't know anything about the legal part.

@szimek
Copy link
Owner

szimek commented Aug 23, 2018

@Springrbua v3 of this library supports pointer events, which can provide such info if a device provides it. So it should be possible, but I don't plan it for the first 3.0.0 release, because of lack of time and devices to test it on.

@probert94
Copy link

Thanks for the fast answer. I just discovered pressurejs lately so I was curious, if it could be used for digital signatures too.

@ntelkedzhiev
Copy link

ntelkedzhiev commented Oct 11, 2018

@szimek awesome library, using it in a react native project and it's smooth as butter!

@mvayngrib How exactly did you manage to make it work with RN?

@mvayngrib
Copy link

mvayngrib commented Oct 11, 2018

@ntelkedzhiev I'm using react-native-signature-pad which is a wrapper around this library. Or rather, I'm using a fork of that wrapper, as I wanted to get width/height back along with the data url (https://github.com/tradle/react-native-signature-pad)

@ntelkedzhiev
Copy link

ntelkedzhiev commented Oct 11, 2018

@mvayngrib Thanks for letting me know! I'm doing the same thing. I also tried updating https://github.com/kevinstumpf/react-native-signature-pad to use the new updated version of this repo but it will take more time than I have right now. I think all modules have to be put inside one file and they should be in CJS/UMD format.

@dhasselberg
Copy link

any updates on this topic?

@javyxx
Copy link

javyxx commented Jul 23, 2019

Hi, I found a biometric implementation standard.
https://github.com/biosignin/bsi-core
It is a bit old but it can be worth as a starting point

@corrrso
Copy link

corrrso commented Sep 15, 2020

Hi! This would be such a great addition to the project. Any news about it?

@bandguyz24
Copy link

bandguyz24 commented Sep 15, 2020 via email

@Bibendus83
Copy link

Bibendus83 commented Sep 16, 2020

you mean like pulse, blood pressure, pupil dialation, body temperature?

On Tue, Sep 15, 2020 at 1:24 AM corrrso @.***> wrote: Hi! This would be such a great addition to the project. Any news about it? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#213 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AK7MVVQPIVBQFFOLU3NGV4TSF4QCBANCNFSM4C5GCIDQ .
-- here's my signature

He means stroke pressure, acceleration, speed, etc. as defnied in ISO 19794-7 specifications.
This is considered biometric data useful to legally validate a signature, the signature image alone is insufficient in some countries like Italy and maybe Spain.

@javyxx
Copy link

javyxx commented Sep 16, 2020

you mean like pulse, blood pressure, pupil dialation, body temperature?

On Tue, Sep 15, 2020 at 1:24 AM corrrso @.***> wrote: Hi! This would be such a great addition to the project. Any news about it? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#213 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AK7MVVQPIVBQFFOLU3NGV4TSF4QCBANCNFSM4C5GCIDQ .
-- here's my signature

He means stroke pressure, acceleration, speed, etc. as defnied in ISO 19794-7 specifications.
This is considered biometric data useful to legally validate a signature, the signature image alone is insufficient in some countries like Italy and maybe Spain.

Yea. It's is true in spain is a requirement for advanced digital signature.

I try to do in https://github.com/javyxx/signature_pad is simple modification to save pressure

https://developer.mozilla.org/en-US/docs/Web/API/PointerEvent/pressure

@aegeavaz
Copy link

Hi...

If someone is interested, I made in my fork a basic implementation to capture pressure when available and also I implemented a function to convert the captured data to the ISO 19794-7 specification.

Any comments, tests, fixes or improvements are welcome.

@certifirm
Copy link

May be very interesting to merge with this project anyway

@certifirm
Copy link

I forked your repo to update with latest changes in szimek original repo.
I think that is interesting to add the accelerations AX and AY, that can calculate like the velocity. (Increment of the velocity / increment time)
What is your opinion?

@aegeavaz
Copy link

What is your opinion?

Hi Certifirm...

I tried to add the accelerations also, but I was having problems with the validation of the xml data results. The ISO specify a maximum value and sometimes I was having values bigger than that maximum. There is a way in the ISO that allows you to scale up and down the values, but it wasn't too much clear for me and I didn't have more time to investigate. Sorry.

@Simolation
Copy link

What is your opinion?

Hi Certifirm...

I tried to add the accelerations also, but I was having problems with the validation of the xml data results. The ISO specify a maximum value and sometimes I was having values bigger than that maximum. There is a way in the ISO that allows you to scale up and down the values, but it wasn't too much clear for me and I didn't have more time to investigate. Sorry.

I wanted to ask if you found a solution for that. Also, is there any software which can validate the generated XML for biometric signatures?

@szimek
Copy link
Owner

szimek commented Jun 8, 2021

Hey! I still have PDF with the standard (BIOMETRIC DATA INTERCHANGE FORMATS - PART 7: SIGNATURE/SIGN TIME SERIES DATA) ;) If anyone creates a PR that captures the minimum of additional data from pointer events (pressure only? tilt? width? height?) then it could be merged. However, I think that the function that converts raw data to ISO format doesn't necessarily have to be part of this library.

There already is #503, but it includes way too many changes.

@UziTech
Copy link
Collaborator

UziTech commented Jun 8, 2021

@szimek are you able to share that PDF? I would be willing to create a PR for this.

@Simolation
Copy link

Hey! I still have PDF with the standard (BIOMETRIC DATA INTERCHANGE FORMATS - PART 7: SIGNATURE/SIGN TIME SERIES DATA) ;) If anyone creates a PR that captures the minimum of additional data from pointer events (pressure only? tilt? width? height?) then it could be merged. However, I think that the function that converts raw data to ISO format doesn't necessarily have to be part of this library.

There already is #503, but it includes way too many changes.

I also used @aegeavaz changes and it works great. Could you maybe share the PDF that is generated, as I am still not 100% sure on how to include it? Is it only an attachment, or how did you do it?

@nicpaola
Copy link

Hey! I still have PDF with the standard (BIOMETRIC DATA INTERCHANGE FORMATS - PART 7: SIGNATURE/SIGN TIME SERIES DATA) ;) If anyone creates a PR that captures the minimum of additional data from pointer events (pressure only? tilt? width? height?) then it could be merged. However, I think that the function that converts raw data to ISO format doesn't necessarily have to be part of this library.

There already is #503, but it includes way too many changes.

Hi! Can u share that pdf?
I wanna find a way to validate the generated XML :)
Thank you

@nicpaola
Copy link

What is your opinion?

Hi Certifirm...

I tried to add the accelerations also, but I was having problems with the validation of the xml data results. The ISO specify a maximum value and sometimes I was having values bigger than that maximum. There is a way in the ISO that allows you to scale up and down the values, but it wasn't too much clear for me and I didn't have more time to investigate. Sorry.

Hi @aegeavaz! Your fork is great.
But what do you use to validate the generated xml?
Thank u for all!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.