Open source tools and auditability

[MatthewCaseres]

There was some discussion about an upcoming article and the point was brought up that open source tools might cause trouble in an audit or something like that. That sounds like a big problem for the premise of open source actuarial software, so I am opening this discussion so we can talk about open source and auditability.

Open source auditable? Yes? No?

[MatthewCaseres]

My perspective is basically that something like chainladder-python might be hooked up to a database and calculate some numbers.

I haven't interacted with auditors and I don't know what they are looking for. But if I was an auditor I would not complain if someone had a codebase that can exactly reproduce the numbers I am looking for.

[kennethshsu]

I think you are right, I think it goes back to Familiarity. If a tool is familiar, to auditor/regulator/or anyone, it's friendlier.

I propose we remove Audibility from the Cons list. Open-source tools are not harder to audit, in fact, it's easier, because of the transparency, which is covered in the Pros.

[patuary84]

I think I was the one who suggested this. I don't feel strongly about keeping it, but a few thoughts to consider:

• I've heard "audit" used in lots of different ways: a financial audit from an external auditor, a SOX audit from an internal auditor, a review from a fellow actuary, etc. So we should probably be clear about what we mean.

• I interact with external/internal auditors frequently in my current role. Typically, these are generalists who don't have any actuarial knowledge, let alone any programming skills or familiarity with GitHub. They frequently want to review reconciliations that verify that the data from this place matches the data from that place, or that the summarized data prepared by one group is matched to summarized data prepared by some independent party, or that the assumptions one party used were reviewed for reasonability by some other party, etc. If you do all this stuff in Excel, that's easy to send them. But if it's done in R/Python, you may have to produce an Excel summary of your work to the auditors for this purpose. This is not difficult, but it may not be part of your existing workflow, so it's an extra step.

• Of course, I agree that familiarity is also a big part of it. External/internal auditors probably won't know what to do with your GitHub repository without involving a specialist. They may be satisfied by simply having a second analyst attest to having reviewed the code run by the first analyst, etc. It seems to vary based on who you get.

• As for situations where "audit" simply means a fellow actuary reviewing your work, if you're the only one you can find at your firm who can code in Python, you may simply not be allowed to code in Python on your project. When I was a consulting actuary, this was the biggest challenge for getting a group of skilled coders off the ground.

[rajesh06]

Good points! On the second, we typically document our R-based analyses in reports through the use of data visualization (typically in figures). One way to handle this issue is to print the counts at the top of the barplot (for example). Then you can easily produce a schedule that shows how the source data reconciles to the figures.

[kennethshsu]

Thanks everyone for sharing their additional thoughts.

I am going to make a call and remove Audibility from our con list. Please let me know if anyone objects!