Concurrency issue with TimeZoneConverter

[jamesemann]

Version

4.19.2

Describe the bug

Intermittently we are seeing a concurrency issue when trying to use date based adaptive functions.

It seems to stem from concurrent access of static dictionaries in TimeZoneConverter.

To Reproduce

To reproduce you can follow the following steps. Note step 3 is not easy to reproduce and requires some load:

1. Having a bot using the adaptive function convertFromUTC in the first turn. e.g. =float(formatDateTime(convertFromUTC(utcNow(), 'Eastern Standard Time'), 'HH'))
2. Then restarting the process hosting the bot code
3. Having concurrent requests hit our bot which both cause the static dictionaries in TimeZoneConverter to be initialized - causing the error below which stops the bot from functioning:

System.InvalidOperationException

AdaptiveExpressions.TimeZoneConverter.LoadData

Operations that change non-concurrent collections must have exclusive access. A concurrent update was performed on this collection and corrupted its state. The collection's state is no longer correct.

Expected behavior

The dictionary access in TimeZoneConverter should be thread-safe.

Screenshots

N/A

Additional context

N/A

[robertcox-msft]

@ceciliaavila This is the business impact of this bug provided by the AE for City of Atlanta:

this can a have a significant impact on the city’s 311 system. We have experienced one outage already which the Zammo team had to develop a work around for. The 311 system is under a broad scrutiny from the Mayor’s office and city Council as well as the citizens. It seems every nuance of change in the performance of 311 gets over exposed and a ripple affect rifles through the city and the CIO his staff and I get a call.
So, my initial answers to your two questions are below:
• Business justification:
o Include the number of people impacted and the actual business impact.
 Atlanta has ~500K people directly impacted and over 6M who access the 311 system in varying degrees. This system receives ~100K calls per month. This was prior to the announcement of the new AI based MSFT IVR, chat bots and virtual agents we’ve installed and went live on 12/16 2022.
 Most citizens and businesses (including MSFT) who live and/or work in Atlanta use this system for the following services on a regular basis for services within the following departments:
• Water Bill
• Garbage, Recycling & Graffiti
• Utilities
• Business
• Supportive Services
• Licensing and Permitting
• Water
• Inmate Services
• Public Safety
• Payments and Credits
• Record Requests
• Right of Way
• Water Services
• Streets and Sidewalks
• Sewer
• Bin Request
• Traffic and Parking
• Sidewalk Defects
• Bridge Repair
• Pothole
• Trash
• Streetlights
• Code Enforcement
• Trees
• Transportation
• Parking Meter
• Homeowner Information
• Parking Tickets
• Emergency Preparedness
• Event Permits
• Police Reports
• Municipal Courts
• Building Permits
• Facility Requests
• Activities
• Zoning
• Illegal Dumping
• Taxation
• Weather
• Parking Violations
• Housing Assistance
• Pay Parking Tickets
• Signs and Signals
• Fire Prevention
• Parks and Trees
• Community-Oriented Policing
• Claims
• Safety Inspections
• Park Maintenance
• Liens and Bankruptcy
• Recreation and Event Planning
• Federal and State Law
Message from the Mayor: (on tweeter)
@ATL311
has a new interactive chatbot to provide 24/7 access to non-emergency services. The artificial intelligent (AI) chatbot is the fastest way for residents and visitors to submit a service or general request. Learn more: https://bit.ly/3S7kHcU

4:15 PM · Feb 20, 2023 10K Views

• Impact to Microsoft’s business if the bug is not prioritized.
o These 311 systems are currently generating over $20K in monthly telephony (Azure ACS) revenue in its infancy.
o Our relationship/partnership with the CIO and Mayor of Atlanta will be significantly impacted if this becomes an issue and does not perform well. This customer is constantly being courted to use the AWS connect service they already own which serves as the city wide IVR system but does not have the robust capabilities of the MSFT/Zammo (partner) AI based solution. This was a highly competitive win for MSFT and could easily be replaced by our super aggressive competitor.
o A slip in our perception, credibility in supporting what we’ve just sold will negatively impact our role as trusted advisor and overall relationship.
o We have several million dollars of new projects in flight at the City that could be impacted. Such as:
 Security MSSP
 Massive cloud migration for the enterprise and E-911
 No code low code innovations within police dept

[ceciliaavila]

Hi @robertcox-msft, we are working on this, we'll let you know as soon as we have news.

[robertcox-msft]

Thank you Cecilia. I am a CSAM with Microsoft on this account. Would you be available for a quick call for us to get a better understanding of where we are on it? We would like to understand if the bug is accepted, if so where is it prioritized, any idea on a best guess resolution, etc. Just need to know so we can set expectations.

[tracyboehrer]

@robertcox-msft The bug is accepted and assigned to an engineer. We'd also likely put it into a patch release. I'll wait for the assessment from the engineers before giving an estimate on when it would land.

[tracyboehrer]

@robertcox-msft We will publish a patched release (4.19.3) if not today, then Monday.

[robertcox-msft]

Thank you Tracy!

[robertcox-msft]

@tracyboehrer Do you know how long after the patch release that the customer will receive it? Are they on a ring deployment, have to do a manual update, etc? Thanks!

[tracyboehrer]

@robertcox-msft It will be available on NuGet within minutes of the release. We will do a Release on GitHub. Anyone watching the repo will get notified. We don't have any other mechanism to notify.

It won't be until Monday at the earliest though.

[jamesemann]

Thanks team for prioritising this. I will keep an eye out for the new release Monday