[HOLD for payment 2023-08-28] [$1000] Profile - Entering country without a diacritic for Côte D'ivoire returns no suggestions

[lanitochka17]

If you haven’t already, check out our contributing guidelines for onboarding and email contributors@expensify.com to request to join our Slack channel!

---

Issue found when executing PR #24140

Action Performed:

1. Open the app
2. Go to Settings > Profile > Personal details > Home address > Country
3. Start entering "cote" or "cote d'Ivoire" without a diacritic

Expected Result:

The suggested country name "Côte d'Ivoire" is displayed

Actual Result:

The suggested country is not displayed, and "no results found" is shown

Workaround:

Unknown

Platforms:

Which of our officially supported platforms is this issue occurring on?

• Android / native
• Android / Chrome
• iOS / native
• iOS / Safari
• MacOS / Chrome / Safari
• MacOS / Desktop

Version Number: 1.3.52.2

Reproducible in staging?: Yes

Reproducible in production?: Yes

If this was caught during regression testing, add the test name, ID and link from TestRail:

Email or phone of affected tester (no customers):

Logs: https://stackoverflow.com/c/expensify/questions/4856

Notes/Photos/Videos: Any additional supporting documentation

Bug6159355_IMG_0917.mp4

Expensify/Expensify Issue URL:

Issue reported by: Applause - Internal Team

Slack conversation:

View all open jobs on GitHub

Upwork Automation - Do Not Edit

• Upwork Job URL: https://www.upwork.com/jobs/~01788d0f20c53c4622
• Upwork Job ID: 1689673663122317312
• Last Price Increase: 2023-08-10
• Automatic offers:
• Pluto0104 | Contributor | 26073309

[melvin-bot]

Triggered auto assignment to @bfitzexpensify (Bug), see https://stackoverflow.com/c/expensify/questions/14418 for more details.

[melvin-bot]

Bug0 Triage Checklist (Main S/O)

• This "bug" occurs on a supported platform (ensure Platforms in OP are ✅)
• This bug is not a duplicate report (check E/App issues and #expensify-bugs)
  • If it is, comment with a link to the original report, close the issue and add any novel details to the original issue instead
• This bug is reproducible using the reproduction steps in the OP. S/O
  • If the reproduction steps are clear and you're unable to reproduce the bug, check with the reporter and QA first, then close the issue.
  • If the reproduction steps aren't clear and you determine the correct steps, please update the OP.
• This issue is filled out as thoroughly and clearly as possible
  • Pay special attention to the title, results, platforms where the bug occurs, and if the bug happens on staging/production.
• I have reviewed and subscribed to the linked Slack conversation to ensure Slack/Github stay in sync

[makiour]

Proposal

Please re-state the problem that we are trying to solve in this issue.

Countries list is not showing all options if diacritics are needed.

What is the root cause of that problem?

As we can see in the search in the country selection modal search options, we are not handling comparison without diacretics:

App/src/libs/searchCountryOptions.js

Lines 10 to 22 in 07571e3

	function searchCountryOptions(searchValue, countriesData) {
	const trimmedSearchValue = searchValue.toLowerCase().replaceAll(CONST.REGEX.NON_ALPHABETIC_AND_NON_LATIN_CHARS, '');
	if (trimmedSearchValue.length === 0) {
	return [];
	}
	const filteredData = _.filter(countriesData, (country) => country.searchValue.includes(trimmedSearchValue));
	// sort by country code
	return _.sortBy(filteredData, (country) => (country.value.toLowerCase() === trimmedSearchValue ? -1 : 1));
	}
	export default searchCountryOptions;

We need to include diacritics and compare without them the two words from the list.

What changes do you think we should make in order to solve the problem?

We can add a function to remove diacritics from the country lists whenever we are comparing it with out word. We also need to remove them from our own word if we want complete comparison without edge cases.

String.normalize() allow us to remove the desired accents when we transform the word into a normalized composition. This is done through:

        function simplifyWordFromDiacritics(country) {
            return country.normalize('NFD').replace(/[\u0300-\u036f]/g, '');
        }

(Reference)

Thus, our search country options would change to:

import _ from 'underscore';
import CONST from '../CONST';

/**
 * Removes diacritics from a string
 * @param {String} text - The input text with diacritics
 * @returns {String} The input text without diacritics
 */
function simplifyWordFromDiacritics(country) {
    return country.normalize('NFD').replace(/[\u0300-\u036f]/g, '');
}

/**
 * Searches the countries/states data and returns sorted results based on the search query
 * @param {String} searchValue
 * @param {Object[]} countriesData - An array of country data objects
 * @returns {Object[]} An array of countries/states sorted based on the search query
 */
function searchCountryOptions(searchValue, countriesData) {
    const trimmedSearchValue = simplifyWordFromDiacritics(searchValue.toLowerCase().replaceAll(CONST.REGEX.NON_ALPHABETIC_AND_NON_LATIN_CHARS, ''));
    if (trimmedSearchValue.length === 0) {
        return [];
    }

    const filteredData = _.filter(countriesData, (country) => simplifyWordFromDiacritics(country.searchValue.toLowerCase()).includes(trimmedSearchValue));

    // sort by country code
    return _.sortBy(filteredData, (country) => (country.value.toLowerCase() === trimmedSearchValue ? -1 : 1));
}

export default searchCountryOptions;

What alternative solutions did you explore? (Optional)

We can also install a new module called diacritics through npm install, and calling it will allow us to the same work.

[jeet-dhandha]

Proposal

Please re-state the problem that we are trying to solve in this issue.

• Without diatric characters, it is difficult to search for Côte d'Ivoire in the country field.

What is the root cause of that problem?

• The root cause is that the country field does not normalize diatric characters.

What changes do you think we should make in order to solve the problem?

• We can use the below code to normalize diatric characters in the country field.

function normalizeDiacritics(str) {
  return str.normalize("NFD").replace(/[\u0300-\u036f]/g, "");
}

• Now we will need to update:
  • searchCountryOptions.js to use above function to normalize diatric characters.
  • We can further use above function in searching for Chats, Rooms, and Users.
    • By adding it to isSearchStringMatch of OptionsListUtils.js.

What alternative solutions did you explore? (Optional)

• There are no alternative solutions. As this is a simple problem, the above solution should be sufficient.

[makiour]

Proposal

Please re-state the problem that we are trying to solve in this issue.

Countries list is not showing all options if diacritics are needed.

What is the root cause of that problem?

As we can see in the search in the country selection modal search options, we are not handling comparison without diacretics:

App/src/libs/searchCountryOptions.js

Lines 10 to 22 in 07571e3

	function searchCountryOptions(searchValue, countriesData) {
	const trimmedSearchValue = searchValue.toLowerCase().replaceAll(CONST.REGEX.NON_ALPHABETIC_AND_NON_LATIN_CHARS, '');
	if (trimmedSearchValue.length === 0) {
	return [];
	}
	const filteredData = _.filter(countriesData, (country) => country.searchValue.includes(trimmedSearchValue));
	// sort by country code
	return _.sortBy(filteredData, (country) => (country.value.toLowerCase() === trimmedSearchValue ? -1 : 1));
	}
	export default searchCountryOptions;

We need to include diacritics and compare without them the two words from the list.

What changes do you think we should make in order to solve the problem?

We can add a function to remove diacritics from the country lists whenever we are comparing it with out word. We also need to remove them from our own word if we want complete comparison without edge cases.

String.normalize() allow us to remove the desired accents when we transform the word into a normalized composition. This is done through:

        function simplifyWordFromDiacritics(country) {
            return country.normalize('NFD').replace(/[\u0300-\u036f]/g, '');
        }

(Reference)

Thus, our search country options would change to:

import _ from 'underscore';
import CONST from '../CONST';

/**
 * Removes diacritics from a string
 * @param {String} text - The input text with diacritics
 * @returns {String} The input text without diacritics
 */
function simplifyWordFromDiacritics(country) {
    return country.normalize('NFD').replace(/[\u0300-\u036f]/g, '');
}

/**
 * Searches the countries/states data and returns sorted results based on the search query
 * @param {String} searchValue
 * @param {Object[]} countriesData - An array of country data objects
 * @returns {Object[]} An array of countries/states sorted based on the search query
 */
function searchCountryOptions(searchValue, countriesData) {
    const trimmedSearchValue = simplifyWordFromDiacritics(searchValue.toLowerCase().replaceAll(CONST.REGEX.NON_ALPHABETIC_AND_NON_LATIN_CHARS, ''));
    if (trimmedSearchValue.length === 0) {
        return [];
    }

    const filteredData = _.filter(countriesData, (country) => simplifyWordFromDiacritics(country.searchValue.toLowerCase()).includes(trimmedSearchValue));

    // sort by country code
    return _.sortBy(filteredData, (country) => (country.value.toLowerCase() === trimmedSearchValue ? -1 : 1));
}

export default searchCountryOptions;

What alternative solutions did you explore? (Optional)

We can also install a new module called diacritics through npm install, and calling it will allow us to the same work.

Result

WhatsApp.Video.2023-08-09.at.22.36.24.mp4

[Pluto0104]

Proposal

Please re-state the problem that we are trying to solve in this issue.

The country name is not displayed when you start entering "cote" or "cote d'Ivoire" without a diacritic, and instead you see "no results found".

What is the root cause of that problem?

The app does not handle diacritics or special characters properly when searching or matching country names.

What changes do you think we should make in order to solve the problem?

We can use _.deburr instead of simplifyWordFromDiacritics to remove diacritics from a string. This is a built-in function from lodash, which is a library that we already use in our app. This will make your code more concise and readable. (I'll import lodash module instead of underscore as _)

For example:

const trimmedSearchValue =_.chain(searchValue).deburr().toLower().value().replaceAll(CONST.REGEX.NON_ALPHABETIC_AND_NON_LATIN_CHARS, '');
/* ... */
const filteredData = _.filter(countriesData, (country) => _.deburr(country.searchValue).includes(trimmedSearchValue));
/* ... */

What alternative solutions did you explore? (Optional)

function simplifyWordFromDiacritics(country) {
    return country.normalize('NFD').replace(/[\u0300-\u036f]/g, '');
}

[melvin-bot]

Job added to Upwork: https://www.upwork.com/jobs/~01788d0f20c53c4622

[melvin-bot]

Triggered auto assignment to @alexpensify (External), see https://stackoverflow.com/c/expensify/questions/8582 for more details.