Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Privacy System to smf #4892

Open
wants to merge 34 commits into
base: release-2.1
from
Open
Changes from 12 commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
5889093
Added UI for Privacy Settings
albertlast Aug 4, 2018
4ebc4dd
Added download and some prep for gui
albertlast Aug 5, 2018
ce5e3b1
formating
albertlast Aug 5, 2018
f0f429e
fix personal message download
albertlast Aug 5, 2018
bb20a4e
frontend of download user data
albertlast Aug 6, 2018
fd5cc1e
wrong text name
albertlast Aug 6, 2018
a9297da
Enable to disable privacy function some prep for policy
albertlast Aug 6, 2018
36cc3f4
policy gui
albertlast Aug 6, 2018
e20db6a
Update and Create for policy
albertlast Aug 6, 2018
7090e38
Better error handling when no data was found
albertlast Aug 6, 2018
ac8bd19
Display policy
albertlast Aug 7, 2018
e3a652e
evaluate the policy state
albertlast Aug 7, 2018
5e26187
Enforce new policy
albertlast Aug 7, 2018
85c5b86
remove old policy
albertlast Aug 7, 2018
b039640
registration integration
albertlast Aug 7, 2018
79372a7
mysql don't like sql
albertlast Aug 8, 2018
bf0235c
mysql keep not liking sql
albertlast Aug 8, 2018
f1a912c
mysql
albertlast Aug 8, 2018
d92ca15
mysql
albertlast Aug 8, 2018
e94880a
Added current txt language reordner mysql and pg query
albertlast Aug 9, 2018
7e3759b
forgot to delete the c
albertlast Aug 9, 2018
f3b103e
fix user redirect, fix maintaince mode
albertlast Aug 9, 2018
11e3a34
Align permission name, added language to reports
albertlast Aug 9, 2018
7e2dbae
fix different implementation of fetch_all
albertlast Aug 9, 2018
a9c416d
Allow the save button to save
albertlast Aug 14, 2018
3d7de77
db_prefix and novalid -> invalid
albertlast Oct 28, 2018
0e06689
do mysql ip convert
albertlast Oct 28, 2018
66f502c
don't ask the admin to validate his policy
albertlast Oct 28, 2018
fb3edfa
invalid
albertlast Oct 28, 2018
0444aae
Never trust sesqui
albertlast Oct 28, 2018
d0265a5
Fixed some scrutinizer issues
albertlast Oct 30, 2018
bcaaf9c
fix mysql broken behavior
albertlast Oct 31, 2018
4061335
fix all other mysql
albertlast Oct 31, 2018
2b7232e
fixed syntax
albertlast Oct 31, 2018
File filter...
Filter file types
Jump to…
Jump to file or symbol
Failed to load files and symbols.
+586 −0
Diff settings

Always

Just for now

Copy path View file
@@ -114,6 +114,8 @@ function AdminMain()
'likes' => array($txt['likes']),
'mentions' => array($txt['mentions']),
'alerts' => array($txt['notifications']),
'privacy' => array($txt['privacy']),
'policy' => array($txt['policy']),
),
),
'antispam' => array(
Copy path View file
@@ -2300,6 +2300,8 @@ function triggerCron()
// Call load theme integration functions.
call_integration_hook('integrate_load_theme');
// Privacy logic
// We are ready to go.
$context['theme_loaded'] = true;
}
Copy path View file
@@ -65,6 +65,8 @@ function ModifyFeatureSettings()
'likes' => 'ModifyLikesSettings',
'mentions' => 'ModifyMentionsSettings',
'alerts' => 'ModifyAlertsSettings',
'privacy' => 'ModifyPrivacySettings',
'policy' => 'ModifyPolicySettings',
);
loadGeneralSettingParameters($subActions, 'basic');
@@ -95,6 +97,10 @@ function ModifyFeatureSettings()
'alerts' => array(
'description' => $txt['notifications_desc'],
),
'privacy' => array(
),
'policy' => array(
),
),
);
@@ -2275,4 +2281,138 @@ function ModifyAlertsSettings()
$context['sub_template'] = 'alert_configuration';
}
/**
* Config array for changing privacy settings
* Accessed from ?action=admin;area=featuresettings;sa=privacy;
*
* @param bool $return_config Whether or not to return the config_vars array
* @return void|array Returns nothing or returns the $config_vars array if $return_config is true
*/
function ModifyPrivacySettings($return_config = false)
{
global $txt, $scripturl, $context;
$config_vars = array(
array('check', 'enable_privacy_userexport'),
array('permissions', 'privacy_userexport_own'),
array('permissions', 'privacy_userexport_others'),
);
call_integration_hook('integrate_privacy_settings', array(&$config_vars));
if ($return_config)
return $config_vars;
// Saving?
if (isset($_GET['save']))
{
checkSession();
call_integration_hook('integrate_save_privacy_settings');
saveDBSettings($config_vars);
$_SESSION['adm-save'] = true;
redirectexit('action=admin;area=featuresettings;sa=privacy');
}
$context['post_url'] = $scripturl . '?action=admin;area=featuresettings;save;sa=privacy';
$context['settings_title'] = $txt['privacy'];
prepareDBSettingContext($config_vars);
}
/**
* Config array for changing policy settings
* Accessed from ?action=admin;area=featuresettings;sa=policy;
*
* @param bool $return_config Whether or not to return the config_vars array
* @return void|array Returns nothing or returns the $config_vars array if $return_config is true
*/
function ModifyPolicySettings($return_config = false)

This comment has been minimized.

Copy link
@Sesquipedalian

Sesquipedalian Oct 23, 2018

Member

This function needs some logic to ensure that enable_policy_function cannot be set to true unless policy_version and any other necessary values are defined.

Right now, if the admin makes the mistake of enabling the policy function without setting up everything else first, the admin will be redirected on every page load to the page where he is prompted to accept the policy. But the admin will be unable to do so because there is no policy to accept, and so he will be stuck in an endless loop and will be unable to fix his forum except by editing his database directly.

This comment has been minimized.

Copy link
@albertlast

albertlast Oct 28, 2018

Author Collaborator

Since i check everywhere if enabled_policy_function is true:

if (!empty($modSettings['enable_policy_function']) && !$user_info['is_guest'] && !$user_info['is_admin'])

I see no need here todo any checks in the settings.

{
global $txt, $scripturl, $context, $sourcedir, $modSettings, $smcFunc;
// Needed for the WYSIWYG editor.
require_once($sourcedir . '/Subs-Editor.php');
$context['sub_template'] = 'edit_policy';
$context['page_title'] = $txt['policy_management'];
$config_vars = array(
array('check', 'enable_policy_function'),
array('text', 'policy_text'),
);
$currentVersion = !empty($modSettings['policy_version']) ? substr($modSettings['policy_version'], 11) : 0;
// Now create the editor.
$editorOptions = array(
'id' => 'policy_text',
'value' => !empty($modSettings['policy_text' . $currentVersion]) ? $modSettings['policy_text' . $currentVersion] : '',
'height' => '250px',
'width' => '100%',
'labels' => array(
'post_button' => $txt['policy_save'],
),
'preview_type' => 2,
'required' => true,
);
create_control_richedit($editorOptions);
// Store the ID for old compatibility.
$context['post_box_name'] = $editorOptions['id'];
call_integration_hook('integrate_policy_settings', array(&$config_vars));
$request = $smcFunc['db_query']('', '
SELECT count( case when th.value is null then 1 end) "not",
count( case when th.value is not null and th.value != {string:policy_version} then 1 end) "old",
count( case when th.value = {string:policy_version} then 1 end) "fresh"
FROM smf_members mem
This conversation was marked as resolved by albertlast

This comment has been minimized.

Copy link
@Sesquipedalian

Sesquipedalian Oct 23, 2018

Member

smf_{db_prefix}

LEFT JOIN smf_themes th ON (mem.id_member = th.id_member AND th.id_theme = 1 AND th.variable = {string:policy_approved})',
This conversation was marked as resolved by albertlast

This comment has been minimized.

Copy link
@Sesquipedalian

Sesquipedalian Oct 23, 2018

Member

smf_{db_prefix}

array(
'policy_version' => 'policy_text' . $currentVersion,
'policy_approved' => 'policy_approved',
)
);
list ($context['policy']['not'], $context['policy']['old'], $context['policy']['fresh']) = $smcFunc['db_fetch_row']($request);
$smcFunc['db_free_result']($request);
if ($return_config)
return $config_vars;
// Saving?
if (isset($_GET['save']))
{
checkSession();
call_integration_hook('integrate_save_policy_settings');
if (!empty($_REQUEST['save_new_policy']))
{
$currentVersion++;
$config_vars[] = array('text', 'policy_text'.$currentVersion);
$_POST['policy_text'.$currentVersion] = $_REQUEST['policy_text'];
$config_vars[] = array('text', 'policy_version');
$_POST['policy_version'] = 'policy_text'.$currentVersion;
unset($config_vars[1]);
}
elseif (!empty($_REQUEST['update_policy']))
{
$config_vars[] = array('text', 'policy_text'.$currentVersion);
$_POST['policy_text'.$currentVersion] = $_REQUEST['policy_text'];
$config_vars[] = array('text', 'policy_version');
$_POST['policy_version'] = 'policy_text'.$currentVersion;
unset($config_vars[1]);
}
saveDBSettings($config_vars);
$_SESSION['adm-save'] = true;
redirectexit('action=admin;area=featuresettings;sa=policy');
}
$context['post_url'] = $scripturl . '?action=admin;area=featuresettings;save;sa=policy';
$context['settings_title'] = $txt['privacy'];
prepareDBSettingContext($config_vars);
}
?>
Copy path View file
@@ -969,4 +969,195 @@ function subscriptions($memID)
$context['sub_template'] = 'user_subscription';
}
/**
* Function to allow the user to todo Privacy stuff
*
* @param int $memID The ID of the member
*/
function getProfileData($memID)
{
global $txt, $user_profile, $context, $smcFunc, $user_info;
if ($memID == $user_info['id'])
$context['pdc']['own'] = true;
else
$context['pdc']['name'] = $context['member']['name'];
if (!empty($_REQUEST['nofound']))
$context['pdc']['nofound'] = true;
if (empty($_REQUEST['activity']))
return;
$mode = $_REQUEST['activity'];
$profileData = array();
if ($mode == 'profile') // profile
{
loadMemberData($memID, false, 'profile');

This comment has been minimized.

Copy link
@Sesquipedalian

Sesquipedalian Oct 23, 2018

Member

It would be easier and better to use loadMemberContext() here. It formats the member data into a nicer form for reading, you wouldn't need to unset security-related fields, and the custom profile fields could easily be presented as distinct columns instead of a blob of JSON data.

$profile = $user_profile[$memID];
$removeFields = array('secret_question','tfa_secret','password_salt');
foreach ($removeFields as $value)
{
unset($profile[$value]);
}
foreach($profile as $key => &$value)
{
if (is_array($value))
$value = $smcFunc['json_encode']($value);
}
$profileData[0] = array_keys($profile);
This conversation was marked as resolved by Sesquipedalian

This comment has been minimized.

Copy link
@Sesquipedalian

Sesquipedalian Oct 23, 2018

Member

I like that you are providing a header row here. Please add one to the post and personal messages CSVs as well.

This comment has been minimized.

Copy link
@albertlast

albertlast Oct 28, 2018

Author Collaborator

Like you see here: #4892 (comment)
they get already added.

This comment has been minimized.

Copy link
@Sesquipedalian

Sesquipedalian Oct 31, 2018

Member

Yes, you are right that there is a header row. Unfortunately, it is filled with numeric keys instead of column names, so it isn't useful at all. Please change that.

This comment has been minimized.

Copy link
@albertlast

albertlast Oct 31, 2018

Author Collaborator

Nope please read the start post,
you need to apply the pr #5048
otherwise the fetch_all is different across pg and mysql.

This comment has been minimized.

Copy link
@Sesquipedalian

Sesquipedalian Nov 1, 2018

Member

Fair enough. 🙂

$profileData[1] = $profile;
call_integration_hook('integrate_getProfile_profile', array(&$profileData));
}
elseif ($mode == 'messages') // messages

This comment has been minimized.

Copy link
@Sesquipedalian

Sesquipedalian Oct 23, 2018

Member

A user could have thousands of posts, and if he does, processing all of them at once like this will overwhelm the server—either it will run out of memory, or it will run out of time, or both. So, this needs some sort of method to work through them in manageable batches. Writing to a temporary file and automatically reloading the page every few seconds, like I did with the XML version, would probably work well here.

This comment has been minimized.

Copy link
@albertlast

albertlast Oct 28, 2018

Author Collaborator

100k messages takes 500 ms -> looks good to me
the file stuff got many weakness
it created by his self a new gdpr topic
it could be infected by evil software
it take more time to proceed -> the cpu load is overall higher

This comment has been minimized.

Copy link
@Sesquipedalian

Sesquipedalian Oct 29, 2018

Member

How much memory was used to process those 100,000 messages all at once?

This comment has been minimized.

Copy link
@CoreISP

CoreISP Nov 1, 2018

Member

@Sesquipedalian That's very important indeed, especially problematic on more aggressively resource-limited shared hosting. Plus hosting many boards on a server executing that kind of queries can become a nuisance. Always assume low resources is probably a good idea.

This comment has been minimized.

Copy link
@albertlast

albertlast Nov 1, 2018

Author Collaborator

It needs to be realistic,
100k message is more than sm.org got
And you will not host setup like this on freehost

This comment has been minimized.

Copy link
@Sesquipedalian

Sesquipedalian Nov 1, 2018

Member

That's not an answer to the question, @albertlast. 🤨

There are very good reasons why SMF uses paging to perform any operation involving large numbers of table rows. Those reasons apply here just as much as they do anywhere else.

This comment has been minimized.

Copy link
@albertlast

albertlast Nov 1, 2018

Author Collaborator

you didn't awnser the question for the xml solution.

I test some solution,
handle the size better.

{
$request = $smcFunc['db_query']('','
SELECT id_msg, id_topic, poster_time, subject, modified_time, modified_name, modified_reason, body, likes, poster_ip
This conversation was marked as resolved by albertlast

This comment has been minimized.

Copy link
@Sesquipedalian

Sesquipedalian Oct 23, 2018

Member

Don't forget to use your nice little int_dtop() function to decode the IP address (here and elsewhere).

FROM {db_prefix}messages
WHERE id_member = {int:memID}',
array(
'memID' => $memID,
)
);
$profileData = $smcFunc['db_fetch_all']($request);
if (!is_array($profileData))
redirectexit('action=profile;area=getprofiledata;u=' . $memID . ';nofound=1');
array_unshift($profileData, array_keys($profileData[0]));
$smcFunc['db_free_result']($request);
call_integration_hook('integrate_getProfile_messages', array(&$profileData));
}
elseif ($mode == 'pmessages')
This conversation was marked as resolved by albertlast

This comment has been minimized.

Copy link
@Sesquipedalian

Sesquipedalian Oct 23, 2018

Member

Exporting personal messages could also overwhelm the server, so this will need a method to process them in manageable batches, too.

This comment has been minimized.

Copy link
@albertlast

albertlast Oct 28, 2018

Author Collaborator

duplicate to message,
we only need to talk at one place.

{
$request = $smcFunc['db_query']('','
SELECT pm.msgtime, pm.subject, pm.body

This comment has been minimized.

Copy link
@Sesquipedalian

Sesquipedalian Oct 23, 2018

Member

This query grabs PMs from the user and PMs to the user, but the exported data doesn't indicate who was the author and who were the recipients. It would be more helpful if each row identified both.

This comment has been minimized.

Copy link
@albertlast

albertlast Oct 28, 2018

Author Collaborator

This is indended,
because otherwise you export userdata from a different user in your data and
never allowed you to download his data.

This comment has been minimized.

Copy link
@Sesquipedalian

Sesquipedalian Oct 29, 2018

Member

The displayed names of other users don't count as private data, so it is perfectly fine to include them in the downloaded information.

This comment has been minimized.

Copy link
@Sesquipedalian

Sesquipedalian Oct 29, 2018

Member

Just to clarify, I'm asking for pm.from_name and GROUP_CONCAT(COALESCE(mem.real_name, mem.member_name)) AS to_names to be added to the SELECT in this query (which will require joining the members table, of course), and for those values to be included in the output columns.

This comment has been minimized.

Copy link
@albertlast

albertlast Oct 29, 2018

Author Collaborator

In my eyes you don't need this,
would be enough to mark if you send or receive the message and maybe the technical user_id from the other.

This comment has been minimized.

Copy link
@Sesquipedalian

Sesquipedalian Oct 29, 2018

Member

As a rule, human beings find it easier to understand who a message was sent from or sent to if the name of the person is given than if an ID number is given. Please use the names.

FROM {db_prefix}personal_messages pm
LEFT JOIN {db_prefix}pm_recipients pmr on (pm.id_pm = pmr.id_pm and pmr.id_member = {int:memID})
WHERE pm.id_member_from = {int:memID} or pmr.id_member = {int:memID}',
array(
'memID' => $memID,
)
);
$profileData = $smcFunc['db_fetch_all']($request);
if (!is_array($profileData))
redirectexit('action=profile;area=getprofiledata;u=' . $memID . ';nofound=1');
array_unshift($profileData, array_keys($profileData[0]));
$smcFunc['db_free_result']($request);
call_integration_hook('integrate_getProfile_pmessages', array(&$profileData));
}
$count = count($profileData);
$csv_data = '';
for($i = 0; $i < $count; $i++)
{
$csv_data .= arrayToCsv($profileData[$i]) . "\r\n";
}
header("Pragma: no-cache");
header('Content-Disposition: attachment; filename="privacySMF'.$mode.'.csv";');
header("Content-Length: " . strlen($csv_data));
header("Content-Transfer-Encoding: binary");
header("Content-Type: application/force-download");
echo $csv_data;
exit;
}
/**
* Function to allow the user to todo Policy stuff
*
* @param int $memID The ID of the member
*/
function getPolicyData($memID)
{
global $txt, $user_profile, $context, $smcFunc, $user_info, $modSettings;
$context['poc']['own'] = false;
if ($memID == $user_info['id'])
$context['poc']['own'] = true;
if (!empty($user_profile[$memID]['options']['policy_approved']) &&
!empty($modSettings[$user_profile[$memID]['options']['policy_approved']]))
{
$context['poc']['approved_text'] = $modSettings[$user_profile[$memID]['options']['policy_approved']];
}
if (!empty($user_profile[$memID]['options']['policy_approved']) &&
!empty($modSettings[$user_profile[$memID]['options']['policy_approved']]) &&
!empty($modSettings['policy_version']) &&
!empty($modSettings[$modSettings['policy_version']]) &&
$modSettings[$user_profile[$memID]['options']['policy_approved']] != $modSettings['policy_version'])
{
$context['poc']['newVersionText'] = $modSettings[$modSettings['policy_version']];
}
if (empty($_REQUEST['activity']))
return;
$mode = $_REQUEST['activity'];
$profileData = array();
if ($mode == 'save') // profile
{
if (!$context['poc']['own']) //only for yourself
exit;
loadMemberData($memID, false, 'profile');
$profile = $user_profile[$memID];
$removeFields = array('secret_question','tfa_secret','password_salt');
foreach ($removeFields as $value)
{
unset($profile[$value]);
}
foreach($profile as $key => &$value)
{
if (is_array($value))
$value = $smcFunc['json_encode']($value);
}
$profileData[0] = array_keys($profile);
$profileData[1] = $profile;
call_integration_hook('integrate_getProfile_profile', array(&$profileData));
}
}
/**
* Formats a line (passed as a fields array) as CSV and returns the CSV as a string.
* from https://stackoverflow.com/questions/3933668/convert-array-into-csv
*
* @param array $fields a line of the array
* @param string @delimiter the delimiter char
* @param string @enclosure the enclosure char
* @param boolean $encloseAll enclose all
* @param boolean $nullToMysqlNull
*/
function arrayToCsv( array $fields, $delimiter = ';', $enclosure = '"', $encloseAll = false, $nullToMysqlNull = false ) {
$delimiter_esc = preg_quote($delimiter, '/');
$enclosure_esc = preg_quote($enclosure, '/');
$output = array();
foreach ( $fields as $field )
{
if ($field === null && $nullToMysqlNull)
{
$output[] = 'NULL';
continue;
}
// Enclose fields containing $delimiter, $enclosure or whitespace
if ( $encloseAll || preg_match( "/(?:${delimiter_esc}|${enclosure_esc}|\s)/", $field ) )
{
$output[] = $enclosure . str_replace($enclosure, $enclosure . $enclosure, $field) . $enclosure;
}
else
{
$output[] = $field;
}
}
return implode( $delimiter, $output );
}
?>
Oops, something went wrong.
ProTip! Use n and p to navigate between commits in a pull request.
You can’t perform that action at this time.