Per query http proxy (proxies) configuration

CHANGELOG.md

@@ -7,7 +7,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Changed
+
 - Updated onto-deside configuration 2025-04-09 (#202).
+- In the configuration file: per query `httpProxies` setting replaces global `httpProxy` setting (#4, #47).
 
 ## [1.7.0] - 2025-04-09
 

README.md

@@ -13,7 +13,7 @@ Table of contents:
 * [Logging in](#logging-in)
 * [Configuration file](#configuration-file)
   * [Specifying sources](#specifying-sources)
-  * [About httpProxy](#about-httpproxy)
+  * [About httpProxies](#about-httpproxies)
   * [Adding variable type](#adding-variable-type)
   * [Templated queries](#templated-queries)
     * [Templated queries with fixed values for the template variables](#templated-queries-with-fixed-values-for-the-template-variables)
@@ -147,7 +147,6 @@ The configuration file must follow the structure shown below.
   "footer": "HTML components or text that will function as the footer (will be placed in the footer div.)",
   "defaultIDP": "The default value used for IDP when logging in, this IDP can be manually changed in the Web app as well. ",
   "queryFolder": "The base location of the queries, all query locations will start from this folder (relative to public folder).",
-  "httpProxy": "Optional http proxy through which the requests will be rerouted - see documentation below.",
   "introductionText": "The text that the app shows on the dashboard, which the app also shows when you first open it.",
   "queryGroups" : [
     {
@@ -166,7 +165,6 @@ The configuration file must follow the structure shown below.
       "icon": "The key to the icon for the query. This is optional and a default menu icon will be used when left empty.",
       "comunicaContext": {
         "sources": "Initial array of sources over which the query should be executed",
-        "useProxy": "true or false, whether the query should be executed through the proxy or not. This field is optional and defaults to false.",
         ... any other field that can be used in the Comunica query engine https://comunica.dev/docs/query/advanced/context/
       },
       "sourcesIndex": {
@@ -184,7 +182,13 @@ The configuration file must follow the structure shown below.
           ...
         ]
       },
-
+      "httpProxies": [
+        {
+          "urlStart": "all sources whose url start with this string will be rerouted",
+          "httpProxy": "http proxy through which these sources will be rerouted - see also documentation 'About httpProxies' below."
+        }
+        ...
+      ],
       "askQuery": {
         "trueText": "The text that is to be shown when the query result is true (in ASK queries).",
         "falseText": "The text that is to be shown when the query result is false (in ASK queries)."
@@ -209,15 +213,22 @@ The (auxiliary) query provided in `sourceIndex.queryLocation` is executed on `so
 If `sourceIndex` is used and there is no `comunicaContext.lenient` property found, one will be created with value `true`.
 This makes sure that the (main) query can succeed if not all obtained sources are accessible.
 
-### About httpProxy
+### About httpProxies
+
+Per query, an optional array of `httpProxies` can be specified.
+An http proxy can be used to solve CORS issues in case CORS headers are not set (correctly) on some queried sources.
+Note that the involved sources can include those specified in `comunicaContext.sources` as well as those described in and found through `sourceIndex`.
 
-Configuration setting `httpProxy` can be used to solve CORS issues in case CORS headers are not set (correctly) on a queried source.
 We support static proxies such as [cors-anywhere](https://www.npmjs.com/package/cors-anywhere) that take the URL from the path.
 
-We simply prepend the `httpProxy` value before the URL of each source in a query that has `comunicaContext.useProxy` set to `true`.
+Each element of such array contains a property `httpProxies` and a property `urlStart`.
+
+We simply prepend the `httpProxy` value before the URL of each source whose URL starts with the string in the corresponding `urlStart` value.
 
-Example: if `httpProxy` is set to `http://myproxy.org/`, source `http://www.example.com/source-xyz`
-will be accessed as `http://myproxy.org/http://www.example.com/source-xyz`.
+Example: if
+`httpProxies[i].urlStart` is set to `http://www.example.com/path-xyz` and
+`httpProxies[i].httpProxy` is set to `http://myproxy.org/`,
+source `http://www.example.com/path-xyz-source-xyz` will be accessed as `http://myproxy.org/http://www.example.com/path-xyz-source-xyz`.
 
 ### Adding variable type
 

main/configs/demo/config.json

@@ -7,7 +7,6 @@
   "titleColor": "black",
   "textColor": "#1976D2",
   "queryFolder": "queries",
-  "httpProxy": "http://localhost:8000/",
   "showMilliseconds": false,
   "defaultIDP": "http://localhost:8080",
   "footer": "<p><a href='https://idlab.technology/'>IDLab</a> - <a href='https://www.imec.be/nl'>imec</a> - <a href='https://www.ugent.be/'>UGent</a></p>",
@@ -33,9 +32,14 @@
       "comunicaContext": {
         "sources": [
           "http://localhost:8001/example/idols"
-        ],
-        "useProxy": true
-      }
+        ]
+      },
+      "httpProxies": [
+        {
+          "urlStart": "http://localhost:8001",
+          "httpProxy": "http://localhost:8000/"
+        }
+      ]
     },
     {
       "id": "1010",

main/configs/test/config.json

@@ -7,7 +7,6 @@
   "titleColor": "black",
   "textColor": "#1976D2",
   "queryFolder": "queries",
-  "httpProxy": "http://localhost:8000/",
   "showMilliseconds": false,
   "defaultIDP": "http://localhost:8080",
   "footer": "<p><a href='https://idlab.technology/'>IDLab</a> - <a href='https://www.imec.be/nl'>imec</a> - <a href='https://www.ugent.be/'>UGent</a></p>",
@@ -38,9 +37,14 @@
       "comunicaContext": {
         "sources": [
           "http://localhost:8001/example/idols"
-        ],
-        "useProxy": true
-      }
+        ]
+      },
+      "httpProxies": [
+        {
+          "urlStart": "http://localhost:8001",
+          "httpProxy": "http://localhost:8000/"
+        }
+      ]
     },
     {
       "id": "1010",
@@ -504,6 +508,23 @@
         ],
         "lenient": true
       }
+    },
+    {
+      "id": "9130",
+      "queryGroupId": "gr-test",
+      "queryLocation": "components.rq",
+      "name": "Http proxy test combined with indirect sources and source verification",
+      "description": "Contents same as those for the 'Source verification' query, but now through http proxy in a worse case test.",
+      "sourcesIndex": {
+        "url": "http://localhost:8080/example/index-example-for-proxy-test-only",
+        "queryLocation": "/sourceQueries/index_example_common_lt.rq"
+      },
+      "httpProxies": [
+        {
+          "urlStart": "http://localhost:8080",
+          "httpProxy": "http://localhost:8000/"
+        }
+      ]
     }
   ]
 }

main/src/components/ActionBar/ActionBar.jsx

@@ -106,10 +106,10 @@ function ActionBar() {
                       <SourceAuthenticationIcon source={source} />
                     </TableCell>
                     <TableCell>
-                      <SourceFetchStatusIcon proxyUrl={config.httpProxy || ""} context={context} source={source} />
+                      <SourceFetchStatusIcon source={source} />
                     </TableCell>
                     <TableCell>
-                      <SourceVerificationIcon proxyUrl={config.httpProxy || ""} context={context} source={source} />
+                      <SourceVerificationIcon httpProxies={query.httpProxies} source={source} />
                     </TableCell>
                   </TableRow>
                 ))}

main/src/components/ActionBar/SourceFetchStatusIcon/SourceFetchStatusIcon.jsx

@@ -8,19 +8,13 @@ import comunicaEngineWrapper from '../../../comunicaEngineWrapper/comunicaEngine
 
 /**
  * @param {object} props - the props passed to the component
- * @param {object} props.context - the query context
  * @param {string} props.source - the source to check
- * @param {string} props.proxyUrl - the proxy url to use if the resource is accessed through a proxy
  * @returns {Component} an icon indicating whether the query was executed succesfully or not
  */
-function SourceFetchStatusIcon({ context, source, proxyUrl }) {
-  let actualSource = source;
-  if (context.useProxy) {
-    actualSource = `${proxyUrl}${source}`;
-  }
-  const status = comunicaEngineWrapper.getFetchStatusNumber(actualSource);
+function SourceFetchStatusIcon({ source }) {
+  const status = comunicaEngineWrapper.getFetchStatusNumber(source);
 
-  if (comunicaEngineWrapper.getFetchSuccess(actualSource)) {
+  if (comunicaEngineWrapper.getFetchSuccess(source)) {
     return (
       <Tooltip title="Fetch was successful">
         <CheckIcon size="small" />
@@ -44,9 +38,7 @@ function SourceFetchStatusIcon({ context, source, proxyUrl }) {
 }
 
 SourceFetchStatusIcon.propTypes = {
-  context: PropTypes.object.isRequired,
-  source: PropTypes.string.isRequired,
-  proxyUrl: PropTypes.string.isRequired,
+  source: PropTypes.string.isRequired
 };
 
 export default SourceFetchStatusIcon;

main/src/components/ActionBar/SourceVerificationIcon/SourceVerificationIcon.jsx

@@ -7,6 +7,7 @@ import GppBadIcon from '@mui/icons-material/GppBad';
 import GppMaybeIcon from '@mui/icons-material/GppMaybe';
 import { coreVerify } from '../../../vendor/vcCore';
 import comunicaEngineWrapper from '../../../comunicaEngineWrapper/comunicaEngineWrapper';
+import { translateUrlToProxiedUrl } from '../../../lib/utils';
 
 const VERIFICATION_STATES = {
   VERIFIED: 'VERIFIED',
@@ -17,16 +18,12 @@ const VERIFICATION_STATES = {
 
 /**
  * @param {object} props - the props passed to the component
- * @param {object} props.context - the query context
  * @param {string} props.source - the source to check
- * @param {string} props.proxyUrl - the proxy url to use if the resource is accessed through a proxy
+ * @param {array} props.httpProxies - array of httpProxy definitions
  * @returns {Component} an icon indicating whether the source was verified or not
  */
-function SourceVerificationIcon({ context, source, proxyUrl }) {
-  let sourceUrl = source;
-  if (context.useProxy) {
-    sourceUrl = `${proxyUrl}${source}`;
-  }
+function SourceVerificationIcon({ source, httpProxies }) {
+  let sourceUrl = translateUrlToProxiedUrl(source, httpProxies);
 
   const [isLoading, setIsLoading] = useState(true);
   const [verificationState, setVerificationState] = useState(undefined);
@@ -108,9 +105,8 @@ function SourceVerificationIcon({ context, source, proxyUrl }) {
 }
 
 SourceVerificationIcon.propTypes = {
-  context: PropTypes.object.isRequired,
   source: PropTypes.string.isRequired,
-  proxyUrl: PropTypes.string.isRequired,
+  httpProxies: PropTypes.array.isRequired,
 }
 
 export default SourceVerificationIcon;

main/src/components/CustomQueryEditor/customConversionButton.jsx

@@ -26,6 +26,7 @@ export default function CustomConversionButton({ id }) {
         convertTemplatedQueriesFixedVariables()
         convertComunicaContextAndSources()
         convertASKquery()
+        convertHttpProxies()
 
         // The id and group have to be deleted because a new custom query is going to be made. Name is modified.
         delete convertedQuery.id
@@ -135,6 +136,13 @@ export default function CustomConversionButton({ id }) {
         }
     }
 
+    // This function handles the logic for http proxies
+    function convertHttpProxies() {
+        if (convertedQuery.httpProxies) {
+            convertedQuery.httpProxiesCheck = "on"
+        }
+    }
+
 
     return (
         <Box display="flex" justifyContent="flex-end" width="100%">

main/src/components/CustomQueryEditor/customEditor.jsx

@@ -29,6 +29,7 @@ export default function CustomEditor(props) {
     comunicaContextCheck: false,
     sourceIndexCheck: false,
     askQueryCheck: false,
+    httpProxiesCheck: false,
     directVariablesCheck: false,
     indirectVariablesCheck: false,
   });
@@ -37,6 +38,7 @@ export default function CustomEditor(props) {
   const [editError, setEditError] = useState(false);
   const [parsingErrorComunica, setParsingErrorComunica] = useState(false);
   const [parsingErrorAsk, setParsingErrorAsk] = useState(false);
+  const [parsingErrorHttpProxies, setParsingErrorHttpProxies] = useState(false);
   const [parsingErrorTemplate, setParsingErrorTemplate] = useState(false);
 
   // Default placeholders for the forms
@@ -61,6 +63,7 @@ ORDER BY ?genre`;
 
   const defaultExtraComunicaContext = JSON.stringify({ "lenient": true }, null, 2);
   const defaultAskQueryDetails = JSON.stringify({ "trueText": "this displays when true.", "falseText": "this displays when false." }, null, 2);
+  const defaultHttpProxiesDetails = JSON.stringify([{ "urlStart": "http://www.example.com/path-xyz", "httpProxy": "http://myproxy.org/" }], null, 2);
   const defaultTemplateOptions = JSON.stringify(
     {
       "variableOne": [
@@ -100,7 +103,7 @@ ORDER BY ?genre`;
   const handleSubmit = async (event) => {
     event.preventDefault();
 
-    if (!parsingErrorComunica && !parsingErrorAsk && !parsingErrorTemplate) {
+    if (!parsingErrorComunica && !parsingErrorAsk && !parsingErrorHttpProxies && !parsingErrorTemplate) {
       setShowError(false);
       const formData = new FormData(event.currentTarget);
       const jsonData = Object.fromEntries(formData.entries());
@@ -190,6 +193,10 @@ ORDER BY ?genre`;
       parsedObject.askQuery = JSON.parse(dataWithStrings.askQuery);
     }
 
+    if (ensureBoolean(dataWithStrings.httpProxiesCheck)) {
+      parsedObject.httpProxies = JSON.parse(dataWithStrings.httpProxies);
+    }
+
     if (ensureBoolean(dataWithStrings.directVariablesCheck)) {
       parsedObject.variables = JSON.parse(dataWithStrings.variables);
     }
@@ -545,6 +552,42 @@ ORDER BY ?genre`;
                 </div>
               }
 
+              <FormControlLabel
+                control={<Checkbox
+                  name='httpProxiesCheck'
+                  checked={!!formData.httpProxiesCheck}
+                  onChange={
+                    () => {
+                      setParsingErrorHttpProxies(false);
+                      setFormData((prevFormData) => ({
+                        ...prevFormData,
+                        'httpProxiesCheck': !formData.httpProxiesCheck,
+                      }));
+                    }
+                  }
+                />} label="Http proxies" />
+
+              {formData.httpProxiesCheck &&
+                <div>
+                  <TextField
+                    required={ensureBoolean(formData.httpProxiesCheck)}
+                    label="Specifying http proxies"
+                    name="httpProxies"
+                    error={parsingErrorHttpProxies}
+                    multiline
+                    fullWidth
+                    minRows={5}
+                    variant="outlined"
+                    helperText={`Write http proxies in JSON-format${parsingErrorHttpProxies ? ' (Invalid Syntax)' : '.'}`}
+                    value={!!formData.httpProxies ? typeof formData.httpProxies === 'object' ? JSON.stringify(formData.httpProxies, null, 2) : formData.httpProxies : formData.httpProxies === '' ? '' : defaultHttpProxiesDetails}
+                    placeholder={defaultHttpProxiesDetails}
+                    onClick={(e) => handleJSONparsing(e, setParsingErrorHttpProxies)}
+                    onChange={(e) => handleJSONparsing(e, setParsingErrorHttpProxies)}
+                    sx={{ marginBottom: '16px' }}
+                  />
+                </div>
+              }
+
             </div>
           </Card>