Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add a POST parameter in the FileDownloader processing algorithm #44867

Merged
merged 1 commit into from Sep 3, 2021
Merged

Add a POST parameter in the FileDownloader processing algorithm #44867

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
1 change: 1 addition & 0 deletions python/core/auto_additions/qgis.py
View file
Open in desktop
Expand Up @@ -276,6 +276,7 @@
Qgis.BrowserDirectoryMonitoring.__doc__ = 'Browser directory item monitoring switches.\n\n.. versionadded:: 3.20\n\n' + '* ``Default``: ' + Qgis.BrowserDirectoryMonitoring.Default.__doc__ + '\n' + '* ``NeverMonitor``: ' + Qgis.BrowserDirectoryMonitoring.NeverMonitor.__doc__ + '\n' + '* ``AlwaysMonitor``: ' + Qgis.BrowserDirectoryMonitoring.AlwaysMonitor.__doc__
# --
Qgis.BrowserDirectoryMonitoring.baseClass = Qgis
Qgis.HttpMethod.baseClass = Qgis
QgsVectorLayerExporter.ExportError = Qgis.VectorExportResult
# monkey patching scoped based enum
QgsVectorLayerExporter.NoError = Qgis.VectorExportResult.Success
Expand Down
4 changes: 3 additions & 1 deletion python/core/auto_generated/network/qgsfiledownloader.sip.in
View file
Open in desktop
Expand Up @@ -35,7 +35,7 @@ An optional authentication configuration can be specified.
%End
public:

QgsFileDownloader( const QUrl &url, const QString &outputFileName, const QString &authcfg = QString(), bool delayStart = false );
QgsFileDownloader( const QUrl &url, const QString &outputFileName, const QString &authcfg = QString(), bool delayStart = false, Qgis::HttpMethod httpMethod = Qgis::HttpMethod::Get, const QByteArray &data = QByteArray() );
%Docstring
QgsFileDownloader

Expand All @@ -46,6 +46,8 @@ QgsFileDownloader
be triggered by a later call to :py:func:`~QgsFileDownloader.startDownload`. This can be useful if connections need
to be made to the downloader and there's a chance the download will emit
signals before these connections have been made.
:param httpMethod: Method for the HTTP request : GET or POST, since QGIS 3.22
:param data: If the request is POST, some data can be added, since QGIS 3.22
%End

signals:
Expand Down
6 changes: 6 additions & 0 deletions python/core/auto_generated/qgis.sip.in
View file
Open in desktop
Expand Up @@ -211,6 +211,12 @@ The development version
AlwaysMonitor,
};

enum HttpMethod
{
Get,
Post
};

enum class VectorExportResult
{
Success,
Expand Down
36 changes: 35 additions & 1 deletion src/analysis/processing/qgsalgorithmfiledownloader.cpp
View file
Open in desktop
Expand Up @@ -16,8 +16,11 @@
***************************************************************************/

#include "qgsalgorithmfiledownloader.h"
#include "qgsprocessingparameters.h"
#include "qgis.h"
#include "qgsfiledownloader.h"
#include "qgsfileutils.h"

#include <QEventLoop>
#include <QFileInfo>
#include <QTimer>
Expand Down Expand Up @@ -63,6 +66,26 @@ QgsFileDownloaderAlgorithm *QgsFileDownloaderAlgorithm::createInstance() const
void QgsFileDownloaderAlgorithm::initAlgorithm( const QVariantMap & )
{
addParameter( new QgsProcessingParameterString( QStringLiteral( "URL" ), tr( "URL" ), QVariant(), false, false ) );

std::unique_ptr< QgsProcessingParameterEnum > methodParam = std::make_unique < QgsProcessingParameterEnum > (
QStringLiteral( "METHOD" ),
QObject::tr( "Method" ),
QStringList()
<< QObject::tr( "GET" )
<< QObject::tr( "POST" ),
false,
QStringLiteral( "GET" )
);
methodParam->setHelp( QObject::tr( "The HTTP method to use for the request" ) );
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Gustry is this a new feature in Processing or just an old feature nobody uses? From a user (and writer) pov, this could be quite helpful in many contexts

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's indeed a new "mini" feature. Before, all requests were "GET". Now, you can use POST and also add a BODY parameter when it's POST.

For instance, you can make a POST query to the OSM Overpass server, with a very long query in the BODY. Before, with GET, you might have reached the limit of characters in a GET request.

Is-that OK ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @Gustry Actually I was talking about the ability to add a description to a parameter (the setHelp method). I'm not sure I noticed it earlier in another algorithm

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah sorry.

It's a new feature from QGIS 3.16:
https://qgis.org/en/site/forusers/visualchangelog316/index.html#add-help-string-for-parameters

Not very used, but quite convenient indeed to add a longer description instead of adding a lot of text in the main algorithm description.

methodParam->setFlags( methodParam->flags() | QgsProcessingParameterDefinition::FlagAdvanced );
addParameter( methodParam.release() );

std::unique_ptr< QgsProcessingParameterString > dataParam = std::make_unique < QgsProcessingParameterString >(
QStringLiteral( "DATA" ), tr( "Data" ), QVariant(), false, true );
dataParam->setHelp( QObject::tr( "The data to add in the body if the request is a POST" ) );
dataParam->setFlags( dataParam->flags() | QgsProcessingParameterDefinition::FlagAdvanced );
addParameter( dataParam.release() );

addParameter( new QgsProcessingParameterFileDestination( QStringLiteral( "OUTPUT" ),
tr( "File destination" ), QObject::tr( "All files (*.*)" ), QVariant(), true ) );
}
Expand All @@ -73,13 +96,24 @@ QVariantMap QgsFileDownloaderAlgorithm::processAlgorithm( const QVariantMap &par
QString url = parameterAsString( parameters, QStringLiteral( "URL" ), context );
if ( url.isEmpty() )
throw QgsProcessingException( tr( "No URL specified" ) );

QString data = parameterAsString( parameters, QStringLiteral( "DATA" ), context );
QString outputFile = parameterAsFileOutput( parameters, QStringLiteral( "OUTPUT" ), context );

QEventLoop loop;
QTimer timer;
QUrl downloadedUrl;
QStringList errors;
QgsFileDownloader *downloader = new QgsFileDownloader( QUrl( url ), outputFile, QString(), true );

Qgis::HttpMethod httpMethod = static_cast< Qgis::HttpMethod>( parameterAsEnum( parameters, QStringLiteral( "METHOD" ), context ) );

if ( httpMethod == Qgis::HttpMethod::Get && ! data.isEmpty() )
{
feedback->pushWarning( tr( "DATA parameter is not used when it's a GET request." ) );
data = QString();
}

QgsFileDownloader *downloader = new QgsFileDownloader( QUrl( url ), outputFile, QString(), true, httpMethod, data.toUtf8() );
connect( mFeedback, &QgsFeedback::canceled, downloader, &QgsFileDownloader::cancelDownload );
connect( downloader, &QgsFileDownloader::downloadError, this, [&errors, &loop]( const QStringList & e ) { errors = e; loop.exit(); } );
connect( downloader, &QgsFileDownloader::downloadProgress, this, &QgsFileDownloaderAlgorithm::receiveProgressFromDownloader );
Expand Down
19 changes: 17 additions & 2 deletions src/core/network/qgsfiledownloader.cpp
View file
Open in desktop
Expand Up @@ -25,9 +25,11 @@
#include <QSslError>
#endif

QgsFileDownloader::QgsFileDownloader( const QUrl &url, const QString &outputFileName, const QString &authcfg, bool delayStart )
QgsFileDownloader::QgsFileDownloader( const QUrl &url, const QString &outputFileName, const QString &authcfg, bool delayStart, Qgis::HttpMethod httpMethod, const QByteArray &data )
: mUrl( url )
, mDownloadCanceled( false )
, mHttpMethod( httpMethod )
, mData( data )
{
mFile.setFileName( outputFileName );
mAuthCfg = authcfg;
Expand Down Expand Up @@ -65,7 +67,20 @@ void QgsFileDownloader::startDownload()
mReply->deleteLater();
}

mReply = nam->get( request );
switch ( mHttpMethod )
{
case Qgis::HttpMethod::Get:
{
mReply = nam->get( request );
break;
}
case Qgis::HttpMethod::Post:
{
mReply = nam->post( request, mData );
break;
}
}

if ( !mAuthCfg.isEmpty() )
{
QgsApplication::authManager()->updateNetworkReply( mReply, mAuthCfg );
Expand Down
7 changes: 6 additions & 1 deletion src/core/network/qgsfiledownloader.h
View file
Open in desktop
Expand Up @@ -21,6 +21,7 @@
#include <QNetworkReply>
#include <QUrl>

#include "qgis.h"
#include "qgis_core.h"

#ifndef QT_NO_SSL
Expand Down Expand Up @@ -56,8 +57,10 @@ class CORE_EXPORT QgsFileDownloader : public QObject
* be triggered by a later call to startDownload(). This can be useful if connections need
* to be made to the downloader and there's a chance the download will emit
* signals before these connections have been made.
* \param httpMethod Method for the HTTP request : GET or POST, since QGIS 3.22
* \param data If the request is POST, some data can be added, since QGIS 3.22
*/
QgsFileDownloader( const QUrl &url, const QString &outputFileName, const QString &authcfg = QString(), bool delayStart = false );
QgsFileDownloader( const QUrl &url, const QString &outputFileName, const QString &authcfg = QString(), bool delayStart = false, Qgis::HttpMethod httpMethod = Qgis::HttpMethod::Get, const QByteArray &data = QByteArray() );

signals:
//! Emitted when the download has completed successfully
Expand Down Expand Up @@ -123,6 +126,8 @@ class CORE_EXPORT QgsFileDownloader : public QObject
QNetworkReply *mReply = nullptr;
QFile mFile;
bool mDownloadCanceled;
Qgis::HttpMethod mHttpMethod = Qgis::HttpMethod::Get;
QByteArray mData;
QStringList mErrors;
QString mAuthCfg;
};
Expand Down
11 changes: 11 additions & 0 deletions src/core/qgis.h
View file
Open in desktop
Expand Up @@ -309,6 +309,17 @@ class CORE_EXPORT Qgis
};
Q_ENUM( BrowserDirectoryMonitoring )

/**
* Different methods of HTTP requests
* \since 3.22
*/
enum HttpMethod
{
Get = 0, //!< GET method
Post = 1 //!< POST method
};
Q_ENUM( HttpMethod )

/**
* Vector layer export result codes.
*
Expand Down