Skip to content

Commit

Permalink
Azure repository: Move to named configurations as we do for S3 reposi…
Browse files Browse the repository at this point in the history 
…tory

We should have the same behavior for Azure repositories as we have for S3 (see #22762).

Instead of:

```yml
cloud:
    azure:
        storage:
            my_account1:
                account: your_azure_storage_account1
                key: your_azure_storage_key1
                default: true
            my_account2:
                account: your_azure_storage_account2
                key: your_azure_storage_key2
```

Support something like:

```
azure.client:
            default:
                account: your_azure_storage_account1
                key: your_azure_storage_key1
            my_account2:
                account: your_azure_storage_account2
                key: your_azure_storage_key2
```

Then instead of:

```
PUT _snapshot/my_backup3
{
    "type": "azure",
    "settings": {
        "account": "my_account2"
    }
}
```

Use:

```
PUT _snapshot/my_backup3
{
    "type": "azure",
    "settings": {
        "config": "my_account2"
    }
}
```

If someone uses:

```
PUT _snapshot/my_backup3
{
    "type": "azure"
}
```

It will use the `default` azure repository settings.

And mark as deprecated old settings.

Closes #22763.
  • Loading branch information
@dadoonet
dadoonet committed Aug 8, 2017
1 parent f0c1e30 commit 80b142d
Show file tree
Hide file tree
Showing 17 changed files with 482 additions and 208 deletions.
70 changes: 26 additions & 44 deletions docs/plugins/repository-azure.asciidoc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,35 +10,28 @@ include::install_remove.asciidoc[]
[[repository-azure-usage]]
==== Azure Repository

To enable Azure repositories, you have first to set your azure storage settings in `elasticsearch.yml` file:
To enable Azure repositories, you have first to define your azure storage settings as
{ref}/secure-settings.html[secured settings]:

[source,yaml]
----
cloud:
azure:
storage:
my_account:
account: your_azure_storage_account
key: your_azure_storage_key
----
[source,sh]
----------------------------------------------------------------
bin/elasticsearch-keystore add azure.client.default.account
bin/elasticsearch-keystore add azure.client.default.key
----------------------------------------------------------------

Where `account` is the azure account name and `key` the azure secret key.

Note that you can also define more than one account:

[source,yaml]
----
cloud:
azure:
storage:
my_account1:
account: your_azure_storage_account1
key: your_azure_storage_key1
default: true
my_account2:
account: your_azure_storage_account2
key: your_azure_storage_key2
----
[source,sh]
----------------------------------------------------------------
bin/elasticsearch-keystore add azure.client.default.account
bin/elasticsearch-keystore add azure.client.default.key
bin/elasticsearch-keystore add azure.client.secondary.account
bin/elasticsearch-keystore add azure.client.secondary.key
----------------------------------------------------------------

`my_account1` is the default account which will be used by a repository unless you set an explicit one.
`default` is the default account name which will be used by a repository unless you set an explicit one.

You can set the client side timeout to use when making any single request. It can be defined globally, per account or both.
It's not set by default which means that elasticsearch is using the
Expand All @@ -53,23 +46,13 @@ before retrying after a first timeout or failure. The maximum backoff period is

[source,yaml]
----
cloud:
azure:
storage:
timeout: 10s
my_account1:
account: your_azure_storage_account1
key: your_azure_storage_key1
default: true
max_retries: 7
my_account2:
account: your_azure_storage_account2
key: your_azure_storage_key2
timeout: 30s
cloud.azure.storage.timeout: 10s
azure.client.default.max_retries: 7
azure.client.secondary.timeout: 30s
----

In this example, timeout will be `10s` per try for `my_account1` with `7` retries before failing
and `30s` per try for `my_account2` with `3` retries.
In this example, timeout will be `10s` per try for `default` with `7` retries before failing
and `30s` per try for `secondary` with `3` retries.

[IMPORTANT]
.Supported Azure Storage Account types
Expand All @@ -89,10 +72,9 @@ https://azure.microsoft.com/en-gb/documentation/articles/storage-premium-storage

The Azure repository supports following settings:

`account`::
`client`::

Azure account settings to use. Defaults to the only one if you set a single
account or to the one marked as `default` if you have more than one.
Azure named client to use. Defaults to `default`.

`container`::

Expand Down Expand Up @@ -153,14 +135,14 @@ PUT _snapshot/my_backup3
{
"type": "azure",
"settings": {
"account": "my_account1"
"client": "secondary"
}
}
PUT _snapshot/my_backup4
{
"type": "azure",
"settings": {
"account": "my_account2",
"client": "secondary",
"location_mode": "primary_only"
}
}
Expand Down
4 changes: 4 additions & 0 deletions plugins/repository-azure/build.gradle
View file
Original file line number Diff line number Diff line change
Expand Up @@ -45,4 +45,8 @@ thirdPartyAudit.excludes = [
integTestCluster {
setting 'cloud.azure.storage.my_account_test.account', 'cloudazureresource'
setting 'cloud.azure.storage.my_account_test.key', 'abcdefgh'
keystoreSetting 'azure.client.default.account', 'cloudazureresource'
keystoreSetting 'azure.client.default.key', 'abcdefgh'
keystoreSetting 'azure.client.secondary.account', 'cloudazureresource'
keystoreSetting 'azure.client.secondary.key', 'abcdefgh'
}
23 changes: 11 additions & 12 deletions ...epository-azure/src/main/java/org/elasticsearch/cloud/azure/blobstore/AzureBlobStore.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,6 @@
import com.microsoft.azure.storage.LocationMode;
import com.microsoft.azure.storage.StorageException;
import org.elasticsearch.cloud.azure.storage.AzureStorageService;
import org.elasticsearch.cloud.azure.storage.AzureStorageService.Storage;
import org.elasticsearch.cluster.metadata.RepositoryMetaData;
import org.elasticsearch.common.Strings;
import org.elasticsearch.common.blobstore.BlobContainer;
Expand All @@ -45,7 +44,7 @@ public class AzureBlobStore extends AbstractComponent implements BlobStore {

private final AzureStorageService client;

private final String accountName;
private final String clientName;
private final LocationMode locMode;
private final String container;
private final String repositoryName;
Expand All @@ -56,7 +55,7 @@ public AzureBlobStore(RepositoryMetaData metadata, Settings settings,
this.client = client;
this.container = Repository.CONTAINER_SETTING.get(metadata.settings());
this.repositoryName = metadata.name();
this.accountName = Repository.ACCOUNT_SETTING.get(metadata.settings());
this.clientName = Repository.CLIENT_NAME.get(metadata.settings());

String modeStr = Repository.LOCATION_MODE_SETTING.get(metadata.settings());
if (Strings.hasLength(modeStr)) {
Expand Down Expand Up @@ -91,7 +90,7 @@ public BlobContainer blobContainer(BlobPath path) {
public void delete(BlobPath path) {
String keyPath = path.buildAsString();
try {
this.client.deleteFiles(this.accountName, this.locMode, container, keyPath);
this.client.deleteFiles(this.clientName, this.locMode, container, keyPath);
} catch (URISyntaxException | StorageException e) {
logger.warn("can not remove [{}] in container {{}}: {}", keyPath, container, e.getMessage());
}
Expand All @@ -103,41 +102,41 @@ public void close() {

public boolean doesContainerExist(String container)
{
return this.client.doesContainerExist(this.accountName, this.locMode, container);
return this.client.doesContainerExist(this.clientName, this.locMode, container);
}

public void deleteFiles(String container, String path) throws URISyntaxException, StorageException
{
this.client.deleteFiles(this.accountName, this.locMode, container, path);
this.client.deleteFiles(this.clientName, this.locMode, container, path);
}

public boolean blobExists(String container, String blob) throws URISyntaxException, StorageException
{
return this.client.blobExists(this.accountName, this.locMode, container, blob);
return this.client.blobExists(this.clientName, this.locMode, container, blob);
}

public void deleteBlob(String container, String blob) throws URISyntaxException, StorageException
{
this.client.deleteBlob(this.accountName, this.locMode, container, blob);
this.client.deleteBlob(this.clientName, this.locMode, container, blob);
}

public InputStream getInputStream(String container, String blob) throws URISyntaxException, StorageException, IOException
{
return this.client.getInputStream(this.accountName, this.locMode, container, blob);
return this.client.getInputStream(this.clientName, this.locMode, container, blob);
}

public OutputStream getOutputStream(String container, String blob) throws URISyntaxException, StorageException
{
return this.client.getOutputStream(this.accountName, this.locMode, container, blob);
return this.client.getOutputStream(this.clientName, this.locMode, container, blob);
}

public Map<String,BlobMetaData> listBlobsByPrefix(String container, String keyPath, String prefix) throws URISyntaxException, StorageException
{
return this.client.listBlobsByPrefix(this.accountName, this.locMode, container, keyPath, prefix);
return this.client.listBlobsByPrefix(this.clientName, this.locMode, container, keyPath, prefix);
}

public void moveBlob(String container, String sourceBlob, String targetBlob) throws URISyntaxException, StorageException
{
this.client.moveBlob(this.accountName, this.locMode, container, sourceBlob, targetBlob);
this.client.moveBlob(this.clientName, this.locMode, container, sourceBlob, targetBlob);
}
}
11 changes: 9 additions & 2 deletions ...sitory-azure/src/main/java/org/elasticsearch/cloud/azure/storage/AzureStorageService.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -45,12 +45,19 @@ public interface AzureStorageService {
ByteSizeValue MAX_CHUNK_SIZE = new ByteSizeValue(64, ByteSizeUnit.MB);

final class Storage {
@Deprecated
public static final String PREFIX = "cloud.azure.storage.";

@Deprecated
public static final Setting<Settings> STORAGE_ACCOUNTS = Setting.groupSetting(Storage.PREFIX, Setting.Property.NodeScope);

public static final Setting<TimeValue> TIMEOUT_SETTING =
Setting.timeSetting("cloud.azure.storage.timeout", TimeValue.timeValueMinutes(-1), Property.NodeScope);
/**
* Azure timeout (defaults to -1 minute)
* @deprecated We don't want to support global timeout settings anymore
*/
@Deprecated
static final Setting<TimeValue> TIMEOUT_SETTING =
Setting.timeSetting("cloud.azure.storage.timeout", TimeValue.timeValueMinutes(-1), Property.NodeScope, Property.Deprecated);
}

boolean doesContainerExist(String account, LocationMode mode, String container);
Expand Down
81 changes: 46 additions & 35 deletions ...ry-azure/src/main/java/org/elasticsearch/cloud/azure/storage/AzureStorageServiceImpl.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -45,39 +45,57 @@
import java.io.OutputStream;
import java.net.URI;
import java.net.URISyntaxException;
import java.security.AccessController;
import java.security.PrivilegedActionException;
import java.security.PrivilegedExceptionAction;
import java.util.HashMap;
import java.util.Map;

public class AzureStorageServiceImpl extends AbstractComponent implements AzureStorageService {

final AzureStorageSettings primaryStorageSettings;
final Map<String, AzureStorageSettings> secondariesStorageSettings;
final Map<String, AzureStorageSettings> storageSettings;
final Map<String, AzureStorageSettings> deprecatedStorageSettings;

final Map<String, CloudBlobClient> clients;

public AzureStorageServiceImpl(Settings settings) {
public AzureStorageServiceImpl(Settings settings, Map<String, AzureStorageSettings> regularStorageSettings) {
super(settings);

Tuple<AzureStorageSettings, Map<String, AzureStorageSettings>> storageSettings = AzureStorageSettings.parse(settings);
this.primaryStorageSettings = storageSettings.v1();
this.secondariesStorageSettings = storageSettings.v2();
if (regularStorageSettings.isEmpty()) {
this.storageSettings = new HashMap<>();
// We have deprecated settings so we need to migrate them to the new implementation
Tuple<AzureStorageSettings, Map<String, AzureStorageSettings>> storageSettingsMapTuple = AzureStorageSettings.loadLegacy(settings);
deprecatedStorageSettings = storageSettingsMapTuple.v2();
if (storageSettingsMapTuple.v1() != null) {
if (storageSettingsMapTuple.v1().getName().equals("default") == false) {
// We add the primary configuration to the list of all settings with its deprecated name in case someone is
// forcing a specific configuration name when creating the repository instance
deprecatedStorageSettings.put(storageSettingsMapTuple.v1().getName(), storageSettingsMapTuple.v1());
}
// We add the primary configuration to the list of all settings as the "default" one
deprecatedStorageSettings.put("default", storageSettingsMapTuple.v1());
} else {
// If someone did not register any settings or deprecated settings, they
// basically can't use the plugin
throw new IllegalArgumentException("If you want to use an azure repository, you need to define a client configuration.");
}


} else {
this.storageSettings = regularStorageSettings;
this.deprecatedStorageSettings = new HashMap<>();
}

this.clients = new HashMap<>();

logger.debug("starting azure storage client instance");

// We register the primary client if any
if (primaryStorageSettings != null) {
logger.debug("registering primary client for account [{}]", primaryStorageSettings.getAccount());
createClient(primaryStorageSettings);
// We register all regular azure clients
for (Map.Entry<String, AzureStorageSettings> azureStorageSettingsEntry : this.storageSettings.entrySet()) {
logger.debug("registering regular client for account [{}]", azureStorageSettingsEntry.getKey());
createClient(azureStorageSettingsEntry.getValue());
}

// We register all secondary clients
for (Map.Entry<String, AzureStorageSettings> azureStorageSettingsEntry : secondariesStorageSettings.entrySet()) {
logger.debug("registering secondary client for account [{}]", azureStorageSettingsEntry.getKey());
// We register all deprecated azure clients
for (Map.Entry<String, AzureStorageSettings> azureStorageSettingsEntry : this.deprecatedStorageSettings.entrySet()) {
logger.debug("registering deprecated client for account [{}]", azureStorageSettingsEntry.getKey());
createClient(azureStorageSettingsEntry.getValue());
}
}
Expand Down Expand Up @@ -107,32 +125,25 @@ void createClient(AzureStorageSettings azureStorageSettings) {

CloudBlobClient getSelectedClient(String account, LocationMode mode) {
logger.trace("selecting a client for account [{}], mode [{}]", account, mode.name());
AzureStorageSettings azureStorageSettings = null;

if (this.primaryStorageSettings == null) {
throw new IllegalArgumentException("No primary azure storage can be found. Check your elasticsearch.yml.");
}

if (Strings.hasLength(account)) {
azureStorageSettings = this.secondariesStorageSettings.get(account);
}

// if account is not secondary, it's the primary
AzureStorageSettings azureStorageSettings = this.storageSettings.get(account);
if (azureStorageSettings == null) {
if (Strings.hasLength(account) == false || primaryStorageSettings.getName() == null || account.equals(primaryStorageSettings.getName())) {
azureStorageSettings = primaryStorageSettings;
// We can't find a client that has been registered using regular settings so we try deprecated client
azureStorageSettings = this.deprecatedStorageSettings.get(account);
if (azureStorageSettings == null) {
// We did not get an account. That's bad.
if (Strings.hasLength(account)) {
throw new IllegalArgumentException("Can not find named azure client [" + account +
"]. Check your elasticsearch.yml.");
}
throw new IllegalArgumentException("Can not find primary/secondary client using deprecated settings. " +
"Check your elasticsearch.yml.");
}
}

if (azureStorageSettings == null) {
// We did not get an account. That's bad.
throw new IllegalArgumentException("Can not find azure account [" + account + "]. Check your elasticsearch.yml.");
}

CloudBlobClient client = this.clients.get(azureStorageSettings.getAccount());

if (client == null) {
throw new IllegalArgumentException("Can not find an azure client for account [" + account + "]");
throw new IllegalArgumentException("Can not find an azure client for account [" + azureStorageSettings.getAccount() + "]");
}

// NOTE: for now, just set the location mode in case it is different;
Expand Down
Loading

0 comments on commit 80b142d

Please sign in to comment.